MantisBT - Issues

0005867: XLSDataReader: Infinite loop when the xlsxReaderDataPolicy test from trunk runs

Fri, 25 Feb 2011 15:02:46 +0100

Repeated output in console:
INFO [XLS_READER0_0] - EmployeeID has incorrect default value

0005868: DataPolicy: Max error count is exceeded, but graph does not fail.

Fri, 25 Feb 2011 15:00:45 +0100

xlsReaderDataPolicy test from trunk.

Max error count is set to 3, there are four errors, but graph finishes OK.

0005858: CTL2: make byte and cbyte types comparable by == and !=

Fri, 25 Feb 2011 12:57:02 +0100

see Summary.

0005866: DBJoin: SQL query vs. metadata checking

Fri, 25 Feb 2011 07:31:04 +0100

When user's query returns more columns than there are fields in DB metadata and there is no 'where field1 = ? and field2 = ? ...' expression, no error or warning is reported and returned data are silently mapped to metadata fields ignoring overflowing columns.

This is dangerous and extremely hard to debug. Particularly, during development users change SQL queries adding or removing columns but they forget to change metadata.

When 'where field1 = ? and field2 = ? ...' expression is present, an error message 'Node DBJOIN2 finished with status: ERROR caused by: java.sql.SQLException: Missing IN or OUT parameter at index::4' is shown. This is at least somehow helpful.

0005865: Selecting: Shift+Click should select multiple graph elements in the Navigator pane.

Thu, 24 Feb 2011 17:54:58 +0100

Now it only marks the first and the last element.
Connection, metadata, etc.

0005864: NPE: When linking a lookup table.

Thu, 24 Feb 2011 17:50:53 +0100

Build # 349.

java.lang.NullPointerException
at com.cloveretl.gui.model.LookupTable.addLookupTable(Unknown Source)
at com.cloveretl.gui.actions.f.run(Unknown Source)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:584)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:501)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:411)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1003)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3880)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3473)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2405)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2369)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2221)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:500)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:493)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:113)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:194)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:368)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:559)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:514)
at org.eclipse.equinox.launcher.Main.run(Main.java:1311)

0005734: HTTPConnector initialization - unclear error message

Thu, 24 Feb 2011 17:12:24 +0100

When HTTP connector is connected its input and output ports, an error message:
WARN [main] - Input file specified will be ignored, because an input port is connected.
WARN [main] - Output file specified will be ignored, because an output port is connected.
DEBUG [main] - HTTP_CONNECTOR0 ...OK

Only when DEBUG logging level is enabled, user finds which component produces the above warning messages.

0005761: XLSDataWriter: sheet overwriting issue (cannot write less records into same file)

Thu, 24 Feb 2011 17:11:51 +0100

When user attempts to read data from XLS file, modify them and write them, back into the same file and same sheet, it is not possible.

Clover does not allow user to clear sheet and write new data. It only allows user
* to remove all sheets (not always possible)
* or to overwrite existing data. However, if there are more records in the XLS file than those to be written down, the difference from the old file remains in the new one.

Possible solutions:
* add attribute 'clear sheet' or 'remove sheet' which would clear/remove the specified sheet before writing data to the XLS file
* change/extend behaviour of 'Append to the sheet' attribute

0005713: IllegalArgumentException when "open file" dialog is shown

Thu, 24 Feb 2011 16:41:20 +0100

An error is reported every time an "Open file" dialog is shown. It does not matter from where the dialog is shown - the error is reported whether selecting the XML schema or input file etc.

Ubuntu 10.10 x64, Sun JDK 1.6.0_22
Runs on 64 bit Java

0005728: Unify ordered key in components

Thu, 24 Feb 2011 16:34:51 +0100

Many components has key attribute (some of them has "key with order"), some components has order attribute, but usually each component handles these attributes in different way. This should be unified and every component should use the same method (algorithm) for comparing records, when using the same key + order.

0005235: DataIntersection does not report error on unsorted inputs

Thu, 24 Feb 2011 16:34:16 +0100

DataIntersection expects input to be sorted. If the inputs are not sorted, the component transofrms data without any error message. The out-of-order records are not matched with the data on the other port.

0005863: Normalizer should handle SKIP return value in different way

Thu, 24 Feb 2011 14:15:03 +0100

Input record processing should be end up by 'SKIP' return value of RecordNormalize.transform() method, RecordNormalize.count() for next input record should be called afterwards.

0005681: UniversalDataReader - 'Number of skipped records' counts lines not records

Thu, 24 Feb 2011 14:02:55 +0100

When having a "quoted strings" data file with a header line containing newline character (correctly quoted) and UniversalDataReader set to skip 1 record then the data file is not parsed correctly. 1 line is skipped instead of 1 record. Subsequently the parsing fails because of invalid format of the remaining part of the record which should have been skipped. This is an issue only for a header containing newline characters.

The quoted newline characters in the rest of the file (not-skipped records) are parsed properly.

0005469: XLSDataReader confusing error message for invalid field mapping

Thu, 24 Feb 2011 14:01:36 +0100

When the field mapping from XLS file to metadata is not set properly, e.g. some of the columns in metadata are not mapped (e.g. due to bug 0005461), the confusing error message is produced (see below Additional Information Section).

I think that the exception message should be more descriptive, possibly include the invalid field or column index.

0005461: XLSDataReader: Field Mapping Editor does not work properly

Thu, 24 Feb 2011 14:01:18 +0100

Please see the attached screenshots, the Field Mapping Editor does not work properly for columns M and N.

0005586: Aggregate devises the pre-aggregated field's datatype based on input metadata

Thu, 24 Feb 2011 13:55:29 +0100

The data type of the field used for aggregating the data is devises from the input metadata rather than on the output. Current behaviour does not work well in (at least) the following cases:
1. the integer field on the input; summing millions of such records may produce 'long' result.
2. same as #1 but for avg()
3. Even if user (logically) expects that the output may be larger than the input data (e.g. integer) and configures output metadata correctly to e.g. long, it has no effect.

It would seem more logical to me to have the datatype devised from the fields in the output metadata.

Moreover, this behaviour is not described in the documentation.

0005585: Aggregate does not report an error on integer overflow

Thu, 24 Feb 2011 13:55:15 +0100

When aggregating the data of the integer field by sum() or avg() and the integer field overflows, there is no error reported and graph finishes successfully.

I have found it by summing 10,0000,000 positive integers and getting the negative result. Same happened for average.

0005701: New component: Pivot

Thu, 24 Feb 2011 13:35:26 +0100

Create a new component that allows for creating a "pivot" table. It's basically a simple form of Denormalizer.

The component would read input records and treat them as groups (either by key or count) and produce a single record from each group. The component setting would instruct it to treat particular input values as output field names and other fields as output values.

Example:
GID, Field, Value, somethingMore
===============================
1, Name, Sebastien, 2312
1, Age, 22, 42354
1, City, Paris, 233
2, Name, John, 543543
2, Age, 37, 6433
3, Name, Peter, 54354

Settings:
[key] = 'GID'
[fieldId] = 'Field'
[valueId] = 'Value'

Result:
someField, Name, City, Age, Note
=====================
null, Sebastien, Paris, 22, null
null, John, null, 37, null
null, Peter, null, null, null

Notes:
- somethingMore in the input is ignored
- output metadata is arbitrary (but fixed of course)
- settings provides a "mapping" which says which input field's value is used to designate the output field and which input field's value is used as a value for that field
- output fields without corresponding input records are simply null

0005775: EmailSender: multiline data in MessageBody not handled correctly in Message property editor

Thu, 24 Feb 2011 13:29:22 +0100

When user enters a multi-line text into MessageBody, after closing the Message property dialog and opening it again, MessageBody is split into separate properties; not preserved as a multiline field.

This is undesirable behaviour.

0005861: CTL2 - left function returns empty string when input is shorter then desired length

Thu, 24 Feb 2011 11:48:21 +0100

see ${subj}

0005776: Metadata naming: Allow non-java-compliant metadata/field names

Thu, 24 Feb 2011 11:22:41 +0100

There is a bunch of request to support metadata names with non-standard characters, spaces, etc.

The main issue with this is using metadata names as data types in CTL2.

Please investigate into possibility of having MSSQL similar standard for literals like:

[Customers-metadata].[field with a space]

0005860: DBOutputTable: byte data not handled correctly in batch mode

Thu, 24 Feb 2011 10:48:31 +0100

When byte data are inserted into database using DbOutputTable in batch mode, the results are wrong - the first byte value is duplicated in whole batch.

Works fine when string is used instead of byte in Clover's metadata.

0005805: Designer does not support PRIMITIVE_SEQUENCE

Thu, 24 Feb 2011 10:19:51 +0100

Designer currently supports only SIMPLE_SEQUENCE. We have also another type of sequence already implemented in engine. Support of this sequence should be implemented.

0005820: DBFDataReader: When Data policy is set to Controlled, Strict or Default, the numbers of rejected records ARE NOT provided

Wed, 23 Feb 2011 17:43:01 +0100

The number of incorrect field is the only information user gets.

0005735: Partition key: The output for Key file tag is different from that for Number file tag

Wed, 23 Feb 2011 15:05:21 +0100

Key file tag seems to ignore upper cases in the key.
from:
abc
klm
ABC
DEF
Key file tag creates 3 files (DEF belongs to the unassigned), it excludes ABC as a key,
whereas
Number file tag creates 4 files with ABC included in one of them.
See attached project.
The same in trunk.