Ghost inputs/outputs, generic implementation

DataStage has an interesting feature called "runtime column propagation". This means that a DS stage (like transfomer) takes whatever is thrown at it, say the result of an SQL query, reads its metadata and propagate it through the stage element. It is possible to use the actual data in the stage like in a transfomer.


You configure it be ticking a checkbox in the output tab. It might be a bit hidden though.

It is, however a double-edged sword. The up is that one can easily implement generic etl jobs. The down is that especially the undiscerning user can get puzzled by ghostly inputs/outputs. Connected to that you need to have memorised the structure getting propagated.

 

Kommentare

Kommentar veröffentlichen

Beliebte Posts aus diesem Blog

Error when checking operator: Could not find input field 'field_name'

Well, you checked and double-checked the existence of respective field and it is there! Yeah, stumbled over it again. And again! Possible solutions You need to compile jobs explicitly. You're done then. Try it. Go! What are you waiting for? If you are really extra sure compiled the job, then double-check the presence of the respective field again, but not in the stage DS tells you but in the very source at the start of the dataflow where this field comes from. It happens that if the column name of your SQL is not identical to the field name propagated. I do absolutely not know why: compilation does not reveal the problem DataStage misses the correct problem location by far
Mehr anzeigen

Missing record delimiter "\n", saw EOF instead

Actually, this warning is quite simple to fix. Apparently, DS expects the last record of a file to terminate with the record delimiter in any case - maybe there is a setting to alter this behaviour but I did not deem worthwhile the effort to look for it. So the solution is just to terminate the file on a single empty line. Obviously, this case is for Unix like line breaks. The others delimiters are handled analogously.
Mehr anzeigen