Product: StreamSets Transformer
Question:
Does Oracle JDBC origin support multiple offset columns?
Answer:
No, The Oracle JDBC Table origin uses a single offset column. The offset column should be a column in the table with unique values, such as a primary key or indexed column, which does not contain nulls.
Why?
Because spark assumes that it will read the whole table each time regardless where we put the min and max since the queries that it generates are open ended, e.g. the first split is ': "id" < 26 or "id" is null' whereas last split is "id" >= 76. We are adding the effective min/max to the query to bound the query and never go beyond of what we have originally set to do.
To sum up spark does split the query into ranges, hence that's the requirement for the PK or offset override column.
