Issue:
When executing an SDC pipeline that uses the Oracle CDC Client origin, the pipeline fails while attempting to ingest a record. The following error is seen in the logs, failing the pipeline:
2021-02-03 10:42:52,069 [user:xxxxxx] [pipeline:yyyyyyy] [runner:] [thread:Oracle CDC Data Generator] [stage:] ERROR OracleCDCSource - Error while reading data
java.lang.NullPointerException
at com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle.OracleCDCSource.addRecordsToQueue(OracleCDCSource.java:1165)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle.OracleCDCSource.generateRecords(OracleCDCSource.java:742)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle.OracleCDCSource.lambda$startGeneratorThread$5(OracleCDCSource.java:463)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-03 10:42:52,070 [user:xxxxxx] [pipeline:yyyyyyy] [runner:] [thread:ProductionPipelineRunnable-yyyyyy] [stage:] ERROR ProductionPipelineRunner - Pipeline execution failed
java.lang.NullPointerException: record cannot be null
Versions affected:
3.14.0 or earlier
Solution:
The true cause of the failure is actually masked by a NullPointerException, thus resulting in a misleading error message. Ultimately, the cause of the issue is the result of the SDC pipeline attempting to ingest record(s) from a table that contain unsupportede1] datatypes.
If the "Unsupported Field Type" configuration for the Oracle CDC Origin stage is set to either "Send Record to Error" or "Discard Record", the NPE seen in the error message above causes the pipeline to fail.
Encountering unsupported datatypes from the CDC tables being monitored should not crash the pipeline. This was tracked as an issue via SDC-1493892], and was resolved in SDC3.17+.
As a workaround, you can either change the configuration of the Oracle CDC Client origin and set the "Unsupported Field Type" property to "Pass Record to Pipeline", or you can skip/remove tables that might contain unsupported datatypes.