Issue:
After upgrading the SDC from the version 2.7 to 3.0, you may see following warning message with cluster pipelines:
VALIDATION_0071 - Stage 'Write to DPM directly' from 'Basic' library does not support 'Cluster Batch' execution mode
Versions affected:
SDC 3.0.x.x
Solution:
This issue has been tracked in https://issues.streamsets.com/projects/SDC/issues/SDC-8069 and was resolved in SDC 3.1.0.0.
As a workaround, you can follow next steps:
1. If this is a production pipeline, you should always back up the pipeline.json file before modifying it.
2. Go to SDC_DATA directory and find pipeline.json file for the respective pipeline.
3. Open the pipeline.json file and search for "statsAggregatorStage"
- There are two places where statsAggregatorStage needs to be changed in the json file of the pipeline. The first place:
From:
"name" : "statsAggregatorStage",
"value" : "streamsets-datacollector-basic-lib::com_streamsets_pipeline_stage_destination_devnull_StatsDpmDirectlyDTarget::1"To:
"name" : "statsAggregatorStage",
"value" : "streamsets-datacollector-basic-lib::com_streamsets_pipeline_stage_destination_devnull_StatsNullDTarget::1"
- And in the second place of "statsAggregatorStage":
From:
"instanceName" : "WritetoDPMdirectly_StatsAggregatorStage",To:
"instanceName" : "Discard_StatsAggregatorStage",
From:
"stageName" : "com_streamsets_pipeline_stage_destination_devnull_StatsDpmDirectlyDTarget",
To:
"stageName" : "com_streamsets_pipeline_stage_destination_devnull_StatsNullDTarget",
From:
"label" : "Stats Aggregator - Write to DPM directly",
To:
"label" : "Stats Aggregator - Discard",
4. After these changes, save the file, refresh the SDC UI, or if you modified the file outside of the SDC_DATA directory, you can import the pipeline back to the SDC.
