Issue:
Getting below error in the SDC logs:
2020-03-23 14:00:39,454 [user:*abc [pipeline:xyz] [runner:0] [thread:ProductionPipelineRunnable-xyz] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '2020-03-23 08:00:32.000000' could not be written out: Server Error
2020-03-23 14:00:39,518 [user:*abc] [pipeline:xyz] [runner:0] [thread:ProductionPipelineRunnable-xyz] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '2020-03-22 15:38:51.424000' could not be written out: java.net.SocketTimeoutException: Read timed out
2020-03-23 12:30:09,047 [user:*abc [pipeline:xyz [runner:] [thread:sdcipc-server-1576408] WARN HttpChannel - /ipc/v1
java.lang.IllegalStateException: Queue full
Solution:
Due to this exception we are not able to write the updated batch and offset values and pipeline goes into RUNNING_ERROR which cause the offset reset
This issue occurs when the SDC RPC Origin or Destination is overloaded and system is not able to handle the requests either we need to do performance tuning for OS and Network
As workaround, we can increase the value for the Connection Timeout and Read Timeout properties in the SDC RPC Target
Also we can add Buffer between the stages which adds the delay.
