Skip to main content

Hello,

We are getting following messages in sdc.log.

What could be an issue here? Pipeline monitoring in GUI doesn't show errors. We tried to increase Connection timeout of SDC RPC Target and issue still persists.

I also noticed, that some messages doesn't contain any offset number (first message). Might it be related?

[thread:ProductionPipelineRunnable-pipelineName1[stage:SDCRPC_1] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '' could not be written out: Server Error

2022-02-09 13:56:14,492 [user:admin] [pipeline:pipelineName2] [runner:0] [thread:ProductionPipelineRunnable-PipelineName2] [stage:SDCRPC_2] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '168' could not be written out: Server Error

Thank you.

 

@DavidC  This WARN message indicates that an RPC endpoint is unable to process the request on that attempt and it will retry that attempt.

if there were too many RPCs being sent against common SDC RPC destinations, it could be resulting in full IPC handler queues.

In order to reduce the number of WARN messages, the concurrent load against the SDC RPC endpoints should be reduced or the number of targeted SDC RPC endpoints should be increased and evenly spread out

Hi @Rishi 

Thank you for reply. This problem occurs on multiple pipelines and some of them are really multiple RPCs sending to single destination RPC.

It’s strange, that there is also pipeline, where isn’t any incoming traffic and SDC is still showing this message:

 

2022-02-25 14:35:06,989 6user:XXXX] Xpipeline:pipeline4] irunner:] nthread:sdcipc-server-266] WARN  HttpChannel - /ipc/v1 java.lang.IllegalStateException: Queue full

 

Pipeline is running for 4 hours and this is it’s statictic.

 

Reply


Terms & Conditions