Skip to main content

Hello,

Does somebody have experience with this issue?

I have pipeline with HTTP server on origin side and Kafka producer on destination side. The pipeline restarts more or less regularly with this error message:

 

2022-09-07 13:24:26,245  user:*admin]  pipeline:rx_syslog/rx_syslog]  runner:]  thread:ProductionPipelineRunnable-rx_syslog-rx_syslog] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, org.apache.kafka.common.errors.TimeoutException: Failed to update metadata aft
er 60000 ms.

There is number of following messages before failure

2022-09-07 13:24:23,594  user:*admin]  pipeline:rx_syslog/rx_syslog]  runner:4]  thread:kafka-producer-network-thread | topicMetadataClient] WARN  N
etworkClient -  Producer clientId=topicMetadataClient] Error while fetching metadata with correlation id 436 : {avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}
2022-09-07 13:24:23,694  user:*admin]  pipeline:rx_syslog/rx_syslog]  runner:4]  thread:kafka-producer-network-thread | topicMetadataClient] WARN  N
etworkClient -  Producer clientId=topicMetadataClient] Error while fetching metadata with correlation id 437 : {avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}
2022-09-07 13:24:23,774  user:*admin]  pipeline:rx_syslog/rx_syslog]  runner:4]  thread:http-receiver-server:HTTPServer_01-19050] ERROR PipeRunner -
 Failed executing stage 'com_streamsets_pipeline_stage_destination_kafka_KafkaDTarget_02': org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedExc
eption
2022-09-07 13:24:23,774  user:*admin]  pipeline:rx_syslog/rx_syslog]  runner:]  thread:http-receiver-server:HTTPServer_01-19050] ERROR ProductionPip
elineRunner - Can't process batch

Other pipeline sending output to the same Kafka, but different topic doesn’t experience this issue.

Topic exists on Kafka and it’s accessible and I can read incoming messages via console-consumer when pipeline works correctly. I tried to delete and recreate topic, but issue still exists.

I also added config suggested in this topic, but it didn’t help either.

{
    "key": "connections.max.idle.ms",
    "value": "60000"
},
{
    "key": "metadata.max.idle.ms",
    "value": "60000"
},
{
    "key": "metadata.max.age.ms",
    "value": "60000"
}

Thank you.

@DavidC I remember seeing this error myself in our case it was a special character in the topic name. Please can you check both the topic names, the one where it works and the one where it doesn’t. Please can you try recreating the topic with only characters.

Thanks for suggestion. On Kafka side, only 2 non-alphanumerical characters in topic name are “.” and “-”

Topic name in Kafka producer on Streamsets side looks like this. Field “var” already exists within incoming messages.

 

avro.${record:value('/var')}.syslog.viptela

 

@DavidC 

The starting . for the topic name could be the problem.

I can see in the log:

 

{avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}

Try removing the first dot and see if it works.

Reply


Terms & Conditions