Question

Pipeline fails with error - Failed to update metadata aft er 60000 ms.

  • 7 September 2022
  • 3 replies
  • 999 views

Hello,

Does somebody have experience with this issue?

I have pipeline with HTTP server on origin side and Kafka producer on destination side. The pipeline restarts more or less regularly with this error message:

 

2022-09-07 13:24:26,245 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:] [thread:ProductionPipelineRunnable-rx_syslog-rx_syslog] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, org.apache.kafka.common.errors.TimeoutException: Failed to update metadata aft
er 60000 ms.

There is number of following messages before failure

2022-09-07 13:24:23,594 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:4] [thread:kafka-producer-network-thread | topicMetadataClient] WARN  N
etworkClient - [Producer clientId=topicMetadataClient] Error while fetching metadata with correlation id 436 : {avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}
2022-09-07 13:24:23,694 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:4] [thread:kafka-producer-network-thread | topicMetadataClient] WARN N
etworkClient - [Producer clientId=topicMetadataClient] Error while fetching metadata with correlation id 437 : {avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}
2022-09-07 13:24:23,774 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:4] [thread:http-receiver-server:HTTPServer_01-19050] ERROR PipeRunner -
Failed executing stage 'com_streamsets_pipeline_stage_destination_kafka_KafkaDTarget_02': org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedExc
eption
2022-09-07 13:24:23,774 [user:*admin] [pipeline:rx_syslog/rx_syslog] [runner:] [thread:http-receiver-server:HTTPServer_01-19050] ERROR ProductionPip
elineRunner - Can't process batch

Other pipeline sending output to the same Kafka, but different topic doesn’t experience this issue.

Topic exists on Kafka and it’s accessible and I can read incoming messages via console-consumer when pipeline works correctly. I tried to delete and recreate topic, but issue still exists.

I also added config suggested in this topic, but it didn’t help either.

{
"key": "connections.max.idle.ms",
"value": "60000"
},
{
"key": "metadata.max.idle.ms",
"value": "60000"
},
{
"key": "metadata.max.age.ms",
"value": "60000"
}

Thank you.


3 replies

Userlevel 4
Badge

@DavidC 

The starting . for the topic name could be the problem.

I can see in the log:

 

{avro..syslog.viptela=UNKNOWN_TOPIC_OR_PARTITION}

Try removing the first dot and see if it works.

Thanks for suggestion. On Kafka side, only 2 non-alphanumerical characters in topic name are “.” and “-”

Topic name in Kafka producer on Streamsets side looks like this. Field “var” already exists within incoming messages.

 

avro.${record:value('/var')}.syslog.viptela

 

Userlevel 4
Badge

@DavidC I remember seeing this error myself in our case it was a special character in the topic name. Please can you check both the topic names, the one where it works and the one where it doesn’t. Please can you try recreating the topic with only characters.

Reply