Question

Handling the error records in the same Kafka topic

  • 25 July 2023
  • 3 replies
  • 81 views
nikkichow
Rank

Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages  . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?

3 replies

Bikram
Rank
Userlevel 5
Badge +1

@nikkichow 

May i know if you need to send the error messages in json format to Kafka immediately or if you can send it later it will fulfill your use case.

If that is the case then write the error messages to s3 as json and later create a pipeline to read the json file from s3 and send it to Kafka topic.

 

I hope you have tried with the below setups while sending error messages back to Kafka in your pipeline.

 

 

nikkichow
Rank

Hello Bikram, we’d like to send the error records in json format immediately to kafka topic.

 

Yes,I’ve used the setup in the screenshots you’ve provided while sending error messages back to Kafka in my pipeline.

Sanjeev
Rank
Userlevel 2
Badge

@nikkichow SDC record is StreamSets proprietary data format and therefore trying to read it directly via an external application is not going to work. Please consider the two pipeline solution outlined below:

  1.  The 1st pipeline writes the error records to a Kafka topic
  2.  The 2nd pipeline reads from the Kafka topic(using ‘SDC record’ format) and push the data into the desired topic as JSON format

Reply


Terms & Conditions