Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?
+1- Headliner
- 486 replies
May i know if you need to send the error messages in json format to Kafka immediately or if you can send it later it will fulfill your use case.
If that is the case then write the error messages to s3 as json and later create a pipeline to read the json file from s3 and send it to Kafka topic.
I hope you have tried with the below setups while sending error messages back to Kafka in your pipeline.
- Fan
- 2 replies
Hello Bikram, we’d like to send the error records in json format immediately to kafka topic.
Yes,I’ve used the setup in the screenshots you’ve provided while sending error messages back to Kafka in my pipeline.
- StreamSets Employee
- 53 replies
- The 1st pipeline writes the error records to a Kafka topic
- The 2nd pipeline reads from the Kafka topic(using ‘SDC record’ format) and push the data into the desired topic as JSON format
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.