Skip to main content
Solved

Reading SDC Error File from S3 Bucket


I have a pipeline that reads Kafka records and I’m writing any errors to an S3 bucket.  Most of the errors are related to tombstone records.  Once my original pipeline runs, I plan on having another pipeline process the error files on S3.  So far so good.  I can get the error code from the SDC file in the record:errorCode() field.  I also need the Kafka key from the S3 file.  What is the highlighted record/field called and how do I obtain the value with an expression evaluator or other processor?  Once I convert it to a map, that value is gone.

 

Best answer by pkandra

No worries @Bikram  I was able to get what I needed using an Expression Evaluator and the below.

 

 

View original
Did this topic help you find an answer to your question?

7 replies

Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • March 7, 2023

@pkandra 

can i get the error file with no sensitive data in it ,so i can try it from my end. .


  • Author
  • Opening Band
  • 19 replies
  • March 7, 2023

@Bikram  attached is a sample S3 error file in SDC format with 50 records


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • March 8, 2023

@pkandra 

Kindly check if the below code snippet works for you. The expression will work for this case only but not for other cases because I try to fix it by using the str:length() function with some hard coded values.

We can handle it by using any of the processors like (Jython or groovy etc) .

Please let me know if it helps.

Result  = ${str:substring(record:value('/text'),str:length(record:value('/text')) - 63,str:length(record:value('/text')) - 27)}

 

 

 

 

Thanks & Regards

Bikram_


  • Author
  • Opening Band
  • 19 replies
  • March 8, 2023

Hi @Bikram.  That doesn’t work for me.  I get the attached error.  My error file is in SDC format, but your looks like a map.

 

 


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • March 8, 2023

@pkandra 

 

I just kept the error file in s3 bucket and fetched all data from s3 with data format as text and by using expression evaluator got the above the result.

Can you please share your pipeline with no sensitive data in it ,so i can check it and will try to help you on the same.

 

Thanks & Regards

Bikram_


  • Author
  • Opening Band
  • 19 replies
  • Answer
  • March 8, 2023

No worries @Bikram  I was able to get what I needed using an Expression Evaluator and the below.

 

 


Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • March 8, 2023

@pkandra 

Nice to know that you managed to fix the issue , please mark the issue as resolved .

 

Thanks & Regards

Bikram_


Reply