Skip to main content
Solved

How to pass Table Name as an attribute while producing events from hadoop fs?

  • January 19, 2022
  • 4 replies
  • 130 views
A
Fan

My pipeline is configured to pick data from JDBC Multitable Consumer Origin and put it in Hadoop FS Destination. My requirement is to rename the output file at destination with TableName_TimeStamp.

Able to achieve TimeStamp using Expression Evaluator. How do I get TableName as an event passed from Hadoop Fs to be used in HDFS File Metadata?

Best answer by Abhishek Sarda

Thanks Pradeep. Achieved it via below method.

My file path was /<DBName>/<SchemaName>/<TableName>/

${file:pathElement(record:value('/filepath'),2)}

View original
Did this topic help you find an answer to your question?

4 replies

Pradeep
StreamSets Employee
Forum|alt.badge.img+1
  • PradeepStreamSets Employee
  • StreamSets Employee
  • 48 replies
  • January 21, 2022

You might want to look at Record Header attribute ‘jdbc.tables’ which has the list of table names referred in JDBC stage query. 

 

${record:attribute('jdbc.tables')} 

https://docs.streamsets.com/portal/datacollector/latest/help/datacollector/UserGuide/Origins/MultiTableJDBCConsumer.html#concept_ofs_p54_rkb

A
Fan

Thanks Pradeep. ${record:attribute('jdbc.tables')} is holding the table name till it reaches Hadoop FS destination. However, that when accessed in the events produced by Hadoop FS, it isn’t available.

Pradeep
StreamSets Employee
Forum|alt.badge.img+1
  • PradeepStreamSets Employee
  • StreamSets Employee
  • 48 replies
  • January 25, 2022

@Abhishek Sarda Since I do not have the pipeline, I am guessing what you can try. You can rename the files in ‘Hadoop FS’ destination itself? Try using record attribute/value for jdbc.tables, timestamp in ‘Files Prefix’ or ‘Files Suffix’ as needed. This way output files written via Hadoop FS will have the table name and timestamp. As the filename is passed in the event record of Hadoop FS you can update file properties as needed in HDFS File Metadata executor? 

Drew Kreiger
1 person likes this
  • Drew Kreiger
    Drew Kreiger
A
Fan

Thanks Pradeep. Achieved it via below method.

My file path was /<DBName>/<SchemaName>/<TableName>/

${file:pathElement(record:value('/filepath'),2)}

R
Pradeep
2 people like this
  • R
    Ranjith P
  • Pradeep
    Pradeep

Reply


Terms & Conditions