Skip to main content

Converting Data to the Parquet Data Format with dynamic path.

  • January 20, 2022
  • 0 replies
  • 58 views

AkshayJadhav
StreamSets Employee
Forum|alt.badge.img

Product: StreamSets Data Collector 

 

Issue:

We have this solution here in order to convert Avro to Parquet with a static output path for the parquets file. If we need, instead, to copy the parquets to store the converted parquet file into the same directory as the input generated dynamically, we won't be able to use record:value('/filepath')

 

Solution:

We can use record:value('/filepath')  with  FileEL. E.g: The output directory for parquet may be the following : 

${file:parentPath(file:parentPath(record:value('/filepath'))}

 

 

This topic has been closed for replies.