Skip to main content

Converting Data to the Parquet Data Format with dynamic path.


AkshayJadhav
StreamSets Employee
Forum|alt.badge.img

Product: StreamSets Data Collector 

 

Issue:

We have this solution here in order to convert Avro to Parquet with a static output path for the parquets file. If we need, instead, to copy the parquets to store the converted parquet file into the same directory as the input generated dynamically, we won't be able to use record:value('/filepath')

 

Solution:

We can use record:value('/filepath')  with  FileEL. E.g: The output directory for parquet may be the following : 

${file:parentPath(file:parentPath(record:value('/filepath'))}

 

 

Did this topic help you find an answer to your question?
This topic has been closed for comments