Skip to main content
Question

Dynamically creating folders in S3 using streamsets

  • January 13, 2022
  • 1 reply
  • 85 views

harshith
Discovered Fame

im using oracle as a source and s3 as destination.

im ingesting the records from the source and adding table name as a column through expression evalutaor. i want to use this table name and create a folder in s3 dynamically before dropping those records in s3 bucket

so the folder name should be created dynamically through streamsets fetching the name of the table. what should be the approach for the same?

for ex: if I'm fetching records from table abc , I need to create a folder called “abc” and drop all the records inside that folder.

1 reply

dima
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • January 13, 2022

If I’m not mistaken, S3 doesn't have a separate abstractions for folders; the name of an object delimited by slashes is what's interpreted as a folder hierarchy. You can set this name by setting the "Common prefix” or the “Object name prefix,” as appropriate for your use case.