Skip to main content

I’m trying to write some simple output from a database to an Amazon S3 bucket.  I have the access key and shared secret setup properly, but what values do I need to use for bucket and path?  The documentation isn’t too clear and the rows seem to all go to Stage Error on a pipeline run.

@pkandra , would you please let us know:

  1. what error/s are you seeing in the stage error?
  2. The link to the documentation you are referring to please?

1: I’m new to StreamSets, so I’m unsure of how to see the exact error but have attached a screenshot.

 

 

2:  

Here’s the documentation I’m referring to, there’s no reference of what values to use for the bucket...is it the URL, the URI or something else?

Portal - StreamSets Docs

@pkandra ,

  1. For the bucket: here is the documentation https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Destinations/AmazonS3.html#concept_bnp_gwp_f1b
  2. Not sure where you are seeing the path (in s3 destination -stage configs)
  3. Related to stage errors: I am not sure how you have configured error-handling: depending on that you need to see errors: Here is documentation on the same https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Pipeline_Design/ErrorHandling.html#concept_pm4_txm_vq

 

Thanks Kirti, but that documentation doesn’t tell me if I should use the bucket URL, URI or something else.  I’ve tried every combination and it doesn’t seem to be working.

@pkandra ,

you can put the name of the bucket e.g. streamsets-bucket

 

Have you tried that? If yes, what is happening?

@pkandra 

Yes , you need to provide the bucket name and prefix and postfix details.

If you want the prefix is csv then dataformat should be csv , if json then data format format should be json etc.

Attached screenshot for you reference.

 

Reply


Terms & Conditions