Configuring AWS S3 Bucket

I’m trying to write some simple output from a database to an Amazon S3 bucket.  I have the access key and shared secret setup properly, but what values do I need to use for bucket and path?  The documentation isn’t too clear and the rows seem to all go to Stage Error on a pipeline run.

6 replies

Userlevel 2

@pkandra , would you please let us know:

  1. what error/s are you seeing in the stage error?
  2. The link to the documentation you are referring to please?

1: I’m new to StreamSets, so I’m unsure of how to see the exact error but have attached a screenshot.




Here’s the documentation I’m referring to, there’s no reference of what values to use for the it the URL, the URI or something else?

Portal - StreamSets Docs

Userlevel 2

@pkandra ,

  1. For the bucket: here is the documentation
  2. Not sure where you are seeing the path (in s3 destination -stage configs)
  3. Related to stage errors: I am not sure how you have configured error-handling: depending on that you need to see errors: Here is documentation on the same


Thanks Kirti, but that documentation doesn’t tell me if I should use the bucket URL, URI or something else.  I’ve tried every combination and it doesn’t seem to be working.

Userlevel 2

@pkandra ,

you can put the name of the bucket e.g. streamsets-bucket


Have you tried that? If yes, what is happening?

Userlevel 5
Badge +1


Yes , you need to provide the bucket name and prefix and postfix details.

If you want the prefix is csv then dataformat should be csv , if json then data format format should be json etc.

Attached screenshot for you reference.