So if you ever designed a Transformer pipeline with Cluster Manager Type EMR you are required to specify S3 Staging URI and directory.
Why you need a Staging Directory?
To run pipelines on an EMR cluster, Transformer must store files on Amazon S3.
Transformer stores libraries in the following location: <S3 staging URI>/<staging directory>.
The location must exist before you start the pipeline. You define the location using the following pipeline configuration properties:
- Staging Directory on the Cluster tab
- S3 Staging URI on the EMR tab
Transformer stores the following files in the specified location:
Files that can be reused across pipelines
Transformer stores files that can be reused across pipelines, including Transformer libraries and external resources such as JDBC drivers, in the following location:
For example, say you use s3://mybucket as the S3 staging URI and the default /streamsets staging directory for a Transformer 3.14.0 pipeline. Then, Transformer stores the reusable files in the following location:
s3://mybucket/streamsets/3.14.0
Files specific to each pipeline
Transformer stores files specific to each pipeline, such as the pipeline JSON file and resource files used by the pipeline, in the following directory:
For example, say you use s3://mybucket as the S3 staging URI and the default /streamsets staging directory to run a pipeline named KafkaToJDBC.
Transformer stores pipeline-specific files in a directory like the following:
s3://mybucket/streamsets/staging/KafkaToJDBC03a0d2cc-f622-4a68-b161-7f2d9a4f3052/run1557350076328
Can we use the same EMR S3 stating location for multiple jobs in Transformer?
Yes, You can use the same staging dir.
However, If you have multiple instances of Transformers that are configured to use different library versions, you might specify a different S3 staging URI or staging directory to avoid using the same staging location. For example, if you have two 3.13.0 Transformers, each using a different Oracle JDBC driver. To allow each Transformer to use its own driver version, specify different staging locations for those pipelines.
