Question:
We have a situation whereby we need to run a particular pipeline (standalone) as a certain user due to a locked down local filesystem permissions. The pipeline simply pulls data from a HTTP client. Basically looking for the creator of the pipeline to be the user that runs the pipeline on the OS rather than the “sdc” user.
How to achieve this?
Answer:
Going that route directly would mean running the pipeline in a different process which is not how SDC is designed to run. The StreamSets Data Collector service account (default SDC user) must have access to the local file system when writing a file using the LocalFS stage destination.
The workaround though is to write the file to a path on the same local file system, for which the SDC user has access. Then in StreamSets use an on close file event (generated by the LocalFS stage destination) to use a Shell stage executor (script process) to move the file to the required path location using the appropriate user and file permissions.