Skip to main content

I want to create a parameter that I can use across multiple pipelines.

For example, I have multiple pipelines that are configured to run on EMR cluster.

Instead of providing the same EMR configuration values for all the pipelines, if there is a way for me to create parameters globally that I can refer to, in each of the relevant pipelines.

Thanks !

@srinivasa_nanduru, the only way I can think of to achieve the effect of a global variable is to store the value in something like Thycotic Secret Server or Azure Key Vault and then use the ${credential:get} command to pull the value. I cannot say this actually works as I have never tried it myself for anything other than credentials, but it may be worth a shot.


@srinivasa_nanduru  in my opinion, the best way to achieve that would be to use a runtime variable, You would reference these keys within your Engine configuration, although i wouldnt store any sensitive values here as they would be in plain text, I would use a Key Vault / Credential:get() in that instance

 

but for non sensitive items you can follow https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Pipeline_Configuration/RuntimeValues.html#concept_fjx_g31_1s


Reply