Skip to main content
Question

Global parameter

  • 12 July 2024
  • 2 replies
  • 26 views

I want to create a parameter that I can use across multiple pipelines.

For example, I have multiple pipelines that are configured to run on EMR cluster.

Instead of providing the same EMR configuration values for all the pipelines, if there is a way for me to create parameters globally that I can refer to, in each of the relevant pipelines.

Thanks !

2 replies

Sperch
Rank

@srinivasa_nanduru, the only way I can think of to achieve the effect of a global variable is to store the value in something like Thycotic Secret Server or Azure Key Vault and then use the ${credential:get} command to pull the value. I cannot say this actually works as I have never tried it myself for anything other than credentials, but it may be worth a shot.

antmcmullen
Rank
Userlevel 2

@srinivasa_nanduru  in my opinion, the best way to achieve that would be to use a runtime variable, You would reference these keys within your Engine configuration, although i wouldnt store any sensitive values here as they would be in plain text, I would use a Key Vault / Credential:get() in that instance

 

but for non sensitive items you can follow https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Pipeline_Configuration/RuntimeValues.html#concept_fjx_g31_1s

Reply


Terms & Conditions