Skip to main content
Solved

Are Connection Objects in DataOps Platform tied to Pipelines ONLY?

S
Headliner
Forum|alt.badge.img+1

Hi there,

We are migrating to DataOps Platform.

We run a Self-Managed Environment.

We use connection objects for our pipelines and our credential store is AWS.

Our structure is such that, ONLY Operations team can deploy pipelines to PROD deployment / engines.

 

We are new to using the DataOps Platform, we noticed that we could NOT change the connection to point to PROD sources at the time of creation of a job instance from an already tested pipeline. 

 

Our “Operations” team had to create a new version of the pipeline pointing to PROD connections and ONLY then they can create a Job instance and deploy. Note - The PROD connections are ONLY accessible by our Operations team.

 

It would have been nice, if we have the option to choose the values for the connection (objects) at the point of job instance creation. 

 

Please advise if there are any other alternatives.

 

P.S. I understand I can always switch to runtime parameters to meet my requirements but then I will have to completely ignore the concept of Connection Objects.

 

Thanks,

 

 

Best answer by wilsonshamim

Hi @Srinivasan Sankar :

 

You can use the same connection in multiple jobs. It can simply be achieved by using parameters

While creating a connection, use parameters which will be defined in the pipelines. 

 

you can override those parameters while creating the jobs. Hope it helps

View original
Srinivasan Sankar
Did this topic help you find an answer to your question?

wilsonshamim
StreamSets Employee
Forum|alt.badge.img+1

Hi @Srinivasan Sankar :

 

You can use the same connection in multiple jobs. It can simply be achieved by using parameters

While creating a connection, use parameters which will be defined in the pipelines. 

 

you can override those parameters while creating the jobs. Hope it helps

S
1 person likes this
  • S
    Srinivasan Sankar
Drew Kreiger
Rock star
Forum|alt.badge.img
  • Drew KreigerRock star
  • Senior Community Builder at StreamSets
  • March 7, 2022

@Srinivasan Sankar were @wilsonshamim suggestions helpful? 

S
Headliner
Forum|alt.badge.img+1

@wilsonshamim , @Drew Kreiger - the suggestion really helped.

 

I’ve tested one pipeline with a parameterized connection object and it worked as expected. I am yet to fully test our security model. This will definitely reduce the effort on our Ops Support team. Thanks. 

Srinivasan Sankar
Drew Kreiger
1 person likes this
  • Drew Kreiger
    Drew Kreiger
Drew Kreiger
Rock star
Forum|alt.badge.img
  • Drew KreigerRock star
  • Senior Community Builder at StreamSets
  • March 9, 2022

Great to hear, @Srinivasan Sankar. 

Reply


Terms & Conditions