Question

Connection management - is it possible to switch environments seamlessly?

  • 26 May 2023
  • 3 replies
  • 67 views
J
Rank

In comparison with Informatica and SSIS, I am puzzled by the connection management in Streamsets.

If I build a pipeline that reads from a MSSQL, and a connection for this server and database is created, I can use it as the source. However, at the beginning, I only want to connect to a DEV instance, and the connection is dedicated to it. After I finish testing and want to get into user acceptance testing (UAT), I can’t use this connection. It feels that my pipeline would have to be modified so that I use a connection that’s dedicated to UAT. But, this isn’t right! The pipeline shouldn’t need to be changed!

In Informatica or SSIS, when I need to switch from DEV to UAT, my “pipeline” doesn’t need to be changed. There is always a mechanism that enables the connection to be switched from DEV to UAT, seamlessly.

I imagine Steamsets also has a way to enable seamless switching of environments with regard to connections. But, I can’t find it. I would appreciate it very much if someone could share a tip with me.

If there isn’t a seamless mechanism to switch between environments, Steamsets should add this feature soon. Do you agree?

3 replies

wilsonshamim
Rank
Userlevel 2
Badge +1

@jyuan :

 

you can create parameters in pipeline for DB url and use those parameters in the connections.  For user and Passwords, you can make use of credential stores. Hope this helps

J
Rank

@jyuan :

 

you can create parameters in pipeline for DB url and use those parameters in the connections.  For user and Passwords, you can make use of credential stores. Hope this helps

We have been using parameters. This process is not seamless at all and that’s the reason for my raising this question here.

We haven’t used the credential store. Thanks for the tip! I will look into it. Since I don’t know if it offers seamless transition from one environment to another, I would like others to chime in. Does it? 

antmcmullen
Rank
Userlevel 2

Another Alternative, is to use runtime parameters, similar concept to Parameters but you set them at the engine level, so when you are in Dev it refers to a Dev Connection path/URL , Prod you get a Prod Connection path, etc

https://docs.streamsets.com/portal/datacollector/latest/help/datacollector/UserGuide/Pipeline_Configuration/RuntimeValues.html

 

But credentials should ideally be stored in credential stores

Reply


Terms & Conditions