StreamSets ControlHub On-prem 3.18.1 script change during upgrade.


Userlevel 4
Badge

With StreamSets Control Hub On-prem version 3.18.1, the database initialization script can take a longer time to run than in previous upgrades. The amount of time depends on the total number of pipeline versions or commits in the pipelinestore database.

 

The 3.18 pipeline upgrade script moves the blob column (PIPELINE_DEFINITION) from the PIPELINE table to a new PIPELINE_BLOB. The upgrade script moves all the blobs from the old table to the new table and alters the old table to add a join column. Depending on the number of pipelines in the table, type, and RDBMS size, the script would take a long time then before.

 

NOTE: Such a large maintenance window will apply only to customers with a vast number of pipelines (>10000). 

 

For Example:

  • With 10,000 pipeline versions, the script can take a few minutes to half an hour to run.
  • With 100,000 pipeline versions, the script can take one to two hours to run. 

 

To know the number of pipelines in the PIPELINE table, one can do “select count(*) from PIPELINE.”

You can find more information in our documentation here.

 

If you have any additional queries or concerns, please contact our support team via our Zendesk support portal or at support@streamsets.com.


0 replies

Be the first to reply!

Reply