Technical Service Bulletin 2023-01-27 (TSB) StreamSets Control Hub - Incident with Engines Provisioned via Kubernetes Provisioning Agent

StreamSets Support team would like to inform you that we have conducted a thorough root cause analysis for the issue with Data Collector engines provisioned via Kubernetes Provisioning Agent after Control Hub release on January 27, 2023.

Products Affected: StreamSets Control Hub Cloud (SCH)

Users Affected: StreamSets Control Hub Cloud (SCH) users using Data Collector engines provisioned via Kubernetes Provisioning Agent

Severity: High

Description: Data Collector provisioned via Kubernetes Provisioning Agent may have been unregistered and deleted from user’s organization after Control Hub release at 10:30 p.m. PT on January 27, 2023. 
  
StreamSets deployed a configuration change to disable this incorrect behavior at 7:30 a.m. PT on January 28, 2023, and will be deploying a code change as part of an emergency release.

Jobs running in these Data Collectors may have not synchronized offsets, because of that, there is a possibility of data duplication.

Immediate action required: Yes. Users using engines provisioned via Kubernetes Provisioning Agent must restart deployments provisioned via the control agent as soon as possible. Affected users could do a gradual rollout:
kubectl rollout restart deployment <deployment name> -n <namespace>

Fix: We have completed our root cause analysis. The underlying issue was related to a change added in the release deployed on January 27, 2023. The change contained an improved retry mechanism to how status events are processed from control agent in case of a transaction failure, which resulted in incorrectly determining that some of Data Collectors were not existing, even though they were active.

The StreamSets team is taking the necessary steps to avoid a recurrence of the issue and is planning to deploy a fix during an emergency release on Monday, January 30, 2023.


For updates and questions, please visit our website trust.streamsets.com or contact our Support team via our Support portal or at support@streamsets.com.