When will StreamSets introduce the ability to use Git repos such as Github or GitLab for version control?
Hi
StreamSets Control Hub has version control for pipelines through its pipeline repository feature.
Not sure I understand. Does that allow me to use my own GitHub repo for version control?
It doesn't, but unlike source code where developers directly modify it and where diffs are human-readable, StreamSets pipelines are JSONs that aren't intended to be inspected by eye. As such, while I know of some Control Hub users who have setup automation that pushes pipeline JSONs to external version control systems, they use them more as a backup. Meanwhile, the pipeline repository feature actually lets you see which changes happened between pipeline versions, lets you branch them off, rollback, etc. in a much more full-featured way.
Can you elaborate a bit more on the use case you're interested in? I could provide more detail if I understand it a bit better.
If I am correct, the idea is to be able to combine the pipeline definitions into the repo with other code and incorporate it all into a CI/CD pipeline for automated testing.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.