Question

Alternate Git Repo Integration

  • 4 January 2022
  • 5 replies
  • 32 views
mblahay
Rank

When will StreamSets introduce the ability to use Git repos such as Github or GitLab for version control?

5 replies

dima
Rank
Userlevel 2
Badge

Hi @mblahay,

StreamSets Control Hub has version control for pipelines through its pipeline repository feature.

mblahay
Rank

Not sure I understand. Does that allow me to use my own GitHub repo for version control?

dima
Rank
Userlevel 2
Badge

It doesn't, but unlike source code where developers directly modify it and where diffs are human-readable, StreamSets pipelines are JSONs that aren't intended to be inspected by eye. As such, while I know of some Control Hub users who have setup automation that pushes pipeline JSONs to external version control systems, they use them more as a backup. Meanwhile, the pipeline repository feature actually lets you see which changes happened between pipeline versions, lets you branch them off, rollback, etc. in a much more full-featured way.
 

Can you elaborate a bit more on the use case you're interested in? I could provide more detail if I understand it a bit better. 

mblahay
Rank

If I am correct, the idea is to be able to combine the pipeline definitions into the repo with other code and incorporate it all into a CI/CD pipeline for automated testing.

dima
Rank
Userlevel 2
Badge

So we actually have a way of doing this kind of thing with functionality that’s already in the StreamSets DataOps Platform. Feel free to DM me on the community Slack (streamsetters.slack.com) and I can walk you through it because it’s just a bit involved for this thread.

Reply