Skip to main content

We want to create an integration test suite and run it in a self-contained world of headless containers

The high level goal is 

docker-compose to start all containers 

  • streamsets
  • kafka

configure streamsets 

send data to kafka topic 

StreamSets processes the data in the topic and send its to other topics 

Application under tests, processes the data (fails / passes) - the test

tear down  

Question:

How do I configure streamset pipeline without using the UI? 

Hi @haroon.said ,

There is SDK for Python available to configure StreamSets pipeline as well as many other UI operations to do programmatically.

 

Documentation for SDK 

SDK Tutorials

In fact, we use this SDK heavily along with STF (StreamSets Test Framework) for internal E2E integration testing. And, if interested, SDC tests are available here at datacollector-tests repo to take a look.

 

Regards-

Kirti

Reply


Terms & Conditions