Skip to main content

Hello,

We’re trying to create a transformer pipeline with Origin: AmazonS3 , Destination: Azure ADLS and compute is on Azure Databricks. 

We tried to break is down into smaller pieces.

  1. S3->ADLS pipeline with Streamsets own cluster worked fine.
  2. Random data source to ADLS with Databricks compute worked fine. 

But when we try to bring in all 3 components together, we are getting stage library specific errors which are conflicting. We have tried different combinations from the list in the attached picture.

In this particular scenario, what are the stage libraries we need to choose? 

 

Query resolved.The pipeline worked well with the following libraries. 

>
    "streamsets-spark-azure-no-dependency-lib:5.2.0",
    "streamsets-spark-aws-no-dependency-lib:5.2.0",
    "streamsets-spark-basic-lib:5.2.0",
    "streamsets-spark-file-lib:5.2.0",
    "streamsets-spark-jdbc-lib:5.2.0"
]

Reply


Terms & Conditions