Skip to main content

Show us your Pipelines

Share an example of your Pipeline pattern. Upload a Screenshot and detailed description. No wrong answers. Do NOT share personal or credential information.

  • 68 Topics
  • 73 Replies
68 Topics
CAVEMAN
Fan
CAVEMANFan
posted in  Show us your Pipelines
CAVEMAN

Hi

40
CAVEMAN
Fan
sujith
Fan
sujithFan
asked in  Show us your Pipelines
HAdoop to Azure Migration

How to read multipipe directory data from Hadoop using single pipeline?The below pipeline i have created but i am not sure how define the multiple directory values in the File tab Hadoop FS. Please help us .  

90
sujith
Fan
L
Fan
lotihig137Fan
asked in  Show us your Pipelines
getting Error: [Local FS 1 - Directory Template] Base directory path could not be created

I am trying to build a pipeline by following the steps mentioned in the streamsets docs- https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/Try.html#task_q3r_p2x_k4b but getting an issue while writing to the local fils system.the issue resides in the directory template [Directory Template info/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}codeHADOOPFS_41 - Base directory path could not be created]Can anyone help? Do we have a set of configuration that needs to be installed before running streamsets?

210
L
Fan
J
Fan
jojodoFan
posted in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.

160
J
Fan
J
Fan
jojodoFan
posted in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.  

80
J
Fan
B
Fan
bozo16Fan
asked in  Show us your Pipelines
"What are pipelines?" box gone.

Doing the tutorial, fragments section (https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals/lessons/172/lab-create-a-new-fragment) I’m invite to import a pipeline, but I don’t have the box “What are pipelines?” with the import link. Maybe I closed the box at some stage, but how doI get it back?Screenshot of my Pipelines 

392
B
Fan
P
Fan
PratikSinhaFan
asked in  Show us your Pipelines
Error: [Local FS 1 - Directory Template] Base directory path could not be created (HADOOPFS_41)

I am trying to build a pipeline by following the steps mentioned in the streamsets docs- https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/Try.html#task_q3r_p2x_k4b but getting an issue while writing to the local fils system.the issue resides in the directory template [Directory Template info/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}codeHADOOPFS_41 - Base directory path could not be created]Can anyone help? Do we have a set of configuration that needs to be installed before running streamsets?

290
P
Fan
M
Fan
malikhas15Fan
asked in  Show us your Pipelines
Help Needed: Issues Connecting Streamsets JMS Producer to Solace

I have a Streamsets pipeline where I am using the SQL Server CDC component and the JMS Producer component. My connection to SQL Server is working perfectly, and I am receiving data in my pipeline from SQL Server. However, I am encountering issues when trying to connect to Solace through the JMS connector.Initially, I received errors when attempting to connect to Solace. I then used the following parameters:- JMS Initial Context Factory: `com.solacesystems.jndi.SolJNDIInitialContextFactory`- JNDI Connection Factory: `/jms/cf/default`- JMS Provider URL: `tcp://pub:admin@172.17.0.4:55555`- JMS Destination Name: `cdc/demo/sqlserver`- JMS Destination Type: `Topic`- Solace JMS VPN: `test`The pipeline builds successfully, but when I go to the Solace "Try Me" tab and attempt to subscribe to this topic using the client "pub" I do not receive any messages.Additionally, I noticed something peculiar: when I use a client name that does not exist in Solace, the connection still builds successfully.

360
M
Fan
J
JnaneshwariNew Member
posted in  Show us your Pipelines
OBRUNNER_63 - Pipeline status: 'START_ERROR', Pool exhausted'

Hi Team, I am facing bellow error when i execute my pipeline . as a work around we increased ClassLoaders = 100 Still facing same issues  OBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'java.lang.RuntimeException: Could not get a private ClassLoader for 'streamsets-datacollector-azure-lib', for 'com_streamsets_pipeline_stage_metadata_gen2_DataLakeGen2MetadataDExecutor', active private ClassLoaders='50': java.util.NoSuchElementException: Pool exhausted'

80
J
moviepages
Fan
moviepagesFan
asked in  Show us your Pipelines
how to create the pipeline for the xml data where it is taken from sftp

how to create the pipeline for the xml data where it is taken from sftp in the streamsets we have getting only 1 record from sftp in the preview data 

402
roma
Fan
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Cannot Connect StreamSets with Python SDK, but Can connect via CURL command with same Credential ID and Token

Hi Team,I generate Credential ID and Token via StreamSets UI → Manage → API Credential, and with the generated ID and Token I can run Curl command, returned status is “HTTP/1.1 200 OK”, also return me a json format that show my organization id, email, extra.. However, When I want to connect with below code snippet (using same credential ID and Token), it returns me error of 403. paste code and error below:>>> from streamsets.sdk import ControlHub>>> sch = ControlHub(credential_id='absd_myid', token="abcd_mytoken.") Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch.py", line 141, in __init__    self.api_client = sch_api.ApiClient(  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch_api.py", line 96, in __init__    raise ValueError('Encountered error while decoding auth token: {}'.format(e))ValueError: Encountered error while decoding aut

581
Bikram
Headliner
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
How to Config to make Pipeline "Stop Event" not run when pipeline failed

Hi team,we have a pipeline, which configured a stop event, it let a sql statement to run once pipeline finish processing data,  however we found, when the pipeline failed due to some reason, this “Stop Event” still run, which is not expected for us. Can you please let me know if some place can be configured, which let the Pipeline "Stop Event" not run when pipeline failed processing data? I actually found if the pipelien “Start Event” failed, the pipeline will not run which is expected, however the “Stop Event” always run even pipeline failed. 

632
D
Discovered Fame
vishalsdc2024
vishalsdc2024New Member
asked in  Show us your Pipelines
Stream set pipeline is not starting

Hi , i am facing an issue while starting my sdc service and sdc is not coming up. it is prod and need to be up. can someone please help on this at the earliest. Caused by: com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0206 - Cannot load details for pipeline 'SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com': java.io.IOException: File '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-tmp' exists, '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-old' should exists we have referred link for solution but not sure what can be done as files are not there on those pipelines.  Thanks & Regards,Vishal Verma     

370
vishalsdc2024
C
ColinNew Member
asked in  Show us your Pipelines
Inquiry Regarding StreamSets Product Versions and Feature Differences

As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?

180
C
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

Hi, I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.can you let know anything wrong here? 

181
D
Discovered Fame
dima
StreamSets Employee
dimaStreamSets Employee
posted in  Show us your Pipelines
Analyzing chess games with batch and streaming pipelines

I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates

1431
R
Fan
Arjo
Fan
ArjoFan
asked in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

101
Bikram
Headliner
Arjo
Fan
ArjoFan
posted in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

40
Arjo
Fan
realmatcha
Opening Band
realmatchaOpening Band
asked in  Show us your Pipelines
JDBC ERROR

 Error:  Source :  hello everyone, I want to pull data from the source to the warehouse, in the source there is a "customer_id" column, when the streamsets are run why is there an invalid customer error? in streamsets I also flag based on the "id" column. please help, thank you

151
Bikram
Headliner
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
About Oracle CDC Client Configure

Hi team,I am configuring Oracle CDC Client now, our oracle version is 19C, non-CDB oracle.I ensure our oracle instance has enabled Database Archiving Mode, cause the return of statement "select log_mode from v$database" is "ARCHIVELOG". My question is for Step "Enable Supplemental Logging", since I have three "NO" for statement "select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;". Is it mandatory to run this "alter database add supplemental log data;" ?I actually do not want to make all database tables' Supplemental Logging, only want to enable it for some related tables, if this, should I only run like below "alter table <schema name>.<table name> add supplemental log data (all) columns;" , then run "alter system archive log current;" or I need to run below in sequence?Step 1:Enable minimal supplemental logging, run "alter database add supplemental log data;"Step 2:alter table <schema name>.<table name> add su

291
D
Discovered Fame
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Transformer for spark (Extract and Load).

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.Right now, I am using.1.JDBC query consumer (origin) 2.Snowflake (Destination)Note. In our sources we don’t have any primary key or Unique key So, not able to define offset column because of that not able achieve partition the data.   

120
hrishikeh132609
Fan
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.Can anyone help me understand this behavior. 

2115
lakshmi_narayanan_t
Discovered Fame
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Load data from Teradata to Snowflake

I am trying to load data from Teradata to Snowflake. I am using Transformer for spark as engine.Origin: JDBC query consumer. Destination:SnowflakeI am able to load 12 million data in 1 hour. but I want to improve the performance.Below is the configuration of my pipeline.   

431
Bikram
Headliner
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
Posting XML data

Hi Team,We are currently working on creating a pipeline which will consume Json data from Kafka and transform it into XML then will post the xml data using a particular REST API.Using Kafka stage we are able to consume the data successfully.Using Jython evaluator we are able to generate a desired XML from Json (can’t use ‘data generator’ as did not find a way add namespaces in the final XML).The output of Jython Evaluator is xml string. Now using HTTP client I have to post that XML to a particular API. In the HTTP Client I don’t see the option for xml in Data Format. How can I output data in XML format from Jython evaluator and how can I set the data format to XML.I have already tried the Content-Type = application/xml in headers but no luck. Regards,Dixit

400
D
Fan
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
how to create multiple job instance

Hi ,I have created Metadata Driven Data Ingestion Framework.Step 1:I am reading the Metadata from source and passing the runtime parameter(Database name and table name) to the job .Step 2:I am able to load one table at one time. Is there any way to create Mutiple job instance for each record? Please reach out to me If you any solution.email: hrishikeshe143143@gmail.com

360
hrishikeh132609
Fan

Badge winners

Show all badges
Terms & Conditions