Skip to main content

Show us your Pipelines

Share an example of your Pipeline pattern. Upload a Screenshot and detailed description. No wrong answers. Do NOT share personal or credential information.

  • 63 Topics
  • 73 Replies
63 Topics
B
Fan
bozo16Fan
asked in  Show us your Pipelines
"What are pipelines?" box gone.

Doing the tutorial, fragments section (https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals/lessons/172/lab-create-a-new-fragment) I’m invite to import a pipeline, but I don’t have the box “What are pipelines?” with the import link. Maybe I closed the box at some stage, but how doI get it back?Screenshot of my Pipelines 

362
B
Fan
23 days ago
P
Fan
PratikSinhaFan
asked in  Show us your Pipelines
Error: [Local FS 1 - Directory Template] Base directory path could not be created (HADOOPFS_41)

I am trying to build a pipeline by following the steps mentioned in the streamsets docs- https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/Try.html#task_q3r_p2x_k4b but getting an issue while writing to the local fils system.the issue resides in the directory template [Directory Template info/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}codeHADOOPFS_41 - Base directory path could not be created]Can anyone help? Do we have a set of configuration that needs to be installed before running streamsets?

140
P
Fan
3 months ago
M
Fan
malikhas15Fan
asked in  Show us your Pipelines
Help Needed: Issues Connecting Streamsets JMS Producer to Solace

I have a Streamsets pipeline where I am using the SQL Server CDC component and the JMS Producer component. My connection to SQL Server is working perfectly, and I am receiving data in my pipeline from SQL Server. However, I am encountering issues when trying to connect to Solace through the JMS connector.Initially, I received errors when attempting to connect to Solace. I then used the following parameters:- JMS Initial Context Factory: `com.solacesystems.jndi.SolJNDIInitialContextFactory`- JNDI Connection Factory: `/jms/cf/default`- JMS Provider URL: `tcp://pub:admin@172.17.0.4:55555`- JMS Destination Name: `cdc/demo/sqlserver`- JMS Destination Type: `Topic`- Solace JMS VPN: `test`The pipeline builds successfully, but when I go to the Solace "Try Me" tab and attempt to subscribe to this topic using the client "pub" I do not receive any messages.Additionally, I noticed something peculiar: when I use a client name that does not exist in Solace, the connection still builds successfully.

200
M
Fan
3 months ago
J
JnaneshwariNew Member
posted in  Show us your Pipelines
OBRUNNER_63 - Pipeline status: 'START_ERROR', Pool exhausted'

Hi Team, I am facing bellow error when i execute my pipeline . as a work around we increased ClassLoaders = 100 Still facing same issues  OBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'java.lang.RuntimeException: Could not get a private ClassLoader for 'streamsets-datacollector-azure-lib', for 'com_streamsets_pipeline_stage_metadata_gen2_DataLakeGen2MetadataDExecutor', active private ClassLoaders='50': java.util.NoSuchElementException: Pool exhausted'

70
J
4 months ago
moviepages
Fan
moviepagesFan
asked in  Show us your Pipelines
how to create the pipeline for the xml data where it is taken from sftp

how to create the pipeline for the xml data where it is taken from sftp in the streamsets we have getting only 1 record from sftp in the preview data 

362
roma
Fan
5 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Cannot Connect StreamSets with Python SDK, but Can connect via CURL command with same Credential ID and Token

Hi Team,I generate Credential ID and Token via StreamSets UI → Manage → API Credential, and with the generated ID and Token I can run Curl command, returned status is “HTTP/1.1 200 OK”, also return me a json format that show my organization id, email, extra.. However, When I want to connect with below code snippet (using same credential ID and Token), it returns me error of 403. paste code and error below:>>> from streamsets.sdk import ControlHub>>> sch = ControlHub(credential_id='absd_myid', token="abcd_mytoken.") Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch.py", line 141, in __init__    self.api_client = sch_api.ApiClient(  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch_api.py", line 96, in __init__    raise ValueError('Encountered error while decoding auth token: {}'.format(e))ValueError: Encountered error while decoding aut

441
Bikram
Headliner
6 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
How to Config to make Pipeline "Stop Event" not run when pipeline failed

Hi team,we have a pipeline, which configured a stop event, it let a sql statement to run once pipeline finish processing data,  however we found, when the pipeline failed due to some reason, this “Stop Event” still run, which is not expected for us. Can you please let me know if some place can be configured, which let the Pipeline "Stop Event" not run when pipeline failed processing data? I actually found if the pipelien “Start Event” failed, the pipeline will not run which is expected, however the “Stop Event” always run even pipeline failed. 

522
D
Discovered Fame
6 months ago
vishalsdc2024
vishalsdc2024New Member
asked in  Show us your Pipelines
Stream set pipeline is not starting

Hi , i am facing an issue while starting my sdc service and sdc is not coming up. it is prod and need to be up. can someone please help on this at the earliest. Caused by: com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0206 - Cannot load details for pipeline 'SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com': java.io.IOException: File '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-tmp' exists, '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-old' should exists we have referred link for solution but not sure what can be done as files are not there on those pipelines.  Thanks & Regards,Vishal Verma     

270
vishalsdc2024
6 months ago
C
ColinNew Member
asked in  Show us your Pipelines
Inquiry Regarding StreamSets Product Versions and Feature Differences

As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?

160
C
6 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

Hi, I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.can you let know anything wrong here? 

141
D
Discovered Fame
6 months ago
dima
StreamSets Employee
dimaStreamSets Employee
posted in  Show us your Pipelines
Analyzing chess games with batch and streaming pipelines

I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates

1061
R
Fan
6 months ago
Arjo
Fan
ArjoFan
asked in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

101
Bikram
Headliner
6 months ago
Arjo
Fan
ArjoFan
posted in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

40
Arjo
Fan
6 months ago
realmatcha
Opening Band
realmatchaOpening Band
asked in  Show us your Pipelines
JDBC ERROR

 Error:  Source :  hello everyone, I want to pull data from the source to the warehouse, in the source there is a "customer_id" column, when the streamsets are run why is there an invalid customer error? in streamsets I also flag based on the "id" column. please help, thank you

141
Bikram
Headliner
7 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
About Oracle CDC Client Configure

Hi team,I am configuring Oracle CDC Client now, our oracle version is 19C, non-CDB oracle.I ensure our oracle instance has enabled Database Archiving Mode, cause the return of statement "select log_mode from v$database" is "ARCHIVELOG". My question is for Step "Enable Supplemental Logging", since I have three "NO" for statement "select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;". Is it mandatory to run this "alter database add supplemental log data;" ?I actually do not want to make all database tables' Supplemental Logging, only want to enable it for some related tables, if this, should I only run like below "alter table <schema name>.<table name> add supplemental log data (all) columns;" , then run "alter system archive log current;" or I need to run below in sequence?Step 1:Enable minimal supplemental logging, run "alter database add supplemental log data;"Step 2:alter table <schema name>.<table name> add su

261
D
Discovered Fame
7 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Transformer for spark (Extract and Load).

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.Right now, I am using.1.JDBC query consumer (origin) 2.Snowflake (Destination)Note. In our sources we don’t have any primary key or Unique key So, not able to define offset column because of that not able achieve partition the data.   

90
hrishikeh132609
Fan
8 months ago
antmcmullen
StreamSets Employee
antmcmullenStreamSets Employee
posted in  Show us your Pipelines
Ordinance Survey - Address Base Premium Processing mutli-schema documents easily

Ordnance Survey (OS) produce a suite of products for the UK Market, One of these is Address Base Premium (ABP), Which is a set of details that describe buildings, businesses, residential and items you’d find on a map in detail, such as Lat/Long data, how the royal mail refers to it, and how the local authorities/councils describe and classify those items.Sounds good right? Well its not so nice of a set of data to deal with, It is shipped in 5 km batches of data in a single csv file, across 10 different schema patterns within it.. Nightmare? Nope! You know the secret of streamsets!Streamsets doesn't care about schema on read, That is the key to unlocking this… In the ABP Files, the first column has a record identifier, This tells us which schema and rules to process.This pipeline makes such a difficult issue to deal with normally, clear and transparent.You can watch a record come in from the raw file, read it with no header, Look at the first column. Process that to a given lane, That l

600
N
8 months ago
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.Can anyone help me understand this behavior. 

1385
lakshmi_narayanan_t
Discovered Fame
8 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Load data from Teradata to Snowflake

I am trying to load data from Teradata to Snowflake. I am using Transformer for spark as engine.Origin: JDBC query consumer. Destination:SnowflakeI am able to load 12 million data in 1 hour. but I want to improve the performance.Below is the configuration of my pipeline.   

291
Bikram
Headliner
8 months ago
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
Posting XML data

Hi Team,We are currently working on creating a pipeline which will consume Json data from Kafka and transform it into XML then will post the xml data using a particular REST API.Using Kafka stage we are able to consume the data successfully.Using Jython evaluator we are able to generate a desired XML from Json (can’t use ‘data generator’ as did not find a way add namespaces in the final XML).The output of Jython Evaluator is xml string. Now using HTTP client I have to post that XML to a particular API. In the HTTP Client I don’t see the option for xml in Data Format. How can I output data in XML format from Jython evaluator and how can I set the data format to XML.I have already tried the Content-Type = application/xml in headers but no luck. Regards,Dixit

320
D
Fan
9 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
how to create multiple job instance

Hi ,I have created Metadata Driven Data Ingestion Framework.Step 1:I am reading the Metadata from source and passing the runtime parameter(Database name and table name) to the job .Step 2:I am able to load one table at one time. Is there any way to create Mutiple job instance for each record? Please reach out to me If you any solution.email: hrishikeshe143143@gmail.com

320
hrishikeh132609
Fan
9 months ago
nuwancml
Fan
nuwancmlFan
asked in  Show us your Pipelines
Init query in the JDBC Query Consumer not working for Delete previous data

  I use Streamsets Data Collector to Load data from Stage 2 database tables (JDBC Query Consumer) using a query and Write loaded data to another Stage 2 Database table (JDBC Producer). I use Init Query as below to delete the previous records before loading data. But this does not delete any record from the table. It would be great if someone can help me. 

502
nuwancml
Fan
1 year ago
knsujith
Fan
knsujithFan
asked in  Show us your Pipelines
Error while connecting solace event broker

HiI am trying to connect to Solace event broker using JMS Producer and  configured all JMS parameters (user name and password ) but still pipeline throwing below error:RETRY: JMS_00 - Could not create initial context 'com.solacesystems.jndi.SolJNDIInitialContextFactory' with provider URL 'tcps://mr-connection-irj89q7fz0j.messaging.solace.cloud:55443' : javax.naming.NamingException: Username must be specified I have already uploaded all solace lib files in external resources section of engine.Please suggest.

261
lakshmi_narayanan_t
Discovered Fame
1 year ago
drvynguyencosmetic
Fan
drvynguyencosmeticFan
asked in  Show us your Pipelines
drvynguyencosmetic

Với cam kết đem đến sự lựa chọn tốt nhất cho khách hàng, Drvynguyencosmetic mang đến một kho lựa chọn ấn tượng từ các thương hiệu hàng đầu như: SVR: Sự ưa chuộng của các chuyên gia da liễu với sản phẩm chất lượng cao. Hãy truy cập ngay https://drvynguyencosmetic.com/ để tìm hiểu rõ hơn về những thông tin trên nhé. Thông tin liên hệ: SĐT: 0931510129Xem thêm:https://drvynguyencosmetic.com/https://www.facebook.com/drvynguyencosmetichttps://www.instagram.com/drvynguyencosmetichttps://twitter.com/dr_vynguyenhttps://www.youtube.com/@drvynguyen.cosmetic/abouthttps://www.flickr.com/people/drvynguyencosmetic/https://www.pinterest.com/drvynguyencosmetic/https://www.tumblr.com/drvynguyencosmetichttps://myspace.com/drvynguyencosmetic

30
drvynguyencosmetic
Fan
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in  Show us your Pipelines
how to use Google adress api to validate adress column or records

I want to validate the address column by using google address api  how to do that .this is my record : I want to validate the address .how to do any one can help this scenario.

241
Bikram
Headliner
1 year ago

Badge winners

Show all badges
Terms & Conditions