Skip to main content

Show us your Pipelines

Share an example of your Pipeline pattern. Upload a Screenshot and detailed description. No wrong answers. Do NOT share personal or credential information.

  • 74 Topics
  • 73 Replies
74 Topics
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
About Oracle CDC Client Configure

Hi team,I am configuring Oracle CDC Client now, our oracle version is 19C, non-CDB oracle.I ensure our oracle instance has enabled Database Archiving Mode, cause the return of statement "select log_mode from v$database" is "ARCHIVELOG". My question is for Step "Enable Supplemental Logging", since I have three "NO" for statement "select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;". Is it mandatory to run this "alter database add supplemental log data;" ?I actually do not want to make all database tables' Supplemental Logging, only want to enable it for some related tables, if this, should I only run like below "alter table <schema name>.<table name> add supplemental log data (all) columns;" , then run "alter system archive log current;" or I need to run below in sequence?Step 1:Enable minimal supplemental logging, run "alter database add supplemental log data;"Step 2:alter table <schema name>.<table name> add su

271
D
Discovered Fame
9 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Transformer for spark (Extract and Load).

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.Right now, I am using.1.JDBC query consumer (origin) 2.Snowflake (Destination)Note. In our sources we don’t have any primary key or Unique key So, not able to define offset column because of that not able achieve partition the data.   

100
hrishikeh132609
Fan
10 months ago
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.Can anyone help me understand this behavior. 

1635
lakshmi_narayanan_t
Discovered Fame
10 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Load data from Teradata to Snowflake

I am trying to load data from Teradata to Snowflake. I am using Transformer for spark as engine.Origin: JDBC query consumer. Destination:SnowflakeI am able to load 12 million data in 1 hour. but I want to improve the performance.Below is the configuration of my pipeline.   

351
Bikram
Headliner
10 months ago
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
Posting XML data

Hi Team,We are currently working on creating a pipeline which will consume Json data from Kafka and transform it into XML then will post the xml data using a particular REST API.Using Kafka stage we are able to consume the data successfully.Using Jython evaluator we are able to generate a desired XML from Json (can’t use ‘data generator’ as did not find a way add namespaces in the final XML).The output of Jython Evaluator is xml string. Now using HTTP client I have to post that XML to a particular API. In the HTTP Client I don’t see the option for xml in Data Format. How can I output data in XML format from Jython evaluator and how can I set the data format to XML.I have already tried the Content-Type = application/xml in headers but no luck. Regards,Dixit

360
D
Fan
11 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
how to create multiple job instance

Hi ,I have created Metadata Driven Data Ingestion Framework.Step 1:I am reading the Metadata from source and passing the runtime parameter(Database name and table name) to the job .Step 2:I am able to load one table at one time. Is there any way to create Mutiple job instance for each record? Please reach out to me If you any solution.email: hrishikeshe143143@gmail.com

320
hrishikeh132609
Fan
11 months ago
nuwancml
Fan
nuwancmlFan
asked in  Show us your Pipelines
Init query in the JDBC Query Consumer not working for Delete previous data

  I use Streamsets Data Collector to Load data from Stage 2 database tables (JDBC Query Consumer) using a query and Write loaded data to another Stage 2 Database table (JDBC Producer). I use Init Query as below to delete the previous records before loading data. But this does not delete any record from the table. It would be great if someone can help me. 

562
nuwancml
Fan
1 year ago
knsujith
Fan
knsujithFan
asked in  Show us your Pipelines
Error while connecting solace event broker

HiI am trying to connect to Solace event broker using JMS Producer and  configured all JMS parameters (user name and password ) but still pipeline throwing below error:RETRY: JMS_00 - Could not create initial context 'com.solacesystems.jndi.SolJNDIInitialContextFactory' with provider URL 'tcps://mr-connection-irj89q7fz0j.messaging.solace.cloud:55443' : javax.naming.NamingException: Username must be specified I have already uploaded all solace lib files in external resources section of engine.Please suggest.

291
lakshmi_narayanan_t
Discovered Fame
1 year ago
drvynguyencosmetic
Fan
drvynguyencosmeticFan
asked in  Show us your Pipelines
drvynguyencosmetic

Với cam kết đem đến sự lựa chọn tốt nhất cho khách hàng, Drvynguyencosmetic mang đến một kho lựa chọn ấn tượng từ các thương hiệu hàng đầu như: SVR: Sự ưa chuộng của các chuyên gia da liễu với sản phẩm chất lượng cao. Hãy truy cập ngay https://drvynguyencosmetic.com/ để tìm hiểu rõ hơn về những thông tin trên nhé. Thông tin liên hệ: SĐT: 0931510129Xem thêm:https://drvynguyencosmetic.com/https://www.facebook.com/drvynguyencosmetichttps://www.instagram.com/drvynguyencosmetichttps://twitter.com/dr_vynguyenhttps://www.youtube.com/@drvynguyen.cosmetic/abouthttps://www.flickr.com/people/drvynguyencosmetic/https://www.pinterest.com/drvynguyencosmetic/https://www.tumblr.com/drvynguyencosmetichttps://myspace.com/drvynguyencosmetic

40
drvynguyencosmetic
Fan
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in  Show us your Pipelines
how to use Google adress api to validate adress column or records

I want to validate the address column by using google address api  how to do that .this is my record : I want to validate the address .how to do any one can help this scenario.

241
Bikram
Headliner
1 year ago
shaik111
Fan
shaik111Fan
asked in  Show us your Pipelines
Issue with TCP Server Origin listening port

Am trying to listen some of the port from tcp server origin but its not giving any results, the same port am when using in linux console which we deploy streamsets server am able to listen the message currently am using Version:StreamSets Data Collector 3.19.1  in log am getting TCP_00 - Cannot bind to port [0.0.0.0/0.0.0.0:******]: java.net.BindException: Address already in usehere am hiding port number for confidential with *Could you please help me how to listen messages from streamsets tcp server origin. Thanks in advance 

385
shaik111
Fan
1 year ago
realmatcha
Opening Band
realmatchaOpening Band
asked in  Show us your Pipelines
Connect SqlServer

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data from sqlserver to hadoop and hive, the jdbc connection has been changed to sqlserver but it still doesn't work, please help me

524
realmatcha
Opening Band
1 year ago
HIMANSHU_SURANA
Fan
HIMANSHU_SURANAFan
asked in  Show us your Pipelines
Consuming MySQL binlog data in JDBC producer

I am trying to build CDC pipeline to migrate MySQL database to MySQL database in different server.Here’s the data collector pipeline I’ve created.  As per the documentation here, JDBC producers should be able to process binlog data. I’ve used Field remover to only use /Data and /Table fields.  When I run the pipeline I am getting error that input record has no data for <schema>.<table> . How can I create a pipeline to consume binlog records?

696
Bikram
Headliner
1 year ago
S
Fan
SubhaFan
asked in  Show us your Pipelines
HTTP Server

Can anyone let me know on how to use the HTTP Server Origin stage to fetch large volume of recordsMy problem statement is, I need to fetch large volume of data from a source using GET call. We can use HTTP Client stage for the same but since the source application doesn't have pagination configured , we will receive all the records at once . Thus i want to use HTTP server stage to achieve the same. Can anyone suggest how to achieve this. Appreciate your help!!Thanks in Advance!!

170
S
Fan
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in  Show us your Pipelines
Sorting a specific column and writing it to a new table

Use Case:-We have a dataset, in which we have columns as follows:-FIRST_NAME, LAST_NAME, EMAIL, PHONE, GENDER, DEPARTMENT, JOB_TITLE, YEARS_OF_EXPEREIENCE, SALARY.  Lets sort the column SALARY in ascending order and write to a new table with just 4 columns, FIRST_NAME, LAST_NAME, YEARS_OF_EXPEREIENCE, SALARY. Pipeline Design:-Snowflake Table (origin) Sort (Processor) Column Remover (Processor) Snowflake Table (Destination) Pipeline Working:-Snowflake Origin will fetch the table and columns and pass the records to Sort processor Sort processor will sort the data based on the configuration and pass it to Column Remover. (SALARY column, Ascending order) Column Remover will keep or remove the columns based on the configuration. Snowflake Table Destination will write the data to a new snowflake table.  

230
A
StreamSets Employee
1 year ago
A
Discovered Fame
AnkurDiscovered Fame
asked in  Show us your Pipelines
Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

Hi,We need to connect AWS S3 Select using Groovy scripting. For that need to upload jar files. While uploading below Jar files, getting the subject line error. The size of these jars is less than 1MB.joda-time.jarhttpclient.jarhttpcore.jaraws-java-sdk-s3.jarCan you please help me resolving this?Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms 

483
Bikram
Headliner
1 year ago
newkanyemerch32
Fan
newkanyemerch32Fan
asked in  Show us your Pipelines
btsmerchstore45

To operate on an EMR cluster, I have developed a straightforward transformer pipeline. I used the required configuration information during cluster configuration.I have the following problem: 

71
Bikram
Headliner
1 year ago
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
asked in  Show us your Pipelines
Transformer

In Transformer, i sent data from file(directory) to file(local fs)but the destination will got errorJOBRUNNER_63 - Pipeline status: 'RUN_ERROR', message: 'Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information'RUN_ERROR: Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information (View Stack Trace... ) how to clear the error and I need  help to clear the errorThanks Tamilarasu 

1786
tamilarasup
Discovered Fame
1 year ago
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
asked in  Show us your Pipelines
Databricks Transformer

Hinow i am working in streamsets data-ops transformer platform. i try to connect databricks cluster in streamsets. But i facing some issues.  JOBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'Failed to start pipeline, check logs for error. The Transformer Spark application in the Spark cluster might not be able to reach Transformer at ‘http://661f1312d1e9:19630’. If this is not the correct URL, update the transformer.base.http.url property in the Transformer configuration file or define a cluster callback URL for the pipeline and restart Transformer.'START_ERROR: Failed to start pipeline, check logs for error. The Transformer Spark application in the Spark cluster might not be able to reach Transformer at ‘http://661f1312d1e9:19630’. If this is not the correct URL, update the transformer.base.http.url property in the Transformer configuration file or define a cluster callback URL for the pipeline and restart Transformer. how to clear this error? and how to connect the cluster to

170
tamilarasup
Discovered Fame
1 year ago
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
asked in  Show us your Pipelines
mongodb

Is possible to perform cdc in mongodb atlas using mongodb oplog? The mongodb oplog connection string will be satisfied for MongoDB atlas but i got time out errorplease give some guidance to clear that error.

190
tamilarasup
Discovered Fame
1 year ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in  Show us your Pipelines
Example pipelines for Delta lake lookup processor

 Input data:[  {    "product_id": 1,    "unit_price": 23.2  },  {    "product_id": 4,    "unit_price": 21.64  },  {    "product_id": 2,    "unit_price": 10.87  }]Delta lake lookup table. This is same for all operations performedReturn all matching rows, generating a record for each match:  Return first matching row:There won’t be change in the output from above since there are no duplicates in input. Return a count of matching rows:Return true if matches exist, otherwise false:Input data:[  {    "product_id": 1,    "unit_price": 23.2  },  {    "product_id": 2,    "unit_price": 21.64  },  {    "product_id": 4,    "unit_price": 10.87  },    {    "product_id": 5,    "unit_price": 99.32  },    {    "product_id": 4,    "unit_price": 20.33  }]Return all matching rows, generating a record for each match:Return first matching row:Return true if matches exist, otherwise false: Return a count of matching rows: 

270
Pradeep
StreamSets Employee
1 year ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in  Show us your Pipelines
what is best of streamsets?

Hello Team, My self Santhos kumari'm Beginner of streamsets so which one is best of the learning in depth of streamsets.

333
S
Discovered Fame
2 years ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in  Show us your Pipelines
JDBC PIPE LINE NOT RUNING,SHOWS ME ERROR LIKE: JDBC_16 - Table '' does not exist or PDB is incorrect. Make sure the correct PDB was specified

I facing a issue on running JDBC pipeline ,it shows the below error ,I tried so many things  from the google but no use anybody can solve . 

1102
lakshmi_narayanan_t
Discovered Fame
2 years ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in  Show us your Pipelines
Convert multiple CSV records into a single JSON and make an HTTP bulk API POST call

Hi,The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.

4825
R
Fan
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines
Orchestrate Initial Bulk Load to Change Data Capture

 This pipeline is designed to orchestrate initial bulk load to change data capture workload from on-prem Oracle database to Snowflake Data Cloud. The pipeline takes into consideration the completion status of the initial bulk load job before proceeding to the next (CDC) step and also sends success/failure notifications. The pipeline also takes advantage of the platform’s event framework to automatically stop the pipeline when there’s no more data to be ingested. 

1071
P
Opening Band
2 years ago

Badge winners

Show all badges
Terms & Conditions