Skip to main content

Show us your Pipelines

Share an example of your Pipeline pattern. Upload a Screenshot and detailed description. No wrong answers. Do NOT share personal or credential information.

  • 72 Topics
  • 73 Replies
72 Topics
J
Fan
jojodoFan
posted in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.

60
J
Fan
21 days ago
J
Fan
jojodoFan
posted in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.  

10
J
Fan
21 days ago
L
Fan
lanek92292Fan
posted in  Show us your Pipelines
50 Cápsulas de Amor Propio PDF Gratis Sara Espejo

Libro 50 Cápsulas de Amor Propio PDF Gratis, 50 Cápsulas de Amor Propio PDF Gratis, 50 Cápsulas de Amor Propio PDF Descargar Gratis. Descubre el Libro 50 Cápsulas de Amor Propio PDF.. ➤➤➤➤ DESCARGAR 50 CAPSULAS DE AMOR PROPIO PDF GRATIS CLIC AQUI .El libro ’50 Cápsulas de Amor Propio’ ofrece una guía integral para redescubrir el sentido de la vida. Al embarcarnos en este viaje de autodescubrimiento, es fundamental reconocer la importancia de encontrar un propósito y una pasión que nos impulsen a seguir adelante. El amor propio juega un rol crucial en este proceso, ya que nos permite entender nuestras necesidades y deseos más profundos, y así orientar nuestras acciones hacia una vida más significativa.Encontrar propósito no es una tarea sencilla, pero ’50 Cápsulas de Amor Propio’ nos proporciona herramientas y reflexiones para identificar nuestras pasiones y metas. El libro nos ayuda a conectarnos con nuestro yo interior y a escuchar nuestra intuición. Este autoconocimiento es esencial

30730
L
Fan
21 days ago
K
Fan
kebig59243Fan
posted in  Show us your Pipelines
El Codigo De La Abundancia PDF Gratis

El Codigo De La Abundancia PDF Gratis, El Codigo De La Abundancia Jorge Muñoz Parral PDF Gratis, Libro El Codigo De La Abundancia PDF Gratis.En busca de la prosperidad económica y el bienestar emocional, muchas personas se han preguntado cuál es el secreto de las personas más ricas del mundo. ¿Cómo han logrado acumular tanto dinero y vivir en abundancia?. ➤➤➤➤ EL CODIGO DE LA ABUNDANCIA PDF GRATIS CLIC AQUI .La respuesta puede encontrarse en el libro «El Código de la Abundancia«. Este libro revela información que ha sido utilizada por las personas más ricas a lo largo de la historia para atraer el dinero y crear una vida llena de abundancia y bienestar.El Código de la Abundancia es un método probado que ha sido utilizado desde hace miles de años. Se basa en la idea de que todos tenemos el potencial de desbloquear nuestra prosperidad y crear una vida llena de abundancia en todas las áreas.. ➤➤➤➤ EL CODIGO DE LA ABUNDANCIA PDF COMPLETO + BONOS GRATIS CLIC AQUI . 

240
K
Fan
21 days ago
H
Fan
haceya8112Fan
posted in  Show us your Pipelines
Te lo Ocultaron PDF Gratis

Libro Te lo Ocultaron PDF Gratis, Ebook Te lo Ocultaron PDF Gratis, Te lo Ocultaron PDF Descargar Gratis.Libro «Te lo Ocultaron»: Descubre la verdad oculta.. ➤➤➤➤ LIBRO TE LO OCULTARON PDF GRATIS CLIC AQUI .Te lo Ocultaron es tu billete a un universo de secretos vetados y conspiraciones entrelazadas. Explora las teorías de conspiración y los eventos históricos alterados.4 increíbles bonos gratis con Te lo Ocultaron: Ellos Gobiernan, Contacto Encubierto, Silenciados y Tu energía ¿A quién se la das?Prepárate para un viaje emocionante que desafiará tus conocimientos y te invitará a pensar críticamente.. ➤➤➤➤ LIBRO TE LO OCULTARON PDF + BONOS GRATIS CLIC AQUI . 

180
H
Fan
21 days ago
T
Fan
tewoho3642Fan
posted in  Show us your Pipelines
Desbloquea Tu Mente Millonaria PDF Gratis

Desbloquea Tu Mente Millonaria PDF Gratis, Libro Desbloquea Tu Mente Millonaria PDF Gratis, Desbloquea Tu Mente Millonaria PDF Descargar Gratis. Desbloquea tu mente millonaria: Pensar, actuar y crear riqueza como los millonarios.. ➤➤➤➤ DESBLOQUEA TU MENTE MILLONARIA PDF GRATIS CLIC AQUI .Descubre cómo cambiar tu mentalidad hacia el dinero y alcanzar el éxito financiero con el libro ‘Desbloquea tu mente millonaria‘.Obtén 5 bonus exclusivos gratis y únete a una comunidad en línea de personas comprometidas con su crecimiento personal y financiero. ¡Transforma tu vida y alcanza la abundancia y la prosperidad que deseas!Una de las claves principales que se revela en «Desbloquea tu mente millonaria» es la importancia de cambiar nuestra mentalidad hacia el dinero. Muchas personas tienen creencias limitantes sobre la riqueza y el éxito financiero, lo que les impide alcanzar su máximo potencial.El libro Desbloquea tu mente millonaria te guiará a través de ejercicios prácticos y reflexiones prof

1610
T
Fan
22 days ago
SeduccionRemasterizada
Fan
SeduccionRemasterizadaFan
posted in  Show us your Pipelines
Seduccion Remasterizada PDF Gratis

Libro Seducción Remasterizada: La obra maestra respaldada por expertos en seducción, persuasión y psicología aplicada.. ➤➤➤➤ SEDUCCION REMASTERIZADA PDF GRATIS CLIC AQUI .Si estás buscando mejorar tus habilidades sociales y dominar el arte de la seducción, el libro «Seducción Remasterizada» es tu mejor opción. «Seducción Remasterizada» ha sido creado por expertos en el campo de la seducción, la persuasión y la psicología aplicada, y te brindará todas las herramientas necesarias para seducir mujeres de manera efectiva.Aprende cómo desarrollar tu carisma y ser más atractivo ante las mujeres.Descubre cómo seducir a cualquier mujer sin importar tu apariencia física o tu situación económica.Y mucho mas…. ➤➤➤➤ SEDUCCION REMASTERIZADA PDF + BONOS GRATIS CLIC AQUI . 

3560
SeduccionRemasterizada
Fan
22 days ago
Jdididis18
Fan
Jdididis18Fan
asked in  Show us your Pipelines
hi there I have a question about a new one for the and

Hi there I have a question about a new one for the and then you can always come over and watch the game with the and then you can always come over and watch the game with the and then you can always come over and 

30
Jdididis18
Fan
25 days ago
T
Fan
TRUCOSPSICOLOGIAFan
posted in  Show us your Pipelines
Trucos Psicologia Oscura PDF Gratis

La psicología oscura es una disciplina fascinante que puede ayudarnos a mejorar nuestras habilidades de comunicación, detectar mentiras y engaños, y influir en las decisiones de los demás.. ➤➤➤➤ TRUCOS PSICOLOGIA OSCURA PDF + BONOS GRATIS CLIC AQUI .Si estás interesado en aprender más sobre la psicología oscura, te recomendamos Trucos de Psicología Oscura.Recuerda que el conocimiento es poder, y mientras más aprendas sobre este tema, más efectivo serás en su aplicación.  

290
T
Fan
26 days ago
B
Fan
bozo16Fan
asked in  Show us your Pipelines
"What are pipelines?" box gone.

Doing the tutorial, fragments section (https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals/lessons/172/lab-create-a-new-fragment) I’m invite to import a pipeline, but I don’t have the box “What are pipelines?” with the import link. Maybe I closed the box at some stage, but how doI get it back?Screenshot of my Pipelines 

372
B
Fan
1 month ago
P
Fan
PratikSinhaFan
asked in  Show us your Pipelines
Error: [Local FS 1 - Directory Template] Base directory path could not be created (HADOOPFS_41)

I am trying to build a pipeline by following the steps mentioned in the streamsets docs- https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/Try.html#task_q3r_p2x_k4b but getting an issue while writing to the local fils system.the issue resides in the directory template [Directory Template info/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}codeHADOOPFS_41 - Base directory path could not be created]Can anyone help? Do we have a set of configuration that needs to be installed before running streamsets?

150
P
Fan
4 months ago
M
Fan
malikhas15Fan
asked in  Show us your Pipelines
Help Needed: Issues Connecting Streamsets JMS Producer to Solace

I have a Streamsets pipeline where I am using the SQL Server CDC component and the JMS Producer component. My connection to SQL Server is working perfectly, and I am receiving data in my pipeline from SQL Server. However, I am encountering issues when trying to connect to Solace through the JMS connector.Initially, I received errors when attempting to connect to Solace. I then used the following parameters:- JMS Initial Context Factory: `com.solacesystems.jndi.SolJNDIInitialContextFactory`- JNDI Connection Factory: `/jms/cf/default`- JMS Provider URL: `tcp://pub:admin@172.17.0.4:55555`- JMS Destination Name: `cdc/demo/sqlserver`- JMS Destination Type: `Topic`- Solace JMS VPN: `test`The pipeline builds successfully, but when I go to the Solace "Try Me" tab and attempt to subscribe to this topic using the client "pub" I do not receive any messages.Additionally, I noticed something peculiar: when I use a client name that does not exist in Solace, the connection still builds successfully.

220
M
Fan
4 months ago
J
JnaneshwariNew Member
posted in  Show us your Pipelines
OBRUNNER_63 - Pipeline status: 'START_ERROR', Pool exhausted'

Hi Team, I am facing bellow error when i execute my pipeline . as a work around we increased ClassLoaders = 100 Still facing same issues  OBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'java.lang.RuntimeException: Could not get a private ClassLoader for 'streamsets-datacollector-azure-lib', for 'com_streamsets_pipeline_stage_metadata_gen2_DataLakeGen2MetadataDExecutor', active private ClassLoaders='50': java.util.NoSuchElementException: Pool exhausted'

70
J
5 months ago
moviepages
Fan
moviepagesFan
asked in  Show us your Pipelines
how to create the pipeline for the xml data where it is taken from sftp

how to create the pipeline for the xml data where it is taken from sftp in the streamsets we have getting only 1 record from sftp in the preview data 

362
roma
Fan
6 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Cannot Connect StreamSets with Python SDK, but Can connect via CURL command with same Credential ID and Token

Hi Team,I generate Credential ID and Token via StreamSets UI → Manage → API Credential, and with the generated ID and Token I can run Curl command, returned status is “HTTP/1.1 200 OK”, also return me a json format that show my organization id, email, extra.. However, When I want to connect with below code snippet (using same credential ID and Token), it returns me error of 403. paste code and error below:>>> from streamsets.sdk import ControlHub>>> sch = ControlHub(credential_id='absd_myid', token="abcd_mytoken.") Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch.py", line 141, in __init__    self.api_client = sch_api.ApiClient(  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch_api.py", line 96, in __init__    raise ValueError('Encountered error while decoding auth token: {}'.format(e))ValueError: Encountered error while decoding aut

481
Bikram
Headliner
6 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
How to Config to make Pipeline "Stop Event" not run when pipeline failed

Hi team,we have a pipeline, which configured a stop event, it let a sql statement to run once pipeline finish processing data,  however we found, when the pipeline failed due to some reason, this “Stop Event” still run, which is not expected for us. Can you please let me know if some place can be configured, which let the Pipeline "Stop Event" not run when pipeline failed processing data? I actually found if the pipelien “Start Event” failed, the pipeline will not run which is expected, however the “Stop Event” always run even pipeline failed. 

532
D
Discovered Fame
6 months ago
vishalsdc2024
vishalsdc2024New Member
asked in  Show us your Pipelines
Stream set pipeline is not starting

Hi , i am facing an issue while starting my sdc service and sdc is not coming up. it is prod and need to be up. can someone please help on this at the earliest. Caused by: com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0206 - Cannot load details for pipeline 'SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com': java.io.IOException: File '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-tmp' exists, '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-old' should exists we have referred link for solution but not sure what can be done as files are not there on those pipelines.  Thanks & Regards,Vishal Verma     

270
vishalsdc2024
7 months ago
C
ColinNew Member
asked in  Show us your Pipelines
Inquiry Regarding StreamSets Product Versions and Feature Differences

As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?

160
C
7 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

Hi, I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.can you let know anything wrong here? 

151
D
Discovered Fame
7 months ago
dima
StreamSets Employee
dimaStreamSets Employee
posted in  Show us your Pipelines
Analyzing chess games with batch and streaming pipelines

I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates

1101
R
Fan
7 months ago
Arjo
Fan
ArjoFan
asked in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

101
Bikram
Headliner
7 months ago
Arjo
Fan
ArjoFan
posted in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

40
Arjo
Fan
7 months ago
realmatcha
Opening Band
realmatchaOpening Band
asked in  Show us your Pipelines
JDBC ERROR

 Error:  Source :  hello everyone, I want to pull data from the source to the warehouse, in the source there is a "customer_id" column, when the streamsets are run why is there an invalid customer error? in streamsets I also flag based on the "id" column. please help, thank you

151
Bikram
Headliner
8 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
About Oracle CDC Client Configure

Hi team,I am configuring Oracle CDC Client now, our oracle version is 19C, non-CDB oracle.I ensure our oracle instance has enabled Database Archiving Mode, cause the return of statement "select log_mode from v$database" is "ARCHIVELOG". My question is for Step "Enable Supplemental Logging", since I have three "NO" for statement "select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;". Is it mandatory to run this "alter database add supplemental log data;" ?I actually do not want to make all database tables' Supplemental Logging, only want to enable it for some related tables, if this, should I only run like below "alter table <schema name>.<table name> add supplemental log data (all) columns;" , then run "alter system archive log current;" or I need to run below in sequence?Step 1:Enable minimal supplemental logging, run "alter database add supplemental log data;"Step 2:alter table <schema name>.<table name> add su

271
D
Discovered Fame
8 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Transformer for spark (Extract and Load).

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.Right now, I am using.1.JDBC query consumer (origin) 2.Snowflake (Destination)Note. In our sources we don’t have any primary key or Unique key So, not able to define offset column because of that not able achieve partition the data.   

100
hrishikeh132609
Fan
9 months ago

Badge winners

Show all badges
Terms & Conditions