Got a Question?

Can't find what you're looking for? Ask it here!

  • 689 Topics
  • 1,772 Replies

689 Topics

R
Fan
rajFan
asked in Got a Question?

Mongo Origin Read

I am creating a simple pipeline to read all from mongo collection and store into different collection,But my mongo DB origin is able to read only one record (even through i have 33 records in source) - what is the issue with my configuration ?  

2
R
Fan
2 months ago
S
Fan
shivaFan
asked in Got a Question?

Skip rows while reading excel

I am trying to read a excel file which has data starting from row 3,So my requirement is to set the columns names from row 3 and data from row 4 and skip column 1 & 2

7
Bikram
Headliner
1 month ago
M
Roadie
mr_maniiRoadie
asked in Got a Question?

Fragments

When I create a new fragment I get an error message as “Invalid Fragment. No open Input or Output lanes”. Anything that I miss here? 

2
M
Roadie
6 months ago
B
Roadie
BharathiMaddelaRoadie
asked in Got a Question?

Authorising transformer update in Transformer Jobs in Control Hub

Hi,is it possible to change the Authorising Transformer of  a job on control hub?we have a scenario to change another Authorising Transformer  in all the Transformer Jobs, when we have changed it  in a underneath pipeline and chose to update the Job while doing check-in, it does not reflect on the job.

3
Rishi
StreamSets Employee
11 months ago
R
Fan
rcanvasFan
asked in Got a Question?

Need help to determine what causes these errors

 Hello,I am looking for insight to what root cause could generate this error in my pipeline. As of a week ago this pipeline which is scheduled to run every 10 mins was running with no issues. On the 16th of December 2022 this pipeline began producing the below error messages. The pipeline grabs files from a local FS and sends them to S3. Under the state history we are given the following for one of the runs: com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0791 - Pipeline lifecycle event stage run error: com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0792 - Pipeline lifecycle event stage generated error record: SHELL_002 - Script execution timed out Any insight on what might be causing this issue would be greatly appreciated. Thanks, (full log file for pipeline error)[runner:] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINE

1
albertfc
Roadie
5 months ago
M
Discovered Fame
MelanieDiscovered Fame
asked in Got a Question?

Send a response to Origin other than REST

HiI need to connect to a WebSocket which will send me hundreds of messages in a TEXT format. Those messages need to be acknowledged so the web socket stops sending the same messages overtime. To acknowledge those messages, I need to send an identifier back to the Origin service in a self contained WebSocket text message.So far, my pipeline is as per below: However, I realized the “Send response to Origin” was not sending back the identifier to the WS.After reading the documentation, it seems that I could only use this block in a microservice pipeline. But the microservice pipeline works on origins which listen/read JSON-formatted data. And that’s where I am blocked. Indeed, my messages from the Web sockets are NOT in JSON and it s not a REST service.Is there a way to send a text message back to a Web socket in Streamset? So I can properly ACK each message that have been processed successfully.Thank you 

8
S
StreamSets Employee
3 months ago
D
Fan
denisaungureanuFan
asked in Got a Question?

Send Data Sequentially

Hello everyone, I have a csv file and I want to send data to a REST API.My final data looks like this:I will give you an example of how I want my final result to be.For 3 records in my csv file, every record will look similar with the one above. For 1 record in my csv file, I want to sent firstly a json with serial, interface and userData_dep fields, and then another json with serial, interface and userData_lan fields.To do that I used 2 http client processors, one for the dep and one for the lan. But I saw that the jsons weren’t sent as I want. My pipeline send 3 json with _dep field, and then another 3 json with _lan field.I want to send 1 json with dep field, then 1 json with lan field, then the next json with the dep field, and the next json with the lan field and so on.I want to send them sequentially.Could you help me with some ideas?Thank you! 

0
D
Fan
1 year ago
R
Fan
ramkumar_bvrFan
posted in Got a Question?

How to Pass Index value to Field Renamer Stage instead of source field name

I’m trying to automate our data restatement process. Based on the source field we are changing it to generic field as field1,field2,field3…..fieldN. But if we use Field Renamer stage it is mandatorily asking the exact source field name. Will it be possible to pass the index value instead of Source Field Name. Please let me know, if you need any further information on this. 

2
Bikram
Headliner
7 months ago
S
Headliner
Srinivasan SankarHeadliner
asked in Got a Question?

DataOps Platform / Control Hub - Data Collector Engine is unreachable

Team, Our DataOps deployment runs on AWS EC2. We have one Data Collector engine running on that deployment.  We are seeing this message frequently, i.e. we are losing connection to our Data Collector engine quite often. How do I check whether my Data Collector is running fine on EC2 machine (Linux) to confirm its the connection issue (Control Hub talking to Data Collector Engine) ? Any idea on why we see lose engine connectivity every few minutes or an hour? Any help would be appreciated. Cheers,Srini

1
Rishi
StreamSets Employee
1 year ago
S
Fan
shobhit_pathakFan
asked in Got a Question?

Using Postgres Metadata to migrate data from oracle to postgres

I have a JDBC Multi-table consumer as a source where it is connected to oracle database.Now I want to migrate the data along with the tables and its constraints, triggers, procedure, sequences etc. to Postgres DB.Currently I was exploring the Postgres Metadata Processor, but it is only reflecting the tables and columns with its respective data type.Can anyone please help me out in configuring the Postgres Metadata processor to migrate the constraints, triggers etc.

4
S
Fan
2 months ago
slaney3
Fan
slaney3Fan
asked in Got a Question?

How to create a sales pipeline

The best marketing techniques to create a sales pipeline for businesses.

0
slaney3
Fan
10 months ago
M
Roadie
mr_maniiRoadie
asked in Got a Question?

Streamset SDK for python

Hi,Does Streamset SDK supports Transformer andd Control Hub? or does it only support data collector?

2
dima
StreamSets Employee
5 months ago
SriniShan
Fan
SriniShanFan
asked in Got a Question?

Connecting to multiple MQTT/JMS servers

HiI am using MQTT Publisher destination. The MQTT server have two server (Active & Standby).  Is there an option to provide multiple MQTT destination end point details, so that this MQTT publisher destination node can automatically connect to the active MQTT server. This same question applicable for JMS Producer destination node as well. Is it possible for “JMS Producer” destination to be configured with multiple endpoint details. I am looking for an alternate solution to go through Load balancer.  My MQTT/JMS server is “Solace”. Since JMS/MQTT clients have option to self manage the server side HA failovers, wanted to check how Streamsets client nodes (MQTT/JMS) can be configured with multiple endpoint details? thanks

2
SriniShan
Fan
1 year ago
J
Fan
jmanoFan
asked in Got a Question?

ODBC driver for OS db2 Z/0s version 12 m500.

helloIs streamset support DB2 database? if yes what version of ODBC driver need for this OS db2 Z/0s version 12 m500.please point me to right download place or provide the right version to download.

0
J
Fan
1 year ago
ashok verma
Discovered Fame
ashok vermaDiscovered Fame
asked in Got a Question?

while launching S3 from SDK giving error

i have declared s3 data format as below but in UI/Control hub data format is showing blank.pipeline_builder = sch.get_pipeline_builder(engine_type='data_collector', engine_url=”XXXX”)s3_origin = pipeline_builder.add_stage('Amazon S3', type='origin')--s3_origin.data_format ='Text'.How to see values allowed for any particular component in SDK.(ex s3_origin.data_format ,s3_origin.delimiter etc..)

3
ashok verma
Discovered Fame
1 year ago
D
Fan
drozdse1Fan
asked in Got a Question?

Second Job Start in pipeline does not get the full list of parameters

Hi I have the following orchestrating job: The purpose of this pipeline is to first synchronize a list of tables via a full load between Oracle and Snowflake. After that, the CDC sync is started for each table passing the SCN that was read at the beginning. All jobs are executed in the background, so we can take advantage of running in parallel. In the first step (SCN & Tables) I extract a list of tables and SCN and batch sizes. The output looks like this:  TableName SCN BatchSize Table1 123321 100 Table2 123321 100 Table3 123321 500 ... ... ...  Then this list is passed to step 3 (Start Bulk Load) and everything works as expected.Step 3 starts per row a job. I have activated the option "Run In Background".Here the expression of the parameters I’m passing in this step: [{"tablename":"${record:value("/TABLENAME")}","batchsize":"${record:value("/BATCHSIZE")}"} Then the same list is passed in step 5 (Start Oracle CDC) but only the first job of the table i

1
saleempothiwala
Opening Band
9 months ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Got a Question?

how to convert timestamp and write into mysql .

I am using field type convertor to convert the column into time stamp and write into mysql but it only write 2023 year details how can i write 1970 year ,fix it by any other processor.this is my configration 1970 year data not writable in mysqlin mysql only 2023 data will be write  

4
Bikram
Headliner
3 days ago
jk56273
Fan
jk56273Fan
asked in Got a Question?

Truncate and load ADSL folder

Hi, I want to truncate and load ADSL folder using Streamsets pipeline. Can you suggest some way to achieve truncate and load feature in Streamsets

2
R
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
published in Got a Question?

Reset password Control Hub on Cloud.

Question:How can I reset the Control Hub password if I forgot?Clicked on Forgot Password and it says to contact System Admin: Answer: Ask someone of  StreamSets who has authorization in order to reset the password, for example, Jerry Yu, and provide them the email address of the user is trying to access

281
Arthidevi Balavignesh
Fan
11 months ago
sr1
Fan
sr1Fan
asked in Got a Question?

Rest API multiple jobs start

Hi All, i am trying to start the multiple jobs, but throwing an error, but stopping of multiple jobs working as expected. Stop: /jobrunner/rest/v1/jobs/stopJobs ==> Workingstart: /jobrunner/rest/v1/jobs/startJobs ==> failling

1
saleempothiwala
Opening Band
6 months ago
himanshu
Fan
himanshuFan
asked in Got a Question?

Skip or Automate registration on 3.22.3

I am upgrading streamsets data collector from 3.1.3.0 to 3.22.3. Initially it was asking for registration and I added SDC_CONF_http_authentication=form as env variable, it worked with admin/admin creds and did not ask for registration. Now I configured it with ldap authentication and it’s asking for registration again.Is there anyway to skip the registration or automate the registration on first run as I’ll be running this on kubernetes and won’t be able to register manually each time the pod  gets deployed. 

0
himanshu
Fan
5 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in Got a Question?

How to get the exact failure reason for failed job/pipeline

Hi team,I define a pipeline with the processer called “Field Remover”, the job failed and can see all Input rows are shown in Error rows.  However when check the log(log severity is ‘ALL’), I cannot see any error or error reason in the log, I can only see an warning “Exception thrown while trying to remove temporary directory /opt/streamsets-datacollector/tmp/sdc1627444042246270257”, how can I get the exact failed reason?Some context of pipleline, it is to load some data from oracle origin to snowflake internal tables. and it can run well several days, but same workflow failed today, however if I remove the process “filed remover” now, I can see the pipeline run successfully. Any suggestions for this trouble shooting.

2
Bikram
Headliner
10 months ago
W
Fan
wasifkabeerFan
asked in Got a Question?

Cloud Anti-virus Scan via ICAP in Streamsets DC

Dear Experts, I have a simple SFTP data pipeline to take text files from an SFTP cloud server and push these files to a local file system.However, I am required to get these files scanned by an anti-virus via ICAP call "icap://${SERVER_IP}:${SERVER_PORT}/virusScan" before writing to the target.Can anyone guide me on how to get this done in Streamsets?FYI: some other tools like Informatica have a built-in option for this. Please refer this link:https://knowledge.informatica.com/s/article/613287?language=en_US&type=external

2
W
Fan
1 year ago
refl_jasmel
Fan
refl_jasmelFan
asked in Got a Question?

PostgreSql CDC client worked only when i make postgre port public. Why ?

I am trying to create a pipeline with origin PostgreSQL CDC client. The connection was not initializing until unless i make the postgre port to public. What is the exact solution for this ? can anyone help on this ?  

2
Drew Kreiger
Rock star
1 year ago
Priyabala
Fan
PriyabalaFan
asked in Got a Question?

CSV to XLS

Is it possible to convert CSV  file to XLS using Streamsets?

1
saleempothiwala
Opening Band
9 months ago

Badge winners

Show all badges
Terms & Conditions