Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1225 Topics
john.durkin
StreamSets Employee
john.durkinStreamSets Employee
posted in StreamSets Academy
"DataOps Platform Fundamentals" - questions from the video course

Hi -Welcome to the StreamSets Academy area of Community!Please post questions related to the self-paced video course “DataOps Platform Fundamentals”. Please use the tag “DataOps Platform Fundamentals Course” when creating your question under the “StreamSets Academy” category. Thank you! 

4330
john.durkin
StreamSets Employee
realmatcha
Opening Band
realmatchaOpening Band
asked in Community Articles and Got a Question?
JDBC Multitable Consumer error

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data to hive and hadoop using jdbc multitable consumer.the problem that occurs is that there is a table name with '-' instead of '_' example: call-outcomme_id should the correct one be call_outcomme_id.I have used the field remover to maintain the table name with 'keep listed fields' when I start an error occurs, but when I select 'remove listed fields' the column that I targeted was deleted and it worked.thanks in advance

12823
realmatcha
Opening Band
bsayed
Roadie
bsayedRoadie
asked in Community Articles and Got a Question?
Field mapper not mapping all fields

I have a field mapper to remap camel case fields to snake case. This is the message i am trying to save:{   "id": "e35fd529-613f-5e79-036f-f37c2801041d",   "batchName": "SIT Axle3+ 9_12_test",   "batchSize": 35,   "numberOfAudited": 35,   "batchType": "TRIP_BASED",   "batchMode": "RANDOM",   "batchFacility": "71",   "batchDirection": "West",   "batchPlaza": "",   "batchLane": "",   "fromTimestamp": 1694538060,   "toTimestamp": 1694566800,   "createdTimestamp": 1694626323,   "createdBy": "mmccammon@etcc.com",   "completedTimestamp": 1697039389,   "completedBy": "asayed@etcc.com",   "auditDetailList": [      {         "id": "a4f274d5-8de5-4c54-ae70-b6a498ab4e9e",         "auditBatchId": "e35fd529-613f-5e79-036f-f37c2801041d",         "eventId": "49dcbe2c-51a8-11ee-9348-3cecefd01508",         "eventType": "TRIP",         "eventTimeStamp": 1694549374,         "auditResult": false,         "auditCorrections": [],         "auditedTimestamp": 1694635817,         "auditedBy": "ppradeep@etcc.co

21919
bsayed
Roadie
M
Discovered Fame
MelanieDiscovered Fame
asked in StreamSets Academy
JDBC Connection not working (Zomato)

HiI am trying to establishing a connection to jdbc:mysql://mysqldb:3306/zomato   but it is not working.I tried 2 differents approach:at first, I did not want to use Strigo so I was not limited to the 8 hours. So I run my engines in containers within my local machine. The engines are running fine on the Control Hub and I can create and run some pipelines. However, I would like to use the connection cited in one of the lab with mysqldb and zomato. I am not sure how to “install” the zomato db and reviews table locally? I have a container running mysql but from there i dont know what to.         If i try to create the connection using my local engine, i get then the below error.JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I could see a post with the same error message but it does

47919
A
Roadie
J
Roadie
jkathawallaRoadie
asked in Community Articles and Got a Question?
Streamsets for processing large CSV files?

We currently have a C# console application for processing large CSV files. We are exploring using Streamsets to perform the same. Below is my high-level requirement.Currently we have 50+ consumers calling our application by placing a CSV file at               on-premises shared drive folder. The file size can be from 10,000 to 2 million rows.  The application should be able to handle processing up to 5 million rows an hour. Each file drop triggers an individual process run. So that multiple files are processed simultaneously. Below are the high-level steps that are involved in processing the file. Write the file to a database table. Enrich every record with additional data attributes. Make an API call for each record. Update the DB record with the response received from the API call. Also write the response of the API call to a result CSV file for the consumer. After all the records are processed write the result CSV file for the consumer to pick from the shared drive.  Thank you.

45919
Bikram
Headliner
S
Roadie
satishRoadie
asked in Community Articles and Got a Question?
Facing issue JDBC Query

Pipeline Status: RUN_ERROR: QUERY_EXECUTOR_001 - Failed to execute query 'UPDATE user SET  created_by='HistoricalLoad', created_date="2020-04-27 15:11:25", modified_by='', modified_date="" WHERE id = '71021903' LIMIT 1': Data truncation: Incorrect datetime value: '' for column 'modified_date' at row 1 UPDATE user SETcreated_by='${record:value('/created_by')}',created_date="${str:isNullOrEmpty(record:value('/created_date'))?NULL: record:value('/created_date')}",modified_by='${str:isNullOrEmpty(record:value("/modified_by"))?NULL:record:value("/modified_by")}',modified_date="${str:isNullOrEmpty(record:value('/modified_date'))?NULL: record:value('/modified_date')}" WHERE id = '${record:value('/id')}' LIMIT 1

14417
Bikram
Headliner
D
Discovered Fame
DolphinDiscovered Fame
asked in Community Articles and Got a Question?
Field renamer failure

Hi team,I am using field renamer for mssql column name rename from origin mssql “regionid” to target snowflake “RegionID”.Attchment 1 is my configure for Field Rename configuration. When I ran “previewer”, did not show any errors.Attachment 2 is log.Cannot figured out what is the exact reason for the failure from the log.Can you suggest?

10415
Bikram
Headliner
Bikram
Headliner
BikramHeadliner
asked in Community Articles and Got a Question?
How to remove special characters from field names in streamsets data collector?

Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc) 

98515
P
Opening Band
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?
aggregate the records into a single array of objects

I have some json objects that looks as follows and i want them to be all merged in an array like :data: [{nstaskid: “...”,awb: “….”},{nstaskid: “...”,awb: “….”},] i tried to use javascript evalutor , loop over the records and push them in an array and write this record array , but it didn’t work.Can you please help me with that .

77415
tamilselg
Fan
Omar
Fan
OmarFan
asked in Community Articles and Got a Question?
Oracle CDC to Snowflake destination throwing Snowflake_56

Hi, I am getting SNOWFLAKE_56 - Key fields not specified for table 'ORCL_EMP' errorWhen Table Auto create is enabled, CDC also enabled and given table key column as ID column 

18614
alex.sanchez
StreamSets Employee
S
Fan
sgrewalFan
asked in Community Articles and Got a Question?
Excel File as Origin Source using HTTPS

I have a bunch of excel (xls) files that are hosted online (HTTPS) and periodically updated. I want to be able to store the excel file to ADLS Gen2. What is the best way to approach this? Data SourceNo authentication/authorization needed. HTTPS XLS formatEnvironment StackStreamsets Azure Shop

14714
Bikram
Headliner
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles and Got a Question?
Error using a directory created in IIS (Windows) and published via FTPS

There is a directory created in IIS (Windows) and published via FTPS.When trying to use this directory in StreamSets using the SFTP|FTP|FTPS component, it returns the following error: "REMOTE_11 - Unable to connect to remote host 'ftps://ftps.hostname.net:921/PLV' with given credentials. Please verify if the host is reachable, and the credentials and other configuration are valid. The logs may have more details. Message: Could not list the contents of "ftps://ftps.hostname.net:921/PLV" because it is not a . : conf.remoteConfig.remoteAddress" Credentials are OK as it allows to open this directory using LFTP service on LinuxIs there any StreamSets configuration missing to fix this issue?

25413
Yago Aparecido
Discovered Fame
Praveen P
Fan
Praveen PFan
asked in StreamSets Academy
how to write avro file in wholefile format from kafka to local fs?

i trying to write avro format as wholefile format from kafka to localfs but i got error.below  i share the pipeline ,can you please suggest how to done this 

32513
Praveen P
Fan
R
Roadie
RavisagaRoadie
asked in Community Articles and Got a Question?
How to process tables with zero records into target location from origin

I would like to process tables with zero records from JDBC Multitable Consumer to my target location,is it possible to do so please help me to find out solution to this query.

12112
Drew Kreiger
Rock star
Akanksha
Fan
AkankshaFan
asked in Community Articles and Got a Question?
While establishing JDBC connection.

I am getting this type of error JDBC_00 - Cannot connect to specified database: com.mysql.cj.jdbc.exceptions.CommunicationsException. Can anybody help me in this. 

9712
Bikram
Headliner
Kartik Medavarapu
Roadie
Kartik MedavarapuRoadie
asked in Community Articles and Got a Question?
DataOps Platform - REST API Origin - How to invoke pipeline?

Hi All,I created a microservice pipeline with REST API Origin using the DataOps platform as per video: https://www.youtube.com/watch?v=wIZWMV1bMl4Now, I am unable to invoke the pipeline using 3rd party tools like Postman. I have chosen default settings in SDC and headers as (X-SDC-APPLICATION-ID:sdc_microservice & Content-Type:application/json) yet I’m facing issue as “Error: connect ETIMEDOUT xx.xx.xx.xx:8000”.However, when I use the same URL in a curl command where SDC container is running, I see, 200 OK response. curl -i -X GET http://xx.xx.xx.xx:8000/rest/v1/user --header "X-SDC-APPLICATION-ID:sdc_microservice"There is no connectivity issue. So, can anyone tell me why I am unable to use Postman tool or if I am missing anything here?Below are the images showing the same: 

20012
AkshayJadhav
StreamSets Employee
tejas_surve
Roadie
tejas_surveRoadie
posted in Community Articles and Got a Question?
Driver not found for MariaDb

I want to Connect MariaDb to the StreamSet by JDBC processors. MariaDB is similar to the MySQL to need to configure the connection, getting error not supported suitable driver found.So Please help me to find that driver for MariaDb 

10912
saleempothiwala
Headliner
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?
JSON Flattening with Nested elements

Hello Team,Could you please assist me with below JSON flattening, [ { "ID1": "Test_value", "ID2": "Test_value2", "ID3": { "ID3_1": 1, "ID3_2": 2, "MAIN_1": { "OUTPUT": [ { "OUT1": "1", "OUT2": "2", "OUT3": "3", "OUT4": "4" }, { "OUT1": "6", "OUT2": "7", "OUT3": "8", "OUT4": "9" }, { "OUT1": "10", "OUT2": "11", "OUT3": "12", "OUT4": "13" } ] }, "MAIN_2": { "OUTPUT": [ { "OUT1": "1", "OUT2": "2", "OUT3": "3", "OUT4": "4" }, { "OUT1": "6", "OUT2": "7", "OUT3": "8", "OUT4": "9" }, { "OUT1": "10", "OUT2": "11", "OUT3": "12", "OUT4": "13" } ] }, "MAIN_3":

21512
Fredlau
Fan
Priyabala
Roadie
PriyabalaRoadie
asked in Community Articles and Got a Question?
How to handle German Characters(Non-English characters) coming from API response?

Receiving German characters from one of the attribute from API response and while joining the response with another API, we are facing issues because of the non-english characters.Can anyone suggest how we can handle this in pipeline?

144612
Ritantara
Fan
pasha001
Opening Band
pasha001Opening Band
asked in Community Articles and Got a Question?
Pipeline fails with java.lang.NullPointerException when there are no more records to read from table with JDBC Query Consumer & JDBC Producuer

Hi,I am new to streamsets here, so please bear with my questions :) Here is my first one. I created a simple pipeline to copy data from a employee table running sqlserver to employee table on postgresql. I used a JDBC Query consumer to pull records in incrementally using the offset value and a JDBC Producer stage to insert records. When I run the pipeline, it starts execution, select all records from sqlserver and inserts into postgres and then return a java.lang.NullPointerException. It goes into retry mode and continues to fail with same error on every retry attempt. I tried adding more records in the table so that the next retry attempt would pull the new records based on the offset value, however, the pipeline is not pulling them. I also tried filtering null records using stream selector, but it didnt work. I also added Pipeline Finisher stage to end the pipeline when a “no more data” event is generated, but it doesnt seem to work as expected either. The pipeline continues to throw

129312
pasha001
Opening Band
sanket.bhendarkar
Roadie
sanket.bhendarkarRoadie
asked in Community Articles and Got a Question?
java.lang.NoClassDefFoundError: Could not initialize class com.datastax.driver.core.Cluster

Hi, community,  I am using Streamset data collector for data migration from PostgreSQL to Cassandra But while testing the Cassandra connection I am encountering the above error. 

42612
AkshayJadhav
StreamSets Employee
SebastienM
Roadie
SebastienMRoadie
asked in StreamSets Academy
Getting error: VALIDATION_0011 - Stage has open output streams

I get this error, following the steps in the lecture “DataOps Platform Fundamentals”, chapter “Build a Pipeline”, lab “Lab: Build a Pipeline”.When clicking on the error, it takes me to the Stream Selector module, Configuration, General tab. There is nothing highlighted in that section and all seems in working order to me.Could you please advise?

56711
Drew Kreiger
Rock star
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?
insert blank values in xlxs file into a Null Values ?

i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails . 

6411
lakshmi_narayanan_t
Discovered Fame
M
Roadie
mstoffelRoadie
asked in Community Articles and Got a Question?
How to configure kinesis shard

Hi,I use the kinesis consumer on DC 5.3. Configuration shows no errors. When I preview the pipeline everything works but no records are read. This seems to be due to that only one shard contains the actual data. How can I configure the shard id to read from. Thanks, Marco

7410
AkshayJadhav
StreamSets Employee
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Where can I find Transpose processor documentation?

Hello,I am looking for documentation for the transpose processor in transformer pipelines.I searched here : https://docs.streamsets.com/portal/platform-transformer/latest/search.html?searchQuery=transpose but did not find it.Can someone share the link to the relevant documentation?-Dhanashri 

10210
Bikram
Headliner

Badge winners

Show all badges
Terms & Conditions