Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1216 Topics
deepikamani007
Fan
deepikamani007Fan
asked in StreamSets Academy
How to create dynamic column creation in mysql and also auto create table

My origin as dymanic columns changes will happen and also load into mysql not manually but i want to achieve in dynamically.

781
Bikram
Headliner
J
Roadie
jkathawallaRoadie
asked in Community Articles and Got a Question?
Streamsets for processing large CSV files?

We currently have a C# console application for processing large CSV files. We are exploring using Streamsets to perform the same. Below is my high-level requirement.Currently we have 50+ consumers calling our application by placing a CSV file at               on-premises shared drive folder. The file size can be from 10,000 to 2 million rows.  The application should be able to handle processing up to 5 million rows an hour. Each file drop triggers an individual process run. So that multiple files are processed simultaneously. Below are the high-level steps that are involved in processing the file. Write the file to a database table. Enrich every record with additional data attributes. Make an API call for each record. Update the DB record with the response received from the API call. Also write the response of the API call to a result CSV file for the consumer. After all the records are processed write the result CSV file for the consumer to pick from the shared drive.  Thank you.

43619
Bikram
Headliner
N
Fan
NileshFan
asked in Community Articles and Got a Question?
How to write data from Multiple Snowflkae Tables to Multiple text files (separate file for each table) on Local Server?

I have scenario to write data from multiple DB tables on daily basis to file server. Each table supposed  to generate separate file on Local Server

341
Bikram
Headliner
deepikamani007
Fan
deepikamani007Fan
asked in Community Articles and Got a Question?
how to configure mongodp connection string

i dont known how to configure the mongodb opblog origin. 

433
Marina
Fan
S
Fan
shobhit_pathakFan
asked in Community Articles and Got a Question?
Using Postgres Metadata to migrate data from oracle to postgres

I have a JDBC Multi-table consumer as a source where it is connected to oracle database.Now I want to migrate the data along with the tables and its constraints, triggers, procedure, sequences etc. to Postgres DB.Currently I was exploring the Postgres Metadata Processor, but it is only reflecting the tables and columns with its respective data type.Can anyone please help me out in configuring the Postgres Metadata processor to migrate the constraints, triggers etc.

634
S
Fan
deepikamani007
Fan
deepikamani007Fan
asked in Community Articles and Got a Question?
how to concat empty string

my input data is like to be a concat employment all columns in single column but shows errorInvalid expression '${str:concat(record:value('/"employement_details.0.CurrentCompany"'),record:value('/"employement_details.0.c2HhPayroll"')(record:value('/"employement_details.0.c2HhPayroll"')))}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0101 - Error parsing expression ${str:concat(record:value('/"employement_details.0.CurrentCompany"'),record:value('/"employement_details.0.c2HhPayroll"')(record:value('/"employement_details.0.c2HhPayroll"')))}: javax.servlet.jsp.el.ELException: Encountered "(", expected one of [".", ">", "gt", "<", "lt", "==", "eq", "<=", "le", ">=", "ge", "!=", "ne", ")", ",", "[", "+", "-", "*", "/", "div", "%", "mod", "and", "&&", "or", "||", "?"]

1874
Bikram
Headliner
pasha001
Opening Band
pasha001Opening Band
asked in Community Articles and Got a Question?
Pipeline fails with java.lang.NullPointerException when there are no more records to read from table with JDBC Query Consumer & JDBC Producuer

Hi,I am new to streamsets here, so please bear with my questions :) Here is my first one. I created a simple pipeline to copy data from a employee table running sqlserver to employee table on postgresql. I used a JDBC Query consumer to pull records in incrementally using the offset value and a JDBC Producer stage to insert records. When I run the pipeline, it starts execution, select all records from sqlserver and inserts into postgres and then return a java.lang.NullPointerException. It goes into retry mode and continues to fail with same error on every retry attempt. I tried adding more records in the table so that the next retry attempt would pull the new records based on the offset value, however, the pipeline is not pulling them. I also tried filtering null records using stream selector, but it didnt work. I also added Pipeline Finisher stage to end the pipeline when a “no more data” event is generated, but it doesnt seem to work as expected either. The pipeline continues to throw

118612
pasha001
Opening Band
blakeorsak
Fan
blakeorsakFan
posted in Community Articles and Got a Question?
Job History Not Available and Pipeline Version Control

The Issue: Your job history is not available Error running the job: “Error reading entity from input stream”. Prerequisites:  The job is not using the latest pipeline version The latest pipeline version has been removed or deleted  The Solution/Best Practices: There are a few best practices you can implement to avoid this undesirable outcome.   - Edit the updated job to use the previous version of the pipeline           - Choose not to delete the unused new pipeline version          - Your job history will remain intact, and your job will continue to run without error.   - When creating new versions for your pipeline, create a new job for each version            - History from each version will be maintained          - You can delete pipeline versions/jobs without affecting older version’s history

250
blakeorsak
Fan
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Community Articles and Got a Question?
SNOWFLAKE_11 - Could not create SQL DataSource

I created a pipeline which send data from oracle cdc client to snowflakes. The origin is working fine but when I am trying to configure the destination and validate the pipeline I am getting the following error:SNOWFLAKE_11 - Could not create SQL DataSource: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: JDBC driver encountered communication error. Message: HTTP status=403.Can anyone help me regarding this.

3405
AkshayJadhav
StreamSets Employee
R
Fan
rajFan
asked in Community Articles and Got a Question?
Mongo DB Lookup

I am trying to pass id from previous step and get lookup information to perform insert / update operation. I am stuck at passing id from previous step as input to lookup value from destination.I am trying to read records from mongo collection and insert into another collection (CRUD Operation)I took some example from - https://www.youtube.com/watch?v=esx2cp6XXdQ https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Processors/MongoDBLookup.htmlbut both of them does not fulfill to complete my task End 2 End.Please share some documents / examples for mongo CRUD pipeline.

290
R
Fan
R
Fan
rajFan
asked in Community Articles and Got a Question?
Mongo Origin Read

I am creating a simple pipeline to read all from mongo collection and store into different collection,But my mongo DB origin is able to read only one record (even through i have 33 records in source) - what is the issue with my configuration ?  

492
R
Fan
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
using python sdk reading particular version of pipeline

Using python sdk can we read particular version of pipeline like if pipeline has two version then can we read first version. this line pipeline=sch.pipelines.get(name="Test")gives latest version only can we get previous version also.

522
saleempothiwala
Headliner
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
how to get count of number of parameter passed in a request

Hi teami have column request_param:0:   a : “11”   b: “12”   c: “12”   d: “12” how i can get count if above request has 4 parameter it can have 3 also like belowrequest_param:0:   a : “11”   b: “12”   c: “12” based on count i need to hit diff url 

1358
saleempothiwala
Headliner
S
Fan
suresh_venkataFan
asked in StreamSets Academy
Unable to pause Strigo "The workspace cannot be paused"

Hi Team!I’m Unable to Pause My Strigo . Iam Facing Error like this“The workspace cannot be paused while labs are still be provisioned. Once the labs are ready, you will be able to pause your workspace.”Requesting To You Please help me on this 

453
john.durkin
StreamSets Employee
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
asked in Community Articles and Got a Question?
File convertion

Hi TeamI need help to convert CSV file to parquet files or whole files using jython Evaluator or grovy evaluator. I need code guidance on how to implement this use case.origin- avro file data incomingjython evaluvator-want to save avro as whole file format whole file transformer -for convert avro to parquet local fs -write parquet file  in local directory suggest any other ideas if you had. Thanks Tamilarasu 

1734
saleempothiwala
Headliner
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
JDBC stage is failing with "Unknown system variable 'query_cache_size'" when connecting to MySQL server via Proxy SQL.

Environment:StreamSets Data Collector any version. MySQL database + ProxySQL LB. JDBC producer or JDBC consumer stage.Issue:It has been noticed that when connecting to the MySQL server when ProxySQL Load balancer is placed, the pipeline validation fails with following error:2023-02-22 12:28:12,112 [user:test] [pipeline:test_mysql/mysqlce] [runner:] [thread:preview-pool-16-thread-4] [stage:] ERROR HikariPool - HikariPool-4 - Exception during pool initialization.java.sql.SQLException: Unknown system variable 'query_cache_size' at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect

12090
AkshayJadhav
StreamSets Employee
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles and Got a Question?
Map the job submitted in transformer to the one run in Databricks SparkUI

How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?A:In the transformer log, we see something like below once the job is submitted to databricks.2023-03-02 12:02:18,099 INFO submit run response run id: 108505820 In spark UI, you would see the job group having the id 108505820Run response id can also be seen as Task Run id in databricks workflow > job.

140
Pradeep
StreamSets Employee
P
Opening Band
pkandraOpening Band
asked in Community Articles and Got a Question?
Reading SDC Error File from S3 Bucket

I have a pipeline that reads Kafka records and I’m writing any errors to an S3 bucket.  Most of the errors are related to tombstone records.  Once my original pipeline runs, I plan on having another pipeline process the error files on S3.  So far so good.  I can get the error code from the SDC file in the record:errorCode() field.  I also need the Kafka key from the S3 file.  What is the highlighted record/field called and how do I obtain the value with an expression evaluator or other processor?  Once I convert it to a map, that value is gone. 

707
Bikram
Headliner
M
Fan
MadhumithaFan
asked in StreamSets Academy
Any references for preparation of the Streamsets White belt Exam

Can I get any references for the preparation for Streamsets white belt exam other than the Streamsets Academy content…?

5274
Praveen P
Fan
G
Fan
groversakshi1998Fan
asked in Community Articles and Got a Question?
Looping over cell content

Hello,I am reading an excel file and my cell contains content like - “A/B/C”I need to seperate out the A, B, C into different records.I was using a for loop which is working fine as a java code, but in streamsets it’s printing out only the last updated value i.e. C in case of the above example. for(String item : splitCellData(cell, indexOfChange, cell.toString().length(), "/")){ ffileRecord.value[‘content’] = item; }  The split data function def splitCellData(Cell cell, int startIndex, int endIndex, String regex){   String res = null;    res = cell.toString().substring(startIndex, endIndex);    String[] result = res.split(regex);    return result; }

1382
saleempothiwala
Headliner
bolgas
Fan
bolgasFan
asked in Community Articles and Got a Question?
Call of Oracle SQL Stored Procedure as Pipeline Destination

Hello, We would like to create a pipeline that performs as a final step the insertion / update of data in an Oracle database via the execution of an Oracle SQL Stored Procedure.Therefore, we would like to ask you if it is possible to call an Oracle SQL Stored Procedure either via a built-in Destination component or a custom Destination component.If the above approach is feasible, could you please provide a list of steps to perform the operation?Thank you,Bolgas

1131
Bikram
Headliner
P
Opening Band
pkandraOpening Band
asked in Community Articles and Got a Question?
Run Pipeline for a specific amount of time

I have numerous Kafka topics that I’m moving to Databricks, but I don’t want the pipelines to continuously run.  Is there a way that I can schedule a pipeline to run for a certain amount of time or trigger it to stop after a certain amount of time...say an hour or 2?

342
saleempothiwala
Headliner
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
Count of total records in previous stage

Hi Team Can someone please help me how to get count from previous stage.Ex: My HTTP client gives 1000 records, streamslector filters x records. How do I get value of (1000-x) in groovy evaluator. sdc.records.size() function is giving only batch size. I require total size of (1000-x) of all batches together.Please help me achieve this in groovy evaluator

1784
Bikram
Headliner
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
how to generate UUID bases on timestamp

Hi Team I have a date datefield storing current datetime in it want to create a UUID using same column how i can create . currently using groovy script to create UUID but want to use same column and create it 

9395
A
Discovered Fame
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
Getting error with HTTP client

Getting below error while passing URL as parameter any input for same  

423
Bikram
Headliner

Badge winners

Show all badges
Terms & Conditions