- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 year ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1209 Topics

Most replies

Most views

Newest first

Drew KreigerSenior Community Builder at StreamSets

asked in Community Articles and Got a Question?

Does StreamSets support automatic propagation of DDL changes when using the CDC Client without data loss?

251

2 years ago

Drew KreigerSenior Community Builder at StreamSets

asked in Community Articles and Got a Question?

Autocreate Tables to Replicate to Snowflake Error

We have hundreds of tables to replicate to Snowflake as our destination, so it makes sense to have StreamSets autocreate these at the destination with the correct DDL but the functionality does not seem to be working. Besides checking the 'Table Auto Create' box are there any other steps we should take?

181

2 years ago

Drew KreigerSenior Community Builder at StreamSets

asked in Community Articles and Got a Question?

Is there any way to sort records?

781

2 years ago

Drew KreigerSenior Community Builder at StreamSets

asked in Community Articles and Got a Question?

Can I use a variable in the 'Request URL' of an HTTP client?

651

2 years ago

matfiorFan

asked in Community Articles and Got a Question?

Unreachable web interface

Whenever I try to log in the web interface I get a 502 error.This is happening when I try to reload a biggish table from a Mysql snapshot.VM is under reasonable load (25% cpu, 50% RAM)Scd service is up and running and apparently the pipelines are running as usual.

454

2 years ago

ashok vermaDiscovered Fame

asked in Community Articles and Got a Question?

refreshing pipeline launch using SDK

After launching pipeline using SDK, if i have any changes in pipeline, i want to do it from SDK instead of UI and those changes has to reflect in UI. how can i achieve this one without again launching pipeline.some more queries1.preview using SDKhow to know stage has no errors in SDK

281

2 years ago

anirbanchFan

asked in Community Articles and Got a Question?

Invoking a SDC job with a set of input parameters

Hi, We know a SDC job be involked through HTTP endpoint and can a set of parameters be passed as input parameter?Effectively we want the following :Originating system X > invoke SDC job with a set of parameters > SDC job looks up a database table with the passed on parameters > Returns the results to a different external system Y[ note its not the originating system X] > Send a response back to Originating system with a pass/fail status.Any demo pipeline will help. Are we advised to use REST Service origin for https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Microservice/Microservice_Title.htmlRegards, Anirban

574

2 years ago

ashok vermaDiscovered Fame

asked in Community Articles and Got a Question?

issue while reading a file

i am reading a fixed width file from s3 and trying to parse with Jython evaluator in Data collector.when i try to run the pipeline i am getting below isue. Script error while processing batch: javax.script.ScriptException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Cannot create PyString with non-byte value in <script> at line number 60i am reading file using UTF-8 option.

582

2 years ago

swayamDiscovered Fame

asked in Community Articles and Got a Question?

Unable to get data from Shopify via HTTP Client Origin using pagination

Hi,I’m trying to fetch data from Shopify using HTTP Clent Origin with pagination set to “Link in HTTP Header”.As per Shopify (refer link here - https://shopify.dev/api/usage/pagination-rest#link-headers) - the parameters that comes back in “link” contain the page_info parameter that we should be using while querying the next page.Using StreamSets HTTP Client Origin, the pagination option using “Link in HTTP Header” is not working. I think this is because SDC is expecting the parameter name as “page” and hence this issue.Has anybody faced this earlier and if so how have you been able to make progress here?Anyone from StreamSets who can comment on this?

2096

2 years ago

shivani2512Fan

asked in Community Articles and Got a Question?

Integrate Bit bucket for already written module based pyspark code

Already have module based pyspark code ..So how can we use the same without creating and adding stage in streamset pipeline.Can we dump or integrate with bitbucket repo?

351

2 years ago

AncutaTitaFan

asked in Community Articles and Got a Question?

NullPointerException when starting a pipeline

We are encountering a NullPointerException error when starting a pipeline which is moving the data into a BigQuery table from a postgres database. After a while (an hour) the pipeline is starting. We are using StreamSets 3.8.0

912

2 years ago

anhducvuFan

asked in Community Articles and Got a Question?

HOW TO FIX ERROR THAT IS NOT CORRESPOND WITH MARIADB/MYSQL VERSION FOR THE RIGHT SYNTAX.. First error: SQLState: 42000 Error Code: 1064

My Streamset loading get this error but i don't know how to fix it. In detail, i paste the error and pineline image here. Many thanks for helping com.streamsets.pipeline.api.StageException: JDBC_77 - SQLSyntaxErrorException attempting to execute query 'SELECT ID AS product_id, AMOUNT AS amount, MODIFIED_DATE FROM TBL_PRODUCT WHERE MODIFIED_DATE > 2021-12-27 17:31:43 ORDER BY MODIFIED_DATE'. Giving up after 1 errors as per stage configuration. First error: SQLState: 42000 Error Code: 1064 Message: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '17:31:43 ORDER BY MODIFIED_DATE' at line 6 This error appear when i have verified and the pineline have run, the Mariadb database received full data successfully. But then this error came

2772

2 years ago

Diana LacatusFan

asked in Community Articles and Got a Question?

Transfer of data

Hi everyone,I would like to know if it is possible to introduce a delay between the transfer of the records, in order to not send the whole information at once. I want to transfer the data record by record.Could you help me, please?

724

2 years ago

swayamDiscovered Fame

asked in Community Articles and Got a Question?

How to ignore the unexpected line in CSV file and proceed to next?

Hi - while processing a CSV file via Directory Origin, I encountered the following error "SPOOLDIR_01 - Failed to process file '/tmp/out/customer_daily_incremental/CUSTOMER/CUSTOMER_20092021.csv' at position '177394': com.streamsets.pipeline.lib.dirspooler.BadSpoolFileException: java.io.IOException: (line 1138) invalid char between encapsulated token and delimiter" - I now know that the records present in line 1138 doesn't comply to the expected format (I see there is an additional double quote). But I don't want SDC to completely abort this file. Is there a configuration available where I can opt to log the record with error message and proceed to next line in the file?

9404

2 years ago

RavisagaRoadie

asked in Community Articles and Got a Question?

How to process tables with zero records into target location from origin

I would like to process tables with zero records from JDBC Multitable Consumer to my target location,is it possible to do so please help me to find out solution to this query.

11812

2 years ago

posamsaitejaFan

asked in Community Articles and Got a Question?

Data doesn't show up in hive

The data pipeline is jdbc → hive_metadata → hadoop file system and hivemetastore. Data moves from oracle database to hadoop file system. Schema is also getting created but there is no data in the tables. Tried changing the number idle time settings to 10 seconds.Any help in this regard will be greatly appreciated.

202

2 years ago

swayamDiscovered Fame

posted in Community Articles and Got a Question?

Swayam's Micro-blogging: Auto detection of date format

Dear Community users,SDC offers a great out-of-box processor “Field Type Converter” to convert an incoming field of one data type to another data type. This is a very useful transformation function that is required before we send the data to final destination. However, sometime we may have to write our own transformation function and today I’m going to share a scenario where I was forced to use a jython evaluator to detect the date format of an incoming field and convert that to a single unified format that our destination is expecting.Problem StatementWe have a need to process the incoming file where there is a field that need to be processed as “date” data type while there is another field which needs to be processed as “timestamp” data type before we generate the final feed for destination. The detaination while accepts the “date” fields is YYYY-MM-DD format, the timestamp fileds need to be of YYYY-MM-DD HH:MI:SS GMT+05:30, the origin fields can be of different formats. The solution

4021

2 years ago

jpreddymFan

asked in Community Articles and Got a Question?

S3 source connection issue

I have configured streamets transformer larger on aws, i am trying to create pipe line , to get data from s3 source , we are getting an error like expecting file:/// we gave it s3://<bucket name> and also tried s3a://<bucket name>

502

2 years ago

harshithDiscovered Fame

asked in Community Articles and Got a Question?

Ingesting xml data using streamsets

We are ingesting a table uisng streamsets which has a column containing xml data.we have added an extra stage to create a hash value for all the records coming through.In our target we could see all xml data is getting suffixed with hash value, and seperate hash column.How to get only the xml value in xml column without suffixing hash value?

651

2 years ago

swayamDiscovered Fame

posted in Community Articles and Got a Question?

Swayam's Micro-blogging: Combine multiple JSON records into a single one

Dear Community users,To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?I often came with requirements where we have to call an external API which suports both single as well as batch requests and most often batch is preffered to avoid the multiple round-trips and increae performance. This topic can be very useful for developers who wants to convert the multiple CSV records coming in a batch to a single JSON list array and send them as a single record to a target destination.Problem StatementConvert the 2 input CSV records into a single JSON.Input CSV Records:NAME|AGE|DEPARTMENT|ADDRESS"John"|"23"|"HR"|"31, 2nd Cross, 4th Street, Wilson-543234""Amanda"|"26"|"HR"|"24, 1st Cross, Wilson Garden-543234"Output Single JSON Record:[ { "data": { "NAME": "John", "AGE": "23", "DEPARTMENT": "HR", "ADDRESS": "31, 2nd Cross, 4th Street, Wilson-543234" } }, { "data": {

8651

2 years ago

casel_chanFan

asked in Community Articles and Got a Question?

Will streamsets dc support other data lake technologies like apache hudi and apache iceberg in roadmap?

We are looking for a GUI tool to support enterprise LakeHouse architecture. Will streamsets dc support other data lake technologies like apache hudi and apache iceberg in roadmap? So that we can leverage it to do business data ingested into hudi lake in “bronze” layer, and then do some ETL into “sliver” layer, finally generate some report into “golden” layer, just said in databricks data lake solution report.

1561

2 years ago

Raman GoyalFan

asked in Community Articles and Got a Question?

Http response - No output records produced.

How we can check the condition in Stream selector for which service is not giving any response because of some exception at service end.If any exception at service end Http client is giving “No output records produced.” message so based on this message how we can further process like if success response comes then we can move to Trash if this message is coming “No output records produced.” we may insert into our sql table. So how we can identify this message in Stream selector or any other way.

733

2 years ago

refl_jasmelFan

asked in Community Articles and Got a Question?

PostgreSql CDC client worked only when i make postgre port public. Why ?

I am trying to create a pipeline with origin PostgreSQL CDC client. The connection was not initializing until unless i make the postgre port to public. What is the exact solution for this ? can anyone help on this ?

292

2 years ago

mblahayDiscovered Fame

posted in Share your Best Practices

Splitting a fixed width array

I’ve been working with some fixed width records recently and one of the tasks was to split a string field into an array of like sized strings. The problem however seemed insurmountable using the field splitter stage as a delimiter is required. I ended up using the Jython evaluator to accomplish the task. Later on, I happened across an article talking about a similar problem in the context of using the Java regular expression engine. The article provided a solution using the \G boundary match combined with a look behind.For example: (?<=\G.{4})When used in the field splitter stage this regex will split the string into equal length parts that are 4 characters in size. The regex essentially creates a zero length delimiter which enforces a 4 character separation between the last delimiter and the current one.Shame on me for loosing the url for the original article. If I come across it again I will be sure to site it here.

1491

2 years ago

sushil.vishwanathanFan

asked in Community Articles and Got a Question?

StreamSets Control Hub LDAP Group Field not visible

I am using the cloud based instance of Control Hub and though I have been able to successfully configure SAML based authentication, I am unable to map the Control Hub groups to LDAP groups as I cannot see that field in the console. Am I missing something in the configuration?

391

2 years ago

Page 44 / 49

Badge winners

Ykyeeywkkwhas earned the badge Eager to help
vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.