30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Get inspired and gain all the knowledge you need
Recently active
Whenever I try to log in the web interface I get a 502 error.This is happening when I try to reload a biggish table from a Mysql snapshot.VM is under reasonable load (25% cpu, 50% RAM)Scd service is up and running and apparently the pipelines are running as usual.
After launching pipeline using SDK, if i have any changes in pipeline, i want to do it from SDK instead of UI and those changes has to reflect in UI. how can i achieve this one without again launching pipeline.some more queries1.preview using SDKhow to know stage has no errors in SDK
Hi, We know a SDC job be involked through HTTP endpoint and can a set of parameters be passed as input parameter?Effectively we want the following :Originating system X > invoke SDC job with a set of parameters > SDC job looks up a database table with the passed on parameters > Returns the results to a different external system Y[ note its not the originating system X] > Send a response back to Originating system with a pass/fail status.Any demo pipeline will help. Are we advised to use REST Service origin for https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Microservice/Microservice_Title.htmlRegards, Anirban
i am reading a fixed width file from s3 and trying to parse with Jython evaluator in Data collector.when i try to run the pipeline i am getting below isue. Script error while processing batch: javax.script.ScriptException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Cannot create PyString with non-byte value in <script> at line number 60i am reading file using UTF-8 option.
Hi,I’m trying to fetch data from Shopify using HTTP Clent Origin with pagination set to “Link in HTTP Header”.As per Shopify (refer link here - https://shopify.dev/api/usage/pagination-rest#link-headers) - the parameters that comes back in “link” contain the page_info parameter that we should be using while querying the next page.Using StreamSets HTTP Client Origin, the pagination option using “Link in HTTP Header” is not working. I think this is because SDC is expecting the parameter name as “page” and hence this issue.Has anybody faced this earlier and if so how have you been able to make progress here?Anyone from StreamSets who can comment on this?
Already have module based pyspark code ..So how can we use the same without creating and adding stage in streamset pipeline.Can we dump or integrate with bitbucket repo?
We are encountering a NullPointerException error when starting a pipeline which is moving the data into a BigQuery table from a postgres database. After a while (an hour) the pipeline is starting. We are using StreamSets 3.8.0
My Streamset loading get this error but i don't know how to fix it. In detail, i paste the error and pineline image here. Many thanks for helping com.streamsets.pipeline.api.StageException: JDBC_77 - SQLSyntaxErrorException attempting to execute query 'SELECT ID AS product_id, AMOUNT AS amount, MODIFIED_DATE FROM TBL_PRODUCT WHERE MODIFIED_DATE > 2021-12-27 17:31:43 ORDER BY MODIFIED_DATE'. Giving up after 1 errors as per stage configuration. First error: SQLState: 42000 Error Code: 1064 Message: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '17:31:43 ORDER BY MODIFIED_DATE' at line 6 This error appear when i have verified and the pineline have run, the Mariadb database received full data successfully. But then this error came
Hi everyone,I would like to know if it is possible to introduce a delay between the transfer of the records, in order to not send the whole information at once. I want to transfer the data record by record.Could you help me, please?
Hi - while processing a CSV file via Directory Origin, I encountered the following error "SPOOLDIR_01 - Failed to process file '/tmp/out/customer_daily_incremental/CUSTOMER/CUSTOMER_20092021.csv' at position '177394': com.streamsets.pipeline.lib.dirspooler.BadSpoolFileException: java.io.IOException: (line 1138) invalid char between encapsulated token and delimiter" - I now know that the records present in line 1138 doesn't comply to the expected format (I see there is an additional double quote). But I don't want SDC to completely abort this file. Is there a configuration available where I can opt to log the record with error message and proceed to next line in the file?
I would like to process tables with zero records from JDBC Multitable Consumer to my target location,is it possible to do so please help me to find out solution to this query.
The data pipeline is jdbc → hive_metadata → hadoop file system and hivemetastore. Data moves from oracle database to hadoop file system. Schema is also getting created but there is no data in the tables. Tried changing the number idle time settings to 10 seconds.Any help in this regard will be greatly appreciated.
Dear Community users,SDC offers a great out-of-box processor “Field Type Converter” to convert an incoming field of one data type to another data type. This is a very useful transformation function that is required before we send the data to final destination. However, sometime we may have to write our own transformation function and today I’m going to share a scenario where I was forced to use a jython evaluator to detect the date format of an incoming field and convert that to a single unified format that our destination is expecting.Problem StatementWe have a need to process the incoming file where there is a field that need to be processed as “date” data type while there is another field which needs to be processed as “timestamp” data type before we generate the final feed for destination. The detaination while accepts the “date” fields is YYYY-MM-DD format, the timestamp fileds need to be of YYYY-MM-DD HH:MI:SS GMT+05:30, the origin fields can be of different formats. The solution
I have configured streamets transformer larger on aws, i am trying to create pipe line , to get data from s3 source , we are getting an error like expecting file:/// we gave it s3://<bucket name> and also tried s3a://<bucket name>
We are ingesting a table uisng streamsets which has a column containing xml data.we have added an extra stage to create a hash value for all the records coming through.In our target we could see all xml data is getting suffixed with hash value, and seperate hash column.How to get only the xml value in xml column without suffixing hash value?
Dear Community users,To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?I often came with requirements where we have to call an external API which suports both single as well as batch requests and most often batch is preffered to avoid the multiple round-trips and increae performance. This topic can be very useful for developers who wants to convert the multiple CSV records coming in a batch to a single JSON list array and send them as a single record to a target destination.Problem StatementConvert the 2 input CSV records into a single JSON.Input CSV Records:NAME|AGE|DEPARTMENT|ADDRESS"John"|"23"|"HR"|"31, 2nd Cross, 4th Street, Wilson-543234""Amanda"|"26"|"HR"|"24, 1st Cross, Wilson Garden-543234"Output Single JSON Record:[ { "data": { "NAME": "John", "AGE": "23", "DEPARTMENT": "HR", "ADDRESS": "31, 2nd Cross, 4th Street, Wilson-543234" } }, { "data": {
We are looking for a GUI tool to support enterprise LakeHouse architecture. Will streamsets dc support other data lake technologies like apache hudi and apache iceberg in roadmap? So that we can leverage it to do business data ingested into hudi lake in “bronze” layer, and then do some ETL into “sliver” layer, finally generate some report into “golden” layer, just said in databricks data lake solution report.
How we can check the condition in Stream selector for which service is not giving any response because of some exception at service end.If any exception at service end Http client is giving “No output records produced.” message so based on this message how we can further process like if success response comes then we can move to Trash if this message is coming “No output records produced.” we may insert into our sql table. So how we can identify this message in Stream selector or any other way.
I am trying to create a pipeline with origin PostgreSQL CDC client. The connection was not initializing until unless i make the postgre port to public. What is the exact solution for this ? can anyone help on this ?
I’ve been working with some fixed width records recently and one of the tasks was to split a string field into an array of like sized strings. The problem however seemed insurmountable using the field splitter stage as a delimiter is required. I ended up using the Jython evaluator to accomplish the task. Later on, I happened across an article talking about a similar problem in the context of using the Java regular expression engine. The article provided a solution using the \G boundary match combined with a look behind.For example: (?<=\G.{4})When used in the field splitter stage this regex will split the string into equal length parts that are 4 characters in size. The regex essentially creates a zero length delimiter which enforces a 4 character separation between the last delimiter and the current one.Shame on me for loosing the url for the original article. If I come across it again I will be sure to site it here.
I am using the cloud based instance of Control Hub and though I have been able to successfully configure SAML based authentication, I am unable to map the Control Hub groups to LDAP groups as I cannot see that field in the console. Am I missing something in the configuration?
Has anyone used Data Collector to extract data from DynamoDB source? How do we connect to a DynamoDB source using Data Collector Stage library?
I have a Job template in Control Hub that gets called 7 times with different set of parameters i.e. 7 different job instances. Can I update the offset file for a single job instance in Control Hub?
The http client processor piece has an area to put the http body, but the http destination doesn’t seem to have that same input. Using a PUT method for both.
Hi,We are processing S3 files with the batch size of 1000, the output also we are planning to store in S3 only. But we have he input file of 10000 records so we are seeing 10 output files in s3. As per client requirement we need to create a single file.Is there a way to create a single S3 output file from Streamsets.
Already have an account? Login
No account yet? Create an account
