30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Whenever I try to log in the web interface I get a 502 error.This is happening when I try to reload a biggish table from a Mysql snapshot.VM is under reasonable load (25% cpu, 50% RAM)Scd service is up and running and apparently the pipelines are running as usual.
After launching pipeline using SDK, if i have any changes in pipeline, i want to do it from SDK instead of UI and those changes has to reflect in UI. how can i achieve this one without again launching pipeline.some more queries1.preview using SDKhow to know stage has no errors in SDK
Hi, We know a SDC job be involked through HTTP endpoint and can a set of parameters be passed as input parameter?Effectively we want the following :Originating system X > invoke SDC job with a set of parameters > SDC job looks up a database table with the passed on parameters > Returns the results to a different external system Y[ note its not the originating system X] > Send a response back to Originating system with a pass/fail status.Any demo pipeline will help. Are we advised to use REST Service origin for https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Microservice/Microservice_Title.htmlRegards, Anirban
i am reading a fixed width file from s3 and trying to parse with Jython evaluator in Data collector.when i try to run the pipeline i am getting below isue. Script error while processing batch: javax.script.ScriptException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Cannot create PyString with non-byte value in <script> at line number 60i am reading file using UTF-8 option.
Hi,I’m trying to fetch data from Shopify using HTTP Clent Origin with pagination set to “Link in HTTP Header”.As per Shopify (refer link here - https://shopify.dev/api/usage/pagination-rest#link-headers) - the parameters that comes back in “link” contain the page_info parameter that we should be using while querying the next page.Using StreamSets HTTP Client Origin, the pagination option using “Link in HTTP Header” is not working. I think this is because SDC is expecting the parameter name as “page” and hence this issue.Has anybody faced this earlier and if so how have you been able to make progress here?Anyone from StreamSets who can comment on this?
We are encountering a NullPointerException error when starting a pipeline which is moving the data into a BigQuery table from a postgres database. After a while (an hour) the pipeline is starting. We are using StreamSets 3.8.0
My Streamset loading get this error but i don't know how to fix it. In detail, i paste the error and pineline image here. Many thanks for helping com.streamsets.pipeline.api.StageException: JDBC_77 - SQLSyntaxErrorException attempting to execute query 'SELECT ID AS product_id, AMOUNT AS amount, MODIFIED_DATE FROM TBL_PRODUCT WHERE MODIFIED_DATE > 2021-12-27 17:31:43 ORDER BY MODIFIED_DATE'. Giving up after 1 errors as per stage configuration. First error: SQLState: 42000 Error Code: 1064 Message: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '17:31:43 ORDER BY MODIFIED_DATE' at line 6 This error appear when i have verified and the pineline have run, the Mariadb database received full data successfully. But then this error came
Hi everyone,I would like to know if it is possible to introduce a delay between the transfer of the records, in order to not send the whole information at once. I want to transfer the data record by record.Could you help me, please?
Hi - while processing a CSV file via Directory Origin, I encountered the following error "SPOOLDIR_01 - Failed to process file '/tmp/out/customer_daily_incremental/CUSTOMER/CUSTOMER_20092021.csv' at position '177394': com.streamsets.pipeline.lib.dirspooler.BadSpoolFileException: java.io.IOException: (line 1138) invalid char between encapsulated token and delimiter" - I now know that the records present in line 1138 doesn't comply to the expected format (I see there is an additional double quote). But I don't want SDC to completely abort this file. Is there a configuration available where I can opt to log the record with error message and proceed to next line in the file?
I would like to process tables with zero records from JDBC Multitable Consumer to my target location,is it possible to do so please help me to find out solution to this query.
The data pipeline is jdbc → hive_metadata → hadoop file system and hivemetastore. Data moves from oracle database to hadoop file system. Schema is also getting created but there is no data in the tables. Tried changing the number idle time settings to 10 seconds.Any help in this regard will be greatly appreciated.
I have configured streamets transformer larger on aws, i am trying to create pipe line , to get data from s3 source , we are getting an error like expecting file:/// we gave it s3://<bucket name> and also tried s3a://<bucket name>
We are ingesting a table uisng streamsets which has a column containing xml data.we have added an extra stage to create a hash value for all the records coming through.In our target we could see all xml data is getting suffixed with hash value, and seperate hash column.How to get only the xml value in xml column without suffixing hash value?
How we can check the condition in Stream selector for which service is not giving any response because of some exception at service end.If any exception at service end Http client is giving “No output records produced.” message so based on this message how we can further process like if success response comes then we can move to Trash if this message is coming “No output records produced.” we may insert into our sql table. So how we can identify this message in Stream selector or any other way.
I am trying to create a pipeline with origin PostgreSQL CDC client. The connection was not initializing until unless i make the postgre port to public. What is the exact solution for this ? can anyone help on this ?
How does JDBC destination handle fields without matching columns? I am encountering a situation where it appears that the fields without matching columns are ignored, but this is not defined in the documentation and seems counter to what I would expect (an exception complaining about a column not existing in the table). Please provide a deeper description of what is happening here.
Existe um diretório criado no IIS (Windows) e publicado via FTPS.Ao tentar usar este diretório em StreamSets usando o componente SFTP|FTP|FTPS, ele retorna o seguinte erro: "REMOTE_11 - Unable to connect to remote host 'ftps://ftps.hostname.net:921/PLV' with given credentials. Please verify if the host is reachable, and the credentials and other configuration are valid. The logs may have more details. Message: Could not list the contents of "ftps://ftps.hostname.net:921/PLV" because it is not a folder. : conf.remoteConfig.remoteAddress" As credenciais estão OK, pois permite abrir este diretório usando o serviço LFTP no LinuxEstá faltando alguma configuração de StreamSets para corrigir esse problema?
im using oracle as a source and s3 as destination.im ingesting the records from the source and adding table name as a column through expression evalutaor. i want to use this table name and create a folder in s3 dynamically before dropping those records in s3 bucketso the folder name should be created dynamically through streamsets fetching the name of the table. what should be the approach for the same?for ex: if I'm fetching records from table abc , I need to create a folder called “abc” and drop all the records inside that folder.
Can you please help me on below points.How we can get to know how many times StreamSets has retried for failure records ?what is their data ?What value we should give in Base backoff interval field ? and what all settings we have to do. bcz for me i am not seeing the incoming data to streamsets is matching with processed records. the difference between both is keeps on increasing. Can you please suggest something on this.
HI, I have tried copy all the files from one folder to another folder with in same s3 bucket using streamsets job. But I am seeing only 1 or 2 files copied into destination folder compared to source folder(like in source folder if 7 files are there, but in destination folder I am seeing 1 or 2 files are copying). Can any one help me on this issue. ThanksMurali
Lookups (into DeltaTable) giving extremely bad performances (sometime it stays in pre-execution stage forever) when used in Transformer with origin of 1000 records, although, it works decent enough in streaming mode which i guess is due to the lesser number of incoming records.
Hi team,how to add new stages to an existing pipelines using streamsets python SDK?
I’m looking for a tool to help prevent missing job version updates when moving code from one environment to another (development / UAT / production).If we update a pipeline and job in our development environment but move ancillary code to our test environment a week later, I’ve seen it is easy for our team to miss the test environment job update, resulting in wasted testing time. It would be helpful if we could see at a glance the differences in job versions before doing a deployment to help validate that we have the correct list of jobs to update as a part of that deployment.I see the REST API and could put together a script to compare versions, but I was wondering if there is any way we could visually see that in Control Hub, or even a way to build a pipeline or report that could be run to give this information. Thanks in advance!-Spyder
I am trying to create transformer pipeline using python sdk , unable to connect transformer engine.. i am getting two id and url for sch.tramsformers command . Please help me
HI, I have tried copy all the files from one folder to another folder with in same s3 bucket using streamsets job. But I am seeing more files copied into destination folder compared to source folder(like in source folder if 7 files are there, but in destination folder I am seeing more than 7 like … 8 or 10 or 12). But this issue is coming only for first time of the day. If I run same job again for the day I am seeing record count matching between source and destination. Can any one help me on this issue. ThanksMurali
Already have an account? Login
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
