30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Can't find what you're looking for? Ask it here or check out the Community articles
Recently active
Hi, I am using a multi table JDBC Origin to connect to Denodo and extract data. The pipeline is running fine but hangs after 5-10 minutes. Getting the below error for offset column I am using.Please let me know if I have to change any settings. The same works perfectly fine when I am using a single table JDBC consumer.Pipeline Status: RUNNING_ERROR: JDBC_75 - Jdbc Runner Failed. Reason java.util.concurrent.ExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NumberFormatException: For input string: "2020-04-05 19:18:11.0"
in session “Run a Job” you are using a pipeline where data is coming form Zomato. you are directly using this pipeline and never told how this was created , how data was pulled from zomato, you are not using pipeline created already in “Build a pipeline”session.Could you please help me to understand how zomato data is fetched in pipeline.
Hi everyone,Is there a way to use a parameter within another parameter?Here’s an example of what I’m trying to do: I’ve also tried this with no luck: ${str:concat("sdc/",${p_sourceEnv})} The idea is that I can include the environment as a parameter within other parameters and then when I want to change the environment only make the change once. Thanks in advance for any help. Much appreciated
We are using offset, where clause initial value is from job parameters. 20 pipelines following the same structure, 1 was done, and we copied to develop all others 19.On DEV and SIT they are working fine, only in prod has this error for 16 pipelines, others 4 are working fine. Giving up, after 1 error as per stage configuration. First error: SQLState: HY000 error code:640 message: [Teradata JSBC Driver][TeraJDBC 16.20.00.10][Error 640][SQÇstare HY000]ResultSet: findColumn — column-- not foundWe got an SQL statement run by Teradata and it works fine.
Job ID from Az Databricks is not an integer data type, whereas Databricks job launcher is expecting an integer value. can anyone please assist?
HiI am using MQTT Publisher destination. The MQTT server have two server (Active & Standby). Is there an option to provide multiple MQTT destination end point details, so that this MQTT publisher destination node can automatically connect to the active MQTT server. This same question applicable for JMS Producer destination node as well. Is it possible for “JMS Producer” destination to be configured with multiple endpoint details. I am looking for an alternate solution to go through Load balancer. My MQTT/JMS server is “Solace”. Since JMS/MQTT clients have option to self manage the server side HA failovers, wanted to check how Streamsets client nodes (MQTT/JMS) can be configured with multiple endpoint details? thanks
In streamsets pipeline, I see the "run history" durations show in the history are created each time when I start and stop even retry the job, but how to isolate the time over the specific period and how to compare the average transform time from last month and previous?
Great to see the Snowflake Transformer being release this week.Are there any near future plans to have a similar engine for BigQuery?
Hi there, Recently, all our Job Instances in the DataOps Platform did not execute due to the following, JOBRUNNER_73 - Insufficient resources to run job. All matching Data Collectors '[https://XXXXXX:18600]' have reached their maximum memory limits. We understand this is because, memory utilization on the engine was more than 80% and Garbage Collection process did not kick-in. We are using Data Collector v4.3; we use Tarball installation type and AWS EC2 instance. We had allocated enough memory for the few job instances that are scheduled to run; the data volume for each job instance was less than 5000 records. We noticed the issue only after 12+ hours since the Job Instance or the pipeline did not fail, The platform is to released to the wider team and how do we get an alert when,A Job instance did not execute in-time. Note - it did not have any errors - it was just in an ACTIVE state but color-coded in RED with the above message. The message was NOT even showing as ERROR. How do we
HiI am using Streamsets to connect an OPC UA server. I am using Streamsets origin node “OPC UA Client” for this purpose. I need to pass across the user name and password to establish the connection to server. I don’t see an option to configure username and password in OPC UA Client origin. Is there a way to configure username and password to connect the server? thanks
There is a directory created in IIS (Windows) and published via FTPS.When trying to use this directory in StreamSets using the SFTP|FTP|FTPS component, it returns the following error: "REMOTE_11 - Unable to connect to remote host 'ftps://ftps.hostname.net:921/PLV' with given credentials. Please verify if the host is reachable, and the credentials and other configuration are valid. The logs may have more details. Message: Could not list the contents of "ftps://ftps.hostname.net:921/PLV" because it is not a . : conf.remoteConfig.remoteAddress" Credentials are OK as it allows to open this directory using LFTP service on LinuxIs there any StreamSets configuration missing to fix this issue?
I am trying to save a single timestamp (latest timestamp from all records) after all batches finished running. However, while the event record gets created it never makes it to an Amazon S3 Destination Stage attached to the Event stream coming out of the Jython Evaluator. This is the code I’m using in the Destroy Script part:last_batch_event = sdc.createEvent("last_batch_ended", 0)last_batch_event.value = {"latestTimeStamp": sdc.state["latestTimeStamp"]}sdc.toEvent(last_batch_event)Any thoughts on what I am doing wrong?
Let us say we have a dataset like below. Requirement is to use JDBC Query Consumer Origin to load id, title, ratings based on whatever is the latest from created_date or updated_date. It means we want to do incremental load but not on primary key ‘id’. popular_moviesWhenever the pipeline runs it should pull in the updated ratings with other fields. In order to incrementally load data based on our requirement we have written query like below.SQL QueryQuery is failing by saying Invalid column name ‘offset_date’ though it is valid per SQL when we do validation.
The script I wrote in a Jython Scripting Origin gives the following error: “Stopping the Pipeline Observer, Reason: java.lang.InterruptedException”After running the sample code in the stage I noticed that it threw the same exception.Could this have an effect on how the custom script runs? Is there a fix for it?Thanks!
I added the “runtimeparam.properties” file in the external resources of my transformer engine.snowUrl=https://xxxxxxx.europe-west4.gcp.snowflakecomputing.comsnowWarehouse=TALEND_XSsnowDatabase=RAW_DEVsnowSchemaOrigin=TMSsnowRole=DATAPREPsnowSchemaTarget=TMS_WRKsnowTable=REF_PRODUCT_PRIORITY2snowWriteMode=APPENDsnowCreateTable=trueproductType=PriorityhamCode=not_appl I also changed the engine configuration in my deployment. BUT, the engine does not want to restart. Do you know what could be the cause of my problem?Thanks in advance.
I want to pass runtime parameters of my snowflake transformer pipeline to a filter condition and to an expression in Expression Evaluator stage (to overwrite the output field’s values).After several tests, I do not see how to do this. Can you help me ?
Hi,I have a requirement to do a truncate and load from SQL Server data base into Snowflake. How can we achieve this.Regards,Poulose
I currently look for a way to perform a typical try catch operation on a pipeline step.In my case on JDBC producer to detect a specific error (e.g. duplicate record, or record does not exists)and based on that to perform a retry by changing the sdc.operation.type.Below a picture to explain that use case.The reason for that is that it seems that no MERGE or often as well named UPSERT is supported by streamsets as it looks. As well i wanted to avoid to check for any record if record exists as that is costing time and resources. (as described here: How to upsert into JDBC Destination? | StreamSets Community)Any hints welcome how to do just a simple try catch.
streamsets:while ingesting data and dropping into s3 bucket in csv format , I'm getting below error:“root field should be list or list-map instead of map”.wanted to know the resolution for this. we wanted the file in s3 bucket as csv
we are ingesting data from oracle to s3 bucket and s3 to data bricks delta table using streamsets.S3 to delta bricks pipeline runs every 15mins. we observed that at times s3 to delta pipeline is not picking up any records form s3 bucket even if has the records present.Just wanted to no, what might be the issue?is their any way to reset origin and start every time on scheduler level as the jobs are scheduled every 15mins?
I have a requirement to write jobs logs to snowflake database table with jobs metrics data like job start time, end time, offset committed, no.of rows processed and errors if any etc.. Can you please help how to achieve this?
i have file which has records delimited by pipe and around pipe i have double quotes.how can is plit this file and store into snowflake or parquet. ex :1”|”USA”|”ABC company“|”Resident2”|”CAN”|”NORther “|”Rsdiendtn
