StreamSets Platform

We are using offset,  where clause initial value is from job parameters. 20 pipelines following the same structure, 1 was done, and we copied to develop all others 19.On DEV and SIT they are working fine,  only in prod has this error for 16 pipelines, others 4 are working fine. Giving up, after 1 error as per stage configuration. First error: SQLState: HY000 error code:640 message: [Teradata JSBC Driver][TeraJDBC 16.20.00.10][Error 640][SQÇstare HY000]ResultSet: findColumn — column-- not foundWe got an SQL statement run by Teradata and it works fine. 

Job ID from Az Databricks is not an integer data type, whereas Databricks job launcher is expecting an integer value. can anyone please assist? 

HiI am using MQTT Publisher destination. The MQTT server have two server (Active & Standby).  Is there an option to provide multiple MQTT destination end point details, so that this MQTT publisher destination node can automatically connect to the active MQTT server. This same question applicable for JMS Producer destination node as well. Is it possible for “JMS Producer” destination to be configured with multiple endpoint details. I am looking for an alternate solution to go through Load balancer.  My MQTT/JMS server is “Solace”. Since JMS/MQTT clients have option to self manage the server side HA failovers, wanted to check how Streamsets client nodes (MQTT/JMS) can be configured with multiple endpoint details? thanks

In streamsets pipeline,   I see the "run history" durations show in the history are created each time when I start and stop even retry the job, but how to isolate the time over the specific period and how to compare the average transform time from last month and previous?

Great to see the Snowflake Transformer being release this week.Are there any near future plans to have a similar engine for BigQuery?

Hi there, Recently, all our Job Instances in the DataOps Platform did not execute due to the following, JOBRUNNER_73 - Insufficient resources to run job. All matching Data Collectors '[https://XXXXXX:18600]' have reached their maximum memory limits. We understand this is because, memory utilization on the engine was more than 80% and Garbage Collection process did not kick-in. We are using Data Collector v4.3; we use Tarball installation type and AWS EC2 instance. We had allocated enough memory for the few job instances that are scheduled to run; the data volume for each job instance was less than 5000 records. We noticed the issue only after 12+ hours since the Job Instance or the pipeline did not fail, The platform is to released to the wider team and how do we get an alert when,

HiI am using Streamsets to connect an OPC UA server. I am using Streamsets origin node “OPC UA Client” for this purpose. I need to pass across the user name and password to establish the connection to server.  I don’t see an option to configure username and password in OPC UA Client origin. Is there a way to configure username and password to connect the server? thanks

There is a directory created in IIS (Windows) and published via FTPS.When trying to use this directory in StreamSets using the SFTP|FTP|FTPS component, it returns the following error: "REMOTE_11 - Unable to connect to remote host 'ftps://ftps.hostname.net:921/PLV' with given credentials. Please verify if the host is reachable, and the credentials and other configuration are valid. The logs may have more details. Message: Could not list the contents of "ftps://ftps.hostname.net:921/PLV" because it is not a . : conf.remoteConfig.remoteAddress" Credentials are OK as it allows to open this directory using LFTP service on LinuxIs there any StreamSets configuration missi

I am trying to save a single timestamp (latest timestamp from all records) after all batches finished running. However, while the event record gets created it never makes it to an Amazon S3 Destination Stage attached to the Event stream coming out of the Jython Evaluator. This is the code I’m using in the Destroy Script part:last_batch_event = sdc.createEvent("last_batch_ended", 0)last_batch_event.value = {"latestTimeStamp": sdc.state["latestTimeStamp"]}sdc.toEvent(last_batch_event)Any thoughts on what I am doing wrong?

Let us say we have a dataset like below. Requirement is to use JDBC Query Consumer Origin to load id, title, ratings based on whatever is the latest from created_date or updated_date. It means we want to do incremental load but not on primary key ‘id’. popular_moviesWhenever the pipeline runs it should pull in the updated ratings with other fields. In order to incrementally load data based on our requirement we have written query like below.SQL QueryQuery is failing by saying Invalid column name ‘offset_date’ though it is valid per SQL when we do validation. 

The script I wrote in a Jython Scripting Origin gives the following error: “Stopping the Pipeline Observer, Reason: java.lang.InterruptedException”After running the sample code in the stage I noticed that it threw the same exception.Could this have an effect on how the custom script runs? Is there a fix for it?Thanks! 

 I added the “runtimeparam.properties” file in the external resources of my transformer engine.snowUrl=https://xxxxxxx.europe-west4.gcp.snowflakecomputing.comsnowWarehouse=TALEND_XSsnowDatabase=RAW_DEVsnowSchemaOrigin=TMSsnowRole=DATAPREPsnowSchemaTarget=TMS_WRKsnowTable=REF_PRODUCT_PRIORITY2snowWriteMode=APPENDsnowCreateTable=trueproductType=PriorityhamCode=not_appl  I also changed the engine configuration in my deployment.  BUT, the engine does not want to restart. Do you know what could be the cause of my problem?Thanks in advance.

I want to pass runtime parameters of my snowflake transformer pipeline to a filter condition and to an expression in Expression Evaluator stage (to overwrite the output field’s values).After several tests, I do not see how to do this. Can you help me ?  

Hi,I have a requirement to do a truncate and load from SQL Server data base into Snowflake. How can we achieve this.Regards,Poulose

I currently look for a way to perform a typical try catch operation on a pipeline step.In my case on JDBC producer to detect a specific error (e.g. duplicate record, or record does not exists)and based on that to perform a retry by changing the sdc.operation.type.Below a picture to explain that use case.The reason for that is that it seems that no MERGE or often as well named UPSERT is supported by streamsets as it looks. As well i wanted to avoid to check for any record if record exists as that is costing time and resources. (as described here: How to upsert into JDBC Destination? | StreamSets Community)Any hints welcome how to do just a simple try catch. 

Hello everyone, I'm a new user, and I got an installation package, but I can't activate it. I can access the community normally, but I can't log in normally and activate streamset. Who can give me some advice? Thank you!

streamsets:while ingesting data and dropping into s3 bucket in csv format , I'm getting below error:“root field should be list or list-map instead of map”.wanted to know the resolution for this. we wanted the file in s3 bucket as csv

we are ingesting data from oracle to s3 bucket and s3 to data bricks delta table using streamsets.S3 to delta bricks pipeline runs every 15mins. we observed that at times s3 to delta pipeline is not picking up any records form s3 bucket even if has the records present.Just wanted to no, what might be the issue?is their any way to reset origin and start every time on scheduler level as the jobs are scheduled every 15mins?

I have a requirement to write jobs logs to snowflake database table with jobs metrics data like job start time, end time, offset committed, no.of rows processed and errors if any etc.. Can you please help how to achieve this?

i have file which has records delimited by pipe and around pipe i have double quotes.how can is plit this file and store into snowflake or parquet. ex :1”|”USA”|”ABC company“|”Resident2”|”CAN”|”NORther “|”Rsdiendtn

i have environment file which has following key values and how can i configure at deployment/engine level and access values in pipelinecluster_id=XXXXXXXpath=XXXXXXXXXschema_name=XXXXXXXX 

Hello, thank you very much for your time to read and help me. :) .I need help because I have a problem that I can't solve.Context:I am developing a CDC integration with the following characteristics:A pipeline that performs the first data load and creates the tables in snowflake (jdbc multi table → snowflake connector). A second pipeline that performs continuous integration using the Stage CDC for Sql Server. ( sql server cdc → snowflake connector) The integration is with hundreds of tables, so it does not support the possibility of manual management and specific configurations at the object level. The update strategy is to configure the snowflake connector that loads the CDC transactions, so that it does a MERGE of the data, saving only the latest version of the row. Problem: The Multitable stage creates the table in snowflake and moves the hi

I used the JDBC Multitable table consumer and the tables were autocreated. I assumed the same would work with the SQL Server CDC Client.

I just wanted to get help how the best to deal with embedded scripts inside a pipeline.What is the best practice:Is there a way to see based on error shown in UI in which line the error occurred in embedded script ? Is there any way to debug into the script ? What is the best way to find root cause of an error (maybe exception handler with own logging - see below) Is there way to emulate the pipeline behavior to test script inside an IDE ? (eclipse, intellij,...) So an easy we to emulate the sdc.records object which is the input for an embedded script ? What i have found out so far:The error shown in Pipeline Builder does not tell anything to relate to where the error occurred in embedded script (at least i could not find any relationship as it just shows streamsets internal classes and not the script itself) I can add my own exception handler inside my script and log info a