StreamSets Platform

we are ingesting data from oracle and it has timestamp format as mm/dd/yyyy hhss AM/PM, while ingesting through streamsets its taking as yyyy-mm-dd hhss. how to change the timestamp format to source format which is mm/dd/yyyy hhss AM/PM in streamsets

Team, Our DataOps deployment runs on AWS EC2. We have one Data Collector engine running on that deployment.  We are seeing this message frequently, i.e. we are losing connection to our Data Collector engine quite often. How do I check whether my Data Collector is running fine on EC2 machine (Linux) to confirm its the connection issue (Control Hub talking to Data Collector Engine) ? Any idea on why we see lose engine connectivity every few minutes or an hour? Any help would be appreciated. Cheers,Srini

we are using multithreading for ingesting bulk data from source using databricks notebook. we are running the notebooks as a job. we have a requirement like pausing the job and resuming the job from where it left off previously(since we can only ingest at particular time).wanted to know how to run the databricks notebook job to pause at certain time and resume it from where it had left off previously instead of starting from the scratch again.  

how i can convert below string date format to date time offset format and load into EDW 2020-10-05 04:52:48455-06:00 to2020-10-05 04:52:48.455-06:00

Hi there,Can we push DataOps / Deployment and Engine logs to CloudWatch LogGroups?Is there any documentation available? Cheers,Srini

Hi there,Is there any documentation available on DataOps Platform / Datadog integration?Is the integration allowed / possible? Cheers,Srini 

Hi All I am reading a CSV file created by a instrument , it produces a csv file with header information , then a collection of fields with values for each element tested for , I want to write each element value set to JDBC as a single rowexamplemachine_id, date, time , run Number,  element, element_value, element_error, element, element_value,element error (50 of these sets )i need to write to the database machine_id,data,time,run Number, element, element_value,element_ error , for each set of elements, values, errors , I can’t figure out  how to loop through the record,So from the 1 csv record I need to write 50 JDBC records , is it possible in streamsetsthanks

There is a function, str:splitKV which creates a map from a string, provided the key value pairs are encoded in the string, but how does one create a map from a group of individual fields?  I suppose one could convert the values into a single string with the necessary key/value pairings and then use the str:splitKV function, but such a conversion seems excessive and might require extra type conversions.Is there a better way?

we are ingesting the data from oracle to databricks,while ingesting i could see some of the staging files (CSV) in s3 bucket are unable to insert into databrciks, its showing as stage errors. is their a way to move these staging files to different bucket and retry it again ?  

Is there a particular scripting language that performs better; Groovy, Jython, Javascript?

 Can you send emails with attachments in a StreamSets pipeline?

How do I set up a Control Hub agent on an EKS cluster?

Can StreamSets deploy an in-house data-center without any public cloud services of StreamSets - completely isolated environment?

How do I add additional empty columns? Can I use a SMB share as a source? How do I delete or move connections between pipeline elements? Can the output destination be an Excel file? Can I control the permissions of the file created (i.e. read by all)?   

How do I connect a REST-API with OAuth 2 Authentication? 

How can I set a schema when the origin is an HTTP server? 

How can I set a schema without having to run the preview?

I can’t run a pipeline, it’s timing out. What do I do? 

Can I use julia scripts in StreamSets? 

How do I connect an Azure MySQL database using the binlog connector?

while ingesting data from oracle to databricks, i could see few staging files(s3 bucket) are moving into stage errorserror:’DELTA_LAKE_32’ -could not copy stage file <filename> Error running query , at least one column must be specified for the tablewhile we inserted the same record manually in databricks , we could see the record got inserted. No issues found on data as such.can you please give suggestions on the error which we are receiving. 

I built a pipeline with SFTP as the origin. The file I want to recover is an Excel file of around 100000 rows and 100 columns but it doesn't work.

How do I downloading a local version of StreamSets on my laptop?

I deployed a StreamSets data collector image from GCP marketplace but can't log in using the default initial username and password (admin, admin). Where can I find the correct initial username/password for my deployment?

What language is the expression editor in?