- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 year ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1197 Topics

Recently active

Most replies Most views Newest first

KavyaStreamSets Employee

asked in Community Articles and Got a Question?

Not able to convert value field when writing to InfluxDB

I’m not familiar with using InfluxDB as a destination, but we have a user having trouble converting the value field in their pipeline. Anyone happen to know how he might be able to solve this?“...it seems that the value field (the one I specify in the value fields input box) are all stored as String, even though I converted it to Double using a Field Type Converter before sending it to InfluxDB. Do you use the line protocol when writing to InfluxDB?”

2 years ago

Philippe REYFan

asked in Community Articles and Got a Question?

How To execute a Snowflake Stored Procedure ?

Is it possible to execute a custom SQL script (dynamic creation with parameters) or a stored procedure in snowflake ?Thanks in advance.

2 years ago

RishiStreamSets Employee

posted in Community Articles and Got a Question?

How to deploy SDC in k8s cluster without provisioning agent?

The Ideal approach via provisioning agent:A Provisioning Agent is a containerized application that runs in a Kubernetes container orchestration framework. The agent communicates with Control Hub to automatically provision Data Collector containers in the Kubernetes cluster in which it runs. Provisioning includes deploying, registering, starting, scaling, and stopping the Data Collector containers. You can configure the Provisioning Agent to provision Data Collector containers enabled for Kerberos authentication. You can find steps here However, if you want to avoid provisionig agent installation due to any reason. you can follow the below steps for reference.This is the bare minimum setup required to deploy SDC in any k8s without control-agent. Step 1: Create a lower-cased UUID and store it in a secret $SDC_ID=`uuidgen | tr "[:upper:]" "[:lower:]"`$echo "Generated sdc.id "${SDC_ID}$kubectl create secret generic sdc-id --from-literal=sdc.id=${SDC_ID}Step 2: Get an SDC auth token from Co

2 years ago

ashok vermaDiscovered Fame

asked in Community Articles and Got a Question?

lookup from list of values

i have a XML which has an element country and i have to check whether this country is part of my list.country in (‘USA’,’AUS’)is there any component where i can configure or i have to write in Evaluator to handle this logic

2 years ago

smangamFan

asked in Community Articles and Got a Question?

Support for Hive3 ACID propeties

With the introduction of Hive3, we are seeing that ACID properties will be supported in a HDFS ecosystem. More information can be found here : Hive3-DocI wanted to check if there are any plans to support this functionality on the Hive Metadata generator module? Thanks.

2 years ago

harshithDiscovered Fame

asked in Community Articles and Got a Question?

converting timestamp to dd/mm/yyyy format through streamsets

we are ingesting data from oracle and it has timestamp format as mm/dd/yyyy hhss AM/PM, while ingesting through streamsets its taking as yyyy-mm-dd hhss. how to change the timestamp format to source format which is mm/dd/yyyy hhss AM/PM in streamsets

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

DataOps Platform / Control Hub - Data Collector Engine is unreachable

Team, Our DataOps deployment runs on AWS EC2. We have one Data Collector engine running on that deployment. We are seeing this message frequently, i.e. we are losing connection to our Data Collector engine quite often. How do I check whether my Data Collector is running fine on EC2 machine (Linux) to confirm its the connection issue (Control Hub talking to Data Collector Engine) ? Any idea on why we see lose engine connectivity every few minutes or an hour? Any help would be appreciated. Cheers,Srini

2 years ago

harshithDiscovered Fame

asked in Community Articles and Got a Question?

pause and resume the job from where we left off

we are using multithreading for ingesting bulk data from source using databricks notebook. we are running the notebooks as a job. we have a requirement like pausing the job and resuming the job from where it left off previously(since we can only ingest at particular time).wanted to know how to run the databricks notebook job to pause at certain time and resume it from where it had left off previously instead of starting from the scratch again.

2 years ago

akanshajain6793Discovered Fame

asked in Community Articles and Got a Question?

how to convert sting to datetime offset

how i can convert below string date format to date time offset format and load into EDW 2020-10-05 04:52:48455-06:00 to2020-10-05 04:52:48.455-06:00

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

DataOps Platform - CloudWatch LogGroup

Hi there,Can we push DataOps / Deployment and Engine logs to CloudWatch LogGroups?Is there any documentation available? Cheers,Srini

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

DataOps Platform - Datadog Integration

Hi there,Is there any documentation available on DataOps Platform / Datadog integration?Is the integration allowed / possible? Cheers,Srini

2 years ago

GavinFan

asked in Community Articles and Got a Question?

read CSV Split some fields into multiple records and write to JDBC

Hi All I am reading a CSV file created by a instrument , it produces a csv file with header information , then a collection of fields with values for each element tested for , I want to write each element value set to JDBC as a single rowexamplemachine_id, date, time , run Number, element, element_value, element_error, element, element_value,element error (50 of these sets )i need to write to the database machine_id,data,time,run Number, element, element_value,element_ error , for each set of elements, values, errors , I can’t figure out how to loop through the record,So from the 1 csv record I need to write 50 JDBC records , is it possible in streamsetsthanks

2 years ago

mblahayDiscovered Fame

asked in Community Articles and Got a Question?

How does one create a map from a set of fields?

There is a function, str:splitKV which creates a map from a string, provided the key value pairs are encoded in the string, but how does one create a map from a group of individual fields? I suppose one could convert the values into a single string with the necessary key/value pairings and then use the str:splitKV function, but such a conversion seems excessive and might require extra type conversions.Is there a better way?

2 years ago

harshithDiscovered Fame

asked in Community Articles and Got a Question?

unable to ingest to databricks using streamsets

we are ingesting the data from oracle to databricks,while ingesting i could see some of the staging files (CSV) in s3 bucket are unable to insert into databrciks, its showing as stage errors. is their a way to move these staging files to different bucket and retry it again ?

2 years ago

mblahayDiscovered Fame

asked in Community Articles and Got a Question?

Which scripting language performs best?

Is there a particular scripting language that performs better; Groovy, Jython, Javascript?

2 years ago