Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,051 Topics
  • 2,324 Replies
1045 Topics
NareshThota
Fan
NareshThotaFan
posted in Community Articles and Got a Question?
Disable Trace and Options for HTTP server origin port

How to disable HTTP trace and options methods for http server origin ports. We have multiple pipelines which are serving api requests but these pipelines are allowing TRACE and OPTIONS methods also which would security vulnerability.We tried with nginx,  it's worked on data collector url but not on http server origin serving urls.

1390
NareshThota
Fan
D
Fan
DavidCFan
asked in Community Articles and Got a Question?
SdcIpcTarget: Batch could not be written out: Server Error

Hello,We are getting following messages in sdc.log.What could be an issue here? Pipeline monitoring in GUI doesn't show errors. We tried to increase Connection timeout of SDC RPC Target and issue still persists.I also noticed, that some messages doesn't contain any offset number (first message). Might it be related?[thread:ProductionPipelineRunnable-pipelineName1[stage:SDCRPC_1] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '' could not be written out: Server Error2022-02-09 13:56:14,492 [user:admin] [pipeline:pipelineName2] [runner:0] [thread:ProductionPipelineRunnable-PipelineName2] [stage:SDCRPC_2] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '168' could not be written out: Server ErrorThank you. 

592
D
Fan
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
How to access or view audit logs from oracle source

we are using CDC(streamsets) for oracle source and , redo logs are enabled from source end(oracle). wanted to know how to view those redo logs so that we get an idea like which record is getting INSERTED/UPDATED/DELETED and for which table.

1333
Dimas Cabré i Chacón
StreamSets Employee
S
Fan
sprkFan
asked in Community Articles and Got a Question?
Couchbase subdocument operation

I am trying to update the array/list of values into the  couchbase document with ‘Allow Sub-Document Writes’ checked. The array value replaces the existing value on the CB instead of just appending them into the existing values already present on the CB document.My parameter in SDC,Sub-Document Path = tracking_numbersSub-Document operation = ARRAY_APPEND("tracking_numbers.USPS", record:value('USPS'))Incoming data to CB destination is “USPS: {LIST[]}CB doc structure,->document      ->tracking_numbers:             → USPS(array):                      [0]: 123                      [1]: 456 

400
S
Fan
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
Azure SSO - map Azure AD Group to Control Hub Groups

Hi, We use DataOps Platform and use Azure SSO provider.Can we map an Azure AD Group to a Control Hub Group? Thanks.

831
Greg
StreamSets Employee
Travis
Fan
TravisFan
asked in Community Articles and Got a Question?
Launching SDC with docker-compose fails

I’ve been spinning up a few docker containers within a docker-compose.yml and recently this has begun to fail with the following error. (my transformer instance has the same issue)Creating collector ... doneAttaching to collectorcollector      | Engine running in Deployment Modecollector      | Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTScollector      | bash: line 1: {ISSUES:[code:SSO_01]}: command not foundcollector      | Engine Pre Start failed with exit code 127collector exited with code 127 Just executing the docker run command still works fine but sorting out the networking will be a pain so getting this functionality back would be helpful. This used to work, but recently has stopped. I’m bringing up a local database and hopefully an instance of a Vault credential store and having the containers linked in the docker-compose file is convenient for development and testing.The service

2606
Drew Kreiger
Rock star
Kavya
StreamSets Employee
KavyaStreamSets Employee
asked in Community Articles and Got a Question?
Not able to convert value field when writing to InfluxDB

I’m not familiar with using InfluxDB as a destination, but we have a user having trouble converting the value field in their pipeline. Anyone happen to know how he might be able to solve this?“...it seems that the value field (the one I specify in the value fields input box) are all stored as String, even though I converted it to Double using a Field Type Converter before sending it to InfluxDB. Do you use the line protocol when writing to InfluxDB?”

470
Kavya
StreamSets Employee
P
Fan
Philippe REYFan
asked in Community Articles and Got a Question?
How To execute a Snowflake Stored Procedure ?

Is it possible to execute a custom SQL script (dynamic creation with parameters) or a stored procedure in snowflake ?Thanks in advance.

5262
P
Fan
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles and Got a Question?
How to deploy SDC in k8s cluster without provisioning agent?

The Ideal approach via provisioning agent:A Provisioning Agent is a containerized application that runs in a Kubernetes container orchestration framework. The agent communicates with Control Hub to automatically provision Data Collector containers in the Kubernetes cluster in which it runs. Provisioning includes deploying, registering, starting, scaling, and stopping the Data Collector containers. You can configure the Provisioning Agent to provision Data Collector containers enabled for Kerberos authentication. You can find steps here However, if you want to avoid provisionig agent installation due to any reason. you can follow the below steps for reference.This is the bare minimum setup required to deploy SDC in any k8s without control-agent. Step 1: Create a lower-cased UUID and store it in a secret $SDC_ID=`uuidgen | tr "[:upper:]" "[:lower:]"`$echo "Generated sdc.id "${SDC_ID}$kubectl create secret generic sdc-id --from-literal=sdc.id=${SDC_ID}Step 2: Get an SDC auth token from Co

1410
Rishi
StreamSets Employee
ashok verma
Discovered Fame
ashok vermaDiscovered Fame
asked in Community Articles and Got a Question?
lookup from list of values

i have a XML which has an element country and i have to check whether this country is part of my list.country in (‘USA’,’AUS’)is there any component where i can configure or i have to write in Evaluator to handle this logic 

3966
swayam
Discovered Fame
S
Fan
smangamFan
asked in Community Articles and Got a Question?
Support for Hive3 ACID propeties

With the introduction of Hive3, we are seeing that ACID properties will be supported in a HDFS ecosystem.  More information can be found here  : Hive3-DocI wanted to check if there are any plans to support this functionality on the Hive Metadata generator module?   Thanks.

150
S
Fan
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
converting timestamp to dd/mm/yyyy format through streamsets

we are ingesting data from oracle and it has timestamp format as mm/dd/yyyy hhss AM/PM, while ingesting through streamsets its taking as yyyy-mm-dd hhss. how to change the timestamp format to source format which is mm/dd/yyyy hhss AM/PM in streamsets

11213
Pradeep
StreamSets Employee
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
DataOps Platform / Control Hub - Data Collector Engine is unreachable

Team, Our DataOps deployment runs on AWS EC2. We have one Data Collector engine running on that deployment.  We are seeing this message frequently, i.e. we are losing connection to our Data Collector engine quite often. How do I check whether my Data Collector is running fine on EC2 machine (Linux) to confirm its the connection issue (Control Hub talking to Data Collector Engine) ? Any idea on why we see lose engine connectivity every few minutes or an hour? Any help would be appreciated. Cheers,Srini

351
Rishi
StreamSets Employee
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
pause and resume the job from where we left off

we are using multithreading for ingesting bulk data from source using databricks notebook. we are running the notebooks as a job. we have a requirement like pausing the job and resuming the job from where it left off previously(since we can only ingest at particular time).wanted to know how to run the databricks notebook job to pause at certain time and resume it from where it had left off previously instead of starting from the scratch again.  

401
Drew Kreiger
Rock star
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
how to convert sting to datetime offset

how i can convert below string date format to date time offset format and load into EDW 2020-10-05 04:52:48455-06:00 to2020-10-05 04:52:48.455-06:00

2362
xavi
StreamSets Employee
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
DataOps Platform - CloudWatch LogGroup

Hi there,Can we push DataOps / Deployment and Engine logs to CloudWatch LogGroups?Is there any documentation available? Cheers,Srini

240
S
Headliner
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
DataOps Platform - Datadog Integration

Hi there,Is there any documentation available on DataOps Platform / Datadog integration?Is the integration allowed / possible? Cheers,Srini 

660
S
Headliner
Gavin
Fan
GavinFan
asked in Community Articles and Got a Question?
read CSV Split some fields into multiple records and write to JDBC

Hi All I am reading a CSV file created by a instrument , it produces a csv file with header information , then a collection of fields with values for each element tested for , I want to write each element value set to JDBC as a single rowexamplemachine_id, date, time , run Number,  element, element_value, element_error, element, element_value,element error (50 of these sets )i need to write to the database machine_id,data,time,run Number, element, element_value,element_ error , for each set of elements, values, errors , I can’t figure out  how to loop through the record,So from the 1 csv record I need to write 50 JDBC records , is it possible in streamsetsthanks

3464
Gavin
Fan
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
How does one create a map from a set of fields?

There is a function, str:splitKV which creates a map from a string, provided the key value pairs are encoded in the string, but how does one create a map from a group of individual fields?  I suppose one could convert the values into a single string with the necessary key/value pairings and then use the str:splitKV function, but such a conversion seems excessive and might require extra type conversions.Is there a better way?

8323
Drew Kreiger
Rock star
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
unable to ingest to databricks using streamsets

we are ingesting the data from oracle to databricks,while ingesting i could see some of the staging files (CSV) in s3 bucket are unable to insert into databrciks, its showing as stage errors. is their a way to move these staging files to different bucket and retry it again ?  

957
Drew Kreiger
Rock star
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
Which scripting language performs best?

Is there a particular scripting language that performs better; Groovy, Jython, Javascript?

682
S
StreamSets Employee
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
asked in Community Articles and Got a Question?
Can you send emails with attachments in a StreamSets pipeline?

 Can you send emails with attachments in a StreamSets pipeline?

1851
Drew Kreiger
Rock star
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
asked in Community Articles and Got a Question?
How do I set up a Control Hub agent on an EKS cluster?

How do I set up a Control Hub agent on an EKS cluster?

491
Drew Kreiger
Rock star
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
asked in Community Articles and Got a Question?
Can StreamSets deploy an in-house data-center without any public cloud services of StreamSets - completely isolated environment?

Can StreamSets deploy an in-house data-center without any public cloud services of StreamSets - completely isolated environment?

131
Drew Kreiger
Rock star
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
asked in Community Articles and Got a Question?
I'm converting an Excel file to another format, with different order and column naming. The things I need help with are: (See Thread)

How do I add additional empty columns? Can I use a SMB share as a source? How do I delete or move connections between pipeline elements? Can the output destination be an Excel file? Can I control the permissions of the file created (i.e. read by all)?   

531
Drew Kreiger
Rock star

Badge winners

Show all badges
Terms & Conditions