Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1209 Topics
S
Fan
sgargFan
asked in Community Articles and Got a Question?
Avro Message Key for kafka producer

Hi,  for Kafka producer, I’m trying to pass both key and value as "avro". while selecting that option, kafka message key shows as "${avro:decode(record:attribute('avroKeySchema'),base64:decodeBytes(record:attribute('<messsage key attribute name')))}documentation says, don't change the first part and second part should be message key attribute.. trying to figure out what would the value for these 2 attributes, as I have tried many options and pipeline is failing and unable to write to kafka.error I receive is, CTRCMN_0100 - Error evaluating expression ${avro:decode(record:attribute('RetailPartnerCase'),base64:decodeBytes(record:attribute('accountmanagement_retailpartner_eps_case_domain_event_v1-key')))}: java.lang.NullPointerException so, trying to understand, what I can pass into those 2 attributes? 

1323
Rishi
StreamSets Employee
2 years ago
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
audit logic for CDC pipelines

we are ingesting data using CDC pipelines(streamsets) from oracle source. so basically it ingests if any INSERT/UPDATE/DELETES are happening at source. Just wanted to how how can audit be performed on the same and its 24*7 and what can be the logic for audit checks.we are basically using databricks for audit checks

1124
Drew Kreiger
Rock star
2 years ago
SebastienM
Roadie
SebastienMRoadie
asked in StreamSets Academy
In Data Collector Fundamental course, Day 2, Lab - Data Drift, I get error 'HIVE_38 - unexpected authorization method SIMPLE, expected KERBEROS' when validating pipeline

Course and chapter url: https://academy.streamsets.com/courses/streamsets-data-collector/lessons/data-drift/topic/data-drift/ Screenshot of errors attached. 

972
A
Fan
2 years ago
R
Roadie
raghuramxlsoftekRoadie
asked in Community Articles and Got a Question?
Truncate and Load into Oracle Table Destination (JDBC)

The data pipeline reads data from a SQLServer Table Origin (JDBC) and loads into Oracle Table Destination (JDBC). i am trying to truncate the table before loading data into oracle destination table (JDBC). Is there any way to truncate the table before load data into destination?

6155
antmcmullen
StreamSets Employee
2 years ago
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
Datetime conversion in Data Collector

Hi there, How do I get to know the Time Zone that my Data Collector is set to?How do I change the Time Zone of my Data Collector? Reason: I use the pipeline:startTime() function and it returns the time in UTC, I would ideally like to have it as Australia - WST. Also, let me know how can I convert a UTC datetime value to a datetime value in AWST timezone. Note : I am on the DataOps platform. Cheers,

8901
S
Headliner
2 years ago
NareshThota
Fan
NareshThotaFan
posted in Community Articles and Got a Question?
Disable Trace and Options for HTTP server origin port

How to disable HTTP trace and options methods for http server origin ports. We have multiple pipelines which are serving api requests but these pipelines are allowing TRACE and OPTIONS methods also which would security vulnerability.We tried with nginx,  it's worked on data collector url but not on http server origin serving urls.

1370
NareshThota
Fan
2 years ago
A
Anonymous
posted in Share your Best Practices
Docker Warning Experienced with a M1 Chip Macbook

If you see the below warning when creating an engine with Docker on a macbook with an M1 chip,WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requestedadd --platform linux/amd64 to your docker run command.

1280
A
2 years ago
D
Fan
DavidCFan
asked in Community Articles and Got a Question?
SdcIpcTarget: Batch could not be written out: Server Error

Hello,We are getting following messages in sdc.log.What could be an issue here? Pipeline monitoring in GUI doesn't show errors. We tried to increase Connection timeout of SDC RPC Target and issue still persists.I also noticed, that some messages doesn't contain any offset number (first message). Might it be related?[thread:ProductionPipelineRunnable-pipelineName1[stage:SDCRPC_1] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '' could not be written out: Server Error2022-02-09 13:56:14,492 [user:admin] [pipeline:pipelineName2] [runner:0] [thread:ProductionPipelineRunnable-PipelineName2] [stage:SDCRPC_2] WARN SdcIpcTarget - Batch for entity '$com.streamsets.datacollector.pollsource.offset$' and offset '168' could not be written out: Server ErrorThank you. 

572
D
Fan
2 years ago
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
How to access or view audit logs from oracle source

we are using CDC(streamsets) for oracle source and , redo logs are enabled from source end(oracle). wanted to know how to view those redo logs so that we get an idea like which record is getting INSERTED/UPDATED/DELETED and for which table.

1253
Dimas Cabré i Chacón
StreamSets Employee
2 years ago
S
Fan
sprkFan
asked in Community Articles and Got a Question?
Couchbase subdocument operation

I am trying to update the array/list of values into the  couchbase document with ‘Allow Sub-Document Writes’ checked. The array value replaces the existing value on the CB instead of just appending them into the existing values already present on the CB document.My parameter in SDC,Sub-Document Path = tracking_numbersSub-Document operation = ARRAY_APPEND("tracking_numbers.USPS", record:value('USPS'))Incoming data to CB destination is “USPS: {LIST[]}CB doc structure,->document      ->tracking_numbers:             → USPS(array):                      [0]: 123                      [1]: 456 

390
S
Fan
2 years ago
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
Azure SSO - map Azure AD Group to Control Hub Groups

Hi, We use DataOps Platform and use Azure SSO provider.Can we map an Azure AD Group to a Control Hub Group? Thanks.

821
Greg
StreamSets Employee
2 years ago
Travis
Fan
TravisFan
asked in Community Articles and Got a Question?
Launching SDC with docker-compose fails

I’ve been spinning up a few docker containers within a docker-compose.yml and recently this has begun to fail with the following error. (my transformer instance has the same issue)Creating collector ... doneAttaching to collectorcollector      | Engine running in Deployment Modecollector      | Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTScollector      | bash: line 1: {ISSUES:[code:SSO_01]}: command not foundcollector      | Engine Pre Start failed with exit code 127collector exited with code 127 Just executing the docker run command still works fine but sorting out the networking will be a pain so getting this functionality back would be helpful. This used to work, but recently has stopped. I’m bringing up a local database and hopefully an instance of a Vault credential store and having the containers linked in the docker-compose file is convenient for development and testing.The service

2537
Drew Kreiger
Rock star
2 years ago
Kavya
StreamSets Employee
KavyaStreamSets Employee
asked in Community Articles and Got a Question?
Not able to convert value field when writing to InfluxDB

I’m not familiar with using InfluxDB as a destination, but we have a user having trouble converting the value field in their pipeline. Anyone happen to know how he might be able to solve this?“...it seems that the value field (the one I specify in the value fields input box) are all stored as String, even though I converted it to Double using a Field Type Converter before sending it to InfluxDB. Do you use the line protocol when writing to InfluxDB?”

470
Kavya
StreamSets Employee
2 years ago
P
Fan
Philippe REYFan
asked in Community Articles and Got a Question?
How To execute a Snowflake Stored Procedure ?

Is it possible to execute a custom SQL script (dynamic creation with parameters) or a stored procedure in snowflake ?Thanks in advance.

5202
P
Fan
2 years ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles and Got a Question?
How to deploy SDC in k8s cluster without provisioning agent?

The Ideal approach via provisioning agent:A Provisioning Agent is a containerized application that runs in a Kubernetes container orchestration framework. The agent communicates with Control Hub to automatically provision Data Collector containers in the Kubernetes cluster in which it runs. Provisioning includes deploying, registering, starting, scaling, and stopping the Data Collector containers. You can configure the Provisioning Agent to provision Data Collector containers enabled for Kerberos authentication. You can find steps here However, if you want to avoid provisionig agent installation due to any reason. you can follow the below steps for reference.This is the bare minimum setup required to deploy SDC in any k8s without control-agent. Step 1: Create a lower-cased UUID and store it in a secret $SDC_ID=`uuidgen | tr "[:upper:]" "[:lower:]"`$echo "Generated sdc.id "${SDC_ID}$kubectl create secret generic sdc-id --from-literal=sdc.id=${SDC_ID}Step 2: Get an SDC auth token from Co

1390
Rishi
StreamSets Employee
2 years ago
ashok verma
Discovered Fame
ashok vermaDiscovered Fame
asked in Community Articles and Got a Question?
lookup from list of values

i have a XML which has an element country and i have to check whether this country is part of my list.country in (‘USA’,’AUS’)is there any component where i can configure or i have to write in Evaluator to handle this logic 

3856
swayam
Discovered Fame
2 years ago
S
Fan
smangamFan
asked in Community Articles and Got a Question?
Support for Hive3 ACID propeties

With the introduction of Hive3, we are seeing that ACID properties will be supported in a HDFS ecosystem.  More information can be found here  : Hive3-DocI wanted to check if there are any plans to support this functionality on the Hive Metadata generator module?   Thanks.

150
S
Fan
2 years ago
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
converting timestamp to dd/mm/yyyy format through streamsets

we are ingesting data from oracle and it has timestamp format as mm/dd/yyyy hhss AM/PM, while ingesting through streamsets its taking as yyyy-mm-dd hhss. how to change the timestamp format to source format which is mm/dd/yyyy hhss AM/PM in streamsets

10623
Pradeep
StreamSets Employee
2 years ago
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
DataOps Platform / Control Hub - Data Collector Engine is unreachable

Team, Our DataOps deployment runs on AWS EC2. We have one Data Collector engine running on that deployment.  We are seeing this message frequently, i.e. we are losing connection to our Data Collector engine quite often. How do I check whether my Data Collector is running fine on EC2 machine (Linux) to confirm its the connection issue (Control Hub talking to Data Collector Engine) ? Any idea on why we see lose engine connectivity every few minutes or an hour? Any help would be appreciated. Cheers,Srini

351
Rishi
StreamSets Employee
2 years ago
harshith
Discovered Fame
harshithDiscovered Fame
asked in Community Articles and Got a Question?
pause and resume the job from where we left off

we are using multithreading for ingesting bulk data from source using databricks notebook. we are running the notebooks as a job. we have a requirement like pausing the job and resuming the job from where it left off previously(since we can only ingest at particular time).wanted to know how to run the databricks notebook job to pause at certain time and resume it from where it had left off previously instead of starting from the scratch again.  

401
Drew Kreiger
Rock star
2 years ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
how to convert sting to datetime offset

how i can convert below string date format to date time offset format and load into EDW 2020-10-05 04:52:48455-06:00 to2020-10-05 04:52:48.455-06:00

2332
xavi
StreamSets Employee
2 years ago
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
DataOps Platform - CloudWatch LogGroup

Hi there,Can we push DataOps / Deployment and Engine logs to CloudWatch LogGroups?Is there any documentation available? Cheers,Srini

240
S
Headliner
2 years ago
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
DataOps Platform - Datadog Integration

Hi there,Is there any documentation available on DataOps Platform / Datadog integration?Is the integration allowed / possible? Cheers,Srini 

550
S
Headliner
2 years ago
Gavin
Fan
GavinFan
asked in Community Articles and Got a Question?
read CSV Split some fields into multiple records and write to JDBC

Hi All I am reading a CSV file created by a instrument , it produces a csv file with header information , then a collection of fields with values for each element tested for , I want to write each element value set to JDBC as a single rowexamplemachine_id, date, time , run Number,  element, element_value, element_error, element, element_value,element error (50 of these sets )i need to write to the database machine_id,data,time,run Number, element, element_value,element_ error , for each set of elements, values, errors , I can’t figure out  how to loop through the record,So from the 1 csv record I need to write 50 JDBC records , is it possible in streamsetsthanks

3374
Gavin
Fan
2 years ago
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
How does one create a map from a set of fields?

There is a function, str:splitKV which creates a map from a string, provided the key value pairs are encoded in the string, but how does one create a map from a group of individual fields?  I suppose one could convert the values into a single string with the necessary key/value pairings and then use the str:splitKV function, but such a conversion seems excessive and might require extra type conversions.Is there a better way?

8063
Drew Kreiger
Rock star
2 years ago

Badge winners

Show all badges
Terms & Conditions