Community Articles

Contribute and publish your articles/blogs here showcasing you usage, architecture, and know how of the StreamSets DataOps Platform. THIS IS NOT THE Q&A CATEGORY. Head over to the "Got a Question?" Category to ask your question.

  • 57 Topics
  • 96 Replies

57 Topics

N
Opening Band
N_aliOpening Band
asked in Community Articles

aggregate the records into a single array of objects

I have some json objects that looks as follows and i want them to be all merged in an array like :data: [{nstaskid: “...”,awb: “….”},{nstaskid: “...”,awb: “….”},] i tried to use javascript evalutor , loop over the records and push them in an array and write this record array , but it didn’t work.Can you please help me with that .

15
tamilselg
Fan
6 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to get count of number of parameter passed in a request

Hi teami have column request_param:0:   a : “11”   b: “12”   c: “12”   d: “12” how i can get count if above request has 4 parameter it can have 3 also like belowrequest_param:0:   a : “11”   b: “12”   c: “12” based on count i need to hit diff url 

7
saleempothiwala
Opening Band
13 days ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

how to disable autosave in pipeline

Hello everybody, how can we remove the "autosave" option in the pipeline?  any changes that are made in the pipeline, the streamsets saves automatically, we need the save to be manual How can we do that?

6
Yago Aparecido
Discovered Fame
8 months ago
Vikki
Fan
VikkiFan
asked in Community Articles

REST API

HI team,    I am having 6 API’s where I have to build 3 pipelines for each i.e., API-KAFKA, KAFKA-FILESYSTEM and FS-Target table. Like this  For 6 API’s I have to build and execute 18 pipelines (6*3). Could anyone suggest me  that is there any approach which reduces the number of pipelines ( to push all 6 APIs data into 1 Kafka topic in 1st pipeline, after that I will generate data file from Kafka to FS and FS to target table).FYI- my target table is same for all APIs.

6
Vikki
Fan
7 months ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

how to check jvm usage of streamsets by command line?

hello streamsets community, the streamsets is installed in cloudera manager, but it does not have graphics that show the memory usage of the jvm with that, i need to know how to query the jvm memory usage of streamsets by the command line.

6
Bikram
Headliner
5 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to generate UUID bases on timestamp

Hi Team I have a date datefield storing current datetime in it want to create a UUID using same column how i can create . currently using groovy script to create UUID but want to use same column and create it 

5
A
Discovered Fame
23 days ago
karansing
Fan
karansingFan
asked in Community Articles

Post-processing capabilities for GoogleCloudStorage_01

Hello everyone,I just wanted to ask that don’t we have post-processing capability for “GoogleCloudStorage_01”?Basically we are using “streamsets/datacollector:3.18.1” for some testing purpose and we are fetching data from GCS and want to move file to some other location post processing.

5
AkshayJadhav
StreamSets Employee
4 days ago
D
Fan
durairajFan
asked in Community Articles

Postgres CDC client is not processing data, when slot volume reaches 1 GB

We are using postgres CDC client to ingest the data from postges to ADLS. It’s working for other pipeline which are getting the CDC records lesser than 1 GB. While we drop and recreate the slots its working fine. But after sometime CDC files are generating from Streamsets. In pipeline logs we are not find error logs and jobs is active and running.while querying pg_replication_slots volume of data reaches more than 1 GB to several. until drop the replication slots CDC pipeline is not streaming.So, please suggest how to fix the issue. Do, we need to change the Streamsets configuration or postgres configuration. please let us know.Our Streamsets config. as below,Max Batch Size : 15000Streamsets engine : 4.4.1Batch Wait Time : 15000Query Timeout : {45 * MINUTES}Poll Interval: {1 * SECONDS}Status Interval : ${30 * SECONDS}CDC Generator Queue Size : 20000

5
Bikram
Headliner
2 months ago
sujatha mogili
Fan
sujatha mogiliFan
asked in Community Articles

The streamsets pipeline fails with java.lang.OutOfMemoryError: Java heap space

The streamsets pipeline fails on a regular basis with the below error. Please advise if you ran into this issue and the resolution. We have increased Heap size couple of times but not helping.“ERROR    A JVM error occurred while running the pipeline, java.lang.OutOfMemoryError: Java heap space”

5
alex.sanchez
StreamSets Employee
7 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

Webservice Array request Json

Hi team I have two json request for singleton[{“a”: “1”“b”:”2”“c”:”3”}]and for batch[{“a”: “1”“b”:”2”“c”:”3”},{“x”: “4”“y”:”5”“z”:”6”}] if singleton come i need to pass first request and if batch comes i need to pick the second one and in the second one i need to pick first value. am using json generator how to convert that into array as a request and then for batch how to pick the first value from multiple values.   

5
Bikram
Headliner
1 month ago
N
Fan
NileshFan
asked in Community Articles

Oracle CDC to Snowflake while processing CLOB datatype

Hi,Facing error in Data collector pipeline Oracle CDC to Snowflake while processing CLOB datatype table. When inserting value >= 4000 then works fine. But the less than that facing error.SNOWFLAKE_28 - Snowflake MERGE for 'sdc-df26c7ce-fd00-4b5d-a3b1-817b9809e0a3.csv.gz' processed '1' CDC records out of '3'Oracle:create table TEST_LOB_VAL(lobid number(10) primary key, lob_value clob );INSERT INTO TEST_LOB_VAL VALUES (1,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',4000,'A')));INSERT INTO TEST_LOB_VAL VALUES (2,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',2000,'A')));INSERT INTO TEST_LOB_VAL VALUES (3,'AAAAAAAAAA');COMMIT;

3
AkshayJadhav
StreamSets Employee
7 hours ago
deepikamani007
Fan
deepikamani007Fan
asked in Community Articles

how to configure mongodp connection string

i dont known how to configure the mongodb opblog origin. 

3
Marina
Fan
7 days ago
A
Fan
akshayp2Fan
asked in Community Articles

How to configure the transformer pipeline to run on AWS EMR cluster?

HiI have created a simple transformer pipeline trying to run on EMR cluster. In Cluster configuration I have used required configuration details.Facing below issue,  Can anyone help me this?

3
wilsonshamim
StreamSets Employee
1 month ago
H
Fan
harshit_agrawal1Fan
asked in Community Articles

Configuring a trigger to activate when a file is created in a directory

I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead? 

3
saleempothiwala
Opening Band
1 month ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to connect and load data to aerospike from streamsets

Hi Team did any one work on aerospike load from streamsets i see the stage is deprecated can we use jdbc aerospike connector ?

3
Bikram
Headliner
1 month ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

Getting error with HTTP client

Getting below error while passing URL as parameter any input for same  

3
Bikram
Headliner
24 days ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

list all streamsets pipelines by rest api

hi streamsets community, we need to export the listing of all pipelines that exist in streamsets by rest api How can we do that?

2
Yago Aparecido
Discovered Fame
5 months ago
V
Fan
vgohilFan
asked in Community Articles

JOBRUNNER_103 error when trying to run complex pipeline (with fragment)

I am following the fragment lesson for which i have imported the pipeline and created a duplicate. With the duplicate pipeline, i managed to create a fragment, edit the pipeline and remove the components and add the new fragment. Upon publishing the pipeline i am receiving the following error below... JOBRUNNER_103:Organization 4441d489-761c-11ed-aae0-e11f88355e74 already has 2 running jobs which is the maximum allowed. Please contact your StreamSets account team or email subscriptions@streamsets.com Can anyone please suggest what to do here to successfully start the job.

1
saleempothiwala
Opening Band
3 months ago
sperandio
Fan
sperandioFan
asked in Community Articles

Global variable to reduce memory usage and API calls

I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all  previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that. Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record? 

1
Dimas Cabré i Chacón
StreamSets Employee
7 months ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in Community Articles

Swayam's Micro-blogging: Combine multiple JSON records into a single one

Dear Community users,To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?I often came with requirements where we have to call an external API which suports both single as well as batch requests and most often batch is preffered to avoid the multiple round-trips and increae performance. This topic can be very useful for developers who wants to convert the multiple CSV records coming in a batch to a single JSON list array and send them as a single record to a target destination.Problem StatementConvert the 2 input CSV records into a single JSON.Input CSV Records:NAME|AGE|DEPARTMENT|ADDRESS"John"|"23"|"HR"|"31, 2nd Cross, 4th Street, Wilson-543234""Amanda"|"26"|"HR"|"24, 1st Cross, Wilson Garden-543234"Output Single JSON Record:[ { "data": { "NAME": "John", "AGE": "23", "DEPARTMENT": "HR", "ADDRESS": "31, 2nd Cross, 4th Street, Wilson-543234" } }, { "data": {

1
Drew Kreiger
Rock star
1 year ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in Community Articles

Swayam's Micro-blogging: Auto detection of date format

Dear Community users,SDC offers a great out-of-box processor “Field Type Converter” to convert an incoming field of one data type to another data type. This is a very useful transformation function that is required before we send the data to final destination. However, sometime we may have to write our own transformation function and today I’m going to share a scenario where I was forced to use a jython evaluator to detect the date format of an incoming field and convert that to a single unified format that our destination is expecting.Problem StatementWe have a need to process the incoming file where there is a field that need to be processed as “date” data type while there is another field which needs to be processed as “timestamp” data type before we generate the final feed for destination. The detaination while accepts the “date” fields is YYYY-MM-DD format, the timestamp fileds need to be of YYYY-MM-DD HH:MI:SS GMT+05:30, the origin fields can be of different formats. The solution

1
Drew Kreiger
Rock star
1 year ago
amarzeet
Fan
amarzeetFan
asked in Community Articles

Streamset data collector 3.x tarball download link not available

I want to use stream set data collector legacy version (3.x) on a linux machine. Only option is to install the tar ball. I couldn't find a link on streamset website to download and use it. Can you please provide this information

1
albertfc
Roadie
2 months ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

jvm memory exceed values defined in xmx and xms

Hello streamset community, I configured xmx and xms with 35GB, like this: SDC_JAVA_OPTS="-Xmx35840m -Xms35840m means that 35G has been configured, but querying the service status with "systemctl status sdc" command, the memory value exceeds 37G, as shown below:  why does this occur? does the memory value that appears with the "systemctl status sdc" command have a different conversion or does this memory add to another type of memory?

1
saleempothiwala
Opening Band
7 months ago
btsmerchstore45
Fan
btsmerchstore45Fan
asked in Community Articles

Is it possible to configure the transformer pipeline to run on an AWS EMR cluster?

I have created a simple transformer pipeline to run on an EMR cluster. In Cluster configuration, I have used the necessary configuration details.I am facing the following issue:

1
saleempothiwala
Opening Band
1 month ago
N
Fan
nick_nilzFan
asked in Community Articles

How do you introduce (write) new columns to an existing hadoop table?

Hello community,My pipeline is being upgraded to ingest more fields and I want to write these new fields to the existing table in Hadoop. I have searched for the metadata processor but it does not exist and I have enabled data drift. None of them are able to write new columns, which means I have to write an ALTER TABLE statement to capture these new fields. With time we will get further requirements to expand our tables and introduce new fields. Is StreamSets able to write new fields automatically?Kind regards,Nick

1
N
Fan
6 months ago

Badge winners

Show all badges
Terms & Conditions