Community Articles

Contribute and publish your articles/blogs here showcasing you usage, architecture, and know how of the StreamSets DataOps Platform. THIS IS NOT THE Q&A CATEGORY. Head over to the "Got a Question?" Category to ask your question.

  • 83 Topics
  • 114 Replies

83 Topics

nicholas.mendoza
Fan
nicholas.mendozaFan
posted in Community Articles

Deployment error with Ingress and authoring Data Collector

Issue:When trying to deploy an authoring Data Collector with Control Agent on Control Hub, the following error is received:io.fabric8.kubernetes.api.model.networking.v1.Ingress cannot be cast to io.fabric8.kubernetes.api.model.extensions.Ingress Cause:This is due to the Ingress apiVersion being set to networking.k8s.io/v1, while Control Agent expects the Ingress to be of the extensions api group (extensions/v1beta1). However, in Kubernetes versions 1.22+, the extensions api group was made unavailable. Solution:The workaround for this would be to create the Ingress resource outside of Control Agent, separately from the SDC deployment. Additionally, ensure that an Ingress controller is set up in your cluster, such as the ingress-nginx Ingress controller.

0
nicholas.mendoza
Fan
2 months ago
karansing
Fan
karansingFan
asked in Community Articles

Post-processing capabilities for GoogleCloudStorage_01

Hello everyone,I just wanted to ask that don’t we have post-processing capability for “GoogleCloudStorage_01”?Basically we are using “streamsets/datacollector:3.18.1” for some testing purpose and we are fetching data from GCS and want to move file to some other location post processing.

5
AkshayJadhav
StreamSets Employee
2 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

How to deploy legacy Kubernates without the use of helm in dataops platform

In this article we will walkthroughs the steps involved to deploy engines in kubernetes using “Legacy Kubernetes” support in DataOps Platform. Note: Legacy Kubernetes integration is enabled only for some paid accounts. For more information, contact your StreamSets account team. Once you have Legacy Kubernetes enabled for your org, you should see the following additional option.Prerequisite:jq installed and of course Kubernates environment of your choice Create a namespace where you are expected to have the control-agent, StreamSets engines(SDC, Transformer for Spark etc.)         Eg command: kubectl create namespace dataopsStep-1: Clone the git repo https://github.com/onefoursix/control-agent-k8s-deploymentStep-2: Set these variables at the top of the file deploy-control-agent-on-dataops-platform.sh:ORG_ID=<Your DataOps Platform Org ID>SCH_URL=<Your DataOps Platform URL> # for example: https://na01.hub.streamsets.comCRED_ID=<Your API Credential CRED_ID>CRED_TOKEN=<

0
Pradeep
StreamSets Employee
2 months ago
pedro
StreamSets Employee
pedroStreamSets Employee
posted in Community Articles

A Cloudera Manager managed Data Collector does not create logs after upgrading to 5.x

After successfully upgrading Data Collector using Cloudera Manager from a previous version to 5.x you notice that the log screen does not get any log entries and there is no sdc.log file generated in the $SDC_LOG directory. Cloudera Manager uses CSD (custom service descriptor) files to determine the configuration, monitoring and file distribution of a custom service (in this case SDC) that will be managed by Cloudera Manager. You can learn more about this in Cloudera’s documentation.  As part of the SDC upgrade steps you need to install the newest CSD file. This has always been a requirement but in previous versions, not updating it caused little to no problems. For this reason some users may have skipped this step for previous SDC upgrades. For upgrades of older SDC versions to 5.x, not updating the CSD file will result in no logs being generated. SDC 5.x uses log4j 2.x while older versions of SDC used log4j 1.x which are not compatible with each other. If the old CSD file is kept in

0
pedro
StreamSets Employee
2 months ago
deepikamani007
Fan
deepikamani007Fan
asked in Community Articles

how to configure mongodp connection string

i dont known how to configure the mongodb opblog origin. 

3
Marina
Fan
2 months ago
N
Fan
NileshFan
asked in Community Articles

Oracle CDC to Snowflake while processing CLOB datatype

Hi,Facing error in Data collector pipeline Oracle CDC to Snowflake while processing CLOB datatype table. When inserting value >= 4000 then works fine. But the less than that facing error.SNOWFLAKE_28 - Snowflake MERGE for 'sdc-df26c7ce-fd00-4b5d-a3b1-817b9809e0a3.csv.gz' processed '1' CDC records out of '3'Oracle:create table TEST_LOB_VAL(lobid number(10) primary key, lob_value clob );INSERT INTO TEST_LOB_VAL VALUES (1,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',4000,'A')));INSERT INTO TEST_LOB_VAL VALUES (2,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',2000,'A')));INSERT INTO TEST_LOB_VAL VALUES (3,'AAAAAAAAAA');COMMIT;

4
N
Fan
1 month ago
N
Fan
NileshFan
asked in Community Articles

How to write data from Multiple Snowflkae Tables to Multiple text files (separate file for each table) on Local Server?

I have scenario to write data from multiple DB tables on daily basis to file server. Each table supposed  to generate separate file on Local Server

1
Bikram
Headliner
2 months ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles

Job History Not Available and Pipeline Version Control

The Issue: Your job history is not available Error running the job: “Error reading entity from input stream”. Prerequisites:  The job is not using the latest pipeline version The latest pipeline version has been removed or deleted  The Solution/Best Practices: There are a few best practices you can implement to avoid this undesirable outcome.   - Edit the updated job to use the previous version of the pipeline           - Choose not to delete the unused new pipeline version          - Your job history will remain intact, and your job will continue to run without error.   - When creating new versions for your pipeline, create a new job for each version            - History from each version will be maintained          - You can delete pipeline versions/jobs without affecting older version’s history

0
blakeorsak
Fan
2 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to get count of number of parameter passed in a request

Hi teami have column request_param:0:   a : “11”   b: “12”   c: “12”   d: “12” how i can get count if above request has 4 parameter it can have 3 also like belowrequest_param:0:   a : “11”   b: “12”   c: “12” based on count i need to hit diff url 

7
saleempothiwala
Opening Band
2 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Map the job submitted in transformer to the one run in Databricks SparkUI

How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?A:In the transformer log, we see something like below once the job is submitted to databricks.2023-03-02 12:02:18,099 INFO submit run response run id: 108505820 In spark UI, you would see the job group having the id 108505820Run response id can also be seen as Task Run id in databricks workflow > job.

0
Pradeep
StreamSets Employee
2 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to generate UUID bases on timestamp

Hi Team I have a date datefield storing current datetime in it want to create a UUID using same column how i can create . currently using groovy script to create UUID but want to use same column and create it 

5
A
Discovered Fame
3 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

Getting error with HTTP client

Getting below error while passing URL as parameter any input for same  

3
Bikram
Headliner
3 months ago
U
Fan
ujeyakumFan
asked in Community Articles

how to merge 2 different pipeline

Hi team,Example pipelinePipeline 1 -- origin--to--kafka producerPipeline 2 -- kafka consumer--to--destinationNeed your support for how to create job to merge pipeline1 and pipeline2. Please let me know the possibility.

1
saleempothiwala
Opening Band
3 months ago
krunchworks
Fan
krunchworksFan
asked in Community Articles

How can i get sample code to perform some data transformation for a sample data using snowflake?

I am new to streamsets and looking to explore more on use case where streamsets will help snowflake. I learned that streamset will help in data transformation. Is there some links or codes available to quickly learn from?

1
Bikram
Headliner
3 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to connect and load data to aerospike from streamsets

Hi Team did any one work on aerospike load from streamsets i see the stage is deprecated can we use jdbc aerospike connector ?

3
Bikram
Headliner
3 months ago
H
Fan
harshit_agrawal1Fan
asked in Community Articles

Configuring a trigger to activate when a file is created in a directory

I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead? 

3
saleempothiwala
Opening Band
3 months ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

Webservice Array request Json

Hi team I have two json request for singleton[{“a”: “1”“b”:”2”“c”:”3”}]and for batch[{“a”: “1”“b”:”2”“c”:”3”},{“x”: “4”“y”:”5”“z”:”6”}] if singleton come i need to pass first request and if batch comes i need to pick the second one and in the second one i need to pick first value. am using json generator how to convert that into array as a request and then for batch how to pick the first value from multiple values.   

5
Bikram
Headliner
3 months ago
btsmerchstore45
Fan
btsmerchstore45Fan
asked in Community Articles

Is it possible to configure the transformer pipeline to run on an AWS EMR cluster?

I have created a simple transformer pipeline to run on an EMR cluster. In Cluster configuration, I have used the necessary configuration details.I am facing the following issue:

1
kanyewestshirt89
19 days ago
A
Fan
akshayp2Fan
asked in Community Articles

How to configure the transformer pipeline to run on AWS EMR cluster?

HiI have created a simple transformer pipeline trying to run on EMR cluster. In Cluster configuration I have used required configuration details.Facing below issue,  Can anyone help me this?

3
wilsonshamim
StreamSets Employee
4 months ago
amarzeet
Fan
amarzeetFan
asked in Community Articles

Streamset data collector 3.x tarball download link not available

I want to use stream set data collector legacy version (3.x) on a linux machine. Only option is to install the tar ball. I couldn't find a link on streamset website to download and use it. Can you please provide this information

1
albertfc
Roadie
5 months ago
V
Fan
vgohilFan
asked in Community Articles

JOBRUNNER_103 error when trying to run complex pipeline (with fragment)

I am following the fragment lesson for which i have imported the pipeline and created a duplicate. With the duplicate pipeline, i managed to create a fragment, edit the pipeline and remove the components and add the new fragment. Upon publishing the pipeline i am receiving the following error below... JOBRUNNER_103:Organization 4441d489-761c-11ed-aae0-e11f88355e74 already has 2 running jobs which is the maximum allowed. Please contact your StreamSets account team or email subscriptions@streamsets.com Can anyone please suggest what to do here to successfully start the job.

1
saleempothiwala
Opening Band
5 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
asked in Community Articles

How to enable ZGC for Streamsets Data Collector ?

What is ZGC ?The Z Garbage Collector, also known as ZGC, is a scalable low latency garbage collector designed to meet the following goals:Sub-millisecond max pause times Pause times do not increase with the heap, live-set or root-set size Handle heaps ranging from a 8MB to 16TB in sizeZGC was initially introduced as an experimental feature in JDK 11, and was declared Production Ready in JDK 15.At a glance, ZGC is:Concurrent Region-based Compacting NUMA-aware Using colored pointers Using load barriersAt its core, ZGC is a concurrent garbage collector, meaning all heavy lifting work is done while Java threads continue to execute. This greatly limits the impact garbage collection will have on your application’s response time. With SDC 5.3.x release, Data Collector supports Java 11 and 17 in addition to Java 8. And hence you can user ZGC if you are using JAVA 17 To enable ZGC just add the following jvm argument -XX:+UseZGC  and  restart the engine.  You can verify this by review $SDC_LOG/g

0
Rishi
StreamSets Employee
5 months ago
D
Fan
durairajFan
asked in Community Articles

Postgres CDC client is not processing data, when slot volume reaches 1 GB

We are using postgres CDC client to ingest the data from postges to ADLS. It’s working for other pipeline which are getting the CDC records lesser than 1 GB. While we drop and recreate the slots its working fine. But after sometime CDC files are generating from Streamsets. In pipeline logs we are not find error logs and jobs is active and running.while querying pg_replication_slots volume of data reaches more than 1 GB to several. until drop the replication slots CDC pipeline is not streaming.So, please suggest how to fix the issue. Do, we need to change the Streamsets configuration or postgres configuration. please let us know.Our Streamsets config. as below,Max Batch Size : 15000Streamsets engine : 4.4.1Batch Wait Time : 15000Query Timeout : {45 * MINUTES}Poll Interval: {1 * SECONDS}Status Interval : ${30 * SECONDS}CDC Generator Queue Size : 20000

5
Bikram
Headliner
4 months ago
subashini
StreamSets Employee
subashiniStreamSets Employee
asked in Community Articles

StreamSets Hive Metadata issue after upgrading to CDP 7.1.x

Issue:    Hive Metadata stage throws error "A managed table's location should be located within managed warehouse root directory or within its database's managedLocationUri" after upgrading to CDP version 7.14 and above. Reason:     In CDP7.x or HDP3.x, the database has 2 locations by default(managed and external) and managed tables are always ACID.Whereas HDP-2.6.x or CDH-6.x or lesser, there is no difference in managed/external table location and managed table will not be created as ACID table by default. In CDP7 or HDP3,When user tries to create managed table with custom location, CDP-Hive validates whether the custom location is outside database's managed location and throws above described exception in StreamSets pipeline. StreamSets will not support ACID table read/write. Workaround:          Since StreamSets pipeline with Hive Metadata stage does not support Hive Managed table creation, the best solution is to choose the External table option and provide Table path template so t

0
subashini
StreamSets Employee
7 months ago
tipharco
Fan
tipharcoFan
asked in Community Articles

CÔNG TY CỔ PHẦN DƯỢC PHẨM TIPHARCO

Tipharco đã phát triển 3 nhà máy sản xuất Dược phẩm bao gồm Non Betalactam, Betalactam và nhà máy sản xuất thuốc Dược liệu đạt tiêu chuẩn GMP-WHO. Thông tin liên hệ: Web: tipharco.vn SĐT: 02733872972, Email: info@tipharco.vn, Địa chỉ: Lô 08, 09 CCN và TTCN Tân Mỹ Chánh, P.9, TP Mỹ Tho, Tiền Giang, Việt NamXem thêm:https://tipharco.vn/https://www.instagram.com/tipharco/https://www.facebook.com/tipharco/https://twitter.com/CTipharhttps://www.linkedin.com/in/tipharco-tg/https://www.youtube.com/channel/UCKn1EwVLYGO0ooLE95xKR7ghttps://www.flickr.com/people/tipharco/https://www.pinterest.com/tipharco/https://www.tumblr.com/tipharco

0
tipharco
Fan
7 months ago

Badge winners

Show all badges
Terms & Conditions