Community Articles

Contribute and publish your articles/blogs here showcasing you usage, architecture, and know how of the StreamSets DataOps Platform. THIS IS NOT THE Q&A CATEGORY. Head over to the "Got a Question?" Category to ask your question.

  • 55 Topics
  • 89 Replies

55 Topics

N
Fan
NileshFan
asked in Community Articles

How to write data from Multiple Snowflkae Tables to Multiple text files (separate file for each table) on Local Server?

I have scenario to write data from multiple DB tables on daily basis to file server. Each table supposed  to generate separate file on Local Server

1
Bikram
Headliner
10 hours ago
deepikamani007
Fan
deepikamani007Fan
asked in Community Articles

how to configure mongodp connection string

i dont known how to configure the mongodb opblog origin. 

3
Marina
Fan
14 hours ago
N
Fan
NileshFan
asked in Community Articles

Oracle CDC to Snowflake while processing CLOB datatype

Hi,Facing error in Data collector pipeline Oracle CDC to Snowflake while processing CLOB datatype table. When inserting value >= 4000 then works fine. But the less than that facing error.SNOWFLAKE_28 - Snowflake MERGE for 'sdc-df26c7ce-fd00-4b5d-a3b1-817b9809e0a3.csv.gz' processed '1' CDC records out of '3'Oracle:create table TEST_LOB_VAL(lobid number(10) primary key, lob_value clob );INSERT INTO TEST_LOB_VAL VALUES (1,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',4000,'A')));INSERT INTO TEST_LOB_VAL VALUES (2,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',2000,'A')));INSERT INTO TEST_LOB_VAL VALUES (3,'AAAAAAAAAA');COMMIT;

1
AkshayJadhav
StreamSets Employee
15 hours ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles

Job History Not Available and Pipeline Version Control

The Issue: Your job history is not available Error running the job: “Error reading entity from input stream”. Prerequisites:  The job is not using the latest pipeline version The latest pipeline version has been removed or deleted  The Solution/Best Practices: There are a few best practices you can implement to avoid this undesirable outcome.   - Edit the updated job to use the previous version of the pipeline           - Choose not to delete the unused new pipeline version          - Your job history will remain intact, and your job will continue to run without error.   - When creating new versions for your pipeline, create a new job for each version            - History from each version will be maintained          - You can delete pipeline versions/jobs without affecting older version’s history

0
blakeorsak
Fan
4 days ago
Pablo
Fan
Pablo
posted in Community Articles

Error after migrating from Oracle CDC Client to the new Oracle CDC stage

Since SDC 5.4.0 was released, there is available a new origin stage Oracle CDC which will replace the old stage Oracle CDC Client. You can have a look into the release notes where it is explained in more detail.StreamSets recommends using the new Oracle CDC origin for all new development. So if you would like to migrate to the new Oracle CDC and take benefit of its improvements, you should take into account two new drivers needed for the new stage: orai18n.jar and orai18n-mapping.jar, so please, check our documentation to install the required Libraries.Otherwise you can get errors running your pipeline as the following:Pipeline initialization error: java.lang.NoClassDefFoundError: oracle/i18n/util/LocaleMapperAnd remember to restart SDC after installing the new drivers.You can always check our documentation for further information.

0
Pablo
Fan
4 days ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to get count of number of parameter passed in a request

Hi teami have column request_param:0:   a : “11”   b: “12”   c: “12”   d: “12” how i can get count if above request has 4 parameter it can have 3 also like belowrequest_param:0:   a : “11”   b: “12”   c: “12” based on count i need to hit diff url 

7
saleempothiwala
Opening Band
6 days ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Map the job submitted in transformer to the one run in Databricks SparkUI

How do we map the job submitted in transformer to the one run in Spark? Or, in other words, what is the corresponding job in the Spark UI for the pipeline/job in question?A:In the transformer log, we see something like below once the job is submitted to databricks.2023-03-02 12:02:18,099 INFO submit run response run id: 108505820 In spark UI, you would see the job group having the id 108505820Run response id can also be seen as Task Run id in databricks workflow > job.

0
Pradeep
StreamSets Employee
11 days ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to generate UUID bases on timestamp

Hi Team I have a date datefield storing current datetime in it want to create a UUID using same column how i can create . currently using groovy script to create UUID but want to use same column and create it 

5
A
Discovered Fame
17 days ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

Getting error with HTTP client

Getting below error while passing URL as parameter any input for same  

3
Bikram
Headliner
17 days ago
U
Fan
ujeyakumFan
asked in Community Articles

how to merge 2 different pipeline

Hi team,Example pipelinePipeline 1 -- origin--to--kafka producerPipeline 2 -- kafka consumer--to--destinationNeed your support for how to create job to merge pipeline1 and pipeline2. Please let me know the possibility.

1
saleempothiwala
Opening Band
1 month ago
btsmerchstore45
Fan
btsmerchstore45Fan
asked in Community Articles

Is it possible to configure the transformer pipeline to run on an AWS EMR cluster?

I have created a simple transformer pipeline to run on an EMR cluster. In Cluster configuration, I have used the necessary configuration details.I am facing the following issue:

1
btshoodieforwoman4
1 month ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

how to connect and load data to aerospike from streamsets

Hi Team did any one work on aerospike load from streamsets i see the stage is deprecated can we use jdbc aerospike connector ?

3
Bikram
Headliner
1 month ago
krunchworks
Fan
krunchworksFan
asked in Community Articles

How can i get sample code to perform some data transformation for a sample data using snowflake?

I am new to streamsets and looking to explore more on use case where streamsets will help snowflake. I learned that streamset will help in data transformation. Is there some links or codes available to quickly learn from?

1
Bikram
Headliner
1 month ago
H
Fan
harshit_agrawal1Fan
asked in Community Articles

Configuring a trigger to activate when a file is created in a directory

I am trying to create a pipeline which gets triggered when I add a file to a directory. Is there any functionality in the streamsets hub or any kind of trigger which I can directly set to the “Directory” origin which can in turn trigger the pipeline ahead? 

3
saleempothiwala
Opening Band
1 month ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles

Webservice Array request Json

Hi team I have two json request for singleton[{“a”: “1”“b”:”2”“c”:”3”}]and for batch[{“a”: “1”“b”:”2”“c”:”3”},{“x”: “4”“y”:”5”“z”:”6”}] if singleton come i need to pass first request and if batch comes i need to pick the second one and in the second one i need to pick first value. am using json generator how to convert that into array as a request and then for batch how to pick the first value from multiple values.   

5
Bikram
Headliner
1 month ago
A
Fan
akshayp2Fan
asked in Community Articles

How to configure the transformer pipeline to run on AWS EMR cluster?

HiI have created a simple transformer pipeline trying to run on EMR cluster. In Cluster configuration I have used required configuration details.Facing below issue,  Can anyone help me this?

3
wilsonshamim
StreamSets Employee
1 month ago
D
Fan
durairajFan
asked in Community Articles

Postgres CDC client is not processing data, when slot volume reaches 1 GB

We are using postgres CDC client to ingest the data from postges to ADLS. It’s working for other pipeline which are getting the CDC records lesser than 1 GB. While we drop and recreate the slots its working fine. But after sometime CDC files are generating from Streamsets. In pipeline logs we are not find error logs and jobs is active and running.while querying pg_replication_slots volume of data reaches more than 1 GB to several. until drop the replication slots CDC pipeline is not streaming.So, please suggest how to fix the issue. Do, we need to change the Streamsets configuration or postgres configuration. please let us know.Our Streamsets config. as below,Max Batch Size : 15000Streamsets engine : 4.4.1Batch Wait Time : 15000Query Timeout : {45 * MINUTES}Poll Interval: {1 * SECONDS}Status Interval : ${30 * SECONDS}CDC Generator Queue Size : 20000

5
Bikram
Headliner
1 month ago
amarzeet
Fan
amarzeetFan
asked in Community Articles

Streamset data collector 3.x tarball download link not available

I want to use stream set data collector legacy version (3.x) on a linux machine. Only option is to install the tar ball. I couldn't find a link on streamset website to download and use it. Can you please provide this information

1
albertfc
Roadie
2 months ago
V
Fan
vgohilFan
asked in Community Articles

JOBRUNNER_103 error when trying to run complex pipeline (with fragment)

I am following the fragment lesson for which i have imported the pipeline and created a duplicate. With the duplicate pipeline, i managed to create a fragment, edit the pipeline and remove the components and add the new fragment. Upon publishing the pipeline i am receiving the following error below... JOBRUNNER_103:Organization 4441d489-761c-11ed-aae0-e11f88355e74 already has 2 running jobs which is the maximum allowed. Please contact your StreamSets account team or email subscriptions@streamsets.com Can anyone please suggest what to do here to successfully start the job.

1
saleempothiwala
Opening Band
3 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
asked in Community Articles

How to enable ZGC for Streamsets Data Collector ?

What is ZGC ?The Z Garbage Collector, also known as ZGC, is a scalable low latency garbage collector designed to meet the following goals:Sub-millisecond max pause times Pause times do not increase with the heap, live-set or root-set size Handle heaps ranging from a 8MB to 16TB in sizeZGC was initially introduced as an experimental feature in JDK 11, and was declared Production Ready in JDK 15.At a glance, ZGC is:Concurrent Region-based Compacting NUMA-aware Using colored pointers Using load barriersAt its core, ZGC is a concurrent garbage collector, meaning all heavy lifting work is done while Java threads continue to execute. This greatly limits the impact garbage collection will have on your application’s response time. With SDC 5.3.x release, Data Collector supports Java 11 and 17 in addition to Java 8. And hence you can user ZGC if you are using JAVA 17 To enable ZGC just add the following jvm argument -XX:+UseZGC  and  restart the engine.  You can verify this by review $SDC_LOG/g

0
Rishi
StreamSets Employee
3 months ago
subashini
StreamSets Employee
subashiniStreamSets Employee
asked in Community Articles

StreamSets Hive Metadata issue after upgrading to CDP 7.1.x

Issue:    Hive Metadata stage throws error "A managed table's location should be located within managed warehouse root directory or within its database's managedLocationUri" after upgrading to CDP version 7.14 and above. Reason:     In CDP7.x or HDP3.x, the database has 2 locations by default(managed and external) and managed tables are always ACID.Whereas HDP-2.6.x or CDH-6.x or lesser, there is no difference in managed/external table location and managed table will not be created as ACID table by default. In CDP7 or HDP3,When user tries to create managed table with custom location, CDP-Hive validates whether the custom location is outside database's managed location and throws above described exception in StreamSets pipeline. StreamSets will not support ACID table read/write. Workaround:          Since StreamSets pipeline with Hive Metadata stage does not support Hive Managed table creation, the best solution is to choose the External table option and provide Table path template so t

0
subashini
StreamSets Employee
4 months ago
tipharco
Fan
tipharcoFan
asked in Community Articles

CÔNG TY CỔ PHẦN DƯỢC PHẨM TIPHARCO

Tipharco đã phát triển 3 nhà máy sản xuất Dược phẩm bao gồm Non Betalactam, Betalactam và nhà máy sản xuất thuốc Dược liệu đạt tiêu chuẩn GMP-WHO. Thông tin liên hệ: Web: tipharco.vn SĐT: 02733872972, Email: info@tipharco.vn, Địa chỉ: Lô 08, 09 CCN và TTCN Tân Mỹ Chánh, P.9, TP Mỹ Tho, Tiền Giang, Việt NamXem thêm:https://tipharco.vn/https://www.instagram.com/tipharco/https://www.facebook.com/tipharco/https://twitter.com/CTipharhttps://www.linkedin.com/in/tipharco-tg/https://www.youtube.com/channel/UCKn1EwVLYGO0ooLE95xKR7ghttps://www.flickr.com/people/tipharco/https://www.pinterest.com/tipharco/https://www.tumblr.com/tipharco

0
tipharco
Fan
4 months ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
posted in Community Articles

UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

0
john.mcavoy
StreamSets Employee
5 months ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

versions of streamsets available for free download

hello cloudera community, what is the latest version of streamsets available for free download? ps: rpm download to install on linux what would be the repository link to download all the free versions?

0
Yago Aparecido
Discovered Fame
5 months ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

how to check jvm usage of streamsets by command line?

hello streamsets community, the streamsets is installed in cloudera manager, but it does not have graphics that show the memory usage of the jvm with that, i need to know how to query the jvm memory usage of streamsets by the command line.

6
Bikram
Headliner
5 months ago

Badge winners

Show all badges
Terms & Conditions