Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1218 Topics
john.durkin
StreamSets Employee
john.durkinStreamSets Employee
posted in StreamSets Academy
"DataOps Platform Fundamentals" - questions from the video course

Hi -Welcome to the StreamSets Academy area of Community!Please post questions related to the self-paced video course “DataOps Platform Fundamentals”. Please use the tag “DataOps Platform Fundamentals Course” when creating your question under the “StreamSets Academy” category. Thank you! 

3950
john.durkin
StreamSets Employee
3 years ago
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
How to read files from s3 bucket using groovy?

Can someone provide working code of groovy to read files from s3 bucket?

2618
Nitika
Fan
9 days ago
T
Fan
tomFan
asked in Community Articles and Got a Question?
How to Install Elasticsearch 8.x Stage Library in StreamSets 3.22.3?

Is the Elasticsearch 8.x stage library officially supported in StreamSets 3.22.3? If not, is there a workaround or manual method to install and use Elasticsearch 8.x with this version of StreamSets? Should I consider upgrading to a newer version of StreamSets, or is there another solution?I appreciate any guidance or insights from the community. Thank you!

80
T
Fan
10 days ago
J
Fan
JMABFan
asked in Community Articles and Got a Question?
Start and Stop Event Connection via Parameter

Hi, I am currently developing a pipeline with Start and Stop Events using JDBC query. We are trying to make it more dynamic by having the connections parameterized, however, Start and Stop Events configuration won’t allow to use a parameterized connection.  Even if I click the “Use parameter” button, it does not work. Can anyone please help me on this?

60
J
Fan
15 days ago
R
Fan
rajasriramFan
asked in Community Articles and Got a Question?
Read Entire JSON data as wholefile format in ADLS and write to VARIANT Field in Snowflake

I have a use case to read a JSON file from ADLS location and write the entire content of the JSON file to a VARIANT field in Snowflake. I have used ADLS origin stage , Jython Processor and Snowflake Destination Stage. In the ADLS stage, I have the data format as “WholeFile” to get /fileRef.I used the Jython processor to read /FileRef and convert it as String before loading this field to Snowflake.I am getting error in Jython Processor Stage Script sent record to error: 'com.streamsets.pipeline.stage.util.scripting.Scrip' object has no attribute 'createInputStreamfileRef : FILE_REF[object Object]How do I read entire JSON file and load the contents to a VARIANT field in SnowflakeSample JSON:{ "MbrDetails" : { "id" : "100001", "ProgramName" : "test"},"NotificationInformation" : { "ProgramStatusDate" : "2024-11-13", "Category" : "Pilot Program, "DateTime" : "2024-11-13", "ProgramStatus" : "Eligible"} } Expected Outcome in Snowflake Table:full_json file_name { "MbrDetails" :  { "

50
R
Fan
18 days ago
dannydineen96
Fan
dannydineen96Fan
asked in Community Articles and Got a Question?
Creating a scheduler via Control Hub API

I want to create a scheduler via the following API request: Creates a new Scheduled Task: PUT {SCH_URL}/scheduler/rest/v2/jobs/{pageId}I have 2 questions: 1) Do I use {pageId} = SchedulerJobListPage or SchedulerJobManagePage, and why?Secondly, I’m passing a request body as per the body suggested on the RESTful API Scheduler page but I’m receiving the error: “ERROR : READ_ONLY_ENFORCER_000 - Page ‘com.streamsets.apps.scheduler.rest.SchedulerJobManagePage’ readOnly values have been modified. 2) Is this error due to the readOnlySignature i have generated or with one of the attributes in the request body?  

293
Sanjeev
StreamSets Employee
19 days ago
junsuk
Fan
junsukFan
asked in Community Articles and Got a Question?
Cannot configure a StreamSets pipeline with OpenSearch version 2.11 or above

Hello. I want to store data in OpenSearch internally, but I keep getting the following error. ELASTICSEARCH_17 - Could not index '1000' records: POST https://testpate.net/_bulk: HTTP/1.1 400 Bad Request {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}I tried configuring the pipeline with the latest version of StreamSets, but I'm still encountering the same issue. I've tested both version 7.9.0 and 8.1.0, and I used _doc for all mapping values. Does this mean I can't use OpenSearch?

160
junsuk
Fan
26 days ago
A
Fan
adarsh01Fan
posted in Community Articles and Got a Question?
How to Find the Correct Voice IDs If a Selected Voice Is Unavailable in Speakatoo?

While using Speakatoo’s API, I encountered an error: "The voice you selected is unavailable to use." I suspect it might be due to an incorrect or unsupported Voice ID.Does anyone know:Where can I access a comprehensive list of correct and available Voice IDs for Speakatoo? Are there specific steps to ensure the selected Voice ID is compatible with the language or settings I’m using?I’d appreciate any guidance or tips from those familiar with Speakatoo’s API. Let’s collaborate to make the most of this fantastic tool!

60
A
Fan
1 month ago
Sahil01
Fan
Sahil01Fan
posted in Community Articles and Got a Question?
How do I select and use language-specific voice IDs in Speakatoo's API?

I’m working with Speakatoo's API and need help with selecting and using language-specific voice IDs. Could someone explain the process or share an example of how to choose and implement a specific voice ID for a given language? Thanks in advance!

92
Sahil01
Fan
1 month ago
G
Fan
GuruFan
asked in Share your Best Practices
Kafka-29 error

Hi,we are getting below error for few of the Kafka consumer Jobs. we fine tuned the Kafka config values like sesson.timeout.ms request.timeout.ms, max poll interval, heart beat value and max poll records. But still we are seeing this issue intermittently. Can you please check below and provide a solution for htis. 'KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.' Thanks,Guru  

90
G
Fan
1 month ago
Anuragiit
Fan
AnuragiitFan
asked in Community Articles and Got a Question?
Amazon S3 bucket - offset using JSON file

Hi Team,I have to read JSON file in Amazon S3 bucket. This JSON file has timestamp folder that are ready to read. Using those values, I need to read Objects in Amazon S3 bucket. This is to avoid reading the object that has files being written simultaneously.Or How Amazon S3 origin, handles file reading that are being written at the same time ?Thanks

160
Anuragiit
Fan
1 month ago
B
Fan
BugalloFFan
posted in StreamSets Academy
Connect Engines with HIVE and Watsonx

Hi, I’ve started working with StreamSets recently and I want a Transformer Engine to process data from IBM Cloud Object Storage (COS) and write it to a Hive table managed by Watsonx.data. However, I’m facing challenges connecting StreamSets to the Spark engine in Watsonx.data.Watsonx.data uses an optimized Spark engine, and from what I understand, it behaves as a Standalone Cluster. The issue is that I don’t have access to the Spark Master URL (spark://<master-host>:<port>), which StreamSets requires to connect the Transformer Engine. StreamSets needs a Master URL to send jobs to the Spark cluster. Without this, I haven’t been able to configure the pipeline to process jobs directly in Watsonx.data’s Spark engine.As well I’ve tried with the data collector engine, that I understand that it’s not the best fit but it requires easier connections. It didn’t work neither, I’ll leave logs at the bottom for this situation.Is it possible to connect StreamSets to the Spark engine in W

200
B
Fan
1 month ago
Sahil01
Fan
Sahil01Fan
asked in Community Articles and Got a Question?
How to manage file format restrictions like the MP3 requirement in Speakatoo’s API when creating a widget?

I am developing a widget and need to handle file format restrictions specified by Speakatoo’s API, such as the requirement for MP3 files. What’s the best way to manage this? Are there any recommended practices or examples for ensuring compliance with the API’s file format rules? Any help would be appreciated!

10
Sahil01
Fan
2 months ago
avivsimibm
Fan
avivsimibmFan
asked in StreamSets Academy
Cannot schedule jobs as requested in streamsets-platform-fundamentals course

Hello,I opened my account in cloud.login.streamsets.com with my IBM id (email@ibm.com) and I am performing this course https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals. My account is not privileged enough to perform all the course lessons. For example I cannot schedule jobs. All the scheduling menus are not seen in my streamsets control hub account. I think this is because I do not have the "Scheduler Operator" role in my user roles:my account organization is “IBM StreamSets Hands On Lab”.I want to be able to complete all the tasks in the course, how can I make sure that my account is compatible with the course tasks?Thanks in advance,Aviv

120
avivsimibm
Fan
2 months ago
D
Fan
Divya J PasalaFan
asked in Community Articles and Got a Question?
StreamSets Data Collector incremental loading from AWS into RDS

Hi, I have done full from AWS athena to RDS Postgresql.Now, i want to do incremental loading using timestamp from table and i’m jdbc query consumaer as origin and jdbc query(executor) as destination as of now and want to merge records at jdbc query side. but, i’m getting validation error from origin ( JDBC_34 - Query failed to execute: 'SELECT * FROM t_target_an.hierarchy WHERE date > TO_TIMESTAMP('0', '2024-09-11 00:00:00') ORDER BY date;' Error: SQLState: HY000 Error Code: 100071 Message: [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. INVALID_FUNCTION_ARGUMENT: Failed to tokenize string [2] at offset [0]) and also please help me with merge query which i have to use at jdbc query.

431
saleempothiwala
Headliner
3 months ago
Fioko
FiokoNew Member
asked in StreamSets Academy
Streamsets Data Collector face-to-face training in France or Morocco

Hi,We are a team of developers. As part of a project we want to use Streamsets Data Collector as a USB between our application and several other third-party applications for Electronic Data Interchanges (EDIs). Therefore, we want to train on this platform in person in France or Morocco. Does anyone know of a training center in these 2 countries?

291
saleempothiwala
Headliner
3 months ago
K
Fan
Karimulla ShaikFan
asked in Community Articles and Got a Question?
spring and log 4j libraries upgrade without upgrading streamsets DC.

 I am using streamsets data collector community edition 3.19.1 and i would like to upgrade only log4j & sping libraries. Can anyone help me with approach.? Also, Would like to understand underlying use cases of these 2 libraries in streamsets.

20
K
Fan
3 months ago
ChangeIt
ChangeItNew Member
asked in StreamSets Academy
SLL error - Path does not chain with any of the trust anchors

Hello,We have an issue with cert, error with amazon s3, trying to replace a new certificate in the truststory.jks file does not help, streamsets was restarted from the Cloudera level. Streamsets Collector 3.10.1.

80
ChangeIt
3 months ago
B
Fan
bozo16Fan
asked in Share your Best Practices
Best New Line Replacement Character in Azure Synapse SQL

Disappointed to get an error when writing strings with new line characters, read from an Azure SQL origin, when written back to an Azure Synapse SQL destination.Have specified, in the New Line Replacement Character, string of “\n” to get the data through, but the string is distorted, and potentially confused if the source string already contains whatever I choose as a replacement. So...has anyone found a better strategy for dealing with this issue? Is an enhancement planned to passed these characters through unchanged, transparently?

280
B
Fan
3 months ago
J
Fan
jessecorp
asked in Community Articles and Got a Question?
StreamSets SDK for Python script issue

Hello,I have developed a Python script based on the SDK version 6.4 documentation, but I got an error when I ran the script:“raise EnginelessError(streamsets.sdk.exceptions.EnginelessError: The SDK does not support publishing engineless pipelines. Please visit the UI to get instructions on how to install a deployment and engine, to proceed further.”. It’s odd becausethe deployment and engine have already been set up. The engine’s information has been included in the script. I’m not sure why I’m getting this error.  # Import the ControlHub class from the SDK.from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom datetime import datetime# Connect to the Control Hub instance you want to interact with.sch = ControlHub(credential_id='example123', token='example123')print("connected")# Get the engine informationsdc = sch.data_collectors.get(engine_url='https://example.org:12345')id = sdc.idprint(f"Found the Data Collector with ID: {sdc.id}")# Build the pipelinepipe

741
wilsonshamim
StreamSets Employee
4 months ago
thejacquemus
thejacquemusNew Member
asked in Share your Best Practices
Jacquemus Pesco T-Shirt - Timeless, Stylish, and Comfortable

The Jacquemus Pesco T-Shirt is an essential piece that combines minimalist design with high-end quality. Crafted with care and precision, this t-shirt seamlessly blends comfort, versatility, and a chic aesthetic, making it a wardrobe staple for any modern fashion enthusiast. Whether you're dressing up for an event or keeping it casual, the Pesco T-Shirt is designed to cater to your style needs with ease.Luxurious Fabric and Quality CraftsmanshipThe Jacquemus Pesco T-Shirt is made from premium cotton, ensuring a soft and breathable feel on the skin. This lightweight fabric allows for all-day comfort, making it ideal for both summer and transitional seasons. Jacquemus prides itself on exceptional craftsmanship, and the Pesco T-Shirt is no exception. Its finely stitched seams, durable construction, and high-quality material make this t-shirt not just fashionable but long-lasting.Effortlessly VersatileOne of the standout features of the Jacquemus Pesco T-Shirt is its versatility. Its minim

60
thejacquemus
4 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in StreamSets Academy
Pipeline load data to Snowflake table, even though the target table does not exit, pipeline still succeeded

I created a pipeline load data from oracle to snowflake. even though the target table in snowflake does not exist in snowflake , the pipline still show succeeded without any error, also without any data loaded. If checked the log, it is said “Error SnowflakeSQLException: SQL compilation errortable XXXX does not exist or not authorized.”How to make the pipeline failed if found target table is incorrect, or column is incorrect, for such kind of normal errors reporting in data load? 

454
saleempothiwala
Headliner
4 months ago
Sperch
Opening Band
SperchOpening Band
asked in Community Articles and Got a Question?
Is it possible to auto increment values of a field using Field Replacer?

The title sums up what I am looking for.The context is that I have a pipeline with a column that is always NULL but the business owner wants it to be populated with an auto incrementing integer to use as a primary key in the Snowflake table destination. I tried permitting the column to accept NULLs in Snowflake while still having it configured to add an integer for each record, but it just uses the NULLs. Taking the field out of the pipeline of course fails due to the missing field.I am hoping to avoid staging this data in another table and then using INSERT to get the auto increment if StreamSets has the capabilities to do this via regex or some other means.Thank you in advance for your time to anyone that reads this.

542
lakshman
Fan
4 months ago
Sperch
Opening Band
SperchOpening Band
asked in Community Articles and Got a Question?
Is there a way to use StreamSets APIs without using the Python SDK for StreamSets?

Is there a way to create a pipeline to successfully call StreamSets API endpoints that does not require the Python SDK for StreamSets to be leveraged? I have tried the REST Service and HTTP Client origins, and even attempted (horribly) a Jython Scripting origin to no avail. I am fairly certain that authentication is the problem since the API works when I use them through the SCH RESTful API section manually, and receive the following as an error when attempting to use a pipeline stage:com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 2]My ask for a UI-only solution is due to 1) it would allow less experienced admins to troubleshoot without immediately calling on me if problems arise, 2) allow us to build sequences for business- or management-oriented users to

391
Sperch
Opening Band
4 months ago
Sperch
Opening Band
SperchOpening Band
asked in Community Articles and Got a Question?
How Does One Pull Stage Configuration of Each Stage in a Pipeline?

I am familiar with this inquiry from a year ago, but I was hoping to be able to do this via API to avoid additional steps in automating the ETL process of that data being ingested into a database. Is there a means to do this via API that I am just missing? I am familiar with /pipelinestore/rest/v1/pipelines and use that already to get data about pipelines, but see nothing that sticks out to me as fulfilling the desired stage configuration information.Thank you in advance. 

201
Sperch
Opening Band
4 months ago

Badge winners

Show all badges
Terms & Conditions