StreamSets Platform

I am developing a widget and need to handle file format restrictions specified by Speakatoo’s API, such as the requirement for MP3 files. What’s the best way to manage this? Are there any recommended practices or examples for ensuring compliance with the API’s file format rules? Any help would be appreciated!

Hello,I opened my account in cloud.login.streamsets.com with my IBM id (email@ibm.com) and I am performing this course https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals. My account is not privileged enough to perform all the course lessons. For example I cannot schedule jobs. All the scheduling menus are not seen in my streamsets control hub account. I think this is because I do not have the "Scheduler Operator" role in my user roles: my account organization is “IBM StreamSets Hands On Lab”.I want to be able to complete all the tasks in the course, how can I make sure that my account is compatible with the course tasks?Thanks in advance, Aviv

Hi, I have done full from AWS athena to RDS Postgresql.Now, i want to do incremental loading using timestamp from table and i’m jdbc query consumaer as origin and jdbc query(executor) as destination as of now and want to merge records at jdbc query side. but, i’m getting validation error from origin ( JDBC_34 - Query failed to execute: 'SELECT * FROM t_target_an.hierarchy WHERE date > TO_TIMESTAMP('0', '2024-09-11 00:00:00') ORDER BY date;' Error: SQLState: HY000 Error Code: 100071 Message: [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. INVALID_FUNCTION_ARGUMENT: Failed to tokenize string [2] at offset [0]) and also please help me with merge query which i have to use at jdbc query.

Hi, We are a team of developers. As part of a project we want to use Streamsets Data Collector as a USB between our application and several other third-party applications for Electronic Data Interchanges (EDIs). Therefore, we want to train on this platform in person in France or Morocco. Does anyone know of a training center in these 2 countries?

I am using streamsets data collector community edition 3.19.1 and i would like to upgrade only log4j & sping libraries. Can anyone help me with approach.? Also, Would like to understand underlying use cases of these 2 libraries in streamsets.

Hello,We have an issue with cert, error with amazon s3, trying to replace a new certificate in the truststory.jks file does not help, streamsets was restarted from the Cloudera level. Streamsets Collector 3.10.1.

Disappointed to get an error when writing strings with new line characters, read from an Azure SQL origin, when written back to an Azure Synapse SQL destination. Have specified, in the New Line Replacement Character, string of “\n” to get the data through, but the string is distorted, and potentially confused if the source string already contains whatever I choose as a replacement. So... has anyone found a better strategy for dealing with this issue? Is an enhancement planned to passed these characters through unchanged, transparently?

Hello, I have developed a Python script based on the SDK version 6.4 documentation, but I got an error when I ran the script: “raise EnginelessError(streamsets.sdk.exceptions.EnginelessError: The SDK does not support publishing engineless pipelines. Please visit the UI to get instructions on how to install a deployment and engine, to proceed further.”. It’s odd becausethe deployment and engine have already been set up. The engine’s information has been included in the script. I’m not sure why I’m getting this error. # Import the ControlHub class from the SDK.from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom datetime import datetime # Connect to the Control Hub instance you want to interact with.sch = ControlHub(credential_id='example123', token='example123')print("connected") # Get the engine informationsdc = sch.data_collectors.get(engine_url='https://example.org:12345')id = sdc.idprint(f"Found the Data Collector with ID: {sdc.id}") # Build the pipeline

The Jacquemus Pesco T-Shirt is an essential piece that combines minimalist design with high-end quality. Crafted with care and precision, this t-shirt seamlessly blends comfort, versatility, and a chic aesthetic, making it a wardrobe staple for any modern fashion enthusiast. Whether you're dressing up for an event or keeping it casual, the Pesco T-Shirt is designed to cater to your style needs with ease. Luxurious Fabric and Quality Craftsmanship The Jacquemus Pesco T-Shirt is made from premium cotton, ensuring a soft and breathable feel on the skin. This lightweight fabric allows for all-day comfort, making it ideal for both summer and transitional seasons. Jacquemus prides itself on exceptional craftsmanship, and the Pesco T-Shirt is no exception. Its finely stitched seams, durable construction, and high-quality material make this t-shirt not just fashionable but long-lasting. Effortlessly Versatile One of the standout features of the Jacquemus Pesco T-Shirt is its versatility. Its m

I created a pipeline load data from oracle to snowflake. even though the target table in snowflake does not exist in snowflake , the pipline still show succeeded without any error, also without any data loaded. If checked the log, it is said “Error SnowflakeSQLException: SQL compilation error table XXXX does not exist or not authorized. ” How to make the pipeline failed if found target table is incorrect, or column is incorrect, for such kind of normal errors reporting in data load?

The title sums up what I am looking for. The context is that I have a pipeline with a column that is always NULL but the business owner wants it to be populated with an auto incrementing integer to use as a primary key in the Snowflake table destination. I tried permitting the column to accept NULLs in Snowflake while still having it configured to add an integer for each record, but it just uses the NULLs. Taking the field out of the pipeline of course fails due to the missing field. I am hoping to avoid staging this data in another table and then using INSERT to get the auto increment if StreamSets has the capabilities to do this via regex or some other means. Thank you in advance for your time to anyone that reads this.

Is there a way to create a pipeline to successfully call StreamSets API endpoints that does not require the Python SDK for StreamSets to be leveraged? I have tried the REST Service and HTTP Client origins, and even attempted (horribly) a Jython Scripting origin to no avail. I am fairly certain that authentication is the problem since the API works when I use them through the SCH RESTful API section manually, and receive the following as an error when attempting to use a pipeline stage: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 2] My ask for a UI-only solution is due to 1) it would allow less experienced admins to troubleshoot without immediately calling on me if problems arise, 2) allow us to build sequences for business- or management-oriented users to e

I am familiar with this inquiry from a year ago, but I was hoping to be able to do this via API to avoid additional steps in automating the ETL process of that data being ingested into a database. Is there a means to do this via API that I am just missing? I am familiar with /pipelinestore/rest/v1/pipelines and use that already to get data about pipelines, but see nothing that sticks out to me as fulfilling the desired stage configuration information. Thank you in advance.

I have a number of pipelines, all of which use the JDBC Multitable Consumer as the Origin. I’m resetting the origins with a pipeline finisher but if I need to stop the pipeline or it doesn’t complete the origins don’t get reset. The documentation says they can be reset from the Job Instances view, but I don’t see that menu option.

Hi , I am unable to access the Lab documents provided in the DataOps Platform Fundamentals Course” and the Lab page is rendered as Blank window. Able to access to Streamset youtube videos for the related lesson, however, could not access the lab. Not sure if this is an issue related to accessing google drive. Please let us know if any workaround to fix this issue. Thanks.

Hi , I am taking up the treamsets Foundational course and unable to open the Lab Instructions for Streamsets Foundational course . Often getting blank page instead of the actual contents. Any idea whether it is a known issue ? Please suggest if any workaround available . Thanks .

Hi, There is a username and password to login to academy portal using personal account. However there is no MFA enabled. I had thought Auth0 was some form of authentication on top of a password but it does not ask for any extra layer of verification of the account holder. Would like to check if my understanding about Auth0 is correct.

Hi, Is the StreamSets Data Collector Version 3.22.3 supported on RHEL8? Thanks, Jay

Hi all, Is there a Streamsets Postman collection available? If yes, would someone please share the download link? Thank you,

I want to create a parameter that I can use across multiple pipelines. For example, I have multiple pipelines that are configured to run on EMR cluster. Instead of providing the same EMR configuration values for all the pipelines, if there is a way for me to create parameters globally that I can refer to, in each of the relevant pipelines. Thanks !

Hi I am following the steps generate API Credential, and run the content in the green frame directly, however it return error like below. curl -X GET https://eu01.hub.streamsets.com/security/rest/v1/currentUser -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $CRED_ID" -H "X-SS-App-Auth-Token: $CRED_TOKEN" -i % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 19 100 19 0 0 22 0 --:--:-- --:--:-- --:--:-- 22HTTP/1.1 403 Forbiddencontent-length: 19content-type: text/plaincontent-security-policy: object-src 'none';script-src 'self' https://cdn.cookielaw.org https://privacyportal.onetrust.com https://geolocation.onetrust.com https://app.intercom.io https://widget.intercom.io https://js.intercomcdn.com https://js.userflow.com https://cdn.userflow.com;style-src 'self' https://fonts.googleapis.com https://js.userflow.com https://cdn.userflow.com 'unsafe-inline';worker-src blob: ;

Hi,Am using streamSet free version. So for CDC, have created a new deployment and started the engine using generated installation script. But as soon as I connect my vpn, engine stop working (or not start) but I need to connect to vpn for my CDC pipeline work (to connect to origin i.e my oracle db). Please help to resolve this issue. Also, Is there any other way for streamSet CDC setup as I want my standalone streamSet setup to be work in k8s env. TIA!!

Good afternoon, I’m currently using Streamsets Data Collector 3.14 (last open source version of sdc). I see that it is possible to connect sdc to a control hub and there is a lot of advantages to do so. Also there is a documentation for installing an on prem control-hub from an archive : https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/InstallingDPM.html#concept_exg_11p_hbb Use the following command to extract the tarball: tar xzvf streamsets-dpm-<version>.tar.gz There was even a tutorial for how to build your own sch on prem from a github repo: Howerver the repo is no longer available : http: //github.com/streamsets/domainserver Could it be possible to have again access to the repository or to the archive ? Thank you.

Question How are the SDC Metrics collected and what do they represent? Answer The graphs on the SDC Metrics page in StreamSets Data Collector and the Metrics tab of the Execution Engine page in Control Hub are system-level metrics collected from standard Java core libraries. They should roughly correspond with metrics reported by other system-level tools, such as top and uptime at the command line, as well as external monitoring tools that show system-level metrics. Note that the numbers reported by different tools won’t match each other exactly due to differences in reporting intervals and other factors, but there should be a correlation.

Hi I have pipeline and it has 1 runtime parameter about table name. I create a job on top of this pipeline, so it also has a runtime parameter, when create job from this pipeline, I provide a default parameter. I try to use python SDK to call the job, if I ran like below, the job will run with the default parameter(TEST_TABLE1) defined in pipeline and job, it is expected: job = sch.jobs.get(job_name = "MY_SDC_JOB") sch.start_job(job) However when I run like this, I want to overwrite the default runtime parameter, but it still using the default parameter(TEST_TABLE1) , how to make it run with the given parameter(TEST_TABLE2)? sch.start_job(job, TABLE_NAME=’TEST_TABLE2')