30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Can't find what you're looking for? Ask it here or check out the Community articles
Recently active
Hello everybody. I'm doing an exercise on streamsets and when creating a pipeline it throws this error.VALIDATION_0096 - This pipeline was created in version 6.1.0 and is not compatible with current version 6.0.0.How to fix it? I'm using the 30-day trial version of streamsets.Thank you very much.
Hello,I am trying to establish a connection from StreamSets to a Redis database on IBM Cloud. This instance requires the use of a Certificate Authority (CA) for authentication and secure communication, but I haven't found a way to properly configure it in StreamSets.I have attempted the following without success: Configuring a Truststore: I created a Truststore and imported the CA certificate using keytool. I updated the StreamSets configuration (sdc.properties), including the https.truststore.path property, to reference the Truststore. Passing parameters in the connection URI: I tried different URI formats, including adding the certificate to the connection string, but the connection still fails. I would appreciate any guidance on the correct way to configure StreamSets to trust the Redis certificate and establish a secure connection.
Dear Community,I hope this message finds you well. I am reaching out to seek your help on an error getting in validation of Mongodb oplog origin stage. Your support would be greatly appreciated.Validation Error: [ MONGODB_04 - Collection isn't tailable because 'oplog.rs' is not a capped collection. : configBean.mongoConfig.collection ]Other detail/info: > Stage Library: MongoDB Java driver 3.12.0 > I tried with another capped collection 'oplog.test' created in development database other than local. But getting the same error. > I checked the older post and found same error is mentioned, however solution is not there.Thank you in advance for your time and assistance.Best regards,Kamakhya
Hi everyone, I would like to trigger an alert when a field “text” is not empty (from Directory stage with Data Format as Text). Nevertheless, even if text is not empty, the alert is not triggered.I don’t understand what is wrong in the data rule. Any idea ?I have read the documentation here : https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Tutorial/BasicTutorial.html#task_ets_dkj_ps.Thanks, Flo
Hi everyone, I would like to trigger alert when processing time through entire pipeline > 2000 ms.I have tried with this, but it is not working well.I have tried several other options but I can’t make it work. Any idea ? Thanks, Flo
Can someone provide working code of groovy to read files from s3 bucket?
Is the Elasticsearch 8.x stage library officially supported in StreamSets 3.22.3? If not, is there a workaround or manual method to install and use Elasticsearch 8.x with this version of StreamSets? Should I consider upgrading to a newer version of StreamSets, or is there another solution?I appreciate any guidance or insights from the community. Thank you!
Hi, I am currently developing a pipeline with Start and Stop Events using JDBC query. We are trying to make it more dynamic by having the connections parameterized, however, Start and Stop Events configuration won’t allow to use a parameterized connection. Even if I click the “Use parameter” button, it does not work. Can anyone please help me on this?
I have a use case to read a JSON file from ADLS location and write the entire content of the JSON file to a VARIANT field in Snowflake. I have used ADLS origin stage , Jython Processor and Snowflake Destination Stage. In the ADLS stage, I have the data format as “WholeFile” to get /fileRef.I used the Jython processor to read /FileRef and convert it as String before loading this field to Snowflake.I am getting error in Jython Processor Stage Script sent record to error: 'com.streamsets.pipeline.stage.util.scripting.Scrip' object has no attribute 'createInputStreamfileRef : FILE_REF[object Object]How do I read entire JSON file and load the contents to a VARIANT field in SnowflakeSample JSON:{ "MbrDetails" : { "id" : "100001", "ProgramName" : "test"},"NotificationInformation" : { "ProgramStatusDate" : "2024-11-13", "Category" : "Pilot Program, "DateTime" : "2024-11-13", "ProgramStatus" : "Eligible"} } Expected Outcome in Snowflake Table:full_json file_name { "MbrDetails" : { "
I want to create a scheduler via the following API request: Creates a new Scheduled Task: PUT {SCH_URL}/scheduler/rest/v2/jobs/{pageId}I have 2 questions: 1) Do I use {pageId} = SchedulerJobListPage or SchedulerJobManagePage, and why?Secondly, I’m passing a request body as per the body suggested on the RESTful API Scheduler page but I’m receiving the error: “ERROR : READ_ONLY_ENFORCER_000 - Page ‘com.streamsets.apps.scheduler.rest.SchedulerJobManagePage’ readOnly values have been modified. 2) Is this error due to the readOnlySignature i have generated or with one of the attributes in the request body?
Hello. I want to store data in OpenSearch internally, but I keep getting the following error. ELASTICSEARCH_17 - Could not index '1000' records: POST https://testpate.net/_bulk: HTTP/1.1 400 Bad Request {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}I tried configuring the pipeline with the latest version of StreamSets, but I'm still encountering the same issue. I've tested both version 7.9.0 and 8.1.0, and I used _doc for all mapping values. Does this mean I can't use OpenSearch?
While using Speakatoo’s API, I encountered an error: "The voice you selected is unavailable to use." I suspect it might be due to an incorrect or unsupported Voice ID.Does anyone know:Where can I access a comprehensive list of correct and available Voice IDs for Speakatoo? Are there specific steps to ensure the selected Voice ID is compatible with the language or settings I’m using?I’d appreciate any guidance or tips from those familiar with Speakatoo’s API. Let’s collaborate to make the most of this fantastic tool!
I’m working with Speakatoo's API and need help with selecting and using language-specific voice IDs. Could someone explain the process or share an example of how to choose and implement a specific voice ID for a given language? Thanks in advance!
Hi Team,I have to read JSON file in Amazon S3 bucket. This JSON file has timestamp folder that are ready to read. Using those values, I need to read Objects in Amazon S3 bucket. This is to avoid reading the object that has files being written simultaneously.Or How Amazon S3 origin, handles file reading that are being written at the same time ?Thanks
I am developing a widget and need to handle file format restrictions specified by Speakatoo’s API, such as the requirement for MP3 files. What’s the best way to manage this? Are there any recommended practices or examples for ensuring compliance with the API’s file format rules? Any help would be appreciated!
Hi, I have done full from AWS athena to RDS Postgresql.Now, i want to do incremental loading using timestamp from table and i’m jdbc query consumaer as origin and jdbc query(executor) as destination as of now and want to merge records at jdbc query side. but, i’m getting validation error from origin ( JDBC_34 - Query failed to execute: 'SELECT * FROM t_target_an.hierarchy WHERE date > TO_TIMESTAMP('0', '2024-09-11 00:00:00') ORDER BY date;' Error: SQLState: HY000 Error Code: 100071 Message: [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. INVALID_FUNCTION_ARGUMENT: Failed to tokenize string [2] at offset [0]) and also please help me with merge query which i have to use at jdbc query.
I am using streamsets data collector community edition 3.19.1 and i would like to upgrade only log4j & sping libraries. Can anyone help me with approach.? Also, Would like to understand underlying use cases of these 2 libraries in streamsets.
Hello,I have developed a Python script based on the SDK version 6.4 documentation, but I got an error when I ran the script:“raise EnginelessError(streamsets.sdk.exceptions.EnginelessError: The SDK does not support publishing engineless pipelines. Please visit the UI to get instructions on how to install a deployment and engine, to proceed further.”. It’s odd becausethe deployment and engine have already been set up. The engine’s information has been included in the script. I’m not sure why I’m getting this error. # Import the ControlHub class from the SDK.from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom datetime import datetime# Connect to the Control Hub instance you want to interact with.sch = ControlHub(credential_id='example123', token='example123')print("connected")# Get the engine informationsdc = sch.data_collectors.get(engine_url='https://example.org:12345')id = sdc.idprint(f"Found the Data Collector with ID: {sdc.id}")# Build the pipelinepipe
The title sums up what I am looking for.The context is that I have a pipeline with a column that is always NULL but the business owner wants it to be populated with an auto incrementing integer to use as a primary key in the Snowflake table destination. I tried permitting the column to accept NULLs in Snowflake while still having it configured to add an integer for each record, but it just uses the NULLs. Taking the field out of the pipeline of course fails due to the missing field.I am hoping to avoid staging this data in another table and then using INSERT to get the auto increment if StreamSets has the capabilities to do this via regex or some other means.Thank you in advance for your time to anyone that reads this.
Is there a way to create a pipeline to successfully call StreamSets API endpoints that does not require the Python SDK for StreamSets to be leveraged? I have tried the REST Service and HTTP Client origins, and even attempted (horribly) a Jython Scripting origin to no avail. I am fairly certain that authentication is the problem since the API works when I use them through the SCH RESTful API section manually, and receive the following as an error when attempting to use a pipeline stage:com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 2]My ask for a UI-only solution is due to 1) it would allow less experienced admins to troubleshoot without immediately calling on me if problems arise, 2) allow us to build sequences for business- or management-oriented users to
I am familiar with this inquiry from a year ago, but I was hoping to be able to do this via API to avoid additional steps in automating the ETL process of that data being ingested into a database. Is there a means to do this via API that I am just missing? I am familiar with /pipelinestore/rest/v1/pipelines and use that already to get data about pipelines, but see nothing that sticks out to me as fulfilling the desired stage configuration information.Thank you in advance.
I have a number of pipelines, all of which use the JDBC Multitable Consumer as the Origin. I’m resetting the origins with a pipeline finisher but if I need to stop the pipeline or it doesn’t complete the origins don’t get reset. The documentation says they can be reset from the Job Instances view, but I don’t see that menu option.
Hi,Is the StreamSets Data Collector Version 3.22.3 supported on RHEL8? Thanks,Jay
Hi all, Is there a Streamsets Postman collection available? If yes, would someone please share the download link?Thank you,
