Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,054 Topics
  • 2,325 Replies
1048 Topics
Sperch
Opening Band
SperchOpening Band
asked in Community Articles and Got a Question?
Possible to query the SCH for the historically longest running jobs?

Is it possible to query the SCH for the longest running jobs? For example, say you have 100 jobs, and you want to know the 10 longest running jobs out of the total (I’d happily take just reporting on averages if that is all there is). It seems like this would be a useful report or API endpoint to have.

30
Sperch
Opening Band
E
Fan
Eric MencariniFan
asked in Community Articles and Got a Question?
Retreaving more metrics fom Jobs(SDK)

Hello!! Currently, I’m using the SDK to extract metrics from some jobs. However, instead of having only {run_count, input_count, output_count, total_error_count} from sch.job.metrics, I would like to have some other metrics as well, such as counting each record for each step of my job. Is this possible? How can I achieve this? I’m also looking to get the last received record (for each pipeline) that we can see in the real-time summary. How can I get this metric from ControlHub using the SDK?

301
palmeida123
Fan
S
Fan
Sri25Fan
asked in Community Articles and Got a Question?
Error while testing a connection and connecting to a SFTP server

Hi , I am creating a new Connection with SFTP protocol and trying to connect to the SFTP server using a private key . I inputted these while creating the Connection: Authentication as Private Key , Private Key provider as plain text Private Key as the text that i copied from the ppk file Username the correct one there was no Passphrase and when i hit Test Connection I get the below error Stage 'com_streamsets_pipeline_lib_remote_RemoteConnectionVerifier_01' initialization error: java.lang.NullPointerException: Cannot invoke "net.schmizz.sshj.userauth.keyprovider.KeyProvider.getPublic()" because "this.kProv" is null (CONTAINER_0701) But alternatively if i input the same in the credentials tab of the pipeline and preview ,I am able to successfully connect to the sftp server and read the file there . Problem is when i create the same via a connection Kindly help me to resolve this issue .

130
S
Fan
Sperch
Opening Band
SperchOpening Band
asked in Community Articles and Got a Question?
Configuration of Postman to Use StreamSets APIs

I am attempting to use Postman to query the runtime history of all jobs so that I can provide a list of the X slowest running jobs. I have been unable to configure Postman correct for this GET call, and would like to know from the hivemind what I am doing wrong here. Trying to use this API endpoint: https://{Control_Hub_URL}.streamsets.com/jobrunner/rest/v1/job/{jobId}/history Auth is set to “No auth” based upon another forum post I read that stated we should include the ID and key for the API credentials used in the header, though I have tried it with API Key and providing the credentials there, as well. I have read through the 14 questions and 4 conversations that return when searching here for “Postman” and tried a few of the proffered solutions but without success. I have no parameters and the headers are where I have been trying to define API credentials. Note these credentials are used by automation in the Control Hub so they are valid. I think my issue lies in what header names

130
Sperch
Opening Band
mithunvs47
Fan
mithunvs47Fan
asked in Community Articles and Got a Question?
Assigning value to runtime parameter in pipeline start event as jdbc query

I am using Data collector 5.8.1 . I am selected JDBC in my pipeline start event I am using this query in my start event: DECLARE @XYZWorkFlowID INT = (SELECT TOP 1 WorkFlowID FROM dbo.RefWorkFlow WHERE WorkFlowName = 'XYZ') INSERT INTO JobTracker(JobName,WorkFlowID,Campaign,LastExecutedTime,JobStatus)VALUES ('XYZ_ABC_Export',@XYZWorkFlowID ,'Dialer',GETDATE(),'InProgress') I need to get the identity value generated by JobTracker table and assign it to a ‘JobTrackerId’ pipeline parameter. I need to use the same parameter in the stop event jdbc to update JobTracker the table. I tried adding: ${JobTrackerId} = SELECT @@Identity I'm pretty sure this is the wrong approach. Can anyone guide me through the right one?

110
mithunvs47
Fan
C
Fan
chrisFan
asked in Community Articles and Got a Question?
Using expression language to get the length of a list

Docs and at least one community thread have suggested that the following expression should work in a stream selector conditional: ${record:exists('/ids') && length(record:value('/ids')) > 0} This returns: ELException: No function is mapped to the name "length". The field in question is a list of integers. What am I not understanding? Thanks

60
C
Fan
S
Fan
SALMA AHMEDFan
asked in Community Articles and Got a Question?
Achieving Pagination 

I have resource URL as “https://api.simpleng.com.au/v2/ActionSheets/SummaryForSite/3?data=bma&changesAfter=2024-11-01T00:00:00&pageNumber=1” And response as :{ "page": 1, "ofPages": 3, "content": [ ] } Where I want to get all records in Page 1,2 and 3.Despite using pagination I am not able to get the records from all the pages and get only from 1st page. I am using Pagination with PageNumber and using ${StartAt} variable.Refer below Screenshots

160
S
Fan
manaso
Fan
manasoFan
asked in Community Articles and Got a Question?
streamsets error crear pipeline

Hello everybody. I'm doing an exercise on streamsets and when creating a pipeline it throws this error. VALIDATION_0096 - This pipeline was created in version 6.1.0 and is not compatible with current version 6.0.0. How to fix it? I'm using the 30-day trial version of streamsets. Thank you very much.

171
Sanjeev
StreamSets Employee
J
Fan
jcuevaFan
asked in Community Articles and Got a Question?
Issue Connecting to Redis on IBM Cloud Using Certificate Authority

Hello, I am trying to establish a connection from StreamSets to a Redis database on IBM Cloud. This instance requires the use of a Certificate Authority (CA) for authentication and secure communication, but I haven't found a way to properly configure it in StreamSets. I have attempted the following without success: Configuring a Truststore: I created a Truststore and imported the CA certificate using keytool. I updated the StreamSets configuration (sdc.properties), including the https.truststore.path property, to reference the Truststore. Passing parameters in the connection URI: I tried different URI formats, including adding the certificate to the connection string, but the connection still fails. I would appreciate any guidance on the correct way to configure StreamSets to trust the Redis certificate and establish a secure connection.

110
J
Fan
K
Fan
kamakhya05Fan
asked in Community Articles and Got a Question?
CDC operation with MongoDB oplog origin

Dear Community, I hope this message finds you well. I am reaching out to seek your help on an error getting in validation of Mongodb oplog origin stage. Your support would be greatly appreciated. Validation Error: [ MONGODB_04 - Collection isn't tailable because 'oplog.rs' is not a capped collection. : configBean.mongoConfig.collection ] Other detail/info: > Stage Library: MongoDB Java driver 3.12.0 > I tried with another capped collection 'oplog.test' created in development database other than local. But getting the same error. > I checked the older post and found same error is mentioned, however solution is not there. Thank you in advance for your time and assistance. Best regards,Kamakhya

50
K
Fan
F
Fan
fdebordeFan
asked in Community Articles and Got a Question?
Data rule : trigger alert when a field (STRING) is not empty

Hi everyone, I would like to trigger an alert when a field “text” is not empty (from Directory stage with Data Format as Text). Nevertheless, even if text is not empty, the alert is not triggered. I don’t understand what is wrong in the data rule. Any idea ? I have read the documentation here : https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Tutorial/BasicTutorial.html#task_ets_dkj_ps. Thanks, Flo

150
F
Fan
F
Fan
fdebordeFan
asked in Community Articles and Got a Question?
Metric rule : trigger alert when processing time through entire pipeline > 2000 ms

Hi everyone, I would like to trigger alert when processing time through entire pipeline > 2000 ms. I have tried with this, but it is not working well. I have tried several other options but I can’t make it work. Any idea ? Thanks, Flo

180
F
Fan
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
How to read files from s3 bucket using groovy?

Can someone provide working code of groovy to read files from s3 bucket?

2938
Nitika
Fan
T
Fan
tomFan
asked in Community Articles and Got a Question?
How to Install Elasticsearch 8.x Stage Library in StreamSets 3.22.3?

Is the Elasticsearch 8.x stage library officially supported in StreamSets 3.22.3? If not, is there a workaround or manual method to install and use Elasticsearch 8.x with this version of StreamSets? Should I consider upgrading to a newer version of StreamSets, or is there another solution? I appreciate any guidance or insights from the community. Thank you!

120
T
Fan
J
Fan
JMABFan
asked in Community Articles and Got a Question?
Start and Stop Event Connection via Parameter

Hi, I am currently developing a pipeline with Start and Stop Events using JDBC query. We are trying to make it more dynamic by having the connections parameterized, however, Start and Stop Events configuration won’t allow to use a parameterized connection. Even if I click the “Use parameter” button, it does not work. Can anyone please help me on this?

150
J
Fan
R
Fan
rajasriramFan
asked in Community Articles and Got a Question?
Read Entire JSON data as wholefile format in ADLS and write to VARIANT Field in Snowflake

I have a use case to read a JSON file from ADLS location and write the entire content of the JSON file to a VARIANT field in Snowflake. I have used ADLS origin stage , Jython Processor and Snowflake Destination Stage. In the ADLS stage, I have the data format as “WholeFile” to get /fileRef. I used the Jython processor to read /FileRef and convert it as String before loading this field to Snowflake. I am getting error in Jython Processor Stage Script sent record to error: 'com.streamsets.pipeline.stage.util.scripting.Scrip' object has no attribute 'createInputStream fileRef : FILE_REF[object Object] How do I read entire JSON file and load the contents to a VARIANT field in Snowflake Sample JSON: { "MbrDetails" : { "id" : "100001", "ProgramName" : "test" },"NotificationInformation" : { "ProgramStatusDate" : "2024-11-13", "Category" : "Pilot Program, "DateTime" : "2024-11-13", "ProgramStatus" : "Eligible" } } Expected Outcome in Snowflake Table: full_json file_name { "MbrDetails" : { "id"

70
R
Fan
dannydineen96
Fan
dannydineen96Fan
asked in Community Articles and Got a Question?
Creating a scheduler via Control Hub API

I want to create a scheduler via the following API request: Creates a new Scheduled Task: PUT {SCH_URL}/scheduler/rest/v2/jobs/{pageId} I have 2 questions: 1) Do I use {pageId} = SchedulerJobListPage or SchedulerJobManagePage, and why? Secondly, I’m passing a request body as per the body suggested on the RESTful API Scheduler page but I’m receiving the error: “ERROR : READ_ONLY_ENFORCER_000 - Page ‘com.streamsets.apps.scheduler.rest.SchedulerJobManagePage’ readOnly values have been modified. 2) Is this error due to the readOnlySignature i have generated or with one of the attributes in the request body?

333
Sanjeev
StreamSets Employee
junsuk
Fan
junsukFan
asked in Community Articles and Got a Question?
Cannot configure a StreamSets pipeline with OpenSearch version 2.11 or above

Hello. I want to store data in OpenSearch internally, but I keep getting the following error. ELASTICSEARCH_17 - Could not index '1000' records: POST https://testpate.net/_bulk: HTTP/1.1 400 Bad Request {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400} I tried configuring the pipeline with the latest version of StreamSets, but I'm still encountering the same issue. I've tested both version 7.9.0 and 8.1.0, and I used _doc for all mapping values. Does this mean I can't use OpenSearch?

170
junsuk
Fan
A
Fan
adarsh01Fan
posted in Community Articles and Got a Question?
How to Find the Correct Voice IDs If a Selected Voice Is Unavailable in Speakatoo?

While using Speakatoo’s API, I encountered an error: "The voice you selected is unavailable to use." I suspect it might be due to an incorrect or unsupported Voice ID. Does anyone know: Where can I access a comprehensive list of correct and available Voice IDs for Speakatoo? Are there specific steps to ensure the selected Voice ID is compatible with the language or settings I’m using? I’d appreciate any guidance or tips from those familiar with Speakatoo’s API. Let’s collaborate to make the most of this fantastic tool!

70
A
Fan
Sahil01
Fan
Sahil01Fan
posted in Community Articles and Got a Question?
How do I select and use language-specific voice IDs in Speakatoo's API?

I’m working with Speakatoo's API and need help with selecting and using language-specific voice IDs. Could someone explain the process or share an example of how to choose and implement a specific voice ID for a given language? Thanks in advance!

102
Sahil01
Fan
Anuragiit
Fan
AnuragiitFan
asked in Community Articles and Got a Question?
Amazon S3 bucket - offset using JSON file

Hi Team, I have to read JSON file in Amazon S3 bucket. This JSON file has timestamp folder that are ready to read. Using those values, I need to read Objects in Amazon S3 bucket. This is to avoid reading the object that has files being written simultaneously. Or How Amazon S3 origin, handles file reading that are being written at the same time ? Thanks

180
Anuragiit
Fan
Sahil01
Fan
Sahil01Fan
asked in Community Articles and Got a Question?
How to manage file format restrictions like the MP3 requirement in Speakatoo’s API when creating a widget?

I am developing a widget and need to handle file format restrictions specified by Speakatoo’s API, such as the requirement for MP3 files. What’s the best way to manage this? Are there any recommended practices or examples for ensuring compliance with the API’s file format rules? Any help would be appreciated!

20
Sahil01
Fan
D
Fan
Divya J PasalaFan
asked in Community Articles and Got a Question?
StreamSets Data Collector incremental loading from AWS into RDS

Hi, I have done full from AWS athena to RDS Postgresql.Now, i want to do incremental loading using timestamp from table and i’m jdbc query consumaer as origin and jdbc query(executor) as destination as of now and want to merge records at jdbc query side. but, i’m getting validation error from origin ( JDBC_34 - Query failed to execute: 'SELECT * FROM t_target_an.hierarchy WHERE date > TO_TIMESTAMP('0', '2024-09-11 00:00:00') ORDER BY date;' Error: SQLState: HY000 Error Code: 100071 Message: [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. INVALID_FUNCTION_ARGUMENT: Failed to tokenize string [2] at offset [0]) and also please help me with merge query which i have to use at jdbc query.

541
saleempothiwala
Headliner
K
Fan
Karimulla ShaikFan
asked in Community Articles and Got a Question?
spring and log 4j libraries upgrade without upgrading streamsets DC.

I am using streamsets data collector community edition 3.19.1 and i would like to upgrade only log4j & sping libraries. Can anyone help me with approach.? Also, Would like to understand underlying use cases of these 2 libraries in streamsets.

20
K
Fan
J
Fan
jessecorp
asked in Community Articles and Got a Question?
StreamSets SDK for Python script issue

Hello, I have developed a Python script based on the SDK version 6.4 documentation, but I got an error when I ran the script: “raise EnginelessError(streamsets.sdk.exceptions.EnginelessError: The SDK does not support publishing engineless pipelines. Please visit the UI to get instructions on how to install a deployment and engine, to proceed further.”. It’s odd becausethe deployment and engine have already been set up. The engine’s information has been included in the script. I’m not sure why I’m getting this error. # Import the ControlHub class from the SDK.from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom datetime import datetime # Connect to the Control Hub instance you want to interact with.sch = ControlHub(credential_id='example123', token='example123')print("connected") # Get the engine informationsdc = sch.data_collectors.get(engine_url='https://example.org:12345')id = sdc.idprint(f"Found the Data Collector with ID: {sdc.id}") # Build the pipeline

801
wilsonshamim
StreamSets Employee

Badge winners

Show all badges

Popular tags

Terms & Conditions