30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Recently active
Hello!! Currently, I’m using the SDK to extract metrics from some jobs. However, instead of having only {run_count, input_count, output_count, total_error_count} from sch.job.metrics, I would like to have some other metrics as well, such as counting each record for each step of my job. Is this possible? How can I achieve this? I’m also looking to get the last received record (for each pipeline) that we can see in the real-time summary. How can I get this metric from ControlHub using the SDK?
Hi , I am creating a new Connection with SFTP protocol and trying to connect to the SFTP server using a private key . I inputted these while creating the Connection: Authentication as Private Key , Private Key provider as plain text Private Key as the text that i copied from the ppk file Username the correct one there was no Passphrase and when i hit Test Connection I get the below error Stage 'com_streamsets_pipeline_lib_remote_RemoteConnectionVerifier_01' initialization error: java.lang.NullPointerException: Cannot invoke "net.schmizz.sshj.userauth.keyprovider.KeyProvider.getPublic()" because "this.kProv" is null (CONTAINER_0701) But alternatively if i input the same in the credentials tab of the pipeline and preview ,I am able to successfully connect to the sftp server and read the file there . Problem is when i create the same via a connection Kindly help me to resolve this issue .
I am attempting to use Postman to query the runtime history of all jobs so that I can provide a list of the X slowest running jobs. I have been unable to configure Postman correct for this GET call, and would like to know from the hivemind what I am doing wrong here. Trying to use this API endpoint: https://{Control_Hub_URL}.streamsets.com/jobrunner/rest/v1/job/{jobId}/history Auth is set to “No auth” based upon another forum post I read that stated we should include the ID and key for the API credentials used in the header, though I have tried it with API Key and providing the credentials there, as well. I have read through the 14 questions and 4 conversations that return when searching here for “Postman” and tried a few of the proffered solutions but without success. I have no parameters and the headers are where I have been trying to define API credentials. Note these credentials are used by automation in the Control Hub so they are valid. I think my issue lies in what header names
I am using Data collector 5.8.1 . I am selected JDBC in my pipeline start event I am using this query in my start event: DECLARE @XYZWorkFlowID INT = (SELECT TOP 1 WorkFlowID FROM dbo.RefWorkFlow WHERE WorkFlowName = 'XYZ') INSERT INTO JobTracker(JobName,WorkFlowID,Campaign,LastExecutedTime,JobStatus)VALUES ('XYZ_ABC_Export',@XYZWorkFlowID ,'Dialer',GETDATE(),'InProgress') I need to get the identity value generated by JobTracker table and assign it to a ‘JobTrackerId’ pipeline parameter. I need to use the same parameter in the stop event jdbc to update JobTracker the table. I tried adding: ${JobTrackerId} = SELECT @@Identity I'm pretty sure this is the wrong approach. Can anyone guide me through the right one?
Docs and at least one community thread have suggested that the following expression should work in a stream selector conditional: ${record:exists('/ids') && length(record:value('/ids')) > 0} This returns: ELException: No function is mapped to the name "length". The field in question is a list of integers. What am I not understanding? Thanks
Already have an account? Login
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
