Is there a prebuild processor/component which captures no. of records processed through stages and other logging events ? We have requirements to capture no. of records processed and other logging events and possibly store them to log files/MySQL stages
Solved
Logging mechanism for Data Transformer and Data Collector pipelines
- Fan
- 1 reply
Best answer by Giuseppe Mura
HI
/jobrunner/rest/v1/metrics/job/{jobId}
You can find the API from the Help menu, the little question mark icon in the toolbar at the top of the page.
When you call the metrics API you get a json document back which contains a number of objects, one is the “counters” which contains the information you’re after, including the following, for pipeline level inputs/outputs:
- pipeline.batchInputRecords.counter
- pipeline.batchOutputRecords.counter
- pipeline.batchErrorRecords.counter
You’ll also find stage level counters in there, if you need to retrieve information at stage level.
In terms of consuming the API, you can use your tool of choice, or indeed build a simple Data Collector pipeline to source from it and write any metrics you’re interested in to logs or MySQL as required.
1 person likes this
- StreamSets Employee
- 37 replies
- Answer
HI
/jobrunner/rest/v1/metrics/job/{jobId}
You can find the API from the Help menu, the little question mark icon in the toolbar at the top of the page.
When you call the metrics API you get a json document back which contains a number of objects, one is the “counters” which contains the information you’re after, including the following, for pipeline level inputs/outputs:
- pipeline.batchInputRecords.counter
- pipeline.batchOutputRecords.counter
- pipeline.batchErrorRecords.counter
You’ll also find stage level counters in there, if you need to retrieve information at stage level.
In terms of consuming the API, you can use your tool of choice, or indeed build a simple Data Collector pipeline to source from it and write any metrics you’re interested in to logs or MySQL as required.
- Fan
- 1 reply
Thanks for the response. Tried calling the API in a Data Collector pipeline using HTTP client origin using JSON as Data Format . However it is giving error:
HTTP_00 - Cannot parse record. HTTP-Status: 200 Reason: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: com.streamsets.pipeline.api.ext.io.OverrunReader@1f4c5afc; line: 1, column: 2]
Noticed that there is an issue open for the same error:
https://issues.streamsets.com/browse/SDC-11039
Any pointers to proceed in this case?
- StreamSets Employee
- 37 replies
Hi
I’ll refer you to the tip on the REST APIs page in Control Hub itself:
Sample script for using Control Hub REST APIs
# Get $CRED_ID & $CRED_TOKEN from API Credentials page # Call Control Hub or Data Collector (Control Hub enabled) REST APIs using API Credentials curl -X GET https://eu01.hub.streamsets.com/security/rest/v1/currentUser -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $CRED_ID" -H "X-SS-App-Auth-Token: $CRED_TOKEN" -i
So, what you need to do is:
- Get the auth token from the UI first
- Make the call to the API passing the CRED_ID and CRED_TOKEN values as in the example above
I hope this helps!
1 person likes this
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.