Is there a way to create a pipeline to successfully call StreamSets API endpoints that does not require the Python SDK for StreamSets to be leveraged? I have tried the REST Service and HTTP Client origins, and even attempted (horribly) a Jython Scripting origin to no avail. I am fairly certain that authentication is the problem since the API works when I use them through the SCH RESTful API section manually, and receive the following as an error when attempting to use a pipeline stage:
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at tSource: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 2]
My ask for a UI-only solution is due to 1) it would allow less experienced admins to troubleshoot without immediately calling on me if problems arise, 2) allow us to build sequences for business- or management-oriented users to execute for results they need without asking an admin to do it, and 3) it would simplify future automation and security of automation for a planned larger ETL process to get the desired information into Snowflake for use in visual analytics presentation by keeping everything “in house”, for lack of a better phrase.
We are on SDC 5.9.1 and a Cloud-hosted SCH (DataOps). We already have API credentials utilized (distinct by environment) but these do not work despite working for orchestration of automated jobs.
I appreciate any tips or suggestions anyone has to offer, and I thank you for your time reading this even if you do not reply.
Sean Perch
