StreamSets Platform

We are working on STF and Pytest. Need help with some resources like flowcharts and sample code?If anyone worked on it or is working on it, please help us out with it.If any document or any link is there, kindly share it.Thank you.

the json data in mongodb had  a date format as a  timestamp in value  but during  process only date shown not timestamp show. how to fix below is my mongodb data:- but  inside of data updatedAt column has timestamp value but in streamsets not shown.how to get timestamp value from mongodb origin.

Hi,I use the kinesis consumer on DC 5.3. Configuration shows no errors. When I preview the pipeline everything works but no records are read. This seems to be due to that only one shard contains the actual data. How can I configure the shard id to read from. Thanks, Marco

How can artificial intelligence be used to enhance cybersecurity and protect  against cyber threats?

How to convert BYTE_ARRAY to String?I wanted to convert BYTE_ARRAY to String.version i am using 3.1.6I have tried using field type converter, but it is changing the valuefield type converter -Preview mode- please see my_value is changed How I can convert BYTE_ARRAY to STRING? please suggest.

  ProblemAfter enabling External JMX Tools for SDC, you are unable to fetch any JMX data from SDC’s JMX socket. For instance, you enabled the JMX socket in your SDC_JAVA_OPTS for 0.0.0.0:3333:export SDC_JAVA_OPTS="-Dcom.sun.management.jmxremote \-Dcom.sun.management.jmxremote.port=3333 \-Dcom.sun.management.jmxremote.local.only=false \-Dcom.sun.management.jmxremote.authenticate=false \-Dcom.sun.management.jmxremote.ssl=false"But connecting to 0.0.0.0:3333 via jconsole, you don’t get any metric data.SolutionThis behavior typically indicates that your Java Security Policy is not granting SDC’s JMX Metrics library access to query the JVM’s runtime information or to connect to the JMX Socket.At SDC startup, you will see if some Java Security Policy is denying some SDC library from loading due to insufficient permissions (java.security.AccessControlException: access denied):Exception in thread "Attach Listener" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAc

inputdata:I want to change all keys are in same:-like 0,1,2, keys are changed to $id for all how to fix this ..suggest . 

When utilizing the HTTP Client stages to connect to resources we can sometimes face issues which default logging within the pipeline may not encompass. However, the HTTP Client stage contains a fantastic feature to help provide context to the issues you might encounter. This feature is “HTTP Request Logging”. From the HTTP Client Stage select “Show Advanced Options”  Selecting “Show Advanced Options” will unlock the advanced tab titled “Logging”  From here check the box at the top of the section to ensure request logging has been enabled. Now that you have enabled request logging we need to ensure that it is configured correctly. The picture below shows the settings you should utilize specifically in order for the logging of verbose request and response information. In order for this setting to work in this context you must ensure that Log Level: INFO and that your Max Entity Size is set high enough to fully encompass the contents which you are attempting to log. Keep in mind when conf

below I  give the required out: suggest how to achieve this..with $id key field remaner{{“$id”:’’12312r2”},{“$id”:”23455”},{“$id”:”34567”}  } but I achieved :by process using field flattener and json generator: -outputincoming data-  

Issue:When trying to deploy an authoring Data Collector with Control Agent on Control Hub, the following error is received:io.fabric8.kubernetes.api.model.networking.v1.Ingress cannot be cast to io.fabric8.kubernetes.api.model.extensions.Ingress Cause:This is due to the Ingress apiVersion being set to networking.k8s.io/v1, while Control Agent expects the Ingress to be of the extensions api group (extensions/v1beta1). However, in Kubernetes versions 1.22+, the extensions api group was made unavailable. Solution:The workaround for this would be to create the Ingress resource outside of Control Agent, separately from the SDC deployment. Additionally, ensure that an Ingress controller is set up in your cluster, such as the ingress-nginx Ingress controller.

Hi Team I am making post call to HTTP to a URL.Post request body Size is 1100KB and sometimes even moreI am getting error like 413, HTTP-payload too large error. Please help me with solution. Any help is appreciated

Above is the Rest API to get the job details for a particular job id which I am able to get it by sending the request through sch or using the URL in any browser, But when I am trying to hit the same URL along with header and params in postman I am not getting any response as shown bellow: 

how to convert list_map data into  json_data:- if any processor to solve this mention Required output or excepted out_put :[{"$oid": "6415a02047038cf77ecc810b"}, {"$oid": "6415a0c247038cf77ecc8120"}, {"$oid": "6415a13947038cf77ecc8135"}, {"$oid": "6415a19e47038cf77ecc8141"}, {"$oid": "6415a1b247038cf77ecc8148"}, {"$oid": "6415a26247038cf77ecc8154"}]  achieved output  but i want some more spec :{"0": "6415a02047038cf77ecc810b", "1": "6415a0c247038cf77ecc8120", "2": "6415a13947038cf77ecc8135", "3": "6415a19e47038cf77ecc8141", "4": "6415a1b247038cf77ecc8148", "5": "6415a26247038cf77ecc8154"}   

below is my error  I want to write these type columns (columns=””)to NULL IN mysqldb 

soure data from mongodb as list_map  to writen in mysql as json_type convertor . mysql data type is json wanted  

I had a JDBC connection change that I want to use in a JDBC stop event query. I am hoping to programmatically update the connection in existing pipelines using the python SDK package. I am getting an error publishing the pipeline after updating the connection in the stop event. Has anyone else run into this, or found a solution? Note: I have been able to successfully update the connection in Hive Query stop events, and publish the pipeline without error. 

Hello,I am looking for documentation for the transpose processor in transformer pipelines.I searched here : https://docs.streamsets.com/portal/platform-transformer/latest/search.html?searchQuery=transpose but did not find it.Can someone share the link to the relevant documentation?-Dhanashri 

Which REST API endpoint to use to check if an engine is offline.

I am facing issue while configuring repartition method attribute for repartition processor using python. What is the right way to set it.

As a part of my requirement I am using http origin to connect to end point and need to perform lookup on file to get relevant data from file.How to perform lookup on a file using streamsets

Hello everyone,I just wanted to ask that don’t we have post-processing capability for “GoogleCloudStorage_01”?Basically we are using “streamsets/datacollector:3.18.1” for some testing purpose and we are fetching data from GCS and want to move file to some other location post processing.

In this article we will walkthroughs the steps involved to deploy engines in kubernetes using “Legacy Kubernetes” support in DataOps Platform. Note: Legacy Kubernetes integration is enabled only for some paid accounts. For more information, contact your StreamSets account team. Once you have Legacy Kubernetes enabled for your org, you should see the following additional option.Prerequisite:jq installed and of course Kubernates environment of your choice Create a namespace where you are expected to have the control-agent, StreamSets engines(SDC, Transformer for Spark etc.)         Eg command: kubectl create namespace dataopsStep-1: Clone the git repo https://github.com/onefoursix/control-agent-k8s-deploymentStep-2: Set these variables at the top of the file deploy-control-agent-on-dataops-platform.sh:ORG_ID=<Your DataOps Platform Org ID>SCH_URL=<Your DataOps Platform URL> # for example: https://na01.hub.streamsets.comCRED_ID=<Your API Credential CRED_ID>CRED_TOKEN=<

After successfully upgrading Data Collector using Cloudera Manager from a previous version to 5.x you notice that the log screen does not get any log entries and there is no sdc.log file generated in the $SDC_LOG directory. Cloudera Manager uses CSD (custom service descriptor) files to determine the configuration, monitoring and file distribution of a custom service (in this case SDC) that will be managed by Cloudera Manager. You can learn more about this in Cloudera’s documentation.  As part of the SDC upgrade steps you need to install the newest CSD file. This has always been a requirement but in previous versions, not updating it caused little to no problems. For this reason some users may have skipped this step for previous SDC upgrades. For upgrades of older SDC versions to 5.x, not updating the CSD file will result in no logs being generated. SDC 5.x uses log4j 2.x while older versions of SDC used log4j 1.x which are not compatible with each other. If the old CSD file is kept in

If i converting string type date to date type means the output show wrong. 

I am trying to export pipeline in json format using python sdk. I am able to do in zip format but not in json format. I am following below links but export_pipeline functions is not available in new version of stramsets. export_pipelines() is available. https://docs.streamsets.com/sdk/latest/usage/sdc/importing_exporting_pipelines.html#:~:text=In%20order%20to%20export%20a,JSON%20representation%20of%20the%20pipeline.