Community Articles

Contribute and publish your articles/blogs here showcasing you usage, architecture, and know how of the StreamSets DataOps Platform. THIS IS NOT THE Q&A CATEGORY. Head over to the "Got a Question?" Category to ask your question.

  • 55 Topics
  • 90 Replies

55 Topics

Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

list all streamsets pipelines by rest api

hi streamsets community, we need to export the listing of all pipelines that exist in streamsets by rest api How can we do that?

2
Yago Aparecido
Discovered Fame
5 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

ERROR StageLibraryUtil - Failed to read repository manifest json

During SDC startup by default SDC try connecting to repository manifest URL 2022-09-12T21:38:54,103 [user:] [pipeline:] [runner:] [thread:ManifestFetcher] [stage:]INFO StageLibraryUtil - Reading from Repository Manifest URL: http://archives.streamsets.com/datacollector/5.0.0/tarball/repository.manifest.jsonNow, if your SDC doesn’t have internet connectivity it will failed with the below error, However, SDC will start without any issue 2022-09-12T21:38:56,714 [user:] [pipeline:] [runner:] [thread:ManifestFetcher] [stage:] ERROR StageLibraryUtil - Failed to read repository manifest jsonjavax.ws.rs.ProcessingException: java.net.SocketTimeoutException: connect timed out at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287) ~[jersey-client-2.25.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:

0
Rishi
StreamSets Employee
6 months ago
N
Opening Band
N_aliOpening Band
asked in Community Articles

aggregate the records into a single array of objects

I have some json objects that looks as follows and i want them to be all merged in an array like :data: [{nstaskid: “...”,awb: “….”},{nstaskid: “...”,awb: “….”},] i tried to use javascript evalutor , loop over the records and push them in an array and write this record array , but it didn’t work.Can you please help me with that .

15
tamilselg
Fan
6 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

STARTING to RUNNING state of a pipeline

When a data collector pipeline is in STARTING state it will validate each stage, create required clients and also do initialization.Each stage has init() method which initializes the objects needed for stages to run.  Validation runs static operations. For example, running the query given in JDBC origin stages with limited results to validate the query, its schema, offset column etc. Do note that validation is specific to each stage.If you notice any slowness in STARTING to RUNNING stage, it is possible that validation is taking sometime and it usually happens when the datasets in configured query are larger. This is an example of slowness for JDBC or Relational DB stages. If the slowness is unacceptable you might consider disabling the query validation. 

0
Pradeep
StreamSets Employee
6 months ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Community Articles

Connecting to Microsoft SQL Server from SDC v5.1 is failing "unable to find valid certification path to requested target"

Environment:Streamsets Data Collector 5.1. Producer or consumer Library for Microsoft SQL Server. Issue: The pipeline starts failing with following messages when try to fetch or produce the data to Microsoft SQL Server.This happens on data collector is 5.1 but the same pipeline is working fine on 4.x versions.com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:af15a6cf-f28d-4e21-adb3-141826becad0 at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3680) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:2113) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.SQLServer

0
AkshayJadhav
StreamSets Employee
6 months ago
N
Fan
nick_nilzFan
asked in Community Articles

How do you introduce (write) new columns to an existing hadoop table?

Hello community,My pipeline is being upgraded to ingest more fields and I want to write these new fields to the existing table in Hadoop. I have searched for the metadata processor but it does not exist and I have enabled data drift. None of them are able to write new columns, which means I have to write an ALTER TABLE statement to capture these new fields. With time we will get further requirements to expand our tables and introduce new fields. Is StreamSets able to write new fields automatically?Kind regards,Nick

1
N
Fan
6 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

(DataOps) Downloading support bundle when Websocket tunneling is enabled

When Websocket Tunneling is enabled, you won’t be able to download support bundle through dataops platform UI. We are shown below message when we try to access it from Engine > Support Bundle page.The support bundle allows you to generate an archive file with the information required to troubleshoot various issues with the engine. To download a support bundle, you must set up direct engine access and turn off Websocket Tunneling in your Browser Settings.Follow below steps to download support bundle:Generate an API Credential from DataOpsDataOps Platform -> Manage -> API Credentials -> Add New API Credential -> (Note Credential ID and Token)Run the following commands on the engine(SDC or Transformer) (replace CRED_ID and CRED_TOKEN values) where the relevant pipeline has been run.export CRED_ID=<xxxxxxxx>export CRED_TOKEN=<xxxxxxxx>curl -X GET '<URL-of-the-engine>/rest/v1/system/bundle/generate?generators=SdcInfoContentGenerator,PipelineContentGenerator,

0
Pradeep
StreamSets Employee
6 months ago
A
Discovered Fame
ana00159Discovered Fame
asked in Community Articles

Reverse pivot

How to do reverse pivot in streamsets? My rows are as follows.ABC,X,,,100ABC,,Y,,100ABC,,,Z,100I want it as one rowABC,X,Y,Z,100 How this can be done?     

0
A
Discovered Fame
6 months ago
deepika
Fan
deepikaFan
posted in Community Articles

DevOps Vs SRE

DevOps and site reliability engineering (SRE) are two approaches that enhance the product release cycle through enhanced collaboration, automation, and monitoring. Both approaches utilize automation and collaboration to help teams build resilient and reliable software—but there are fundamental differences in what these approaches offer and how they operate. About HKR TrainingHKR Trainings excel at providing you the best online classes with high quality facilities at a low price without any compromise on quality. What can you expect from us? A dedicated learning platform with 24*7 support, best in class training materials to help you learn advanced techniques and practical knowledge of all IT Technologies. Our courses are specifically curated for both professionals as well as job-seekers. Online classes conducted by the best knowledgeable and certified trainers helps you earn certification at your convenience. Key featuresFlexible TimingsHands On Experience24/7 SupportCertified & In

0
deepika
Fan
6 months ago
Vikki
Fan
VikkiFan
asked in Community Articles

REST API

HI team,    I am having 6 API’s where I have to build 3 pipelines for each i.e., API-KAFKA, KAFKA-FILESYSTEM and FS-Target table. Like this  For 6 API’s I have to build and execute 18 pipelines (6*3). Could anyone suggest me  that is there any approach which reduces the number of pipelines ( to push all 6 APIs data into 1 Kafka topic in 1st pipeline, after that I will generate data file from Kafka to FS and FS to target table).FYI- my target table is same for all APIs.

6
Vikki
Fan
6 months ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

jvm memory exceed values defined in xmx and xms

Hello streamset community, I configured xmx and xms with 35GB, like this: SDC_JAVA_OPTS="-Xmx35840m -Xms35840m means that 35G has been configured, but querying the service status with "systemctl status sdc" command, the memory value exceeds 37G, as shown below:  why does this occur? does the memory value that appears with the "systemctl status sdc" command have a different conversion or does this memory add to another type of memory?

1
saleempothiwala
Opening Band
6 months ago
xagavin
Fan
xagavinFan
asked in Community Articles

can not register sdc by error 403 message Forbidden

after install StreamSets Data Collector 3.16.1  , and open it by chrome , i have show register page.when i submit infomation , the page is reload and show Register again ,  check network, i fount register request get 403 error . how to do ?

0
xagavin
Fan
7 months ago
sujatha mogili
Fan
sujatha mogiliFan
asked in Community Articles

The streamsets pipeline fails with java.lang.OutOfMemoryError: Java heap space

The streamsets pipeline fails on a regular basis with the below error. Please advise if you ran into this issue and the resolution. We have increased Heap size couple of times but not helping.“ERROR    A JVM error occurred while running the pipeline, java.lang.OutOfMemoryError: Java heap space”

5
alex.sanchez
StreamSets Employee
7 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

How to perform a SAML Trace

Learn to perform a SAML trace and generate SAML Trace JSON file using browser plug-ins. Streamsets support team uses the file to trace the SAML assertions occurring between your identity provider and Control-Hub to troubleshoot your sign-in issues.  Use the links below to download and install the SAML tracer plug-in for your browser:    Note: The links and steps provided here were correct at time of publishing. Mozilla Firefox: https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/ Google Chrome: https://chrome.google.com/webstore/detail/saml-tracer/mpdajninpobndbfcldcmbpnnbhibjmch When the plug-in is added, Click the newly added SAML tracer icon on the upper-right add-in menu for your respective browser. This opens the SAML Tracer dialog box. The SAML tracer dialog box records and displays details as shown below. Note the occasional SAML tags shown at the right indicating SAML assertions being passed. Re-produce the issue. use the SAML Tracer dialog to navigate to Export tab

0
Rishi
StreamSets Employee
7 months ago
sperandio
Fan
sperandioFan
asked in Community Articles

Global variable to reduce memory usage and API calls

I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all  previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that. Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record? 

1
Dimas Cabré i Chacón
StreamSets Employee
7 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment spec

Kubernetes Deployment fails with the below errorAck Error for Event '64627d8f-ccc8-494e-bbd3-e9750f361a68:dpmsupport'. Reason : java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment specControl-agent (Provisioning agent ) cannot create config-maps or secrets. The only resources control agent can create are deployments, services, ingress and hpa.If you want to use these resources then you need to create this separately and call it in deployment manifest.     

0
Rishi
StreamSets Employee
7 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Use of Surrogate Key Generator(monotonically_increasing_id() in spark)

Comparing with relational databases, at times we want to have some id field in our dataset which is always increasing(usually for every record insertion). In general, this is useful for indexing, ordering, to distinguish each record or in simple words to maintain integrity of the database. This might not be the case with distributed systems. For Spark, data is mostly written to destination in partitions(that’s how we achieve parallelism). Since these partitions are written in parallel, we can not have consecutive id generation capability which is monotonically increasing. Spark has a function monotonically_increasing_id() which is helpful to generate increasing id at the partition level.Below is what Spark documentation has to say about it.A column that generates monotonically increasing 64-bit integers.The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record numb

0
Pradeep
StreamSets Employee
7 months ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

how to disable autosave in pipeline

Hello everybody, how can we remove the "autosave" option in the pipeline?  any changes that are made in the pipeline, the streamsets saves automatically, we need the save to be manual How can we do that?

6
Yago Aparecido
Discovered Fame
8 months ago
A
StreamSets Employee
ajinkyaStreamSets Employee
posted in Community Articles

ERROR:- failed to store credentials: credentials cache permissions incorrect(krb5cc_cldr)

ENV:-SDC is integrated with CDH/CDP. Kerberos is enabled. start event script/end event scripts are being used to get the kerberos credentials.ISSUE:-start event script/end event scripts try to kinit and end up with following error message:-ERROR stderr: kinit: Pre-authentication failed: Permission denied while getting initial credentials Resolution:-By default CDH/CDP enabled clusters has set environment variable KRB5CCNAME  add following line to at the beginning of the script to unset the variable temporarilyunset KRB5CCNAME

0
A
StreamSets Employee
9 months ago
Victor
StreamSets Employee
VictorStreamSets Employee
posted in Community Articles

How to ingest multiple Data Formats with Options using runtime parameters and a Job Template for job Instance creation

This document aims to explain how to create a pipeline with an origin configuration capable of ingesting multiple types of data format files in the DataOps platform. Once the pipeline is built, we will be able to create a job template capable of generating job instances for the desired data format ingestion. This methodology works on any stage capable of ingesting multiple types of data. For explanatory purposes we are going to use Amazon S3 as an example to show how to set up the model. First, we create the Pipeline:  Create the pipeline: create a pipeline with an origin S3 stage  Configure the origin S3 stage: Click on S3 stage > Configuration > Amazon S3 tab. Configure it to be able to connect to the desired server and bucket (it can also be parameterized if desired). Parameterize the Data Format and Options: Set the names of the parameters we want to use for data format: Click on the canvas > click on show advanced options > configuration > Parameters tab  

0
Victor
StreamSets Employee
9 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

SDC deployment is not working [Data Collector not showing up in SCH]

Issue: While deploying SDC in kubernetes cluster via provisioning agent. SDC deployment is up and running but data collector doesn’t show up in control hub.   To troubleshoot such issue check the provisioning agent logs$kubectl logs $<provisioning-agent-pod>if you see the error like below KubernetesDeploymentCoordinator - Syncing Deployment Status2022-05-14 03:29:17,810 [thread:Deployment Status Refresher] DEBUG AbstractPodManager - Handling Bootstrap New Pod http://172.24.51.2:186302022-05-14 03:29:17,811 [thread:Deployment Status Refresher] ERROR SDCCredentialsDeploymentAssistant - Error accessing ping path for SDC at address http://172.24.51.2:186302022-05-14 03:29:17,811 [thread:Deployment Status Refresher] DEBUG SDCCredentialsDeploymentAssistant - SDC Not Accessible Error {}java.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at

0
Rishi
StreamSets Employee
10 months ago
nicolettean
Fan
nicoletteanFan
posted in Community Articles

Let’s Scratch the Surface and Get to Understand the Value of MLOps

Hi StreamSets Community! I’m Drew Nicolette, a data engineer in the public sector. I have a passion for teaching, learning and contributing to data engineering communities. I’ve recently published my first blog through the StreamSets Write for Us program, “MLOps: How Data Teams Can Give ML Algorithms Life and Longevity” and I’d love everyone’s thoughts. Feel free to leave any comments to open up a conversation. In this post, I discuss the three steps of the Machine Learning model workflow, MLOps, and the four main reasons I see the need of and value in MLOps. As I look into the future, I see MLOps becoming the de facto way enterprise ML is conducted, so if you’re not already, let’s get familiar with it and its value. I only scratch the surface in this post, so I’d love to keep the discussion going here! Please feel free to share the post on your own channels as well. 

0
nicolettean
Fan
10 months ago
Maryann
Fan
MaryannFan
posted in Community Articles

Data Orchestration Pipelines Guide the Entire Data Journey

Hi StreamSets Community! I’m Maryann Agofure, a cloud engineer and technical writer. I’m currently on a journey to gain a deeper understanding of data pipelines for better analysis. I’ve recently published my first blog through the StreamSets Write for Us program, “The Importance of Data Orchestration Pipelines for Organizations” and I’d love your feedback. In this post, I discuss the data workflow pre and post data orchestration functionalities available in the market. I touch on the benefits of and building a stack for data orchestration. I then walk through a data orchestration pipeline in action. Feel free to leave comments below to open up a conversation.And, please feel free to share the post on your own channels as well.

0
Maryann
Fan
10 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

How to delete header attribute in JavaScript Evaluator

You can use the JavaScript Evaluator processor to read, update, or create record header attributes.Use a map when creating or updating a header attribute. If a header attribute exists, the script updates the value. If it does not exist, the script creates the attribute and sets it to the specified value. However, when it comes to delete due to the underlying Java interpretation (Nashorn engine API) and hence you need to use  Map API within Nashorn  to delete/remove elements. i.erecords[i].attributes.remove('delete_this_attribute')  

0
Rishi
StreamSets Employee
10 months ago
A
StreamSets Employee
ajinkyaStreamSets Employee
posted in Community Articles

How to change labels of jobs using python SDK?

Please make sure all the jobs are in inactive state. Changes following parameters as per the environment:-<CONTROL_HUB_URL> = Controlhub URL <USERNAME> =  admin username <PASSWORD> = password <OLD_LABEL> = old label <NEW_LABEL>= new label from streamsets.sdk import ControlHub,DataCollectorDataCollector.VERIFY_SSL_CERTIFICATES = Falsesch = ControlHub('<CONTROL_HUB_URL>', username='USERNAME', password='PASSWORD')jobs_with_existing_label = sch.jobs.get_all(data_collector_labels=['<OLD_LABEL>'])print(f'jobs_with_existing_label = \n {jobs_with_existing_label}')for job in jobs_with_existing_label: job.data_collector_labels = ['<NEW_LABEL>'] sch.update_job(job)  

0
A
StreamSets Employee
11 months ago

Badge winners

Show all badges
Terms & Conditions