Community Articles

Contribute and publish your articles/blogs here showcasing you usage, architecture, and know how of the StreamSets DataOps Platform. THIS IS NOT THE Q&A CATEGORY. Head over to the "Got a Question?" Category to ask your question.

  • 18 Topics
  • 17 Replies

18 Topics

N_ali
Roadie
N_aliRoadie
asked in Community Articles

aggregate the records into a single array of objects

I have some json objects that looks as follows and i want them to be all merged in an array like :data: [{nstaskid: “...”,awb: “….”},{nstaskid: “...”,awb: “….”},] i tried to use javascript evalutor , loop over the records and push them in an array and write this record array , but it didn’t work.Can you please help me with that .

8
Bikram
Opening Band
7 days ago
Yago Aparecido
Discovered Fame
Yago AparecidoDiscovered Fame
asked in Community Articles

how to disable autosave in pipeline

Hello everybody, how can we remove the "autosave" option in the pipeline?  any changes that are made in the pipeline, the streamsets saves automatically, we need the save to be manual How can we do that?

6
Yago Aparecido
Discovered Fame
20 days ago
sperandio
Fan
sperandioFan
asked in Community Articles

Global variable to reduce memory usage and API calls

I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all  previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that. Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record? 

1
Dimas Cabré i Chacón
StreamSets Employee
9 days ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in Community Articles

Swayam's Micro-blogging: Combine multiple JSON records into a single one

Dear Community users,To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?I often came with requirements where we have to call an external API which suports both single as well as batch requests and most often batch is preffered to avoid the multiple round-trips and increae performance. This topic can be very useful for developers who wants to convert the multiple CSV records coming in a batch to a single JSON list array and send them as a single record to a target destination.Problem StatementConvert the 2 input CSV records into a single JSON.Input CSV Records:NAME|AGE|DEPARTMENT|ADDRESS"John"|"23"|"HR"|"31, 2nd Cross, 4th Street, Wilson-543234""Amanda"|"26"|"HR"|"24, 1st Cross, Wilson Garden-543234"Output Single JSON Record:[ { "data": { "NAME": "John", "AGE": "23", "DEPARTMENT": "HR", "ADDRESS": "31, 2nd Cross, 4th Street, Wilson-543234" } }, { "data": {

1
Drew Kreiger
StreamSets Employee
6 months ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in Community Articles

Swayam's Micro-blogging: Auto detection of date format

Dear Community users,SDC offers a great out-of-box processor “Field Type Converter” to convert an incoming field of one data type to another data type. This is a very useful transformation function that is required before we send the data to final destination. However, sometime we may have to write our own transformation function and today I’m going to share a scenario where I was forced to use a jython evaluator to detect the date format of an incoming field and convert that to a single unified format that our destination is expecting.Problem StatementWe have a need to process the incoming file where there is a field that need to be processed as “date” data type while there is another field which needs to be processed as “timestamp” data type before we generate the final feed for destination. The detaination while accepts the “date” fields is YYYY-MM-DD format, the timestamp fileds need to be of YYYY-MM-DD HH:MI:SS GMT+05:30, the origin fields can be of different formats. The solution

1
Drew Kreiger
StreamSets Employee
6 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Use of Surrogate Key Generator(monotonically_increasing_id() in spark)

Comparing with relational databases, at times we want to have some id field in our dataset which is always increasing(usually for every record insertion). In general, this is useful for indexing, ordering, to distinguish each record or in simple words to maintain integrity of the database. This might not be the case with distributed systems. For Spark, data is mostly written to destination in partitions(that’s how we achieve parallelism). Since these partitions are written in parallel, we can not have consecutive id generation capability which is monotonically increasing. Spark has a function monotonically_increasing_id() which is helpful to generate increasing id at the partition level.Below is what Spark documentation has to say about it.A column that generates monotonically increasing 64-bit integers.The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record numb

0
Pradeep
StreamSets Employee
14 days ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment spec

Kubernetes Deployment fails with the below errorAck Error for Event '64627d8f-ccc8-494e-bbd3-e9750f361a68:dpmsupport'. Reason : java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment specControl-agent (Provisioning agent ) cannot create config-maps or secrets. The only resources control agent can create are deployments, services, ingress and hpa.If you want to use these resources then you need to create this separately and call it in deployment manifest.     

0
Rishi
StreamSets Employee
12 days ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

How to perform a SAML Trace

Learn to perform a SAML trace and generate SAML Trace JSON file using browser plug-ins. Streamsets support team uses the file to trace the SAML assertions occurring between your identity provider and Control-Hub to troubleshoot your sign-in issues.  Use the links below to download and install the SAML tracer plug-in for your browser:    Note: The links and steps provided here were correct at time of publishing. Mozilla Firefox: https://addons.mozilla.org/en-US/firefox/addon/saml-tracer/ Google Chrome: https://chrome.google.com/webstore/detail/saml-tracer/mpdajninpobndbfcldcmbpnnbhibjmch When the plug-in is added, Click the newly added SAML tracer icon on the upper-right add-in menu for your respective browser. This opens the SAML Tracer dialog box. The SAML tracer dialog box records and displays details as shown below. Note the occasional SAML tags shown at the right indicating SAML assertions being passed. Re-produce the issue. use the SAML Tracer dialog to navigate to Export tab

0
Rishi
StreamSets Employee
1 day ago
rvlozano
StreamSets Employee
rvlozanoStreamSets Employee
asked in Community Articles

How to connect JDBC Consumer Stage to MS SQL Server via AAD?

Question:How to connect JDBC Consumer Stage to MS SQL Server via AAD or Active Directory? Answer:Our product teams are currently evaluating to see this could be added natively. Also, our Product Teams has this on the future Product Road Map for native support.For now you will get the following error message if you try to use “authentication=ActiveDirectoryPassword” in your JDBC URL.JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Failed to load MSAL4J Java library for performing ActiveDirectoryPassword authentication.: hikariConfigBean.connectionStringThis is because the out of box Microsoft SQL Server Drivers comes directly from Microsoft. Microsoft does not package support directly for the authentication method ActiveDirectoryPassword.  For this to work you must acquire MSAL4J and it’s dependencies and add them to SDC’s java classpath for JDBC.This is currently classified under custom implementat

0
rvlozano
StreamSets Employee
3 months ago
A
StreamSets Employee
ajinkyaStreamSets Employee
posted in Community Articles

How to change labels of jobs using python SDK?

Please make sure all the jobs are in inactive state. Changes following parameters as per the environment:-<CONTROL_HUB_URL> = Controlhub URL <USERNAME> =  admin username <PASSWORD> = password <OLD_LABEL> = old label <NEW_LABEL>= new label from streamsets.sdk import ControlHub,DataCollectorDataCollector.VERIFY_SSL_CERTIFICATES = Falsesch = ControlHub('<CONTROL_HUB_URL>', username='USERNAME', password='PASSWORD')jobs_with_existing_label = sch.jobs.get_all(data_collector_labels=['<OLD_LABEL>'])print(f'jobs_with_existing_label = \n {jobs_with_existing_label}')for job in jobs_with_existing_label: job.data_collector_labels = ['<NEW_LABEL>'] sch.update_job(job)  

0
A
StreamSets Employee
3 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

How to delete header attribute in JavaScript Evaluator

You can use the JavaScript Evaluator processor to read, update, or create record header attributes.Use a map when creating or updating a header attribute. If a header attribute exists, the script updates the value. If it does not exist, the script creates the attribute and sets it to the specified value. However, when it comes to delete due to the underlying Java interpretation (Nashorn engine API) and hence you need to use  Map API within Nashorn  to delete/remove elements. i.erecords[i].attributes.remove('delete_this_attribute')  

0
Rishi
StreamSets Employee
3 months ago
Maryann
Fan
MaryannFan
posted in Community Articles

Data Orchestration Pipelines Guide the Entire Data Journey

Hi StreamSets Community! I’m Maryann Agofure, a cloud engineer and technical writer. I’m currently on a journey to gain a deeper understanding of data pipelines for better analysis. I’ve recently published my first blog through the StreamSets Write for Us program, “The Importance of Data Orchestration Pipelines for Organizations” and I’d love your feedback. In this post, I discuss the data workflow pre and post data orchestration functionalities available in the market. I touch on the benefits of and building a stack for data orchestration. I then walk through a data orchestration pipeline in action. Feel free to leave comments below to open up a conversation.And, please feel free to share the post on your own channels as well.

0
Maryann
Fan
2 months ago
nicolettean
Fan
nicoletteanFan
posted in Community Articles

Let’s Scratch the Surface and Get to Understand the Value of MLOps

Hi StreamSets Community! I’m Drew Nicolette, a data engineer in the public sector. I have a passion for teaching, learning and contributing to data engineering communities. I’ve recently published my first blog through the StreamSets Write for Us program, “MLOps: How Data Teams Can Give ML Algorithms Life and Longevity” and I’d love everyone’s thoughts. Feel free to leave any comments to open up a conversation. In this post, I discuss the three steps of the Machine Learning model workflow, MLOps, and the four main reasons I see the need of and value in MLOps. As I look into the future, I see MLOps becoming the de facto way enterprise ML is conducted, so if you’re not already, let’s get familiar with it and its value. I only scratch the surface in this post, so I’d love to keep the discussion going here! Please feel free to share the post on your own channels as well. 

0
nicolettean
Fan
2 months ago
javierdallamore
Fan
javierdallamoreFan
posted in Community Articles

Our Journey Finding the Perfect-Fit Data Integration Tool

Overview Since November 2014 we have been implementing a Data Lake platform to collect, process, filter, enrich, store, analyse and visualize logs and business events using Big Data and streaming technologies. The main goal is to detect important events, generate statistics and produce alerts to solve problems in a proactive way.Our first pilot on a new customer is usually related to application logs but once this platform is up and running it naturally becomes the centralized repository not only for logs but also for any type of sources and data formats. With dashboards on display in big screens other areas and departments start requesting dashboards of their own that require new data sources to be connected.To adapt to different use cases and new requirements we needed a more modular, flexible and extensible architecture. As the Data Lake gained prominence within our firsts customers and other project opportunities materialized, challenges and constraints emerged that led us to rethi

0
javierdallamore
Fan
8 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

SDC deployment is not working [Data Collector not showing up in SCH]

Issue: While deploying SDC in kubernetes cluster via provisioning agent. SDC deployment is up and running but data collector doesn’t show up in control hub.   To troubleshoot such issue check the provisioning agent logs$kubectl logs $<provisioning-agent-pod>if you see the error like below KubernetesDeploymentCoordinator - Syncing Deployment Status2022-05-14 03:29:17,810 [thread:Deployment Status Refresher] DEBUG AbstractPodManager - Handling Bootstrap New Pod http://172.24.51.2:186302022-05-14 03:29:17,811 [thread:Deployment Status Refresher] ERROR SDCCredentialsDeploymentAssistant - Error accessing ping path for SDC at address http://172.24.51.2:186302022-05-14 03:29:17,811 [thread:Deployment Status Refresher] DEBUG SDCCredentialsDeploymentAssistant - SDC Not Accessible Error {}java.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at

0
Rishi
StreamSets Employee
2 months ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in Community Articles

Building a No-code Data Connector Platform Using StreamSets Data Collector REST APIs

PrefaceFetching online and offline data across different channels to understand your end-users behavior is a primary topic for any enterprise-level brand. Customer 360 view of an end customer gives enormous insight to marketing leaders who can then prepare their strategy to engage and entice these users. A full-stack CDP (Customer Data Platform) helps meet the needs of marketing leaders and analytic folks.A stable, reliable, and scalable data processing stack is the de facto need for any enterprise CDP solution, and to set up that, data ingestion and extraction architecture need to be solid. StreamSets Data Collector (SDC) has become the preferred tool for many data engineers and data stewards to achieve the ETL and ELT-related needs in this area.In the traditional approach, the ETL engineers normally have definitive requirements in a place where they know the origin, transformation rules, and destination. Based on the requirements, engineers develop their pipelines and deploy them to

0
swayam
Discovered Fame
8 months ago
Victor
StreamSets Employee
VictorStreamSets Employee
posted in Community Articles

How to ingest multiple Data Formats with Options using runtime parameters and a Job Template for job Instance creation

This document aims to explain how to create a pipeline with an origin configuration capable of ingesting multiple types of data format files in the DataOps platform. Once the pipeline is built, we will be able to create a job template capable of generating job instances for the desired data format ingestion. This methodology works on any stage capable of ingesting multiple types of data. For explanatory purposes we are going to use Amazon S3 as an example to show how to set up the model. First, we create the Pipeline:  Create the pipeline: create a pipeline with an origin S3 stage  Configure the origin S3 stage: Click on S3 stage > Configuration > Amazon S3 tab. Configure it to be able to connect to the desired server and bucket (it can also be parameterized if desired). Parameterize the Data Format and Options: Set the names of the parameters we want to use for data format: Click on the canvas > click on show advanced options > configuration > Parameters tab  

0
Victor
StreamSets Employee
2 months ago
A
StreamSets Employee
ajinkyaStreamSets Employee
posted in Community Articles

ERROR:- failed to store credentials: credentials cache permissions incorrect(krb5cc_cldr)

ENV:-SDC is integrated with CDH/CDP. Kerberos is enabled. start event script/end event scripts are being used to get the kerberos credentials.ISSUE:-start event script/end event scripts try to kinit and end up with following error message:-ERROR stderr: kinit: Pre-authentication failed: Permission denied while getting initial credentials Resolution:-By default CDH/CDP enabled clusters has set environment variable KRB5CCNAME  add following line to at the beginning of the script to unset the variable temporarilyunset KRB5CCNAME

0
A
StreamSets Employee
2 months ago

Badge winners

  • Community Champion 🏅
    Bikramhas earned the badge Community Champion 🏅
  • Eager to help
    Bikramhas earned the badge Eager to help
  • Eager to help
    Pradeephas earned the badge Eager to help
  • Community Champion 🏅
    Srinivasan Sankarhas earned the badge Community Champion 🏅
  • Community Champion 🏅
    swayamhas earned the badge Community Champion 🏅
Show all badges
Terms & ConditionsCookie settings