- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 year ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1189 Topics

Recently active

Most replies Most views Newest first

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

How to connect to Azure dedicated SQL pool via Azure Synapse SQL?

In this article, we'll configure the Azure Synapse Dedicated SQL pool and insert some records from Mysql.Platform:Streamsets Data Collector version 5.0 MySql version 8.0.28.Pre-requisite:Mysql connector driver for JDBC query consumer stage. Install Azure Synapse Enterprise Library 1.1.0 from PackageManager to enableAzure Synapse SQL destination. Configuration:Step1: When you create the workspace in Azure Synapse, you will be creating a storage account and workspace however it does not create a SQL pool. You would need to create the SQL pool from:Azure Analytics » Click on Workspace web URL » Manage and then click on “new”. This will create the Azure SQL pool.Once done, you will be able to see the SQL pool in the overview tab. “akshaytest” is the new sql pool here. Step2: Develop the pipeline with JDBC Query Consumer origin and Azure Synapse SQL destination.In the JDBC consumer, we would be reading all the data from the “Azure” mysql database which is hosted on the local machine.These a

2 years ago

jmanoFan

posted in Community Articles and Got a Question?

DC 5.0 load data from db2 to Oracle DB using multitable consumer example

Hellois it possible to use DC pipeline to move data from DB2 to Oracle DB using multitable consumner.? is there any example pipe line ?oris there any other way? thanksJames

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

HTTP Client Origin - Max Object Length Exceeded

Is there a way to increase the amount of data that a HTTP client can return in StreamSets? It seems to be limited (I have maxed out the Max Object Length field and still get this error). The call returns the data OK in postman - the source file is only a 1.9Mb Excel with 29k rows.WE ARE ON DATA COLLECTOR VERSION 5.0.0.

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

StreamSets 5.0.0 Documentation - Discrepancy

Hi there, I was reading the below documentation,https://docs.streamsets.com/portal/platform-datacollector/5.0.x/datacollector/UserGuide/Processors/WaitJob.html#concept_xv5_xbd_zlband I could not find a processor stage named “Start Pipelines” in version 5.0.0. Note - I am on the latest version of DataOps Platform and I am building my Data Collector pipeline on the DataOps Platform / Control Hub GUI. Could you please verify and advise whether I need any additional stage libraries to be installed on my 5.0.0 engine for “Start Pipelines” processor stage? I am able to see other Orchestration stages as described in the below documentation,https://docs.streamsets.com/portal/platform-datacollector/5.0.x/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html#OrchestratorStages I can only assume this is a discrepancy in the documentation (refer first link), please advise otherwise. Cheers,Srini

2 years ago

RishiStreamSets Employee

posted in Community Articles and Got a Question?

SDC deployment is not working [Data Collector not showing up in SCH]

Issue: While deploying SDC in kubernetes cluster via provisioning agent. SDC deployment is up and running but data collector doesn’t show up in control hub. To troubleshoot such issue check the provisioning agent logs$kubectl logs $<provisioning-agent-pod>if you see the error like below KubernetesDeploymentCoordinator - Syncing Deployment Status2022-05-14 03:29:17,810 [thread:Deployment Status Refresher] DEBUG AbstractPodManager - Handling Bootstrap New Pod http://172.24.51.2:186302022-05-14 03:29:17,811 [thread:Deployment Status Refresher] ERROR SDCCredentialsDeploymentAssistant - Error accessing ping path for SDC at address http://172.24.51.2:186302022-05-14 03:29:17,811 [thread:Deployment Status Refresher] DEBUG SDCCredentialsDeploymentAssistant - SDC Not Accessible Error {}java.net.ConnectException: Connection refused (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at

2 years ago

shicaiyuFan

asked in Community Articles and Got a Question?

Linking the registered account error

2 years ago

nicoletteanFan

posted in Community Articles and Got a Question?

Let’s Scratch the Surface and Get to Understand the Value of MLOps

Hi StreamSets Community! I’m Drew Nicolette, a data engineer in the public sector. I have a passion for teaching, learning and contributing to data engineering communities. I’ve recently published my first blog through the StreamSets Write for Us program, “MLOps: How Data Teams Can Give ML Algorithms Life and Longevity” and I’d love everyone’s thoughts. Feel free to leave any comments to open up a conversation. In this post, I discuss the three steps of the Machine Learning model workflow, MLOps, and the four main reasons I see the need of and value in MLOps. As I look into the future, I see MLOps becoming the de facto way enterprise ML is conducted, so if you’re not already, let’s get familiar with it and its value. I only scratch the surface in this post, so I’d love to keep the discussion going here! Please feel free to share the post on your own channels as well.

2 years ago

MaryannFan

posted in Community Articles and Got a Question?

Data Orchestration Pipelines Guide the Entire Data Journey

Hi StreamSets Community! I’m Maryann Agofure, a cloud engineer and technical writer. I’m currently on a journey to gain a deeper understanding of data pipelines for better analysis. I’ve recently published my first blog through the StreamSets Write for Us program, “The Importance of Data Orchestration Pipelines for Organizations” and I’d love your feedback. In this post, I discuss the data workflow pre and post data orchestration functionalities available in the market. I touch on the benefits of and building a stack for data orchestration. I then walk through a data orchestration pipeline in action. Feel free to leave comments below to open up a conversation.And, please feel free to share the post on your own channels as well.

2 years ago

Dongmao ZhangFan

asked in Community Articles and Got a Question?

JDBC writes to MySQL Decimal type field missing decimal places

source data: csv format data fileThe data in the red box has two decimal placesMySQL DDL:CREATE TABLE test."LINEITEM" ( "L_ORDERKEY" BIGINT NOT NULL, "L_PARTKEY" INTEGER NOT NULL, "L_SUPPKEY" INTEGER NOT NULL, "L_LINENUMBER" INTEGER NOT NULL, "L_QUANTITY" DECIMAL(15, 2), "L_EXTENDEDPRICE" DECIMAL(15, 2), "L_DISCOUNT" DECIMAL(15, 2), "L_TAX" DECIMAL(15, 2), "L_RETURNFLAG" VARCHAR(1), "L_LINESTATUS" VARCHAR(1), "L_SHIPDATE" DATE, "L_COMMITDATE" DATE, "L_RECEIPTDATE" DATE, "L_SHIPINSTRUCT" VARCHAR(25), "L_SHIPMODE" VARCHAR(10), "L_COMMENT" VARCHAR(44));I use Directory read csv data file and use JDBC Producer write the data to MySQLbut the result missing decimal places:

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

START_JOBS orgin throws error - DataOps Platform

Hi there,I have a simple pipeline that uses “Data Generator” as the source and writes to Trash. I published this pipeline and I have created a job instance. I then created an orchestration pipeline with “Start_Jobs” origin as the source and I call the job instance created in the previous step. There are no parameters required for the job as it is a simple job (data generator to trash). I use API based authentication and I have provided the API cred. ID and Token. I use the following URL, https://cloud.login.streamsets.com for my Origin stage. I get the following error message when I try to test run my orchestration pipeline, com.streamsets.pipeline.api.StageException: START_JOB_03 - Failed to start job for job ID: 287e3a1a-c08b-4a1e-916d-2c2d9cc2068c:XXXXXXXXXXXXXXXXXX, status code '405': <html><head><title>405 Not Allowed</title></head><body><center><h1>405 Not Allowed</h1></center><hr><center>nginx</center&g

2 years ago

mblahayDiscovered Fame

asked in Community Articles and Got a Question?

UUID version

What version UUID is generated by the uuid:uuid() function?

2 years ago

datinfiFan

asked in Community Articles and Got a Question?

SDC Tarball intallation - activation key can't received

Hi there!We are one of the user of streamset datacollector product and also we have data collector tarball (tar.gz) backup.In previously we can able to install new cloud servers to using that tarball, but now we can’t.during the SDC installtion with existing tarball in RHEL OS, Installation are successfully completed and also SDC UI is loading with default port of [hostname:18630] but the registration was not completed,While the registration we have entered all the details and final stage asking activation key for registration we have entered the email but we are not receiving the registration key.we have checked your product license till you are providing open licensesCould you please give me the solution of my problem ASAP?

2 years ago

meghrajFan

asked in Community Articles and Got a Question?

Pivot stage on empty json list

I have created the simple SDC pipeline with json as input data.When i pivot the list item from json, it works fine when list has the elements but its eliminates the records when list is emptyPivot on revision works but requirements does not produce recordAppreciate any help . Thanks in advance. { "id": 22, "active": true, "code": "210_34", "productcomponents": [], "category": { "id": 21, "code": "SOP", "name": "Standard Operating Procedures" }, "revisions": [ { "id": 27, "active": true, "created": "2020-08-19 15:45:18" } ], "requirements": []}

2 years ago

MasoomFan

asked in Community Articles and Got a Question?

Missing Zomato directory

Hi ,I have found that zomato directory is not available .only three directory is available. please check the below screenshots.Screenshot 1:Screenshot 2:after running the command set given in the step 9 & step 10 .

2 years ago

meghrajFan

asked in Community Articles and Got a Question?

HTTP client with batch mode goes in the loop

Hi,my sdc pipeline with http client origin goes in the loop. ar source has only 67 records but it doesnot stop unless manually aborted.I could not find event like no more data to add as pre condition to finisherany help is appreciated. Thanks in advance.

2 years ago

rajneesh75Fan

asked in Community Articles and Got a Question?

How to parse array or array of JSON objects

HelloHow to parse a JSON structure like below and store in database/tables {"_id": "12345","machineId": "1001001366","activation": [{"activationId": 15589783,"activationActionCode": "ACTIVATE","activationDetail": [{"activationDetailId": 1001001665,"productInfo": {"activationProductId": "138591","skurr": "109-000-034"},"licMovCountLimit": "5","entitlementInfo": {"dlfEntitlementId": "DLF000028770","lac": "CW943GTXK37QSQ29355M","siteInfo": {"siteNumber": 673333,"companyDuns": "G005358932"}},"companyInfo": {"companyName": "abc CORPORATION","regUser": [{"userId": "10536","email": "10533@gggggg.com"}]},"isOriginal": "-1"}],"activationKey": [{"activationKeyId": 1001001398,"type": "licr455"}]}],"status": "ACTIV"} Does it need custom coding or off the shelf components can do it

2 years ago

sreenivasaraovakaFan

asked in Community Articles and Got a Question?

Control HUB REST API for Authorization

I’m running below command in my windows machine for Auth token and getting below error message. curl -s -X POST -d '{"userName":"userid", "password": "password"}' https://cloud.streamsets.com/security/public-rest/v1/authentication/login -H "Content-Type:application/json" -H "X-Requested-By:SCH" Invoke-WebRequest : Cannot bind parameter 'Headers'. Cannot convert the "Content-Type:application/json" value of type "System.String" to type "System.Collections.IDictionary".At line:1 char:168+ ... st/v1/authentication/login -H "Content-Type:application/json" -H "X-R ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest], ParameterBindingException + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeWebRequestCommand Can you please help?

2 years ago

sevtiandy

asked in Community Articles and Got a Question?

Missing redo log files for time range - Oracle CDC

Issue:Oracle CDC Origin status can’t running because it got error message Missing redo log files for time range.But when I checked the log at TRACE level, it got the redo log files:TRACE LogMinerSession - Find logs (2022-04-28T04:58:59, 2022-04-28T04:58:59): +FRA_DG/ONLINELOG/group_1.18814.1103108341, seq: 1/46137, start: 11638474806971 (2022-04-28T02:02:25), end: 11638474807092 (2022-04-28T02:02:28), status: INACTIVE, online: true, archived: true, dictionary: no, discardedTRACE LogMinerSession - Find logs (2022-04-28T04:58:59, 2022-04-28T04:58:59): +FRA_DG/ONLINELOG/group_2.16414.1103109069, seq: 1/46138, start: 11638474807092 (2022-04-28T02:02:28), end: 11638475179615 (2022-04-27T21:58:59.367866), status: CURRENT, online: true, archived: false, dictionary: no, discardedTRACE LogMinerSession - Find logs (2022-04-28T04:58:59, 2022-04-28T04:58:59): +FRA_DG/ONLINELOG/group_3.16418.1103108361, seq: 1/46135, start: 11638436593218 (2022-04-27T16:55:48), end: 11638437222272 (2022-04-27T20:09

2 years ago

sevtiandy

asked in Community Articles and Got a Question?

Oracle CDC Can't Running - Missing redo log files for time range

2 years ago

deepuFan

asked in Community Articles and Got a Question?

Job Monitor

I want to shot one mail when my specific jon change its status from Active to Inactive or Inactive error.Please help

2 years ago

RishiStreamSets Employee

posted in Community Articles and Got a Question?

How to delete header attribute in JavaScript Evaluator

You can use the JavaScript Evaluator processor to read, update, or create record header attributes.Use a map when creating or updating a header attribute. If a header attribute exists, the script updates the value. If it does not exist, the script creates the attribute and sets it to the specified value. However, when it comes to delete due to the underlying Java interpretation (Nashorn engine API) and hence you need to use Map API within Nashorn to delete/remove elements. i.erecords[i].attributes.remove('delete_this_attribute')

2 years ago

sevtiandy

asked in Community Articles and Got a Question?

Missing redo log files for time range on Oracle CDC

2 years ago

FrankHuoFan

asked in Community Articles and Got a Question?

JDBC_23 - Can't coerce '2022-04-25 09:46:50' of type 'STRING' to column 'last_update_date(DATETIME)'

my configuration:Use Multi-Row Operation is closeand Max Cache Size Per Batch (Entries) is “-1”origin:{ "table": "tableName", "op_type": "I", "op_ts": "2022-04-25 09:01:56.000000", "current_ts": "2022-04-25T09:02:01.014000", "pos": "00000000020000939589", "after": { “ROW_ID”:”123456”, "LAST_UPD": "2022-04-25 01:01:54" }}Destinations is jdbc producerI can not update the LAST_UPD into mysql column “last_update_date” please give me some advice

2 years ago

kapil24nagarFan

asked in StreamSets Academy

Lab: Build a JDBC Pipeline (zomato database is not found)

where can I get the SQL used to create the review tables under zomato database? or is there a way/script by which it can be made available in mysqldb docker?

2 years ago

rajkarFan

asked in Community Articles and Got a Question?

Guidance with authentication token job

We have two Streamsets jobs. One job generates token continuously. Another job moves data from a processed Kafka topic to API endpoint. Alongwith, it sends a token in its header. Endpoint validates if token is valid, then process data. If token isn’t valid, data is lost. Token is regenerated every 8 mins. So if data reaches endpoint after 8 mins, token it carried will be invalid. We want a solution that provides near ZERO data loss. One Possible solution we are thinking is as below. Please advise if there is any better option. Retry on Kafka topic to API endpoint job using ‘HTTP Client’ Processor. Write errored records to a local file/S3/ Oracle/HDFS table. Run a daily job every 3 hours to move data from file/table to API endpoint.

2 years ago

Page 38 / 48

Badge winners

vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.