StreamSets Academy

StreamSets Academy course topics and questions

  • 85 Topics
  • 215 Replies

85 Topics

S
Fan
shivrajFan
asked in StreamSets Academy

STF

In STF 3.x we have used "sch_authoring_sdc.get_history(pipeline)"  to get all the information, for 4.x do we have anything like that for accessing the history? as get_history is not working with STF 4.x?

1
Kirti
StreamSets Employee
7 hours ago
pinkbapehoodie22
Fan
pinkbapehoodie22Fan
asked in StreamSets Academy

3. x - Job runner API - Obtaining jobs for a particular pipeline - returns jobs that do not meet the requirement

Hello, I'm utilizing Control Hub 3. x to recreate the customer's environment. Based on the pipeline name (given as a parameter) and the number of days old, I need to clear up old tasks. I made the pipeline shown below, which is also attached.

1
Rishi
StreamSets Employee
8 days ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy

3.x - Job runner API - Getting jobs for a specific pipeline - returns jobs that do not match the criteria

Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached)  When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri

0
D
Opening Band
19 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to covert file and write into localfs as a XLSX file Format.

can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration  after  execution of my pipeline :-out put file like - But my requirement is like  how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.

1
M
Fan
6 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

I want to correct the datatypes of all of my input field values ?

I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all  fields  that contains blank values means feed with  NULL . any other processor can replace these scenario

1
Bikram
Headliner
18 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to Write XLSX file format in localfs.

My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?

1
Bikram
Headliner
20 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to separate pincode into specific field?

I had Pincode like alpha numeric how can i separate numeric value into specific fieldthis is field value -4900 B ICI RD BUT I WANT Output like -4990 and remainig another field like-B ICI RD  

1
Rishi
StreamSets Employee
28 days ago
M
Fan
MalathiFan
asked in StreamSets Academy

Getting error while importing pipeline and cant able to import connections.

Hi,I am trying to import the pipelines in another user account but, i am facing the below issue.Cannot import pipeline D_AmazonSQS_Origin_JDBC_Dest_Pipeline_CredStr, pipeline with the same commit Id 6f1628b3-d3b4-4d49-9a77-d80fce22717b:13f66132-5e7d-11ed-b52a-23b2c9b2adfb already exists. : PIPELINE_STORE_30I don’t have any pipelines in that user account where I am trying to import.Also, I have created separate predefined connections which is used inside those pipelines. When I am trying to export, it got downloaded with only name and type in connections metadata.json file which is inside pipelines.zip. But credentials details are not there in the json connection file. And also, I cant able to import those connections.json file. Do you know how to import the connections?Can you help me what could be the issue? Thanks,Malathi S

1
M
Fan
1 month ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

directory not able ready xml file .

ERROR LIKEin my directory xml file but not able to fetch records and also i increased max record length-but file length is less than 1mb but i don’t know why unable to fetch my xml data any one can find solution and share it. 

6
saleempothiwala
Headliner
1 month ago
V
Fan
vigneshmFan
asked in StreamSets Academy

Possibility to connect DB2 as source in StreamSets

Hi All,There is a plan to read records from DB2 in StreamSets. Can we use JDBC Multitable consumer Origin, if so, is any other additional steps that we need to take care to start reading records?Please confirm the possibilities in StreamSets connecting to DB2. Thank you,

2
V
Fan
1 month ago
Vip
Fan
VipFan
asked in StreamSets Academy

Whole file transfer from a GCP bucket to another GCP bucket

I need to copy the directory or folder that contains multiple types of files, including CSV and PNG, from one GCP bucket and paste it into a folder in another bucket. How can I do this?

1
saleempothiwala
Headliner
1 month ago
U
Fan
unifyStreamSet
asked in StreamSets Academy

Stream set upgrade issues with 3.19 version

Hi Team, We are trying to upgrade the stream set upgrade from 3.16.x to 3.19.X version using the docker container. Queues are not coming up.After doing the analysis, we got to know activation code concept is introduced. Now what is the procedure for getting the activation code using sdc id incase if we don’t have the admin access. Thanks. 

3
Sanjeev
StreamSets Employee
2 months ago
nikkichow
Fan
nikkichowFan
asked in StreamSets Academy

Handling the error records in the same Kafka topic

Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages  . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?

3
Sanjeev
StreamSets Employee
2 months ago
mikewatton
Fan
mikewattonFan
asked in StreamSets Academy

External resource unable to find http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.   http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

0
mikewatton
Fan
2 months ago
HariUmesh
Fan
HariUmeshFan
asked in StreamSets Academy

How to get the list of pipeline names from control hub using python sdk ?

Hi All,i am building a unittest framework for validating the pipeline names from the streamsets control hubFor that i need to fetch pipeline names from streamsets using the python platform sdk as seen belowfrom streamsets.sdk import Controlhubsch = Controlhub(credential_id="my_id", token ="Mytoken", use_webscoke_tunneling=False)pipelines = sch.pipeline.get_allthe above code return list of attributes and name of pipeline with the type of class ‘Method’.  i actually need to get only those pipelines that already deployed into the streamsets through our CI platformso what would be the right approach to filter and get the pipeline names from control hub?I don't see any example specific to this scenarioCan you please help on this? 

2
HariUmesh
Fan
2 months ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

HOW TO CONFIGURE SNOWFLAKE ACCOUNT URL

 BELOW IS MY SNOWFLAKE ACCOUNT URL ERROR SHOWS    ERROR LIKE THIS: 

1
K
StreamSets Employee
3 months ago
R
Fan
RadhaMuthukrishnanMFan
asked in StreamSets Academy

Streamset kubernetes autoscaling : VPA

Hi, I am able to implement HPA (Horizontal Pod Autoscaling) in streamsets by integrating with kubernetes cluster. i can see engines are getting scaled up.  It is possible to do VPA in streamsets?

0
R
Fan
3 months ago
R
Fan
rohit6488Fan
asked in StreamSets Academy

Missing Fragment used in Academy Course for CDC from Mysql to Snowflake

Hi All,i am looking for Fragment which has been used in Academy Course for CDC Pipeline Demo, they have mentioned they will share with us. But i cant find it anywhere. Can someone help me to share that fragment. Regards,Rohit

4
R
Fan
3 months ago
Rhudehoodie52
Fan
Rhudehoodie52Fan
asked in StreamSets Academy

Can the transformer pipeline be configured to run on an AWS EMR cluster?

I created a pipeline for primary transformers that would run on an EMR cluster. I used the crucial setup information when configuring the cluster.I'm experiencing the following issue:Transformer's pipeline Engine Transformer 

0
Rhudehoodie52
Fan
4 months ago
A
Roadie
akshayp2Roadie
asked in StreamSets Academy

Subscriptions created for create incident for failed job in ServiceNow has not triggered.

I have created subscription as suggested in related topic but that subscription is not triggered.  I am able to create incident at pipeline level when pipeline fails, using the notification tab by giving webhook details for incident table.Also, I want some information about job failure in incident table for e.g. Short description like job_name is ‘abc’ failed and their error details. Is this possible using subscription? If yes then please help me on the same.

6
XavierV
StreamSets Employee
4 months ago
A
Discovered Fame
AnkurDiscovered Fame
asked in StreamSets Academy

HTTP Client origin pagination config

Hi,If the field given in Result Field Path does not exist in the response, then we get an error, which is correct, but how to catch and handle that error to stop the pipeline smoothly.  

1
AkshayJadhav
StreamSets Employee
5 months ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy

Need more info on the two events for SFTP destination?

Hello,I am referring to this documentation : https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Destinations/SFTP.html#concept_sgt_m2m_xhbThere are two Event types - Uses one of the following types:file-closed - Generated when the destination closes a file. wholeFileProcessed - Generated when the destination completes streaming a whole file.If I have to add an executor to move the file from this destination then to some other location upon receiving an event, which event I have to use? Are there any more steps between wholeFileProcessed and file-closed? Is there anything more after the event file-closed is triggered? Dhanashri

2
D
Opening Band
5 months ago
A
Discovered Fame
AnkurDiscovered Fame
asked in StreamSets Academy

Metric Rule giving Alert on False Condition.

We have a Metric Alert in a JDBC Producer Input Counter with below Condition:${value() < 35}The above rule with “less than operator” is giving alert in both the conditions True or False. If we change the operator from “less than “< to “greater than”>. This it is behaving as expected.${value() > 35} Please provide the suggestion or resolution for ${value() < 35} to work as expected i.e. only on True condition.        

2
A
Discovered Fame
5 months ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to get data time format from mongodb to mysql .

the json data in mongodb had  a date format as a  timestamp in value  but during  process only date shown not timestamp show. how to fix below is my mongodb data:- but  inside of data updatedAt column has timestamp value but in streamsets not shown.how to get timestamp value from mongodb origin.

1
AkshayJadhav
StreamSets Employee
5 months ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy

Where can I find Transpose processor documentation?

Hello,I am looking for documentation for the transpose processor in transformer pipelines.I searched here : https://docs.streamsets.com/portal/platform-transformer/latest/search.html?searchQuery=transpose but did not find it.Can someone share the link to the relevant documentation?-Dhanashri 

10
Bikram
Headliner
6 months ago

Badge winners

Show all badges
Terms & Conditions