Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,026 Topics
  • 2,316 Replies
1026 Topics
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?
Cannot Rename Files in SFTP dst Server

Hello , I have a simple pipeline where i want some csv files to be moved from s3 bucket to SFTP server,However the files saved in the SFTP server should have different naming that includes the current date, something like:“mer_YYYYMMDDHHmmss.csv”i tried in the file name expression field in the sftp server destination to come up with expression evaluating the above but it always produces errors that functions like YYYY() is not a function.

6
N
Opening Band
2 years ago
P
Opening Band
pedromanuelOpening Band
asked in Community Articles and Got a Question?
Exception when importing fragment in Control Hub

Hi, When trying to import a fragment I get this error:“The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. “Unfortunately I have no idea as to where I can check this exception. I guess it happens on the Control Hub and there would be no way to access the log file?Any ideas?  

2
P
Opening Band
2 years ago
Priyabala
Roadie
PriyabalaRoadie
asked in Community Articles and Got a Question?
Same data in two fields

How to populate the same field value in two different fields of target?src-->csvfield1-->tgt1field1-->tgt2

1
antmcmullen
StreamSets Employee
2 years ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
Groovy evaluator

How do I write else if loop in groovy evaluator Example: the if-else loop in expression evaluator is written in below way: ${record:value('/phone_home[0]') != ' ' ? record:value('/phone_home[0]'):(record:value('/phone_home[1]') != ' ' ? record:value('/phone_home[1]'):(record:value('/phone_home[2]') != ' ' ? record:value('/phone_home[2]'): ' '))} How do I implement Same-Example thing in groovy evaluator,Since there are dynamic number of rows coming each time, [sometimes number is 10, sometimes 100 etc, so need to write for-loop]

6
P
Discovered Fame
2 years ago
Priyabala
Roadie
PriyabalaRoadie
asked in Community Articles and Got a Question?
Multiple Header lines

Can we populate multiple header lines(3) with different header names using data collector?I am using csv as source, which will contains one header.Need to convert csv to XLS, and then the xls should have 3 headers which ae not present in source.3 headers followed by data from the source.

2
Priyabala
Roadie
2 years ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
json to list

How to convert json array into list 

1
Pradeep
StreamSets Employee
2 years ago
P
Opening Band
pedromanuelOpening Band
asked in Community Articles and Got a Question?
Start Pipline (orchestration) origin/processor not available in Data Collector?

Hi, while getting familiar with orchestration stages found that the “Start Pipeline” origin/processor are not available at Data Collector engines (according to https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html#OrchestratorStagesActually they not seem to be available anywhere? Where are they actually available if not at the collector engine? Is this a documentation issue?

2
P
Opening Band
2 years ago
P
Opening Band
pedromanuelOpening Band
asked in Community Articles and Got a Question?
Creating a job instance from template without immediatly starting it?

Hi, just wondering, shouldn’t it be possible to create a job instance from a job template without having it started instantly? Make it a choice to either start it right away or leave the start to a later time?

2
P
Opening Band
2 years ago
P
Opening Band
pedromanuelOpening Band
asked in Community Articles and Got a Question?
Authentication issue with Control Hub API credentials

Hi, when trying to run a pipeline with the “Start Job” origin the pipeline fails and the log shows:com.streamsets.pipeline.lib.ControlHubApiError: Control Hub security token expired or invalid: {"ISSUES":[{"code":"SSO_01","message":"User not authenticated"}]} The credentials I used are newly generated credentials and the stage is configured as can be seen in the screen shot with the credential id and token. Why are these credentials rejected by the Control hub?  If I use the exact same credentials in a call to CURL I get a perfectly fine response showing my user details(request like: curl -X GET https://na01.hub.streamsets.com/security/rest/v1/currentUser -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $cred_id" -H "X-SS-App-Auth-Token: $cred_token" -i)Any ideas? 

2
P
Opening Band
2 years ago
K
Fan
karthikFan
asked in Community Articles and Got a Question?
Aggregation Chart not generated

Hi,I have a pipeline which is using windowing aggregator to perform couple of window aggregations. When I run the pipeline, I don’t see the chart generated but instead I see the below. Am I missing something?Regards,Karthik

1
S
StreamSets Employee
2 years ago
D
Discovered Fame
DolphinDiscovered Fame
asked in Community Articles and Got a Question?
Load Date column from Oracle to Snowflake Failure

Hi team we built a pipeline to load oracle table to Snowflake using StreamSets, there is one date column, and in oracle the value is always like "2012-04-25 00:00:00.000", we create this column as date in snowflake too, however it failed the load, only after I changed the data type of Snowflake corresponding column as TIMESTAMP_NTZ(9) and then it can be loaded.So how can I convert this column to Date in StreamSets when loading, which processor I can use and what is the evaluation of this convert? by the way what is the root cause of failure for this direct date to date column loading from oracle to snowflake with StreamSets?

2
D
Discovered Fame
2 years ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
Converting [ to {

In a json file, I need to replace [  with{ in between json file, How to achieve that using exp evaluator Example: { "student": [ { "id":"01", "name": "Tom", "lastname": "Price" }, { "id":"02", "name": "Nick", "lastname": "Thameson" } ]} to{ "student": { { "id":"01", "name": "Tom", "lastname": "Price" }, { "id":"02", "name": "Nick", "lastname": "Thameson" } }}

1
saleempothiwala
Headliner
2 years ago
D
Discovered Fame
DolphinDiscovered Fame
asked in Community Articles and Got a Question?
DC Pipeline Validation pass but Failed for draft run, log no clear message for rootcause of failure

Hi Team, I am building a DC pipeline to load a small table data from Oracle origin to Snowflake, validation pass but failed for draft run (I click “Reset Origin and Start”), I searched the log, there is only warningWARN Exception thrown while trying to remove temporary directory /opt/streamsets-datacollector/tmp/sdc148432276299548840 ”can you please guide hwo to debug it? Thank you.

4
D
Discovered Fame
2 years ago
A
Discovered Fame
ana00159Discovered Fame
asked in Community Articles and Got a Question?
Reverse pivot

How to do reverse pivot in streamsets? My rows are as follows.ABC,X,,,100ABC,,Y,,100ABC,,,Z,100I want it as one rowABC,X,Y,Z,100 How this can be done?     

0
A
Discovered Fame
2 years ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
Date-issue

Hi Team Can someone help me how to extract only date from  date-timestamp coming from source expression ?

2
saleempothiwala
Headliner
2 years ago
Priyabala
Roadie
PriyabalaRoadie
asked in Community Articles and Got a Question?
Date Conversions in Data Collector

Getting date (YYYY-MM-DD) in a string field from source, need to convert into date type of format(DD/MM/YYYY).Tried Fieldtype converter, but of no use.Any suggestions?

1
antmcmullen
StreamSets Employee
2 years ago
R
Fan
ramakrishna_raoFan
asked in Community Articles and Got a Question?
Orchestration records schema for start jobs

Can you please share the schema for orchestration record for start jobs, I am not seeing job name in the schema which is required for triggering the downstream jobs. All I can see is the attached image of orch record schema.

0
R
Fan
2 years ago
deepika
Fan
deepikaFan
posted in Community Articles and Got a Question?
DevOps Vs SRE

DevOps and site reliability engineering (SRE) are two approaches that enhance the product release cycle through enhanced collaboration, automation, and monitoring. Both approaches utilize automation and collaboration to help teams build resilient and reliable software—but there are fundamental differences in what these approaches offer and how they operate. About HKR TrainingHKR Trainings excel at providing you the best online classes with high quality facilities at a low price without any compromise on quality. What can you expect from us? A dedicated learning platform with 24*7 support, best in class training materials to help you learn advanced techniques and practical knowledge of all IT Technologies. Our courses are specifically curated for both professionals as well as job-seekers. Online classes conducted by the best knowledgeable and certified trainers helps you earn certification at your convenience. Key featuresFlexible TimingsHands On Experience24/7 SupportCertified & In

0
deepika
Fan
2 years ago
P
Fan
PeteFan
asked in Community Articles and Got a Question?
Error when inserting list type to Snowflake

Hello, I have a response from a HTTP client stage that has been pivoted. Some of the fields are lists and these return errors when trying to load into Snowflake.Ideally I’m trying to get the lists (array) inserted as JSON in variant columns for flattening later.I can use a JSON parser however I need to do that for every field and there are some cases where the fields are dynamic.   

2
P
Fan
2 years ago
ravikumar
Fan
ravikumarFan
asked in Community Articles and Got a Question?
How to perform a cyberarkpassword lookup for all hosts in an inventory group and write the keys out to individual pem files?

Ansible version 2.7.9I'm writing an ansible playbook to deploy an piece of software to a linux environment. SSH access to these systems is protected by a CPM (Cyberark), used as an ssh key manager.I've got most of the logic figured out, save for one piece. The playbook needs to loop through hosts in an inventory group, lookup the ssh private key in Cyberark for each host and then use each key to ssh into each host in the inventory group to install the software. I'm struggling with how to make that work in ansible.I've read through the add_host and cyberarkpassword documentation, as well about 4 hours worth of searching stackoverflow and blogs, and couldn't find a single example even close to what I'm trying to do.As far as how I think it should work:Using the cyberarkpassword lookup module, loop through hosts in inventory group specified by {{ env }}. Value for this will be passed in through --extra-args. Retrieve the ssh private key for each host. Register the output from the lookup,

0
ravikumar
Fan
2 years ago
V
Fan
vasanthiFan
asked in Community Articles and Got a Question?
How to generate row numbers

Hi team,Need to create rank/row number function for each row just like how we do in SQL.How to implement using expression evaluator/groovy etc.can anyone  please help me with this .

1
Pradeep
StreamSets Employee
2 years ago
V
Fan
vasanthiFan
asked in Community Articles and Got a Question?
ksqlDB

can we use ksqlDB in streamset?

1
alex.sanchez
StreamSets Employee
2 years ago
S
Fan
SubhaFan
asked in Community Articles and Got a Question?
How to send Form-data in http client for POST Call

I am new to Stremsets.I am looking to perform a POST call using HTTP client and the POST Body is a form-data with file upload . We are able to achieve this in POSTMAN. Is there a way we can do this in streamsets data controller pipeline.

0
S
Fan
2 years ago
P
Opening Band
pedromanuelOpening Band
asked in Community Articles and Got a Question?
runtime.conf.location=<filename> vs expected location of the file

Hi, running data collector 5.1 on Docker under Windows there seems to be a mismatch between the locations of where I can upload a resource file to (Engine => External Resources => Upload) and the location where the collector engine looks for the resource file.Uploading stores the file under /resources/<filename>But when I restart the engine I get a java.io.FileNotFoundException which shows that the collector expects the file in /etc/sdc/<filename>:java.io.FileNotFoundException: /etc/sdc/runtime.properties For more clarification, I followed the steps as outlined here:https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Pipeline_Configuration/RuntimeValues.html#concept_fjx_g31_1s  What do I miss here? Is there a way to configure the resources directory?Thank you!

2
P
Opening Band
2 years ago
Vikki
Fan
VikkiFan
asked in Community Articles and Got a Question?
REST API

HI team,    I am having 6 API’s where I have to build 3 pipelines for each i.e., API-KAFKA, KAFKA-FILESYSTEM and FS-Target table. Like this  For 6 API’s I have to build and execute 18 pipelines (6*3). Could anyone suggest me  that is there any approach which reduces the number of pipelines ( to push all 6 APIs data into 1 Kafka topic in 1st pipeline, after that I will generate data file from Kafka to FS and FS to target table).FYI- my target table is same for all APIs.

6
Vikki
Fan
2 years ago

Badge winners

Show all badges
Terms & Conditions