StreamSets Platform

Environment: Streamsets Data Collector All version. Oracle Java 1.8u351.Issue:It has been observed that data collector and Kafka pipeline[with SASL protocol] starts failing after Java upgrade to 1.8u351 version.This error appears on data collector logs:abnormal exit: java.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginException: Unable to obtain password from userCheck STDERR for more detailsjava.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginException: Unable to obtain password from user at com.streamsets.datacollector.security.SecurityContext.login(SecurityContext.java:160) at com.streamsets.datacollector.main.Main.doMain(Main.java:94) at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:41)Kafka pipeline errors:GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: KDC has no support for encryption type (14))]) occurred wh

I am using kafka consumer as an origin and the event is expected at particular time. I want to send the notification if the event is not published at the expected time. How can this be achieved in streamsets? 

I have a SQL Script with multiple drop & create DDL(Create tables As Select *), I want to run them at one go. I am quite new to informatica powercenter, can some one provide the process of using SQL transformation for BigQuery in informatica.Sample Query:- informatica transformation

Hi All,Is there any way to get the response body for other http code in HTTP client processor? The API (POST method) I am calling, it gives error reason as response for error records with http code  400 in postman. In streamsets I want to get the error reason as response but getting 400 and record is going in error files.Thanks in advance for your help. Regards,Anjalee

How to get the data collector 3.22.3 version's activation key ?

Our requirement is to collect all records and do some processing. That we have done using JavaScript array. We want to write that array to the output processor post completion of all batch. sdc.output.write() function is not working in the destroy block. Is there anyway to achieve the above requirement? Is there any way to hold writing output and only do output once all batches are finished.

I have input coming from source as records. Record-1 is xyz, record 2 is abc…I need to merge all these values into a list by checking few conditions in groovy.How do I achieve that. Finally output should come in groovy when reading all records is over.like [xyz,abc,pqr] But I am getting code output asOutput for record-1 [xyz]output for record2 -[abc] and so onCan someone please help me how its possible?

How we can capture AWS S3 Events in streamsets on cloud,which service we can use here? 

Hello,I am going through the troubleshooting training material on the StreamSets Academy website. Course link: https://academy.streamsets.com/courses/troubleshooting-and-support/lessons/generating-a-support-bundle-in-the-streamsets-cloud-control-hub/Unable to locate this UI (in the screenshot) shown in the training video at 1:00 onwards till 3:11. Is it not accessible during training?  

Hi All, I have created the installation script and tried running the docker image in strigo environment. However, it instantly throws the below error. docker: Got permission denied while trying to connect to the docker daemon socket  Could someone please let me know what is that I am doing it wrongly? Any remedy? Thank you!Yuvaraj Thiyagarajan

My JDBC producer stage is giving the error java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils Has anyone had a similar issue before?

I want my pipeline to automatically trigger only when the source file is placed on the server. How can we achieve this in streamsets?

Hi Team,As, I am new to Streamsets want to have more information about Rest Service.We have databases on on-prem, We run procedure on database that initiate the rest API call to tomcat(service published on this) web server on port 8908 redirectedNow we receive one url which contain different parameter. I need to configure the same in Streamsets.Can you please let me know, How we can configure same in rest service.  

Hi team,I am using field renamer for mssql column name rename from origin mssql “regionid” to target snowflake “RegionID”.Attchment 1 is my configure for Field Rename configuration. When I ran “previewer”, did not show any errors.Attachment 2 is log.Cannot figured out what is the exact reason for the failure from the log.Can you suggest?

Hello,I am going through the Data Ops Platform Fundamentals course, and currently on Lab: Deploy a Transformer Engine with a Sample Pipeline. I have made sure my deployment is up and running, transformer engine is up. Created a pipeline by using sample pipeline as per lab steps. I am getting these validation errors (attached) - Error: Directory path does not exist: file:///data/store_retail Am I missing any steps? Some more info:I have not changed any parameter settings.  The runtime env for the pipeline runtime conf - It shows origin_directory=/data/store_retailThe origin shows: Directory Path = file://${origin_directory}The transformer engine is running. It does not show any directory /data/store_retailbash-5.0# hostname46ab0f17ee4fbash-5.0# ls /data/transformerbash-5.0# ls /data/transformer/blobstore                      meteringSnapshot.json          pipelines                      runInfo                        stats.jsoncontrol-hub-pushed.properties  opt-stats.json                

Hi, community,  I am using Streamset data collector for data migration from PostgreSQL to Cassandra But while testing the Cassandra connection I am encountering the above error. 

How to initiate the emr cluster through streamset

When checked at path Computer Configuration -> Administrative Templates -> Windows Components -> Windows Update, I don't see any detail for WSUS. Please tell what's the other way to check WSUS server detailsCyberArk Vault 

Has anybody used GraphQL as data source? I’ve been tried to connect it using StreamSets, but, no luck! 

I’m looking to make a pipeline fragment where the query to be run will be passed in as a pipeline fragment parameter.It looks like it is not possible specify the SQL Query section as something like ${query} since these values get sanitized. You can only do something like SELECT * FROM TABLE WHERE ID=${id_param}. Is it possible to set a value in an expression evaluator like/query=”SELECT * FROM DUAL”and have it run by referencing the /query variable on the record?

Issue:When using the StreamSets Test Framework and trying to run a test it fails with the following error:$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py 2019-01-28 09:03:16 AM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ... INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ... ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0 benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /root/tests, inifile: plugins: benchmark-3.1.1 collected 7 itemsstage/test_dev_stages.py 2019-01-28 09:03:27 AM [INFO] [streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ... 2019-01-28 09:05:41 AM [ERROR] [streamsets.testframework.sdc] Error reaching Data Col

Hi Team,while logging to stream sets i am facing error with this email id himabindu_nukaraju@persistent.comi need to login my course. Please help on this 

In the build jdbc pipeline the step where we need to upload the driver in the external libraries after selecting the mysql-connector-java-8.0.26 when I am clicking on upload a red line in appearing as error but there nothing mentioned of which error it is.The same thing is happening while importing a pipeline, when I am browsing the zip file and clicking on the import button the same error as mentioned above is appearing. 

Hello Team,I need to update one job config which is “globalMaxRetries” using rest call.I refer Restful section in Control Hub but that Rest call for “updateJob” requires many other configs also needs to mention in Request Body. could you please assist me with this ?

HI Team, Please help me to find the week number from given date.how to use weeknum functions and lookup’s ThanksMadhavan P