Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,025 Topics
  • 2,315 Replies
1025 Topics
mblahay
Discovered Fame
mblahayDiscovered Fame
asked in Community Articles and Got a Question?
How does the JDBC Lookup Cache work?

The JDBC Lookup documentation explains how to configure the cache and how the eviction works, but there is no discussion about how the cache actually works. Is the query being evaluated and then hashed, with the hash value being used as a key? or is something else happening?

1
Dimas Cabré i Chacón
StreamSets Employee
2 years ago
P
Fan
PriyaFan
asked in Community Articles and Got a Question?
Sample java script origin pipeline

 While i try to run any java script code (hello world) in java script scripting origin , i am facing below error. Also if any one had done sample java script pipeline please share  please could any one help me with this. 

1
Pradeep
StreamSets Employee
2 years ago
ravikumar
Fan
ravikumarFan
asked in Community Articles and Got a Question?
Protocols For Informatica Architecture

Before I can gain VPN access to run Informatica from home, I need to identify all the protocols Informatica uses for an ETL job to run. I found that it uses a native protocol for the repository, source, and target databases (so sqlnet for Oracle). I also know that I'll need to identify all protocols that could potentially be used to extract data from non-RDBMS sources (such as ftp).But, what protocol would be used to connect to the server to run the job? Are there any other protocols I'd need to be aware of that it might use as part of the whole process of opening up Designer, creating a job, and running it?

1
Giuseppe Mura
StreamSets Employee
2 years ago
C
Fan
cruaineFan
asked in Community Articles and Got a Question?
Why hasn't my Amazon EC2 deployment created a data collector engine?

I've followed the StreamSets documentation to create an AWS Environment and then an Amazon EC2 Deployment, for a single Data Collector engine. I activated the deployment and can view the provisioned EC2 instance within my AWS account. However, in Control Hub, when I navigate to ‘Engines in Deployment’, it has said ‘No Engines Found’, even when the deployment has been active for > 2.5 hrs. I’ve tried restarting the deployment but this didn’t help.Environment configAWS region: eu-west-2Credential Type: Cross-Account RoleDeployment configDeployment Type: Amazon EC2Engine for Deployment: Data Collector 5.1.0Engine Instances: tried 1, then 2EC2 Instance Type: c4.largeDeployment Event LogAug 1, 2022, 2:34:57 PM    INFO        Finished creating stackAug 1, 2022, 2:33:25 PM    INFO        Creating stackAug 1, 2022, 2:33:24 PM    INFO        Created SSM Parameter Token StreamSets-Deployment-Token-******************************************************************************Aug 1, 2022, 2:33:

1
C
Fan
2 years ago
sperandio
Fan
sperandioFan
asked in Community Articles and Got a Question?
Global variable to reduce memory usage and API calls

I have a pipeline migrating oracle db to salesforce which gets a couple of product attributes from salesforce for all  previously migrated products in salesforce. Each record in the pipeline is processing one product and needs to reference this map, which uses local caching to only call salesforce once to build the product map of everything (rather than call for every product being processed - which is tioo slow and uses too many API calls).With this approach, every record gets the map of all products. The approach works with small batch sizes of 50, but runs out of heap memory with batch sizes bigger than that. Is there a way for every record to access this large map of Prouct information loaded from salesforce without duplicating it inside of every record? 

1
Dimas Cabré i Chacón
StreamSets Employee
2 years ago
C
Roadie
collidRoadie
asked in Community Articles and Got a Question?
Custom Parameters within a pipeline

If I use a Jython script inside Data Collector to create a simple function which returns a “Last business day” of the previous month, is there anyway for me to then pass that result into a parameter that can be called by other nodes further down the pipeline?  Thanks

3
Pradeep
StreamSets Employee
2 years ago
A
Fan
AJ20Fan
asked in Community Articles and Got a Question?
Keep all fields creating duplicate records in http client origin

I have an http client origin which gives a json response. Pipeline uses pagination (by page number). When I am enabling ‘Keep all fields’ option in http client it is creates the duplicates of first record in every page. Say I have 10 records in the file, it writes first record 10 times in the same, 1st record in the second page 10 times and so on. basically it repeats first record for the entire page. Any way to fix this issue? We need to ‘Keep all fields’ enabled to get page properties for processing within the job. 

2
Bikram
Headliner
2 years ago
D
Discovered Fame
DolphinDiscovered Fame
asked in Community Articles and Got a Question?
How to get the exact failure reason for failed job/pipeline

Hi team,I define a pipeline with the processer called “Field Remover”, the job failed and can see all Input rows are shown in Error rows.  However when check the log(log severity is ‘ALL’), I cannot see any error or error reason in the log, I can only see an warning “Exception thrown while trying to remove temporary directory /opt/streamsets-datacollector/tmp/sdc1627444042246270257”, how can I get the exact failed reason?Some context of pipleline, it is to load some data from oracle origin to snowflake internal tables. and it can run well several days, but same workflow failed today, however if I remove the process “filed remover” now, I can see the pipeline run successfully. Any suggestions for this trouble shooting.

2
Bikram
Headliner
2 years ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles and Got a Question?
java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment spec

Kubernetes Deployment fails with the below errorAck Error for Event '64627d8f-ccc8-494e-bbd3-e9750f361a68:dpmsupport'. Reason : java.lang.IllegalStateException: ConfigMap is not a supported kind in the deployment specControl-agent (Provisioning agent ) cannot create config-maps or secrets. The only resources control agent can create are deployments, services, ingress and hpa.If you want to use these resources then you need to create this separately and call it in deployment manifest.     

0
Rishi
StreamSets Employee
2 years ago
P
Fan
PriyankaFan
asked in Community Articles and Got a Question?
Reading files through Groovy

Hi TeamI have a requirement to read files using groovy evaluator and join 2 files using groovy-code.Can someone help me on syntax how to read files using groovy evaluator.?

3
P
Fan
2 years ago
P
Fan
PriyankaFan
asked in Community Articles and Got a Question?
Reusing a component in streamsets

Hi Team,I want to reuse a shell-script across mutiple pipelines in data collector.How to acheive this Without using fragments concept.  

5
dima
StreamSets Employee
2 years ago
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
Using functions within an Auth ID property

Hi there, Can I use pipeline:user() in auth ID property in START JOBS Stage (Origin or Processor) ?Example: ${pipeline:user() == "819339ee-e7bc-XXX-bdef-XXXXXXXXX145@XXXX" ? credential:get("aws","all","ch/dev/secret&key") : credential:get("aws","all","ch/prod/secret&key")} Validation is successful but the condition block always goes to retrieve “ch/prod/secret&key” even when the condition should evaluate to true.====Maybe I shouldn’t use the pipeline:user() function when I use a credential function as discussed in the documentation here,https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Configuration/CredentialStores.html#concept_bt1_bpj_r1b Snippet from the documentation,Important: When you use a credential function in a stage property, the function must be the only value defined in the property. For example, you cannot include another function or a literal value along with the credential function. Can someone confirm this ? Cheers,Sri

3
Drew Kreiger
Rock star
2 years ago
Anupam Kushwah
Fan
Anupam KushwahFan
asked in Community Articles and Got a Question?
Unable to run snowflake multiple queries in Parallel using Snowflake Executor

Hi All, I am trying to run multiple queries in parallel using snowflake executor. I have set the property to run all the queries in parallel. but still on snowflake it is running one at a time. also increased connection pool size to 10. Thanks,Anupam

3
alex.sanchez
StreamSets Employee
2 years ago
D
Fan
DimpleFan
asked in Community Articles and Got a Question?
Mongodb origin error

Hi, I am trying to read from the mongo db. I have set as below in the scala code for the below pipeline:val df=spark.read.format("com.mongodb.spark.sql.DefaultSource").option( "spark.mongodb.input.uri", "mongodb://mu:mp@mh:port/md.cname?authSource=mad&readPreference=primary&ssl=false").load()where the abbreviations stand formad - mongodb.authentication-databasemu - mongodb.usernamemh - mongodb.hostmd - mongodb.databasecname - collection namemp - mongodb.password Error:TRANSFORMER_00 - TRANSFORMER_00 - Failed at operator: Operator: [Type: class com.streamsets.pipeline.spark.transform.scala.ScalaTransform id: Scala_01]: Class: Failed to find data source: com.mongodb.spark.sql.DefaultSource. Please find packages at http://spark.apache.org/third-party-projects.html was not found. This usually happens when running on an unsupported version of Spark, The offsets for failure occurred was 'Unknown' 

2
D
Fan
2 years ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles and Got a Question?
Use of Surrogate Key Generator(monotonically_increasing_id() in spark)

Comparing with relational databases, at times we want to have some id field in our dataset which is always increasing(usually for every record insertion). In general, this is useful for indexing, ordering, to distinguish each record or in simple words to maintain integrity of the database. This might not be the case with distributed systems. For Spark, data is mostly written to destination in partitions(that’s how we achieve parallelism). Since these partitions are written in parallel, we can not have consecutive id generation capability which is monotonically increasing. Spark has a function monotonically_increasing_id() which is helpful to generate increasing id at the partition level.Below is what Spark documentation has to say about it.A column that generates monotonically increasing 64-bit integers.The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record numb

0
Pradeep
StreamSets Employee
2 years ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
asked in Community Articles and Got a Question?
How to load or unload empty field as non nulls into snowflake destination

Empty values are loaded as NULLs when loading the data into a snowflake table. For example, I have an input field of varchar type and sometimes the data in it is empty(‘’). I would want the input data ‘’ to be loaded into snowflake same as ‘’ and not NULL. This will be helpful in cases where I have defined the destination table field as NON NULL.

1
Pradeep
StreamSets Employee
2 years ago
G
Fan
gowrinadhFan
asked in Community Articles and Got a Question?
Passing null values in the JDBC Query

Hi,We are using JDBC Query to insert data into the sql server database. using the following kind of query:INSERT INTO ${SCHEMA_NAME}.${TABLE_NAME}(a,b,c) VALUES ('${record:value("/a")}', '${record:value("/b")}',  '${record:value("/c")}')As we are enclosing in the single quotes even for null values it is inserting empty strings. if I remove the quotes getting exception by saying invalid query as the String are not enclosed within the  quotes.So looking for a check which adds the quotes if the values is present otherwise to return the null.Tried ${record:value("/a") != NULL ?record:value("/a") : NULL} but query failing.

4
Pradeep
StreamSets Employee
2 years ago
V
Fan
vasanthiFan
asked in Community Articles and Got a Question?
Need to join datasets using kafka and join them

Hi team,how to join datasets using kafka and how to join them?Please help me this one. 

5
Rishi
StreamSets Employee
2 years ago
A
Discovered Fame
akanshajain6793Discovered Fame
asked in Community Articles and Got a Question?
How to handle single quotes in between data

Hi Team am using query executor stage and its generating the insert statement below way but when the single  INSERT INTO ${SCHEMA_NAME}.${TABLE_NAME}(a,b,c) VALUES ('${record:value("/a")}', '${record:value("/b")}', '${record:value("/c")}') if records are getting single quotes the insert fails how to handle such case

3
saleempothiwala
Headliner
2 years ago
Anirudh
Fan
AnirudhFan
asked in Community Articles and Got a Question?
jython script gave an error

Dear Team, when i preview the pipeline its says validation errors  SCRIPTING_08 - Script error while running init script: javax.script.ScriptException: NameError: name 'record' is not defined in <script> at line number 28Can someone assist me on these

3
Bikram
Headliner
2 years ago
suresh_sukumaran
Fan
suresh_sukumaranFan
asked in Community Articles and Got a Question?
Facing issues while trying to pull data from soapAPI.

 Hi,  We are trying  to connect to Oracle UCM server  using HTTP Client Plugin in Streamsets with  SOAP API endpoint. Facing issue while connecting via http client. Please help to resolve the issue.error message :: 415 unsupported media typeAPI config Details:auth type is Basic authorizationMethod :: POSTurl :: WSDL detailtsRequest Body:: Soap Requestcontent type :: application/xmlauthentication :: Basic Authdata format     ::Text/XML

3
Bikram
Headliner
2 years ago
A
Fan
Anusha SunkavalliFan
posted in Community Articles and Got a Question?
Http Client connection to Oracle UCM using SOAP API

Hi, We are trying to retrieve data from Oracle UCM server using Http client as source, the configuration we use is POSTMAN tool we are are trying to replicate in http client by we are facing below issues. Steps Followed.Http client is source component WSDL details are provided in reource url path HTTP method is POST SOAP request is provided in Request body. Soap Action details are provided header tab. Data Format is XML                                                                                                              Error message is "HTTP_01 - Error fetching resource. HTTP-Status: 415 Reason: Unsupported Media Type :"Steps followed in Postman ::Method :: POST WSDL is passed as API end point. Authorization is set to Basic. SOAP request is given in body.Request is successful in POSTMAN whereas in HTTP client the same configuration is throwing “HTTP-Status: 415 Reason: Unsupported Media Type”.  

0
A
Fan
2 years ago
P
Fan
PriyaFan
asked in Community Articles and Got a Question?
How to read and write files in Groovy/java ?

Hi Team,I am not able to read and write flies in groovy scripter and groovy evaluator as opening local files in java/groovy when being run in a browser is forbidden. which processor allows me to read/write files? Can anyone help me through this please.

3
wilsonshamim
StreamSets Employee
2 years ago
P
Fan
PaawanFan
asked in Community Articles and Got a Question?
Need help with getting data from API and writing it in Parquet format.

Hi, I am trying to get some data from a SOAP API, I get the data in the from of an XML response. The requirement is to convert the data into Parquet format and store it on an ADLS GEN 2 storage.As far as I understand, I can use the data collector to write files in AVRO and then convert the AVRO to Parquet using a whole file transformer.I know that transformer can write in parquet directly, so is there any way for me to skip the intermediate AVRO file creation? 

2
suresh_sukumaran
Fan
2 years ago
S
Fan
sevtiandy
asked in Community Articles and Got a Question?
How to generate nested JSON

Hi,I want to generate multiple nested json array from multiple tables (in this case from table HEADER and table DETAIL) and I want DETAIL’s records to fetch under 1 json array objects like this example:[{ "ID_HEADER": "header01", "DETAIL": [{ "ID_DETAIL": "detail1", "VALUE": 21521, "UNIT": EA }, { "ID_DETAIL": "detail2", "VALUE": 24, "UNIT": EA }, { "ID_DETAIL": "detail3", "VALUE": 25135, "UNIT": EA }, { "ID_DETAIL": "detail4", "VALUE": 135234, "UNIT": EA } ] }, { "ID_HEADER": "header02", "DETAIL": [{ "ID_DETAIL": "detail1", "VALUE": 23, "UNIT": EA }, { "ID_DETAIL": "detail2", "VALUE": 353, "UNIT": EA } ] }] 

2
S
Fan
2 years ago

Badge winners

Show all badges
Terms & Conditions