I have multiple API’s to be invoked. Can I use the multiple clients within same pipeline and integrate all the result set into 1 csv file or generate multiple csv files within the same pipeline? How can this be accomplished?
If I read from an api and generate a csv file, how can I add the number of records read as a footer in the csv file?
I am trying to create a deployment in na01.hub.streamsets.com. I have downloaded docker desktop to run the deployment script. Can you guide me how to proceed with the installation process?
we are ingesting data from oracle and it has timestamp format as mm/dd/yyyy hhss AM/PM, while ingesting through streamsets its taking as yyyy-mm-dd hhss. how to change the timestamp format to source format which is mm/dd/yyyy hhss AM/PM in streamsets
Hi, for Kafka producer, I’m trying to pass both key and value as "avro". while selecting that option, kafka message key shows as "${avro:decode(record:attribute('avroKeySchema'),base64:decodeBytes(record:attribute('<messsage key attribute name')))}documentation says, don't change the first part and second part should be message key attribute.. trying to figure out what would the value for these 2 attributes, as I have tried many options and pipeline is failing and unable to write to kafka.error I receive is, CTRCMN_0100 - Error evaluating expression ${avro:decode(record:attribute('RetailPartnerCase'),base64:decodeBytes(record:attribute('accountmanagement_retailpartner_eps_case_domain_event_v1-key')))}: java.lang.NullPointerException so, trying to understand, what I can pass into those 2 attributes?
we are using CDC(streamsets) for oracle source and , redo logs are enabled from source end(oracle). wanted to know how to view those redo logs so that we get an idea like which record is getting INSERTED/UPDATED/DELETED and for which table.
I am installing Control Hub and when I run 01-iitdb.sh, I get error of DbUtils.DBType.POSTGRESSLF4J: Found binding in [jar:file:/opt/streamsets-dpm/app-lib/slf4j-log4j12-1.7. 7.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/streamsets-dpm/server-lib/slf4j-log4j12-1 .7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Error while running 'BuildSchemaCommand': java.lang.IllegalArgumentException: No enum constant com.streamsets.lib.util.DbUtils.DBType.POSTGRES----------------------------------------java.lang.IllegalArgumentException: No enum constant com.streamsets.lib.util.DbU tils.DBType.POSTGRES at java.lang.Enum.valueOf(Enum.java:238) at com.streamsets.lib.util.DbUtils$DBType.valueOf(DbUtils.java:180)
Hi there, Can I use pipeline:user() in auth ID property in START JOBS Stage (Origin or Processor) ?Example: ${pipeline:user() == "819339ee-e7bc-XXX-bdef-XXXXXXXXX145@XXXX" ? credential:get("aws","all","ch/dev/secret&key") : credential:get("aws","all","ch/prod/secret&key")} Validation is successful but the condition block always goes to retrieve “ch/prod/secret&key” even when the condition should evaluate to true.====Maybe I shouldn’t use the pipeline:user() function when I use a credential function as discussed in the documentation here,https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Configuration/CredentialStores.html#concept_bt1_bpj_r1b Snippet from the documentation,Important: When you use a credential function in a stage property, the function must be the only value defined in the property. For example, you cannot include another function or a literal value along with the credential function. Can someone confirm this ? Cheers,Sri
Hi, I am getting SNOWFLAKE_56 - Key fields not specified for table 'ORCL_EMP' errorWhen Table Auto create is enabled, CDC also enabled and given table key column as ID column
Hi I need your help here.I have a requirement to build a pipeline where I need to read data from one oracle DB and load the data into another oracle DB. Now for doing this everything I run my job, I should pick only those records from the source database which have been inserted or updated since the last run of my job. Please help me with your inputs. Regards,Ashish
I have run into a problem while working with timezones. It seems I cannot use the JDBC destination to dump ZONE_DATETIME fields into a PostgreSQL timestamptz column. The error received in StreamSets is a bit ambiguous, but the problem is clearly the ZONE_DATETIME typed fields as it works if I exchange them with DATETIME fields. Based on the name, I am assuming that ZONED_DATETIME are actually Java ZonedDateTime types. Some research into the PostgreSQL JDBC driver found that it does not support ZonedDateTime, but rather OffsetDateTime (Not sure why the jdbc driver cannot simply convert the ZonedDateTime to OffsetDateTime since they both describe points in time, but I digress). Do you have any suggestions for handling this situation?
Hi When i run my pileine I m still getting errors, please see below the screenshot. Can you pls let me know what i m doing wrong?
I want to pass custom header fields along with the JMS message to JMS producer.I just found the way to include headers which start with: jms.header.*That would be so far file but i don’t like that it takes over the jms.header. prefix in the header field.I want to be able to steer what name the parameter has as often fields are defined on enterprise withcertain meanings and these name will not include jms.header.*So what i expected was:jms.header.MyCustomHeaderFieldwill turn to:MyCustomHeaderField Propertybut it turned into:jms.header.MyCustomHeaderField
I have a requirement to write jobs logs to snowflake database table with jobs metrics data like job start time, end time, offset committed, no.of rows processed and errors if any etc.. Can you please help how to achieve this?
Hi TeamI have a requirement to read files using groovy evaluator and join 2 files using groovy-code.Can someone help me on syntax how to read files using groovy evaluator.?
How to join 2 files via javascript evaluator (or) groovy (or) jython.Can someone show simple usecase on this
Hi Team,I am not able to read and write flies in groovy scripter and groovy evaluator as opening local files in java/groovy when being run in a browser is forbidden. which processor allows me to read/write files? Can anyone help me through this please.
Dear Team, when i preview the pipeline its says validation errors SCRIPTING_08 - Script error while running init script: javax.script.ScriptException: NameError: name 'record' is not defined in <script> at line number 28Can someone assist me on these
Hi Team am using query executor stage and its generating the insert statement below way but when the single INSERT INTO ${SCHEMA_NAME}.${TABLE_NAME}(a,b,c) VALUES ('${record:value("/a")}', '${record:value("/b")}', '${record:value("/c")}') if records are getting single quotes the insert fails how to handle such case
Hi, We are trying to connect to Oracle UCM server using HTTP Client Plugin in Streamsets with SOAP API endpoint. Facing issue while connecting via http client. Please help to resolve the issue.error message :: 415 unsupported media typeAPI config Details:auth type is Basic authorizationMethod :: POSTurl :: WSDL detailtsRequest Body:: Soap Requestcontent type :: application/xmlauthentication :: Basic Authdata format ::Text/XML
Hi All, I am trying to run multiple queries in parallel using snowflake executor. I have set the property to run all the queries in parallel. but still on snowflake it is running one at a time. also increased connection pool size to 10. Thanks,Anupam
There is a function, str:splitKV which creates a map from a string, provided the key value pairs are encoded in the string, but how does one create a map from a group of individual fields? I suppose one could convert the values into a single string with the necessary key/value pairings and then use the str:splitKV function, but such a conversion seems excessive and might require extra type conversions.Is there a better way?
Hello new to Streamsets. my question : In one server can i install 2 DC by creating 2 Env’s . ? (DevEnvDC and UATEnvDC .) or 1 server 1 DC installation only? thanksJames.
I’m trying to deploy Streamsets in a AWS EKS cluster with several approaches, but none of them seem to be working.First I tried to deploy a control agent following the steps here: https://streamsets.com/blog/deploy-dataflow-pipelines-kubernetes-streamsets-control-agent/When I run the startup script for EKS with the required parameters I receive the following message:Running common-login.shFailed to authenticate with SCH :( If I debug the script, I can see the following error returned by the login service:{ "EXCEPTION": { "rawMessage": "java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'", "className": "java.lang.IllegalArgumentException", "message": "Invalid/unsupported hash version 'UNKNOWN'", "stackTrace": "java.lang.IllegalArgumentException: Invalid/unsupported hash version 'UNKNOWN'\n\tat com.streamsets.lib.security.http.PasswordHasher.getValueToHash(PasswordHasher.java:131)\n\tat com.streamsets.lib.security.http.PasswordHas
i have declared s3 data format as below but in UI/Control hub data format is showing blank.pipeline_builder = sch.get_pipeline_builder(engine_type='data_collector', engine_url=”XXXX”)s3_origin = pipeline_builder.add_stage('Amazon S3', type='origin')--s3_origin.data_format ='Text'.How to see values allowed for any particular component in SDK.(ex s3_origin.data_format ,s3_origin.delimiter etc..)
Already have an account? Login
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
