30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets
Hello, I'm utilizing Control Hub 3. x to recreate the customer's environment. Based on the pipeline name (given as a parameter) and the number of days old, I need to clear up old tasks. I made the pipeline shown below, which is also attached.
Hi, I am using the HTTP client origin, and since I need to load a lot of records, I need to use pagination.my url ends with ‘auditLogs?offset=${startAt}&pagesize=5&isascorder=false&order=id’ cannot give more for confidentiallity reasons. This is a post requestwhen I set pagination to offset or page number, all I get is the first 5 records, offset is actually used to denote the start record of the data returned, not a page in this instance.If I set pagesize to 1, which I think is how I need to do this, I only get 1 result it doesn’t seem to continue, I have tried with both page and offset set as the pagination method.any ideas on what is wrong are welcomed
I want to validate the address column by using google address api how to do that .this is my record : I want to validate the address .how to do any one can help this scenario.
Hi,I need to use a calculated date in a pipeline, more specifically I need to make an HTTP call, where a date range of the form of a ‘From’ and ‘To’ date, of the form “2023-09-10”.I can calculate this date as a field in a record using ${time:extractStringFromDate(time:dateAddition("DAY",-1,time:now()),"yyyy-MM-dd")} But this does not work in the ‘required data’, what would normally be the request body in a post requestIt seems I can use a parameter in this section as follows {"entity":"user","operation":"Login","userName":null,"fromDate":${FDate},"toDate":${TDate},"toDateVal":"2022-07-21T00:00:00.000Z","fromDateVal":"2022-01-01T00:00:00.000Z"}but when I try to create a parameter that calculates the date, using the above formula, I get an EL expression error Can someone suggest where I am going wrong?I have also tried using start jobs, to pass the dates as a parameter, but that also doesn’t seem to work, I can pass a manually entered date to this ‘child job’ but it seems to conv
I want to create a CDC pipeline for DB2 Database, as there is no origin present for the implementation,Is there any alternative to perform CDC for this Database?
Problem statement: If you are doing DR testing or one of your Kafka broker is not available, you might observe an error message in the job log as below. Error getting metadata for topic.error: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadataExplanation: Let's say you have 3 brokers in the bootstrap list and first broker is went down during DR or some other reason, if request made from the client to the first broker is timing out then pipeline will not make a retry to the next available broker , instead it will fail with an above “timeout” error. In ideal scenario client should traverse through all the brokers in the list before marking it as fail. server1:9092,server2:9092,server3:9092This could happen in the kafka client lib 2.6 or below versions. More details can be found in KIP-601 article. Solution: Upgrade kafka client lib to 2.7 or above and tune socket timeouts accordingly. . In this version they have introduced below two configura
i have here an sftp origin that i want to pull the files from do some formatting on it and then push it as a bulk post request to HTTP client destination.i keep getting this error 504 gateway timeout with the files with size 200KB or bigger.it is posting the data on the system but returns this error. i tried setting the read timeout configuration and maximum time out values to a much higher values like 15 mins or so in the HTTP client destination but i keep getting the same error though the data is posted in the destination.what should i do to prevent this error?
this is my datetime values comes from xlsx file and i want to convert into proper datetime VENDOR_CREATION_DATE COLUMN AS A decimal data type --->want to change datetime I used field type convertor :-to achieve this but its not working ,any one can provide the configuration
I am trying to create a master flow that calls another pipeline but am having issues in that when I pass parameters, they seem to be converted to values, or at least they lose the quote marks { "TDate": "2023-09-17", "FDate": "2023-09-14"}In the status window of the job called, they show up as 2023-09-17 and 2023-09-14, instead of “2023-09-17” and “2023-09-14”, and also update the job parameters without the quotes If I try and calculate the dates { "TDate": "${record:value('/TDateY')}", "FDate": "${record:value('/FDateY')}"}I get the following errorSTART_JOB_05 - Failed to parse runtime parameters for job ID: START_JOB_05 - Failed to parse runtime parameters for job ID: 6f86b611-bfbe-4882-a47f-3aa41f0daeab:0f246ae0-4677-11ec-a298-93161f808487, error: com.fasterxml.jackson.core.JsonParseException: Unexpected character (' ' (code 160)): was expecting double-quote to start field name at [Source: (String)"{ "TDate": "2023-09-17", "FDate": "2023-09-17" }";
Problem statement: If you have enabled security manager ( In Java 8 , security manager is enabled by default ) and using non-default external resources directory, then you might observe errors in the Job while loading external libraries. This is a known issue in the Platform August release. Example error: Provider com.mysql.cj.jdbc.Driver could not be instantiated Affected Versions: August, 2023 Platform release Solution: Add the external resource path to the security policy in the “Advanced Configuration” dialog of the deployment. Example: For SDC, you should add below policy: grant codebase "file://${sdc.dist.dir}/externalResources/-" { permission java.security.AllPermission; }:And for transformer, the equivalent is:grant codebase "file://${transformer.dist.dir)/externalResources/-" { permission java.security.AllPermission; };
I would like to download StreamSets Full RPM package of version 3.14.0, but I am unable to do so through the official website. Could someone please send me a copy? I would be very grateful. My email address is 1028259318@qq.com. Thank you in advance.
I have a bunch of excel (xls) files that are hosted online (HTTPS) and periodically updated. I want to be able to store the excel file to ADLS Gen2. What is the best way to approach this? Data SourceNo authentication/authorization needed. HTTPS XLS formatEnvironment StackStreamsets Azure Shop
I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all fields that contains blank values means feed with NULL . any other processor can replace these scenario
Requirement: I want to create an upgraded version of new deployment from an old deployment using SDK. The 2 deployments may have different set of stage libraries depending on their engine versions. So, I want to fetch list of all libraries for each version, compare the old deployment’s libraries with that of the obtained repository comprising upgraded version’s libraries, to check for any deprecated/upgraded versions, and then create the new deployment with the updated stage library list. I want to access all possible stage libraries (common + enterprise) of a particular engine version using SDK. Is there an API to fetch this?
Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached) When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri
can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration after execution of my pipeline :-out put file like - But my requirement is like how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.
I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this. thanks
Hi Team,I have scenario where I am reading delimited file from S3, in next step I need to compare header of this file. I am unable to get header as new field along with other attributes. @Bikram @Ranjith P @Rishi Any idea on this ? TIA
Hi Team,I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC. Is there any workaround we have in SDC. Below are my datasets,Dataset 1 - CSVDataset 2 - Snowflake table Any way to join Dataset 1 and Dataset 2 in SDC. TIA@Rishi @Bikram @Maria Vila @XavierV @AkshayJadhav
I want to evaluate the job metrics field of orchestrator tasks JSON, which is the output of start job origin in orchestration pipeline. The start job origin contains pipeline which moves 5 records from csv file from directory to snowflake table.I need to confirm from the job metrics field that the pipeline output count equals that of input count.But when I preview the orchestration pipeline, both the input and output records count show 0.Is there any additional configuration required that I have missed? I have attached the preview SS along with the job instance pipeline.
Am trying to listen some of the port from tcp server origin but its not giving any results, the same port am when using in linux console which we deploy streamsets server am able to listen the message currently am using Version:StreamSets Data Collector 3.19.1 in log am getting TCP_00 - Cannot bind to port [0.0.0.0/0.0.0.0:******]: java.net.BindException: Address already in usehere am hiding port number for confidential with *Could you please help me how to listen messages from streamsets tcp server origin. Thanks in advance
I would like to download the StreamSets software package, but when I accesshttps://archives.streamsets.com/index.html, it shows Error 405 Not allowed. How can I resolve this issue?
Hello, I had a post yesterday: Python SDK - How to detect new commits. | StreamSets CommunityHowever, I cannot find how to reply with another question to the response I got so I will post it here in a new question: Just to clarify what I am asking; I would like to see all new commits across all pipelines. So, say in the last day, 2 existing pipelines have new commits, and one new pipeline is created. I would like to be able to see a log that reports there 3 pipelines have had new commits made, and what their ID /commit_id is. Would the SDK happen to support some sort of logging mechanism like this? I was not able to find it in the docs.Thank you,-Nate
I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?
Hello, I have been looking through the SDK docs looking for a functionality that would let me see all new pipeline commits made in a certain time frame. I wasn’t having much luck finding anything and I was wondering if there are any functions that support this. Thank you, -Nate
Already have an account? Login
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
