Product Updates

We’re excited to share what’s new with StreamSets! Check it out below.Enhanced Teradata Connection with New Destination Origin The new Teradata destination writes data to one or more tables in a Teradata database. Learn more here. New Notification Capabilities in Transformer for Snowflake The Snowflake Notification executor sends an email after the pipeline completes all writes. Learn more here.   Automatically Configure AWS Credentials In addition to manually configuring the AWS credentials in Control Hub to access and provision resources in an AWS account, StreamSets can automatically configure a cross-account role. Learn more here.  Improved Data Preview and Error Resilience in Transformer for Snowflake Easily preview, identify, and respond to errors at each stage in your transformation pipelines and generate views as your destination. Learn more here.  Programmatically Replace Stages Replace stages programmatically with the latest Python SDK enhancement. Learn more here.     

We’re thrilled to introduce two game-changing enhancements to StreamSets Data Collector (SDC):Support for Parquet Data: Unlock the Power of Efficiency In SDC 5.7, we’re unveiling Parquet as a data type across multiple destinations. While it’s in “Technical Preview,” it brings optimization to data storage, enabling future data analysis improvements. Benefits: Optimizes data storage for enhanced performance Sets the stage for advanced data analysis  Allows you to use Parquet in destinations like Local FS, Hadoop File System (HDFS), Databricks, AWS S3, ADLS2, Azure Blob Storage, Google Cloud Storage (GCS), BigQuery, and SnowflakeWhile Parquet is in “Technical Preview,” we encourage you to explore its potential and share your feedback. Your insights will help us fine-tune this feature for seamless production use. n MongoDB Atlas Lookup Processor: Simplifying Data Access If you are a MongoDB Atlas user, now you will be able to perform streamlined data lookup. Say goodbye to complex data retrieval scripts. Benefits: Simplifies MongoDB Atlas data lookup Compatible with on-premise MongoDB (older versions)  Eliminates the need for intricate data retrieval processesAt StreamSets, we’re committed to providing solutions that empower you to make the most of your data. Login to StreamSets now to explore these new features! 

What’s the latest StreamSets Python SDK 6.0 you ask? Well it’s a blend of StreamSets capabilities with Python’s simplicity. Designed for those aiming to build, integrate, and maintain custom solutions, the SDK offers a gateway to unlock untapped potential. 🙌Let’s dive into its new features: Migration Made Easy: Overcome data migration challenges across platforms. Previously, changing origins or destinations required users to work in the UI, manually adjusting pipelines - a tedious process. Now, a few code lines can programmatically adjust origins and destinations in numerous pipelines.  Combat Fragmentation: With the SDK, you can now modify or expand existing pipelines at scale, preventing the need to duplicate or create new assets and reducing the potential for error.  Enhanced Connection Control: Adjusting stage connections at scale was previously frustrating and time-consuming. The SDK now enables users to find connection patterns across pipelines and adjust them programmatically, bypassing manual, tedious alterations.  Smoother UI - SDK Transition: Transitioning between the UI and SDK can be challenging. The recent update introduces enhanced ‘stage.configuration’ attribute, making top-level UI fields more accessible with simplified property names, simplifying your workflow.Learn how to get started here! 🎉

Check out Data Engineer and StreamSets Developer Evangelist, Brenna Buuck, demonstrating how the StreamSets data integration platform has a consistent user experience, regardless of the pattern, engine, origin, or destination. Inconsistent data patterns can slow data teams down. With StreamSets, the user experience remains the same. Watch the video now

We are excited to announce our new 30-day trial of the full StreamSets Platform! 🎉 Anyone who is interested in trying out our cloud-native platform for data integration can sign up for the trial - no restrictions. If you’re interested in sharing this news with a colleague or want to learn more, check out our latest blog post.

Transform your raw data into a usable format, directly within the Snowflake ecosystem. By utilizing the transformative capabilities of StreamSets, you’ll no longer need to worry about manual hand coding and the time required for complex data transformations. Brush your shoulders off for unlocking the full potential of your Snowflake data and empowering your organization to make informed decisions.

StreamSets has added support to run Transformer for Spark on a new type of cluster: Amazon EMR (Elastic MapReduce) Serverless. Choose it or any of the other available supported cluster types. What is Amazon EMR Serverless?Amazon EMR Serverless is a feature of Amazon EMR that allows users to run big data processing workloads without having to provision or manage any compute resources. Which means you don’t have to know a lot about starting, stopping and managing clusters to get started. With Amazon EMR Serverless, users can focus on their data processing tasks without worrying about managing clusters or paying for idle resources.Amazon EMR Serverless is a flexible option for processing big data workloads, where users only pay for the processing time that their jobs require. Meaning, the clusters automatically spin down when not in use. It also easily scales up or down as needed to meet workload demands.The Benefits of Transformer for Spark + EMR ServerlessBy choosing an Amazon EMR Serverless cluster to run Transformer for Spark pipelines, data teams can benefit from ease of use and scalability. For example, with EMR serverless you don’t have to provision and manage your own spark clusters, which in turn gives users access to the scalability and power of spark without being experts in the underlying hardware. In addition,Amazon EMR Serverless can also easily scale up or down as needed to meet the demands of a workload. This can be especially important for data integration workloads, which can be highly variable in terms of data volume and processing requirements. As a cloud-native platform, StreamSets has scalability built-in and can adjust to clusters of almost any scale. Finally, users of Transformer for Spark and EMR Serverless can leverage both technologies to automatically spin down clusters when they are out of use or their associated pipelines finish.  Which means that data teams aren’t on the hook for potentially pricey idle time where no actual processing is being done. Scheduling pipeline jobs in StreamSets allows data teams to carefully consider, orchestrate, and monitor jobs to maximize this possible cost savings.StreamSets is continually improving by adding new features, supported platforms, origins and destinations. Please join us in the StreamSets community for updates, use cases and more.

 The advanced query language platform enhancement enables users to create filters on more components to find jobs and pipelines by either users or status tags. This is great for both collaboration and individual efficiency, since users can quickly find the they’ve work done in the past.  Learn More