Confluent debuts ‘point-and-click’ canvas for building streaming data pipelines

California-based Confluent, which offers an Apache Kafka-based data streaming platform, today announced Stream Designer, a visual interface to help developers build, test and deploy real-time data pipelines in a matter of minutes. The company showcased the tool at its ongoing Current 2022 conference.

In the last few years, real-time data has become crucial to business growth and success. Batch data is still in widespread use, but it fails to answer many challenges and use cases that enterprises encounter in their routine operations. As a result, streaming data technologies that were once at the edges are witnessing a surge in adoption, including Apache Kafka.

Over 80% of Fortune 100 companies use Kafka to handle large volumes of data in real-time. However, when it comes to building streaming data pipelines on open-source Kafka, teams have to bring in highly specialized engineering talent and deal with long development timelines spread across multiple tools. This puts pervasive data streaming out of reach for many organizations.

Enter Stream Designer

Confluent’s new Stream Designer, which is now generally available, offers a point-and-click visual canvas to describe data flows and business logic within the Confluent Cloud UI. This way, instead of dealing with the headache of managing individual components on Kafka (each requiring their own boilerplate code), developers can build pipelines with the complete Kafka ecosystem accessible in one visual interface. They can also iterate and test before deploying into production in a modular fashion.

“This is the industry’s first visual interface for building streaming pipelines natively on Kafka. While there are other GUI-based stream processors out there, they are powered by other, often closed and proprietary, messaging technologies. Stream Designer is built on Kafka, which is widely known as the standard for data streaming. It is the developers’ platform of choice because of its highly scalable and resilient architecture as well as strong open source following,” Jon Fancey, principal product manager at Confluent, told VentureBeat.

Fancey stated that they held a closed beta program for Stream Designer with select customers and witnessed “very positive” feedback.

“It cut down the ramp time for developers new to Kafka and boosted productivity for more experienced Kafka developers. With a radically easier way to build, test, and deploy streaming data pipelines, they were able to transform traditional ETL pipelines to support a real-time paradigm,” he noted.

End-to-end view

Once a pipeline built using Stream Designer is deployed, teams can also use the tooling to get an end-to-end view of their streaming data pipeline as well as maintain and update it over its lifecycle. Plus, these pipelines can also be exported as SQL source code for sharing with other teams, deploying to another environment, or fitting into existing CI/CD workflows – enabling seamless collaboration and knowledge transfer.

“Data streaming is quickly becoming the central nervous system of our infrastructure as it powers real-time customer experiences across our 12 countries of operations,” Enes Hoxha, enterprise architect at Raiffeisen Bank International, said. 

“Stream Designer’s low-code, visual interface will enable more developers, across our entire organization, to leverage data in motion. With a unified, end-to-end view of our streaming data pipelines, it will improve our developer productivity by making real-time applications, pipeline development, and troubleshooting much easier,” he added.

What’s more at Current 2022?

Confluent also used the Current conference to announce Stream Governance Advanced, a new tier of its fully managed governance suite for Apache Kafka and data in motion. 

The offering, as the company explained, will provide enterprises with point-in-time lineage to resolve issues within complex pipelines, business metadata to discover and understand topics faster, and Schema Registry to enforce quality controls globally. 

Moreover, the company announced a private service connect for Google Cloud, enabling enterprises to set up a simple and secure connection from a Google Cloud virtual private cloud (VPC) to Confluent Cloud. This will minimize the complexity and burden of manually connecting virtual networks in the public cloud while keeping all details about a customer’s network private. 

Notably, the development means Confluent Cloud now supports private endpoints across all three major cloud service providers.

Originally appeared on: TheSpuzz