Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More
InfluxData is advancing its InfluxDB time series database product portfolio today to version 3.0. The major rewrite aims to provide more performance and capabilities to enable real time data analytics.
Founded in 2012, InfluxData has been developing an open source based time series database, written in the Go programming language. A time series database by definition is focused on the core task of keeping track of time-stamped data over time, which is useful across a broad range of use cases and industries. With the move to the InfluxDB 3.0 product portfolio, InfluxData is expanding beyond being just a time series database to providing what the company claims is a real time analytics database.
A foundational part of the InfluxDB 3.0 product launch is a complete rewrite of the database to enable better analytics capabilities. The rewrite is not just in terms of new code; it also uses a different programming language. The storage engine for InfluxDB 3.0 is code that was formerly referred to as InfluxDB IOx and is written in the open source Rust programming language. The storage engine isn’t the only part of the database update written in Rust — InfluxDB 3.0 also benefits from the open source Apache Arrow DataFusion SQL query engine.
“This is one of the biggest changes we’ve had from the core technology perspective, because it was a ground up rewrite of the core of the database in a completely different language,” Paul Dix, co-founder and CTO of InfluxData told VentureBeat. “It has been two and a half years of solid effort, so this has been a long term project, and to finally bring it to market is nerve wracking but also exciting at the same time.”
Event
Transform 2023
Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.
Register Now
Why InfluxDB needed a new time series engine to enable real time analytics
Dix, who wrote the first versions of the InfluxDB database, noted that what became apparent over time were the limitations of the time series engine.
Users of InfluxDB were trying to use the software for various use cases that the database wasn’t able to handle particularly well. One such common use case is real time analytics, which can have certain architectural requirements.
Among the challenges that InfluxDB had was the ability to address the need for ‘infinite cardinality’ in the database. Infinite cardinality is the ability of a database column to contain an unlimited number of unique values, which requires a storage and query engine that can scale in a way that InfluxDB could not, prior to version 3.0.
With the InfluxDB 3.0 database, there is also a data format change that will make it easier to work with data overall. Previously, InfluxDB used a time series data format that wasn’t particularly conducive to data analytics and usage in common deployments such as data lakes. With InfluxDB 3.0, the database is now making use of the open source Apache Parquet file format. Because all the data is in Parquet, Dix said he expects that InfluxDB 3.0 will have much better integration with data lake systems.
Why Rust in a database is a good thing
In the physical world, rust is a form of oxidation that is generally considered to be harmful to materials like steel.
With technology, Rust (as a programming language) is actually a good thing — or at least that’s what Dix hopes, as InfluxDB 3.0 will be a trailblazer as one of the first databases that rely on that language. Dix explained that Rust is a powerful systems language that also has higher level abstractions that make it easier to work with and helps developers to be more productive.
“The tooling that Rust has as a language for writing multi-threaded server based applications is really quite powerful,” said Dix. “It’s also optimized for performance and control.”
