Dbt launches next generation semantic layer to solve trust in data

Philadelphia-based dbt Labs announced today the next version of its open semantic layer — the bridge between data platforms and business intelligence (BI) tools, enabling enterprises to serve the single, verified version of data for driving insights. The company is the same one behind the dbt analytics engineering tool, which prepares data for analytics.

The new version is available today for all users of dbt Cloud. The revamped layer brings support for more data platforms, including Databricks and BigQuery. It also introduces new capabilities that make it easier for organizations to define and access complex metrics, at scale. The features come from dbt Labs’ recent acquisition of Transform Data and makes the semantic layer suitable for a broader range of organizations and use cases.

“The new generation of dbt semantic layer is much more sophisticated, providing a broader range of organizations with more (complex) metrics and dimensions, a wider array of needs around metric types, and different data platform configurations,” Tristan Handy, CEO and founder of dbt Labs, told VentureBeat.

What is dbt Semantic Layer and how is it getting better?

The current architecture of the modern data stack sees information flow from warehouses and lakehouses to artificial intelligence (AI) and BI tools.

However, in this approach, organizations, especially big ones with complex structures and different tools for different analytical needs, have to move different copies of data from warehouses and lakehouses. This not only takes time and effort but can also affect downstream results.

To solve this challenge, dbt offers the semantic layer, a bridge of sorts where business metrics and concepts can be defined and made universally accessible.

It uses existing programming constructs that dbt authors express — refs, macros, sources — and offers the same, consistent version of the truth to all the BI and analytic tools. This will help simplify the whole process of producing analytics.

Dbt introduced the semantic layer in October 2022. However, it was limited to fewer, less complex metrics and dimensions captured from Snowflake.

With the latest release, the company is expanding support to include data from multiple platforms, including Databricks, Google BigQuery and Amazon Redshift. The company is also offering relevant performance optimization for each of these platforms.

Moreover, the upgraded layer enables more complex metric definition and querying with MetricFlow — a state-of-the-art SQL query generation engine — sitting under the hood. It was acquired as part of the Transform Data deal and helps analysts create metrics by constructing appropriate queries for different granularities and dimensions that are useful for various business applications.

Dbt notes MetricFlow brings multiple capabilities for complex metric generation and querying, including dynamic join support across any number of tables to create a semantic graph of data and the generation of joins, filters and aggregations leading to legible and performant SQL.

“We’ve also built new APIs, including an entirely rebuilt Java Database Connectivity (JDBC) interface built with ArrowFlight, as well as a GraphQL API allowing for more seamless integrations and applications to be built on top of the new Semantic Layer,” Handy said.

Finally, through its Semantic Layer Ready Integration Program, dbt is expanding the downstream connection of the layer with AI and BI tools. 

The company said it will now allow users to connect the semantic layer with Tableau, Google Sheets, Hex, Klipfolio, Lightdash, Mode and Push.ai. This will give users access to business-critical metrics that are consistent and reliable, drawn from a single, verified source of truth. In the coming months, the list is expected to grow, giving even more users access to trusted data.

“We believe that the dbt Semantic Layer will help power the next generation of data analytics, and are happy that our customers will now be able to easily experience the benefits of reliable, consistent metrics — backed by dbt — across the organization. Too often we hear from customers that a major barrier to wider adoption is a lack of trust in data. This is one more step in our mission to solve this,” Nicolas Brisoux, senior director of product management at Tableau, said in a statement.

While players like Mozart Data and Datameer also offer tools to prepare data for analytics, dbt has outgrown its rivals to become what Handy describes as ‘an industry standard’. Currently, around 30,000 businesses use dbt, with 900 certified analytics engineers and a community of 90,000 members worldwide. The company’s cloud service makes up 12% of the total user base and is growing at a rapid pace. 

Just over the last year, more than 1,000 enterprises signed up for dbt Cloud, including Airservices Australia, Anheuser-Busch, British Red Cross, ThermoFisher Scientific and Sequoia Capital.

Originally appeared on: TheSpuzz