Databricks Lakehouse Gets a Sibling
A couple of years ago, I wrote a brief review about the evolution of data lakehouse architecture, which at the time was a significant milestone in the world of data architectures and laid the foundations for unifying both analytics and machine learning (ML) workloads for many organisations123. The lakehouse architecture, originally pioneered by the team at Databricks, is now widely adopted and offered by other platforms such as Fabric, Snowflake and more. The capability of the lakehouse to extend the data lake with the governance and structure of a traditional relational data warehouse has opened doors to unlocking insights from structured and unstructured batch and streaming data sources, facilitating both business intelligence (BI) and ML workloads on a single source of truth.
We are now entering a new era in the world of data, where decisions that used to be made by a human are being replaced or enriched by agents that are powered by large language models (LLMs). We are living through a time where data-centric workflows are being re-shaped in real-time by the agents, transitioning humans from doing the analysis into more supervisory and decision-making roles. It was not that long ago when Databricks reported a 327% increase in agentic workloads over a period of 4 months, derived from the telemetric data from 20,000 organisations4. This unprecedented rise in agentic workloads, however, is becoming a burden on traditional data stores, demanding systems that can organise and expose heterogeneous data at the latency and scale it needs reliably5.
The Great Divide
Once you move past simple use cases (chatbots, text-to-SQL generation etc), agentic workloads require highly concurrent random point-lookups with millisecond latency, which is incompatible with lakehouse architectures that are optimised for large, sequential, columnar analytical scans. In other words, you need two separate systems that are optimised for update heavy and short-lived operations (OLTP; online transaction processing), and read-heavy and bandwidth-intensive operations (OLAP; online analytical processing). The OLTP and OLAP split is not new, and has been the norm for decades. For a very long time, the split made sense, given the transactional and analytical workloads have very different needs, query patterns, and performance requirements. Engineers are used to building pipelines that kept both systems in sync. However, the main drawbacks with the split-system are data duplication, stale data, and the overhead to manage the pipelines.
With the proliferation of agentic workloads, we are at a crossroads, needing systems that can support both OLTP and OLAP workloads. As a sector, we have been here in the past, in fact Gartner6 back in 2014 coined the term Hybrid Transactional and Analytical Processing (HTAP) with the thesis that HTAP architecture is envisioned to be a unified engine that can handle both OLTP and OLAP efficiently. This led to the emergence of a number of HTAP databases such as SingleStore and SAP HANA. HTAP databases can be broadly classified into two based on the underlying architectural model; co-located and separated with trade-offs being that co-located architectures sacrifice performance from CPU contention while separated architectures suffer from stale data7. Over the years, HTAP databases have made steady progress but the problem is far from being solved8. Moreover, HTAP systems we have today are highly proprietary, and lack the ecosystem, adoption, and community needed to become the default choice.
At this point in time, there is a general consensus that the lakehouse, for its ACID capabilities, interoperability, and reliability, is a great fit for OLAP workloads. Similarly, the winner on the OLTP front is without doubt Postgres with the production deployments growing from 54% to 60% in the last 2 years alone9. On the surface, it may seem that Postgres is ripe for being the default database for agentic workflows. However, the design for Postgres like many other databases assumed human-paced applications, and we now have evidence that at agent-scale, those assumptions no longer hold10. The interaction between an agent and databases is evolving to the point that databases are no longer treated as a single shared global resource but are instead becoming programmable substrate that can be created, tested, and deleted on-demand. For example: an agent should be able to spin up a new database, perform some tests, and throw it away instead of running it against production at the risk of a possible database contention. An additional constraint on Postgres is imposed by its shared compute and storage layer, making it inefficient and expensive at agent-scale. Lastly, ignoring the drawbacks, if you still use vanilla Postgres, you are once again left with a split-system where your analytics and transaction data are constantly drifting from each other.
In recent years, there have been a few noteworthy efforts to modernise Postgres. A key player in this space is Neon. Neon brought serverless capabilities to Postgres while decoupling both compute and storage with auto-scaling and git-like native branching. The team at Neon achieved this by re-architecting Postgres to decouple both compute and storage through routing the transaction logs from Postgres through a specialised distributed engine, storage system, and eventually into object storage11. Decoupling storage from compute means creating a branch on Neon takes seconds with storage just being a pointer to the storage bucket and any storage consumption on the branch is for specific data blocks that diverge or change on that branch. Ultimately, the durable object storage and git-like branching brought hyper-elasticity and rapid provisioning capabilities to Postgres, closing two of the major friction points for adopting Postgres for agent workloads. Neon was acquired by Databricks in 2025, and has since been integrated into the platform as Lakebase, bringing modern Postgres capabilities to Databricks.
The Unified Layer
Most enterprises are swamped with tools and technologies for creating and managing data; across software subscriptions, applications, data platforms, document stores, spreadsheets and more. The overarching vision with Databricks is to unify this data sprawl, which is admittedly a challenging endeavour, but has tremendous value for generating insights. More than that, it is a step in the right direction for creating rich context for agents. The lakehouse already allows you to integrate data from a number of batch and streaming sources, serving the enterprise data for downstream analytics and ML use cases. Integrating Lakebase into the platform opens a new opportunity to build on top of these capabilities. Given that the underlying engine in Lakebase is Postgres, there is now an opportunity to migrate OLTP workloads to Databricks, thereby having those workloads live under the same shared governance model. The one that I am most excited about is the bi-directional sync between the lakehouse and Lakebase, which essentially means you no longer need to actively maintain a pipeline to keep them in sync, no data staleness between analytics and application layer, and feature serving is no longer a burden on engineering. Above all, you can build applications and agents on top of Lakebase, leveraging the data from analytics while reaping the benefits of the agile Postgres engine under the hood for managing application data, memory, and logs with seamless database branching for running experiments and scenarios by humans and agents.
There is an argument to be made here whether it makes sense to unify everything under one roof. The alternative is what we have today: fragmented data, no single source of truth, poor governance, inconsistent metric definitions, shadow data initiatives, and a data strategy that never became a reality. There are a number of tools out in the market that are likely capable of doing what parts of Databricks can do in isolation but the assessment will have to be made holistically and consider parts (engineering costs, proprietary technology, learning curve, community, and long term maintenance) that never make it to the product specs. Moving into a unified platform is not an easy decision but if you are about to embark on a multi-year data transformation program, there is a benefit to place the bets on a team that is pushing the boundary and has built on open-source frameworks from day one.
Closing Thoughts
The key takeaway here is that the lakehouse has a new sibling, Lakebase. Together, they close the gap between OLTP and OLAP workloads. Databricks is evolving into a platform that can truly unify the data that flows through an organisation, and with the advances of AI by our side and strong data foundations, creating the data intelligence layer for enterprises is within our reach.
Footnotes
-
Hybrid Transaction/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation ↩
-
HeraDB: Towards Real-time Analysis of Transaction-Centric HTAP with CPU-GPU Hybrid Query Execution ↩