Lakebases are the New Databases with Reynold Xin, Co-Founder at Databricks

Richie Cotton: Hi, Reynold, welcome to the show.

Reynold Xin: Hey Richie, thanks for having me.

Richie Cotton: Yeah, great to have you here. I'm looking forward to the discussion. So I guess to begin with, I want to talk about self-service analytics, because this has been something that's been promised for many years now. It's not quite become a reality, but it looks like it's getting closer. So where are we up to?

Reynold Xin: Yeah, I think we're actually getting closer than we ever have and maybe already here, depending on how widely adopted some of the things are. As a matter of fact, at Databricks right now, it's just not just me. Me personally have completely changed the way I run analytics. It used to be the case that whenever I have a question, for example, hey, why did a product adoption metric suddenly spike up or spike down? I would ask somebody who report to me and they talk to somebody on the data team, and then that person talked to somebody who reports to them. And then before you know it, it would be two weeks pass, and then some answers come back. These days, I mostly just open genie and then ask, hey, what's going on? And I would say 95%, maybe even 99% of the time, you can tell me what is happening. But I think in order to get to that point, there was a lot of groundwork that would lay down, both in terms of product development. As well as from the adoption of building up the proper data model, the ontology and all that. And it's not just me that's been doing it. I think even the physical security team or the real estate team at Databricks are doing the same thing. They're not technical people at all. So I think to some extent, the future is already here, it's just not evenly distributed.

Richie Cotton: Yeah, absolutely. And you make a good point that this cross team communication is so hard. So when you ask someone to do something, then they're asking someone in the data team. And then, yeah, there's this huge lag between I've asked a question and I've got a response about data. So I love that speeding things up. You mentioned Jeannie there. So this is the Databricks data agent. Do want to just give a bit more background context on what this is?

Reynold Xin: Yeah, so we did a recent branding to consolidate a bunch of different things under the genie umbrella. But I think the easiest way to think about it is genie is an agent that really helps you with a lot of your enterprise work. And how it differs from I would say a lot of other agents out there is a lot of other agents last year, they try to memorize what exists, but genie actually computes. because I think vast majority of the enterprise questions are to Some decision making and it's actually somewhat quantitative in nature. It needs to be grounded on data. So GNE is designed specifically to address that. You can do a lot of what general agents do also, but that's where it really excels is asking questions like: hey, build me a revenue model that can forecast how my revenue will grow, or hey, how are my sales going across the different geodistributions? Are there any special things I have to be paying attention to? And this change how we work internally, also change, I would say, a lot of how our customers work. And you can think of it as I think it's basically what BI really should be. and instead of just going to dashboards and asking team of armies, building dashboards, Gini is the way to actually answer business intelligence.

Richie Cotton: Absolutely. Certainly if you're building a review model, you really don't want the answer to be built in on your own complex numbers, not just fabricated like a lot of chatbots would do. Yeah, I love that. So talk me through what's the new workflow then? Where do you want the AI to do things? Where do you want humans to do things? What's the best flow for analytics now?

Reynold Xin: Exactly. Yeah, I think there are many pieces that to get to as I was saying, there's actually a lot of work required to lay down the foundation to get to this point of self-analytics. And I think humans are extremely critical aspect of that. So we're not advocating, hey, all the data engineers, data scientists, analysts, you're out of a job. as a matter of fact, we think they are essential in creating the foundation and this foundation includes about a very important part of why Gini can work is this thing called ontology, it's genie ontology. So what it does is try to actually index every computation ever happened in the past. So for example, if your company cares a lot about revenue, like every company does, but revenue is a very difficult metric to actually compute, and it's certainly something you don't want to get wrong in. but At the same time, there's probably tens of thousands of different metrics that different teams and different departments care about in a company. you don't want to manually create all of them, but largely I would say anything that really matter, you have a lot of historic workloads that have run. You probably have lot of SQL queries, a lot of dashboards that reference revenue one way or another. So the genial oncology goes and indexes all of the past workloads and then trying to define, hey, here's how you compute the different metrics, and here's what you mean by, for example, ARR. Because every company might define annual recurring revenue slightly differently. and it puts all of that together and creates a graph. it also allows humans to actually augment the graph. And by combining this Curation-based augmentation and auto-discovery of what exists out there, we can actually really power the AI to do this best work. Because the AI models now have very good reasoning capabilities. What they don't actually have is the context for everything. So gene ontology creates that provides a context to all the AI. And with the context, now one, it can actually do it faster because it doesn't have to. Slip go, hey, let me crawl something and then see if there's a definition here. this links now to somewhere else. Let me go crawl the next thing. It's like one of my co-founders, Ali and CEO of Databricks, has this analogy of imagine you go back to 2000 and then the way search engines work is it opens one website first and searches the website to see if there's anything on there. And if there's something that matches the link, it opens the link next and goes to visit that link and just keep recurse. And then after two hours, it shows the 10 links on Google.com search result. That's how, in a way, stated art how many agents work. The ontology breaks that in a way give you the search index, the knowledge index, knowledge graph of your entire enterprise. So it substantially improved the velocity and substantially improved the accuracy. Because you don't have to go figure out what exactly is the revenue. So with all of this, I think humans has a very important part. One is data modeling. At the end of the day, the AI agent, if you feed it junk, junk will come out. If you actually feed it really great data with the ontology, you can get amazing results. The second one is there is this a little bit of a manual curation component, and we have found actually. Many organizations are religious about whether they should do any manual creation. Some are on the camp of absolutely no manual creation ever. Some are, hey, manual creation would be the most important thing to get us to the, whatever, from 95% accuracy to 99% accuracy. we're not trying to force our customers one way or another. We feel hey, it's a very useful tool. You want to use it, you should use it. Internally Database we do. So obviously humans have a very important part in that manual creation, maybe for your tier zero. So metrics and assets. and the other thing is I think once you have your data set actually get to what we call the go layer, so ready for business metrics, all the systems work pretty well. But there's a massive, typically massive amount of data engineering is required to go from your raw data, which might be collected from Tens of thousands of different systems into the gold metric. And any of this middle layer could break at any given point. we are actually working on many things to improve and self remediate if for example pipeline could break. But in practice, I think you still need humans to actually babysit and think about hey, what happens if something breaks? What's the implication of that? How do I actually fix it? How do I make it more reliable in the future? So we don't think humans will go away. We actually think humans who more and more focus on the more important tasks and be relieved from, hey, can you help me answer this question by writing a specific query? Or can you help me answer this question because I don't know where the data is? I think the humans who spend the time to build the ontology and infrastructure to the point that it becomes largely self-service for any business questions.

Richie Cotton: I love that. And yet certainly a very common business problem that every single team is defined revenue or something like that, or ARR slightly differently. And you really don't want that to happen because you got to that consistency across teams. So I love the fact that you've got this ontology layer underneath. I guess this is closely related to the idea of a semantic layer where you've got these canonical business definitions.

Reynold Xin: Yeah. So the semantic layer is actually now part of ontology. Ontology actually builds on that. it's one of the signals that the ontology pulls from.

Richie Cotton: Okay, that's brilliant. And so it seems like a lot of the work now to be done is just creating that data foundation layer. And then once you've got that, the analytics part is almost easy, because at that point, the AI can take over and you're guaranteed that it's grounded in good data. Okay. All right. So there has been a persistent problem though, with generative AI that sometimes it makes mistakes, sometimes it hallucinates. Do you want to expand on how you deal with that? What do do when it gets things wrong?

Reynold Xin: Yeah, exactly. Yeah, so we actually talked a little bit about it right now, which is ontology. but it is true the AI makes mistakes. But about it, so does humans. how many times do you just trust blindly, hey, this person's gonna come back with the right answer, but it turned out the answer was wrong? that happens all the time too. So I think it's not necessarily the case that hey, AI has to be completely perfect, where humans make mistakes all the time. But certainly, I think if you just were to ask a coding agent, you just give the coding agent or a generic agent some access to all of your maybe this day by plumbing through APIs, there are too much context that it would confuse the agent a lot. And then you start hallucinating fairly frequently. So, what we have found, we actually have a whole AI research team whose focus, their focus isn't on, hey, how do I conquer and build models for AGI that are far smarter than human beings? Their whole focus is on how do we actually create the system and the product that can give the best answers on enterprise-like questions. And we even created a standard benchmark or Office QA. which is now used by many of the frontier models also. They actually run eval against that specific benchmark when they announce their new models. and those are basically grounded on actual questions we see that enterprise work, like knowledge workers would be asking. So the ontology one is it finds all the context, but more importantly than having all the context is actually it finds the right context for any specific question. And with the right context that are hopefully actually small, you can actually substantially improve the accuracy of the model and substantially reduce the possibility of hallucination. Now nothing is perfect, just like humans. I don't think it will ever behave. You have 100% correctness rate and it would never be wrong. So we're not saying you should be asking Genie a question and put it into your maybe quality filings and submit it to that CC. but what our experience is for vast majority of the questions these days, you don't need to ask another person of the data team to help you answer. It's largely self analytics.

Richie Cotton: Okay, that's pretty cool. I love that you are making analytics available to everyone. That begs the question, I guess, traditionally, Databricks, was originally targeted to machine learning scientists, machine learning engineers, and you've got broader. Do you need a different user interface or a different product for technical data people versus everyone else?

Reynold Xin: Yeah. So the way we think about this problem is that definitely you need a different interface for people that are not very technical. Now that specific interface could actually be used by very technical people too. But the more technical people often they have a different jobs to be done. For example, they want to see code, they want to be writing code. Whereas if you show massive amount of code up front to maybe somebody who have never written a single line of code, doesn't never even heard of the word SQL, that would be too daunting. so what we ended up doing is we actually created a whole new interface in Databricks called Genie1. And when you open Genie1, there's nothing else. All it does just, hey, here's a box for you to type in, like Google search. Type in your question. and then Genie One will actually go help you generating a report or answer or do whatever you want to do. and that interface is used heavily by everybody, including technical users and non-technical users. But we also leverage the same underlying technology, like Gini ontology and all of this. And we have a separate product called Gini Code. And what that does is Gini Code is the AI interstation in the traditional Databricks interface for far more technical users. And it understands, for example, here's a notebook, and here's some Smith's code. And the user is trying to do, for example, build a linear model to predict some revenue. And understands, hey, here's Spark traces, and there's a failure in the pipeline. How do I remediate that? So give you far more technical detail. So we built two different products. where the more simplified one is usable by everybody is more for simple analytics. and which is Gini one. And then Gini Co. is the one that's facing more technical users and heavily integrated with existing or pre existing Databricks interface.

Richie Cotton: Okay, all right. So this is everyone can use a chat bot, but then if you want more, you want to use, you want to see them go, you want the other interface. And so you mentioned notebooks. this has been the staple of data people for more than a decade now. What do think the future of notebooks is?

Reynold Xin: Yeah, yeah. Yeah, ironically, I think notebooks gonna become even more important. And if you think about it, every chat interface, like chatbot interface, is a notebook. they don't position it as a notebook, but it's really a notebook. A question is a command you run more or less, and then it comes back with some results. the results could be semi-structured that weave in with charts and Data structure is not very different from notebooks. As a matter of fact, we're seeing many same feature requests that our customers were asking for on the notebook product from 10 years back. Now they're asking for exactly the same set of feature sets for the chatbot product. For example, they want to be able to share a report or an answer generated by the chatbot. Makes sense. And they want to be able to reorder sometimes the questions. Again, that's one of the big things we added very early on on the notebook side. And they want to be able to actually turn maybe an answer into a recurring thing. So turning into a production job because this is something they want to see maybe every day or want to send it out to a mailing list every day. That's something we actually added 10 years ago in our notebook project. So we think there's gonna be a lot of similarities. Now maybe it's not going to be the identical user interface because notebooks target the slightly more or more technical persona. Notebooks are a little bit more coat heavy. but we think the form of the notebook is gonna become even more popular. And maybe we won't call it a notebook, but it's effectively a notebook form.

Richie Cotton: That's hilarious that people want to turn just chat conversations into a notebook in some sense that's something I've not really considered before. Yeah, fascinating stuff.

Reynold Xin: It makes sense. If you mean every enterprise works, some collaborative work. notebooks are great for collaboration and notebooks great for interactivity and looking at inspecting results as you're writing code. that's not very different once the programming language becomes English.

Richie Cotton: Yeah, okay. That's cool. I guess copy and paste features from the notebook product to the chat. It's not quite that easy. yeah, seems useful. Okay. So the other big pain point with cell service analytics seems to be around permissions for things in general, not everyone should have access to the HR data, things like that. How do you deal with that data breaks, that seems to be a blocker.

Reynold Xin: Yeah, I think this is actually one of the most overlooked points when people think about AI, is they think, hey, if I just have access to all the data, everything would be great. But then often you don't want to give the AI access to all the data because you don't know exactly what's going to happen and who's going to be using it. so one of the things, a very early decision we made in Databricks is there's only going to be a single governance layer, which is what we call UND catalog. Every permission is defined in the unique dialogue layer. So if, for example, I've given Ricci access to this but not the other thing, there's no way for you to actually circumvent that, regardless of what interface you come in from. And this is very different from I think a lot of the past BI tubes, which is, hey, because of the extract process. BI tubes often do in order to speed up performance, they carry a separate permission model. So you end up being, hey, maybe Richie should not have access in the actual underlying platform. But then in this specific BI tube, suddenly now he can actually get in. and then before you know it, somebody forgets to actually update the permission policy. And maybe there was intentional in the beginning, but later something changed and they forget to update the permission policy. so in Databricks, there's only one single layer for Governance, which is Unity. And in Unity you set the policies and then it's enforced everywhere. Your AI agents basically act on your behalf. And as a result, it can only see what you can see. If you can see the HR data, you can see or maybe more likely you can maybe see the company employees or work chart, but you can't actually see, for example, on the more sensitive PII stuff. everything will be respected and you just can't get around it.

Richie Cotton: Okay, that does seem sensible that you got all the permissions in one place and it's a separate foundational tool that all the agents know all the stuff built on top of. Okay, so if you'd like making use of agents and things, do have to take this in mind? Do have to, I guess test with different user permissions all this stuff? If you're building with this or if you're doing analysis on top of this, how do you go about making sure it works for all these different scenarios when?

Reynold Xin: Yeah.

Richie Cotton: You do affirmation so you don't.

Reynold Xin: Yeah, the way we did it is actually the agentic layer always access data through the data foundation layer. so the agentic layer never opens something on its own. It goes through this narrow ways. As a result, the narrow ways enforces everything. I think that's how a security system needs to be built. by the way, in general, I think because

Richie Cotton: Okay.

Reynold Xin: Agents are a pretty new concept. There's a lot of innovation happening. I think the industry as a whole have overlooked how important governance is and how important security is. And especially on the personal side. If you look at Open Claw and Hermit agents, it's effectively, once you give it access, it can do anything. And that's super scary. And I think a lot of best practices maybe enterprises have learned in the last 30, 40 years. Of building data systems are largely where we're learning a lot of those in the industry. And having schema is a good concept because it can help you identify what road columns are PII, you can enforce access control on those. Just having a single markdown file for everything is not a great idea. Nothing wrong with markdown files, but you shouldn't treat markdown files a database. I think rest of industry. We'll have to relearn a lot of this, unfortunately. But the way we've been doing it is we spent a lot of time basically building data governance in the last few years. we just thought, hey, there's such a critical aspect of making agents actually work and deployable. so we build it in a way that hey, you just can't circumvent. There's no separate path to go in. You always go through a govern data foundation layer.

Richie Cotton: Okay. This seems absolutely essential for an enterprise or for a business context. Since you mentioned Dope and Claw and all these other AI personal assistants, do think there is a way to solve security for these tools then? Or are they just fundamentally chaos and you're not going to get security there?

Reynold Xin: I think we will have to figure out a way. and I think a lot of it has to do with also introducing maybe different governance layer. You cannot have the agent itself police itself because it could actually hallucinate, it could decide to do things. it's very difficult to explain exactly what happened. Just the same way we don't trust A single human being to enforce themselves 100%. There's always some other guardrails or access control. I think the same thing will have to happen with agents. So, in a way, basically the governance layers can't be in the agent layer. It needs to be below the agent layer or has to have some other agent enforcing it. But one of the things that we actually announced was this thing called Omigen, which is actually Matei's He's personally driving a lot of this. And what Omigen does, it's actually allows you, it's less about data access control, but allows you to declare policies for agentic governance. For example, you're allowed to do this but not the other thing. and it has a separate supervisor agent that's trying to enforce and look at, hey, is this agent now going outside of the guardrail that I've already set up? I think that's one approach, and it looks pretty promising so far, but we'll see. So how I'm sure there will be more approaches that come up. But it just can't be the case that five years from now or three years from now, everybody just decides to give their personal agent or enterprise agent permission to everything and then hope for the best.

Richie Cotton: Yeah, so I agree, it's something we have to solve eventually. But yeah, I suspect it may be quite a while before we get to the point where we're happy with all these things. Yeah, I guess the societal equivalent you mentioned, you shouldn't be policing yourself in real life. We have police forces to police the population and there's other groups to police the police and so forth. Okay. All right. So the other thing I want to talk about is databases. And I know for about as long as I've been alive, page one of every textbook on databases is there are two kinds of databases. You've got transactional databases and you've got analytical databases. These are different. I know you've been working to try and blur this boundary a bit. Talk me through what you've been doing here.

Reynold Xin: Yeah, so at Data N Summit last week, we talked about this thing called LTAP, late transactional analytical processing. And maybe take one step back. As you said, the last 40 years, effectively databases are split into two different categories. There's transactional databases, which is hyper-optimized for point lookups and point updates. For example, hey, Reynolds depositing some money into a bank account. I need to find a row that talks about his balance and very quickly update that one row. and then the other size analytic databases, which are hey, I want to reason on my data, I want to understand, for example, what's the behavior of my aggregate customer base? How much are they buying across different stores? how much are they spending? And those require scanning across large amounts of data and running analytical queries on those. And they historically been avoided two different databases. Because the workloads are just so different. And what happens is usually you will start with for any application, you start with having OLTP a transactional database. And this was examples like Postgres, MySQL, Oracle. and you build an application. And when it becomes successful enough, you want to start reasoning on that data and analyze that data. So you start moving the data. From the transactional database into an analytical system. And the analytical one could be modern days would be like Databricks, Snowflake, Reship, BitCory. Back in the days could be Teradata or Hadoop. That process is very, very painful because it relies on this thing called CDC, which changed data capture, effectively takes little deltas out of the transactional database and ships the delta into the analytic side and then reconstructs the state of the transactional database. it's a very brittle thing. and at the keynote I actually asked the audience, hey, how many of you really love your C V C pipebook? I think some of them thought I asked how many of you hate your CEC pipeline? There's a few. But mostly, nobody loves CEC pipelines. But many data engineers who've woken up at 2 a.m., 3 a.m. because the CEC pipeline break. And we even made to show at Databricks internally that CDC changed data capture does not stand for change data capture. It stands for continuous data corruption. just because of how brittle and how fragile it is. So LTAP effectively, but it's a play on the name HTAP. I don't know if I should get into that here, but it effectively solves this problem by making sure every single table in your transactional database appears actually in the lake in the iceberg format. And once it appears in the lake in ISPR format, you can run all your analytic compute against that data. And the data will be up to date, it will be fresh. And as a result, you don't have to worry about hey, do I have a pipeline to ship my data? All the data will just be automatically available. And that's, I think, a very, very big innovation because now you don't have to worry about building CEC pipelines. And more importantly, you don't have to worry about pre-designing do you need a pipeline. Because sometimes your business change, you don't actually know exactly what you need in the future. this just makes sure everything is always available for analytics. And the key there is why hadn't this been done before? And one of the biggest reasons is performance. The two systems just have such big different requirements and workloads that it's very, very difficult to engineer a shared storage system that works for both. OLTP databases want role-oriented storage because it needs to update individual rows very quickly. Analytical ones wants columnar storage because scans a lot of columns at the same time. And we figured out how to do it in the last year or so. and that's largely built on the general lake-based architecture. so the link-based architecture enabled this innovation, but from a user's perspective, really a big thing is now you can run analytics on your OLTB data as if they were just native to the analytics system. And you don't have to have any pipelines connecting them.

Richie Cotton: Okay, so this is really shortening the time from we've got new transactions coming in to we can now start querying against those transactions.

Reynold Xin: Yeah, immediately. You don't even need to, and you don't have to figure it out, do I even need that? You can just at some point you realize you need it, you can just start running queries. You don't have to spend any time doing data engineering, designing how do you transport the data.

Richie Cotton: Okay, so yeah, no data engineering work in order to set this up. just magically happens. Do you want to talk us through some examples of when this might be really useful? If you've got some examples of queries.

Reynold Xin: Yeah, I think one of the simplest one would be hey, what if I want to understand what's my sales per store or what products are trending? you can now just start running sub analytic queries directly against that data. you don't have to worry about anything. I think in historically, this is probably a many months, multi-months project for most data engineering teams. because they have to figure out how to actually get the CDC out, how to get it in and then set up a separate system. There's often still lags. all of this would just be it shrinks that whole multi-months effort into now just run a query.

Richie Cotton: Okay, love that. Yeah, anything trending is you talked about products trending, you've got, yeah, looking at social media data, looking at news data, I guess, financial stock data, all this, I guess we're getting into real time analytics here. Do you want to talk us through? Is real time analytics becoming more important then? Is that related to your database efforts?

Reynold Xin: Yeah, I think it's gonna become more important. There's actually many, many things, but the term real time is a little bit confusing. Everybody defines in their own way. But I think broadly speaking, we're talking about how do we get really fast analytical queries on very fresh data. so searching LTAB is a very big enabling factor here, which is now you can analyze your latest data without even setting up anything and all of your latest data. but the other effort we announced at the Data NI Summit is this thing called Lakehouse RT, where RT stands for real time. And what it does is if you think about LTAP as the, hey, how do I make the OLTP or transactional data be available immediately for analytics? Lakehouse RT is the analytical engine to now query all those data and more, so all the Lake House data in super low latency. historically, if you want to build, for example, a custom app that runs some analytics, for example, some dashboard observability tracing, and you want that and it's because it's customer facing, you want it to be super responsive on the freshest data. For example, you want tens of milliseconds query time. You have to build a separate serving system in order to do that. Lakehouse RT is built on a new engine, and this engine when this product is actually capable of answering many, many queries in tens of milliseconds. and you can actually answer a lot of them. I think the demo we showed was we have what 1,000 dashboards on the screen, and each of them loading about eight queries. And I just click start and basically simulates all the 10,000 dashboards loading all those queries immediately, and it finishes in less than two seconds. for all of this hitting the lakehouse RT at the same time. so this is a game changer because it enables for the first time the lake houses where most enterprises have almost all their data enables the first time for them to actually serve those data in very, very low latency and very high throughput, throughput here meaning queries per second. And we think this is basically it reduces further data silos. You don't have to copy your data elsewhere into a separate serving system. Now you have separate governance, you have separate again, another data pipeline that might fail and cause latency increase. And so once you combine, I think, LTAP and Lickhouse RT, it really consolidates this. Hey, you just need a single copy of your data in the Lake House. Whether it's your transactional data, whether it's your observability data. and this is just one copy and one copy to govern and you have different systems that can actually handle different workloads all sitting on top of this one copy of the data.

Richie Cotton: OK, 1,000 databases at once. I'm going have to buy some more monitors.

Reynold Xin: Yeah, and now we show a very little thumbnail of the dashboard at that point where we show a thousand of them. Yeah, but more often than not, it's not a single person looking at thousand dashboards. It's actually a thousand different people or ten thousand, hundreds of thousand different people opening different dashboards at the same time.

Richie Cotton: Yeah, so now we've got self-service analytics, your whole company's looking at dashboards. yeah, you're going to need to four of them.

Reynold Xin: And by the way, the agents, we demo in dashboards more visually easier to understand. But honestly, what we've been seeing is agents are generating far more because of agents, our customers collectively are issuing far more queries. so this whole thing about concurrency and latency is mattering even more.

Richie Cotton: OK, yeah, so it's nice to do some of that because I'm curious. How do you see agents hitting databases then? Do change what you need in a database? And yeah, what's going on there?

Reynold Xin: Yeah, I think there's some things common across both, again, maybe I have to go back, even though I have a single copy of data, I think the workloads still look very different between transactional databases and analytical databases. I think for both, we're seeing far more activities. and agents, because agents are issuing more queries, it's not just token costs that's going up across the board. It's also what we're seeing is the infrastructure cost because of the consumption of the tokens generating more queries actually incurring higher infrastructure costs. So I think the future systems and what we've been building also just in the last couple of years is to help customers reduce that spend by having systems that can scale better, by having systems that can run faster on the same set of hardware. and that's part of what Big House RT is. But the other aspect is I think when it comes to transactional databases, agents are becoming the actual primary persona. so it's I would say vast majority of the Silicon Valley engineers, probably 99% of Silicon Valley engineers these days, don't write code manually and don't provision the database manually. They actually use some agent to do it. and that reflects in the numbers we see in production. And agents want because It used to be humans and the humans writing code and the cost of writing code is actually pretty high. But the cost of writing code have dramatically gone down. It's not going to go to zero because the tokens cost something, but have gone down dramatically. And this actually opened up a lot of opportunity for experimentation and innovation. But one of the big things with infrastructure and databases in particular is that databases are viewed as this clunky, expensive. Heavyweight thing. The term infrastructure reminds you of PGE and utility companies. They're not nimble. And as a result, they in the past actually inhibit innovation because maybe you can write code very quickly, but if you can't actually provision the database very quickly and you can't run a database very cheaply, you might decide not to run the experiment at all. So we want to enable experimentation at scale. That are very, very low cost. For example, maybe if something doesn't, if you don't know if something that's gonna take off or not, it better not cost you a fortune. It should cost you maybe a dollar or two in order to run the experiment, maybe even less. But once it takes off, you should not require to re-architect everything because you wouldn't have time to do that. So we think infrastructure in general and databases in particular, database is one of the most important. piece of infrastructure needs to enable rapid experimentation at scale cheaply, but at the same time for any of the experiments actually takes off, and it should be able to scale to mission critical on exactly the same set of infrastructure without having to do any re architecture. And so what we're doing with the lake base, and there's many things that go into it and many benefits that come out of it. For example, you can Provision a database in less than a second. So now your agents don't have to wait. Everything is auto-scale. It can scale down to zero automatically if there's no load. So if your experiment fails, you don't really pay anything. And because of all, you can auto-scale. If your experiment actually becomes successful, you can actually provision it to be much larger resource pools. So you now can actually support, for example, millions of transactions per second. As a matter of fact, we even talked about we will support multi-cloud disaster recovery, which is probably the industry first managed multi-cloud disaster recovery. So you can really go to mission critical even if the whole cloud goes down. And that we think is going to be incredibly important to enable that agente development innovation.

Richie Cotton: Okay, so yeah, from an infrastructure point of view, you got all these kinds of scaling capabilities, you got the resilience. You mentioned the cost of things. I think that's a hot topic at the moment. I think a lot of CEOs have been giving the talk. There's change from, we must use AI as much as we can to actually, this is quite expensive. Tell me through how you think about cost control from the database side of things.

Reynold Xin: Yeah, from the database side of things, it's one of the things we think databases are a fleet-wide problem. you should not think about database as a single instance of database. You should always think about databases as hey, you have a collection of databases and fleets and how do you actually manage them? and each of them we think auto scaling is gonna be extremely important here. because I think most databases in the past or you probably even now are done in a way that hey you provision one, you just it's up and running 24-7. It never shuts down. And that's where a lot of the costs are coming from. because you have to provision for the peak. and more often than not, for something that's not super successful, you provisioned it, but then you forget to shut it down. And before you know it, it racks up hundreds of dollars per month, and you have a thousand of them. The auto scaling takes away that issue because if there's no traffic, it's not up. It doesn't cost you anything. and if there's actually a spike in traffic, it auto-scales to have more resources so it can handle that load. I think that's a very critical part of it. The other thing is the infrastructure spend that pool forget by agents is largely because of experimentation and CI CD cost. LakeBase also solved the problem with this branching. basically LakeBase has this thing called branching, it's actually really, really unique to Lake Base. what it does is it takes a complete clone of your database, logical. So it doesn't actually really copy, it doesn't really clone the database. It just marks, it does a copy on write clone of the underlying storage layer. And basically have a pointer saying, hey, this is the state of the database. And if you go updates, it will now start recording the updates. But with this copy on write branching, you could actually clone a production database or test database in less than a second without paying for extra storage. So this becomes a great way to run CI CD by combining auto scaling and this branching because every time you create a new pull request to, for example, your application code, this is actually very common scenario among our customers now. They will create a branch of their database and they run the entire CI CD on that pull request and that branch. And then when CI CD is done, it shuts down. it incurs maybe one cent of storage cost because that pull request try insert another row or two. and then incurs very little compute because it just runs for the duration of that little CI CD pipeline and then it's gone. This is not something you can by the way, it has perfect isolation because it's the actual new Postgres instance that's running just for this. So this is something that's impossible to do outside of Lake Base right now, because If you actually want to do a clone of the database, you have to clone the data outside of LakeBase. And as a result, it becomes very expensive. It takes a long time. But the cloning of your production database could take down the production database itself. That's another horror story that happens all the time. So LakeBase just solves all of these problems and really makes it much easier to control the cost and have much lower TCO while enabling all of this possible experimentation innovation.

Richie Cotton: That's actually pretty wild. think the traditional way of doing things, have one production copy of all your data. You have one staging copy of your data and that's it. So the idea of just setting up an ephemeral copy of, Hey, this is the variation on your data set. we just didn't have the database to exist until the PRs merged and then get rid of it. That seems like a fundamental, it's a new way of working.

Reynold Xin: Yeah, and the most important part is it's not an actual physical copy. It just appears a logical copy. A logical independent copy is isolated. So you can run all your experiments on it. It has no impact on rest of the other PRs of your production system.

Richie Cotton: All right, so you're not just copying hundreds of gigabytes of data all over the places. OK, yeah. Yeah, that'll be a bit slow and a bit expensive. All right, so I'd also like to talk about Spark. This is, course, where Databricks started. Yeah, what's happening with Spark these days?

Reynold Xin: Otherwise it wouldn't be. Very fun. Yeah, Spark's actually growing like crazy. I think there's many billions to downloads every month right now. I think maybe three billion. so it's growing. It's a super healthy community. There's a lot of pull requests. new releases coming. We actually increased the release velocity. and we also just recently added a couple other major innovation features. one is Spark Declarative Pipeline, which we think is a better way to be building data pipelines. it's another API layer sitting on top of the data frame API that give you more declarative data pipelines. There's also the real-time mode in streaming, which for now for the first time you can actually run again, real time is very overloaded here, but you could actually run streaming jobs and stream processing on records in milliseconds. those are pretty major innovations that's been happening to Spark in the last few months.

Richie Cotton: Okay, so the decorative pipeline seems interesting, because you're not having to specify everything, I guess at a lower level, you decide what you want, and then all the details get figured out for you. Okay, nice. It's nice that the community there is still pretty healthy.

Reynold Xin: Exactly. Yeah.

Richie Cotton: just on subject. because Spark is pretty much a predictive AI machine learning tool. It seems like generative AI, agentic AI have stolen a bit of the limelight here. So do think there are still innovations going on in predictive AI?

Reynold Xin: Yeah, it's really funny. Yeah. so what we have by the way, it is true that everybody's talking about LOMs and GNAI right now, but what we have found with the actual production telemetry is because of the talk on GNAI and the attention on GNAI, they're actually far more classic machine learning, which I assume what you meant by predictive AI is the more classic machine learning, more deterministic, more statistical approach.

Richie Cotton: Yeah.

Reynold Xin: The consumption and usage on those have actually taken off also quite a bit. I think the reason is now there's more budget in every enterprise on AI. And so at 20,000 feet level, there's no difference between what is classic machine learning versus what gen AI. It all got lumped in together. It's like giant AI bucket. and now there's a very good question, which is hey, but then what does classic machine learning, is it still needed? I actually think GNAI is great for a lot of use cases, but for many of the use cases that classic machine learning were designed to do in the past, classic machine learning is still the way to actually do them. For example, you want to build a predictive model on how your revenue might trend in the future. it's actually fundamentally a statistical problem based on, hey, what How have you been doing what's the seasonal variability? What's maybe the country variability? And are there any new products that might be coming in? How would that influence your future? All those are classic machine learning features and signals. And I think basically classic methods like linear methods, decision trees, all this I think are more important than ever. And a big mistake would be: hey, everything is just GNAI. By the way, GNIs are terrible at math. GI can be really good at writing a program that uses classic machine learning to solve a statistical problem. But GNI itself is a terrible solution to statistical problems. so I really think of GNAI as a way to reason, but then often many of the solutions. Is actually a classic machine learning approach. And that's one of the things Welchie pushed on this product. And part of Gini is now you can actually ask Genie to do a lot of the forecast, do a lot of explanation of what had happened, and all this actually calling back into classic methods. And what we're trying to do is to alleviate the users from having to spend a lot of time thinking: hey, should I be using a decision tree here? Should be using a logistic regression model here? Now how do I do hyperparameter tuning? Many of those can be automated now with Gen AI. But outchummining is actually running the classic machine learning methods under the hood.

Richie Cotton: Yeah, I'm right there with you that machine learning is still incredibly important. But I do love that tip that if you just call AI, then I guess your CFO doesn't know the difference. And those are fun to anyway. So yeah, good trick to.

Reynold Xin: Yeah, also but keep in mind many AI systems are honestly sometimes just a bunch of if statements. and you could argue a complicated set of if statements are not very different from a single layer neural network. They are more different.

Richie Cotton: That's true. suppose, yeah, the same. Nice. All right. So just finally, I always want more people to learn from. So whose work are you most excited about right now?

Reynold Xin: I think one of the thing, maybe very self-serving here, but Armi Gen is actually the latest thing that Matei's doing. and I think we briefly talked about it when we're talking about governance, but one of the things with the Army Gen is that it's a meta harness and it solves a very concrete problem, which is hey, you have all these harnesses in AI right now. but they're all independent, they're all different, they all come with different UIs. they're all kinds of challenges, but honestly ultimately you don't want to only use one harness. You want to use, actually you want different harness to collaborate. But it's the same thing as humans. You don't want to just replicate. rental a hundred times and only work with this one person and which thinks in a very specific way. You actually want different people coming from different backgrounds to collaborate to arrive at the best solution. And I think that's not very different from AI. And Omigen is like a what we call a, or Mate calls a, meta harness that sits on top of a lot of little harnesses. you can use Armigen to orchestrate clock code. You can use Omigen to orchestrate Pi, you could use Omigen to orchestrate or open code and actually have them work with each other. And what we have found is actually for a lot of different tasks, you can actually get to better result by having the different coding harnesses collaborate with each other. And I think it's somewhat of a new idea. Omigen is not even the only thing that's trying to do that. I think just in the last few months, there are a few that's emerging now. Omigen's probably having the largest traction. And I think this is gonna be a new layer that's created in the whole AI landscape. And it'll be exciting to see how that layer will evolve. And my guess is over time, more and more people will be programming against the meta harnesses instead of directly against the harnesses.

Richie Cotton: We're from prompt engineering to context engineering to harness engineering now better harness engineering. Okay. All right.

Reynold Xin: Yeah. And every step give you some step function change in terms of capabilities.

Richie Cotton: Nice. Okay. That seems to be the next thing to watch out for then. Wonderful. Thank you so much for your time, Reynold.

Reynold Xin: Yeah, it was nice talking to you, Richie.