Don't Build on Jell-O: How to Make Agentic AI Reliable with Dan Klein, CTO at Scaled Cognition

o if you look at things like horizontality of intelligence or contextual understanding of human language, these are areas where the systems have grown so fast.

There's this explosive increase in capability. But if you look at things like reliability, that hasn't kept pace. And this is a challenge because humans, when humans are intelligent, the shape of our intelligence is different. And humans have a very different profile. If they don't know something, they're gonna tell you, or if they are gonna quiz bowl context where they're gonna make a guess, they know they're making a guess.

We're metacognitive. We know what we know, we know what we don't know, and we think about those things. Systems today do not, and that's why you have systems which are essentially designed... They're not truth engines, they're plausibility engines. They give you output that has captured as many correlations from its training data as possible, and that means that the lies are indistinguishable from the truth.

And that's a real challenge for all of us who are using it. There's information in front of you. Is it right? Is it wrong? And that's very different from, say, when you do a web search where we have built up tools and best practices and instincts for deciding whether information is good or not.

Richie Cotton: Absolutely.

I suppose as humans, yeah, you spend your whole life trying to work out, is this person telling me the truth or not? You should at least have some kind of context for this. But you mentioned this phrase, the shape of intelligence. I've not heard that before. Talk me through what does this mean?

Dan Klein: For example, if you look at when humans write, and that's a skill we have to develop. When humans write and they're trying to communicate We are used to understanding the output of that human process. So for example, if you, back in the day, went to go get a book from the library to learn about a topic, you would go to the library, you would look up books on that topic, and all of those books, the library actually chose to purchase them.

And for each of those books, a publisher chose to publish it. An author put in a lot of time building that book. An editor edited it, and there was this whole infrastructure of vetting information to make sure that by the time it got into your hands, you had a way of being confident in this book. It cited its sources, it went through this whole process.

Then search comes along and okay, now you can get any webpage back, and we all know that some webpages are more reliable than others, but we made it through with some of those instincts, and the reason why was because... Some of your audience might be familiar with this wonderful idea of code smells in software engineering, and this is this lovely idea introduced by Kent Beck in the '90s and popularized by Martin Fowler in the Refactoring book.

And it's this idea that when there's something deeply wrong with your code, some deep issue with the design, you can see surface indicators of that. Maybe the function signatures are getting too long or this is a sign of insufficient modularity. Errors, information errors, misinformation, these things have smells too.

So you go to some webpage and it's slow to load. There's typos. The images are missing. The design looks like it's from the '80s, and there's all of these signs. They're not always gonna be there and they're not perfect, but you have these indications that maybe this is not the most trustworthy thing. And it wasn't perfect, but it was something, and it matched relatively well with the longer tradition of instincts we have.

Now you go to a chat system and you ask a question and you get an answer, and it might be right and it might be wrong, and either way it's fluent, it's confident, it's telling you you asked a great question, so you feel good about it. And that's a real challenge because we don't have these tools to help push our digital literacy along.

Richie Cotton: That's really interesting, the idea of code smell. I suppose it goes back to thousands of years ago, humans developed this instinct where you need to know if a place is suspicious in a few seconds, like am I about to be eaten by a lion? And then code smells is a more modern version of this.

So is there an equivalent then for generative AI output, for agentic AI output where you can just go, "This AI is wrong"?

Dan Klein: You asked me at the very first question, what is the thing that bothers me most? The thing that bothers me most is by design, these systems hide the error smells.

So you take the core of it, and of course there's many stages after the core pre-training, but the core of it is it's seen a bunch of data, it's done autoregressive training to capture as many correlations as possible in that data. Which means that when it is generating information, it's doing it token by token, and whether that information is right or wrong, it's gonna be very plausible.

And you asked about the shape of human versus machine intelligence today. I would say one big axis where they're different is that humans are metacognitive. We think about our thinking. And so an example of this would be if I ask you, "What's the population of Berkeley?" Maybe you'll say, "I don't know."

Most people wouldn't know the answer to that. You know you don't have that information. You wouldn't just make a wild guess unless you were in some specific context, you're taking the SAT or something, and you've been told, "Guess." What does a system do? A system puts together tokens that captures as many correlations as possible.

It's gotta be a number. It's gotta be of this order of magnitude. And maybe it will turn out that it's seen this information and it reconstructs the information correctly, or maybe it just reconstructs information that isn't correct but is very close. And to the system, the difference between getting the right number and getting a number that's indistinguishable from right, there's no explicit representation of that, whereas we do have explicit thinking about information being present or absent, and that is a very different shape of an intelligence.

Richie Cotton: Absolutely, yeah. I hadn't really thought about that before, but, yeah, if you were to ask me that, I don't know what the answer is to how many people there are in Berkeley.

Dan Klein: I could make a guess. But that's the right answer. If you don't know, that's the right answer. We should absolutely demand of our systems that when they don't know, they fess up, as opposed to producing something plausible but wrong.

Richie Cotton: Okay. So I guess we're gonna try and figure out how to make that happen. But maybe to begin with, when is this important? When do you absolutely need the correct information to be there?

Dan Klein: The easy answer would be any time the AI's errors would have consequences. So people immediately think of things like banking, healthcare, cases where the system is making decisions that are gonna actually impact people's lives, where a mistake has a real cost.

And so certainly on one extreme are these critical applications, regulated industries, things like that. What I think is interesting is, at least in my opinion, AI's decisions almost always have consequences. We don't see, for example, a lot of enterprises going, "Good thing mistakes don't matter to my customers."

Usually if you're, whatever you're doing, you think it's important. And so if an AI is gonna come in, is going to augment or automate some of that work, you want it to get things right. And sometimes that's life or death stakes. But even when it's not, I would say any time an AI is taking an action, any time it's giving information that's important, reliability is so critical.

And it has not been the central design goal of the systems that have been deployed today.

Richie Cotton: Yeah. So I think certainly when you want a quantitative answer, it has to be the right number, otherwise, what's the point? And yeah, as you mentioned, if there's a consequence of getting the answer wrong, then you really need to make sure that it is correct there.

So how do you measure then how often you're giving the right answer? 'cause you mentioned if you've got plausible answers, it's sometimes difficult to tell. So is there a way to say, "It gets it right this many times"?

Dan Klein: Yeah, and it's really important to do, and it's getting more important to do. So I think of hallucinations and modelers in general, I think of them a bit like an iceberg.

There are the ones you see. We see examples of LLMs face-planting in the news. And if you've used these systems, you've seen hallucinations. But as systems get better and better, even the mistakes, as we talked about, start looking more and more plausible. For somebody to catch a hallucination, that's the tip of the iceberg.

That's the cases where the system was wrong. And somebody noticed. There are so many cases where the system is wrong, but it's subtle. It's either wrong but only a little wrong, or it's wrong in some subtle way that if you don't already know the answer, maybe you don't have a good way to tell. For example, you see examples out there of people trying to build customer service bots where it'll say, "Follow a refund policy," that isn't actually the company's refund policy, but it's a plausible refund policy.

And so there's this whole rest of the iceberg of cases where systems are constantly a little bit off, but it's hard to tell because it looks right. And if you interact with, say, a chat system in an area where you are an expert, you'll catch a lot of these. And so I'm always thinking in the back of my head, "I'm not an expert in this area that I'm asking an LLM a question about, but I know that there's a pretty high rate of subtle hallucinations or confusions or mismatches in areas where I am an expert.

I'm gonna assume that's true here." And I think having to keep up those instincts that it looks right, but maybe I should verify this, I think is something we're gonna really have to develop.

Richie Cotton: Absolutely. Yeah, I do like the idea that you need that instinct, the habit of checking stuff that's going there.

But yeah, when it's a rare case that it gets something wrong, that's tricky. So if it gets it wrong one time out of 100, then spotting that's gonna be more difficult. So how do you approach those rare hallucinations?

Dan Klein: So I don't think it's that rare. I don't think it's as rare as one in 100.

It may be one in 100 that people notice. When we go in, at Scale Cognition and we go into an enterprise and they're talking to us so we build high reliability models. If they're talking to us, it's because they're unhappy with their hallucination rate. We ask them what they think that is, and then we actually look at logs, and it's 5X, maybe more what they think, because most hallucinations do not rise to that level where people are noticing them.

They're still there, and they still have a cost. I think you asked how you measure. If you are deploying an AI-backed system, I think first of all, the important thing is that you do measure. That you have in your head that before I deploy this, after I deploy this, I'm gonna be constantly vigilant, constantly auditing.

This idea that aggressively pre-auditing things before you launch, a constant real-time monitoring once they're launched to make sure that the system is doing what you want it to do. That is so important. You can't just try a couple times, feel like it's fluent and natural, and it feels like a human, so its accuracy is gonna be like a human.

You have to test the heck out of these things. And so, for example, in our agent development tools, that test-driven, audit-driven development during development, before launch, after launch, is just a critical piece of what we offer and what everyone should be doing to make sure that everything doesn't just look plausible, but it's actually right.

Richie Cotton: Okay, yeah. So lots of testing. Who does that, though? So I think the first thing everyone comes up with, you need to keep a human in the loop. A human's gotta review all the output from AI. When is this a good idea and when's it a bad idea?

Dan Klein: As you say, it can be both. So certainly there are times when putting a human in the loop is critical, like when you've got a system where a human is ultimately the decision-maker.

There are some upsides to having humans in the loop that humans can... If humans are doing part of the process and you are taking those traces, you can use that to improve future systems. So there are a lot of times where you want humans in the loop or where that's a good solution. But there are a lot of times where you don't want that, and part of this gets into a human checking the work of an AI system is a different skill for the human than a human doing that work in the first place.

And typically, we have this idea that verification is easier than production. And so actually doing the work takes longer, it's harder. Just have the human check it. And there are definitely cases where verification is easier than doing the work in the first place. But it's also true that when humans see plausible completed work, that work may have mistakes that they won't catch as an editor, as a reviewer, but they also would never make that mistake.

It's very hard to jump into something completed, especially something that looks plausible and fluent, and catch the errors. And a good example of this is, say self-driving cars, where you might wanna hand control to a human if things get dicey. Humans take a reasonable amount of time to gain contextual awareness to the point where they can drive.

You're tracking all kinds of things, and you may make a good split-second decision, but you are making that decision with contextual awareness that takes time to build. So you can't just throw a person in at the last second and expect them to perform as well as if they had been in control all along.

Similarly, you can't expect somebody who has skills as a writer to instantly be an editor at the same level they were as a writer.

Richie Cotton: Just reading other people's reports and trying to figure out what's gone on there. Reading stuff from a human is tricky. Reading stuff from an AI is like, oof, it can be hard work figuring out what's gone wrong with it.

I

Dan Klein: think humans are so much better at revealing in our writing that we're not sure here or that we're confused, and the underlying confusion has a surface-level smell associated with it. Whereas the AI, you read it and you're like, "Wow, that was so fluent," but I'm not actually sure what that meant or if it meant anything, and that doesn't typically happen with people in the same way.

Richie Cotton: That seems fair. So if it's difficult for humans to review the output from AI, I guess the other thing that's been proposed a lot is having one AI check the other AI. So this is the idea of an LLM as a judge. So talk me through when is this a good idea or a bad idea?

Dan Klein: Yeah, my first reaction is, okay, so you have this noisy system that is unreliable and that's a problem.

So you bring in another noisy system to check it and to crib the famous Jamie Zawinski quote about regular expressions. Now you have two problems. And so now you've got two models to keep up, and you've gotta think about their interactions. And people love to have this idea that if you have two machine-learned models, one makes some mistakes and the other makes other mistakes, but those mistakes are independent and so together they work much better.

Decades of machine learning experience says machine-learned systems do not tend to make independent mistakes. When something's hard for the one model, it's gonna be hard for the other model. You're gonna have to reason about how they interact. What if the first one does the right thing and the second one breaks it?

There are other issues with doing this model checking a model thing, which is, for example, you can't do anything till the second model's completed, which means you've added latency, you've added tokens. In fact, a lot of the things that we would think to do to take an unreliably architected system and make it reliable end up having these sorts of performance costs.

So that's a real issue.

Richie Cotton: Having humans check the work of AIs is problematic. Having other AI check the work of AI, also problematic. What else can we do? How can you make things more reliable?

Dan Klein: There are times when humans, as you say, can be a good tool. You just have to know that even if a human would do the work well, they may not supervise the work well.

Systems checking systems is very fraught, especially when it starts to get very complicated, like these big constellations of systems that are very hard to reason about, and they're very slow and they burn a lot of tokens. There are other patterns that you can use to try to make things reliable.

One pattern that people use is just really limit the scope. You put some kind of guardrails on top of a system. For example, you often see people taking... In a way it's painful to see people taking these wonderfully broad LLMs and because they can't control them effectively, they'll build a system, say for a customer service system, where it's like an IVR system.

You just have a couple branching options. You've designed the conversation by hand, and the LLM is just there to do intent recognition, to decide which branch am I gonna go down. And this to me feels like model lobotomy. This is not using the potential and power of these models. It's shrinking them down to the point where they can be controlled.

Each of these has situations where you might wanna do that. I think what I would prefer to see us all doing as a field and what we do at Scale Cognition is go back and say, "All right, the problem here is essentially the underlying model is this Token-based model, it's noisy, it's unreliable, it's hard to control.

We talk a lot about hallucinations on the output side, but there's also the problem that what if you don't like its behavior? What are you gonna do? All you can really do is change the prompt. It's hard to reason about how a change in the prompt will change the downstream behavior. So we all have this experience of, okay, you reword the thing and it still doesn't do the right thing, so now you move that instruction to the end and it still doesn't do it right, and you put it in all caps and with an exclamation point and maybe another exclamation point.

Somewhere around the third exclamation point, you realize that maybe this isn't the way, and that's because it is not a control surface that has any kind of crisp semantics. So what I think we're gonna see increasingly in the future is architectures and even models that have, instead of the guardrails retrofitted onto them, that are built around these guardrails.

And I think that is, back to that first question about how intelligence is multifaceted, a system where reliability is the weakest of its facets is gonna be hard to control. If you build systems for reliability, you can maybe get on a different curve to begin with.

Richie Cotton: Okay. That's going pretty deep there, 'cause I think the approach has been for a lot of businesses, okay, yeah, I'm building a chatbot or whatever.

I take a standard issue- Mm ... general large language model and just- Mm-hmm ... say, "Okay, just talk about these specific business problems." Yeah. So what you're talking about is basically, I guess, is this throw out LLMs and start again with a different architecture? How deep do we have to go with a rebuild?

Dan Klein: So I think the deeper you go, the better the outcome will be, and that's our hypothesis at Scale Cognition. We build our own models, and they have different architectures and different performance characteristics as a result. But regardless of what you're gonna do, I think the more of the architecture you focus on reliability, the better, including things like just aggressive testing.

In terms of our approach, there are areas of AI where there has been a big increase in systems that turn out to have reliability guarantees. An example of that would be in math or in coding, where, for example, the system may be noisy, it may generate a bunch of incorrect proofs, but as soon as it stumbles upon a correct one, Lean can tell you that, yes, this one's right.

And this tradition goes back to reinforcement learning things like games, for example. So if you want to build a good system that's good at chess or Go, you know the rules of the game, you know who wins in the end. And so even if you don't know how to play yet, you can just play yourself or play at random even, and then you see which side won, and you double down on that.

Reinforcement learning, in a nutshell, is just trying a lot of things and doubling down on the ones that worked. That is most powerful when you can tell what worked. In chess and Go, you can tell who won in the end. In theorem proving, you can tell when you finally stumble on a proof that Lean will accept.

In coding, you can tell when you finally made a change that passes your unit tests. And so I think this idea of verifiable AI in general is very powerful. So we've taken this approach of, first of all, training our model on data which is generated through a verifiable process. Then the model itself, you mentioned how much do you have to go back to the drawing board.

I think there are inherent challenges with models that only have prompts going in and tokens coming out, because tokens don't have semantics. They aren't objects that mean anything until they've been assembled. So you're cranking out a couple digits at a time, and then at the end you're claiming this is the account balance or the population of Berkeley.

The pieces you put together don't really mean anything. It's only once it's assembled that you can even start to bring in a model to check it or something. So our approach instead is to do, there are different names you might use. You could talk about action models, you can talk about structured neural models, but our models operate over pieces of information and actions, APIs, as first-order objects.

And because they are operating over objects that have semantics, that gives you a way to architect around the requirements you have.

Richie Cotton: Wow. So I think basically all the models that are popular at the moment, they're based on this unit of the token. Mm-hmm. So this seems like a fairly fundamental reimagining of things.

Can you just give me an example of some of those things you mentioned? So you mentioned an action or a piece of information as a first-class object. Can you give examples of these?

Dan Klein: Yeah. So—

What we're talking about here is agentic interactions and what exactly is that? One way to think about an agentic interaction, or at least a certain class of them, is that it's a conversation where you've got a human on one side, and humans are gonna speak human. They're gonna speak in natural language, whether that's text or voice, and it's gonna be contextual, and it's gonna have all the properties that human language has, and of course, contextual understanding of those things is something that we understand really well now as a field and we can extract really well from data, from web pages and things like that.

On the other side of that conversation is a bunch of APIs, a bunch of policies, a bunch of pieces of information flowing through those, and that does not look really like a web page. It doesn't look like human conversation. And so I would say that's really the key place where you want to think not in terms of little pieces of words being put together, but rather these objects that you're building up to do a sequence of actions that will accomplish somebody's goal.

Because ultimately, if you don't have the ability to really understand sequences of actions, information flowing through them, and those larger, more sophisticated computations, if you don't have the ability to understand that, you're really gonna be stuck with systems that are gonna tend to be more like question answering systems.

You mentioned RAG at the beginning. Systems that will give you token-based human language answers to your human language questions, which certainly can be useful, but ultimately I think we want our systems to actually take actions.

Richie Cotton: Okay, yeah. And certainly if you're dealing with agents, then an action ought to be represented there somewhere.

Dan Klein: Exactly.

Richie Cotton: It sounds a little bit like the idea behind world models where you've got some physics or underlying real world structures built into the model. Is there a relationship there?

Dan Klein: There is. Actually, on the Berkeley side, my academic group has done research on world models as well.

I would say the big difference is when you say world models, and none of these things are completely crisply described, and these ideas overlap. When does something stop becoming a world model? When does it start? But to me, the core of a world model, if you're thinking about... You're talking about models that are focused on perception and the dynamics of the world, the physics as you mentioned, and that aspect, and that is a great example of an architecture that is focused directly on this embodied context.

This is a little different in the sense that it's not the physics of the real world. It's not mechanics. It's not anything like that. It's not geometry. It's the physics of computation and how information flows through a computational system, and how sequences of actions are put together to accomplish larger goals.

Richie Cotton: Okay. So you've got this structure and, I guess, is it... I'm not sure what goes into it. Sequences of actions are coming out. Are the inputs different then, or is it same as an LLM where it's, I guess, scrape the whole internet and—

Dan Klein: Mm-hmm ...

Richie Cotton: spit something else out?

Dan Klein: You're talking about what we use to train?

Richie Cotton: Or, yeah, or maybe- Yeah ... even if you're trying to just interact with the model, I guess at the inference stage, what goes into it?

Dan Klein: Yeah. Okay, so to the training question, we generate training data through reinforcement learning, through simulation, and the trick there, in order to make this work well, is to be able to do that in a way that's verifiable.

If agentic conversations were like Go, we would know exactly how to do that from the beginning, but it's something that we had to spend a lot of time doing research on to make that a verifiable domain. When you actually then deploy the system at inference time, because it's the two sides, there is a control service that's basically the same as a prompt where you can put natural language instruction in, but there's also a control service that looks more like how you register tools except richer.

And so it's, again, inherently this two-sided system because there's a human interaction, there's also a code and action side to the interaction, and those have the commensurate control structures as well.

Richie Cotton: Okay. And from this, we talked originally about improving reliability of- Mm-hmm

of models. Is that the main goal around this, or are there other benefits to this new model structure as well?

Dan Klein: That was certainly the mission that got me started. So even going back, go back three years, we have large language models today. I was doing research on language models when they only had the one L.

They were not large yet. And so understanding what information goes into a language model, what these systems can capture under what kinds of representations, it was just very clear to me that their ability to capture a broad range of language and knowledge correlations was going to greatly outstrip their ability to act in a reliable way.

And so at the beginning, our idea at Scale Cognition was, let's go take a step back. Let's think about the cases where reliability is really going to be the most important thing to design for, because intelligence without reliability is gonna be limited. And so my mission personally is to build systems you can trust, systems you can control, systems that you can actually start to say things about what they will and sometimes more importantly will never do.

And I think that's the goal. That's what started us down this road. However, it did also turn out that the models that we built pursuant to this had other nice properties, and a lot of this is because essentially they are doing exactly what they were designed to do, rather than being coerced into doing something that they don't fit particularly well with.

Some of those benefits are they are faster, they're able, for a given amount of intelligent behavior, they're able to be much smaller. And so inference costs are much lower. The kinds of compute requirements are lower. They have other advantages in terms of how we build them, in terms of data sovereignty.

And so it just turned out there was this nice alignment between building models that were specialized towards reliable agentic interactions, and a bunch of side benefits that we didn't go in as focused on, but it's turned out our customers find very important. Particularly things like low-cost compute and very low latency.

And part of that is when you don't need to wait for a second model to check your first model, that's a huge advantage in speed.

Richie Cotton: So even if the original model maybe, it doesn't need to be as performed if you're not having to have several layers of checks on top of it, that's gonna give you an overall speed and cost advantage.

Okay. Exactly. So suppose I'm an AI builder, I'm like, "Okay, I need to create an agent that's reliable." Mm-hmm. What do I have to do differently now?

Dan Klein: It depends on the tools you're working with. I would say the first thing you start with is you say, get into your mindset that the important thing is going to be reliability, and that you cannot confuse fluency or flexibility of the interaction with reliability.

So you go in, you're gonna pick all of your tools for reliability first, then you're gonna be very test-driven. You're gonna wanna make sure that you know what you want your system to do, and this is actually harder than you might think because a lot of times people in the context of building, say, a customer support- But they may not actually know what the humans currently doing the task, what they're doing, what they're doing well, what they're having trouble with, what tools they're using.

There's a lot to understand about what you're trying to do, and are you trying to just mimic that? Are you trying to do something larger than that? Are you trying to build a system that handles a certain thing and then hands off to a human? These are very different designs. So reliability first, test before you release, monitor after you release, and you need good tools for all of these things.

We are obviously very partial to our models because they give you the ability to get rid of a lot of the structures you would otherwise have to retrofit on. You can think of it in a nutshell as we have the guardrails in the model, so you don't have to build them on top of the model, and that means fewer things to build and higher performance coming out.

But I think critically just going in with that mindset about reliability testing, and finding the right tools that will let you do that, and not just do it once, but do it in an ongoing way, because as you change your mind, as you wanna grow the system, as you detect behaviors you wanna change, you need to be able to go back and actually have the system be responsive

Richie Cotton: Okay.

So it seems like building testing into the workflow seems essential. So I know in software development, the idea of a test-driven development where you write the test first- Mm-hmm ... is very common. Is that the approach you would take here?

Dan Klein: Yeah, I think the notion of tests is gonna depend on, again, the evaluation tools you're using.

So in our system, we give this very rich ability for people to provide scenarios and then run those scenarios and make sure that the outcomes then get verified to be the outcomes that they're looking for. This is actually very important. Maybe even, I don't wanna say it's more important than it is with coding, 'cause it's very important for coding too.

But one of the things that we've seen is that when you are building large model-backed systems, you make some change, and it is very hard to predict what regressions might happen if you're just, say a lot of people out there build to one model, then they swap in the next version of that model.

Some things get better, some things get worse. Maybe the things that get worse aren't even things you were testing for. And so I think you have to be very intentional about getting those tests right, getting those scenarios right, having tools that can reliably reproduce them. If you're just typing in a couple conversations in by hand and doing a sniff test, it's not necessarily gonna go that well in deployment when you hit real customers.

So I think you want tools that simulate those customers very well, tools that in deployment audit what's happening in the live deployment, and then tools that let you come back and close the control loop and change behaviors if you see things you don't want.

Richie Cotton: Oh my... That feels like quite a lot of infrastructure that's needed then just to test that agentic stuff is working correctly. Do you wanna talk me- Yeah ... through what that tech stack consists of?

Dan Klein: Yeah. So I would say that

At Scale Cognition, our offering really is designed around what we think that stack should look like. And so what does that mean? So at the core, obviously, a lot of our research is around APT, our model, and the rest of the stack takes advantage of the properties that model has from its speed to its reliable, guaranteeable, verifiable aspects But it's also very important that you have this way of building up agents.

Now, you can do that in more pro code ways, more low or no code ways. We have technology that will look at what agents are doing today and surface that, and build agents that will replicate that behavior. But we also give tools for people to go in and really look at what has been discovered so they can go in and change it.

I don't think it's enough to be like, "Oh, do what those people are doing," 'cause there'll be contradictions, there'll be cases where things change. What was happening at the time of that transcript isn't even the right policy anymore. And so there's spinning up your model, we make that fast, but then there's testing it, there's deploying it, there's monitoring it, and real-time monitoring.

And then being able to feed that real-time monitoring back to continuous development. And I think one of the things that we're fortunate to have is that the same tools that we had to build to be able to generate synthetic training data, they actually look very much like the auditing tools you need to make sure that the system is performing the way you want it to.

And so we put all that time into building those tools, and then we found actually, you know what? This is a really critical part of not just training a model, but in authoring an agentic system.

Richie Cotton: That's interesting that the tools for assessing the performance of these agents or these models is useful for obviously creating better models.

So using this as the training stage, but also useful for inference for people who are using these as well. Okay. That's right. And in terms of, if you want to use this model, your APT model, Hmm ... can you just swap that in if you started with an LLM? Is it just a case of swap out the model being used, or do you need to do more?

Dan Klein: There's no... It depends what you're doing right now. There's no one format. There are several standards for how agentic systems look and how you specify them, and how you register APIs. We do support all of that, and you can just swap things in. Our model, because it has a different structure, also has additional control surfaces that you can use as well in order to essentially specify what the guardrails should be.

And so I think to get the full mileage out of the model, you wanna be using all of its control surfaces, but you absolutely can just swap it in and leave those blank, and you'll still get great performance.

Richie Cotton: Okay. So swap the model and then I guess there's all those extra tooling around just making sure the- Yeah performance and monitoring's in place. Yep. Okay. And for people who are interested in building reliable agents, what kind of skills do you need?

I think that's gonna be evolving. So I think if you are using models that do not have the performance characteristics that you need, you're gonna spend a lot of time adding superstructure to them to add those performance characteristics, and that might mean building the second model to judge the first model, or it might mean chaining together smaller calls so that each one is of a more limited scope.

And I think when you're doing that, it's gonna involve skills about really understanding these cascades of systems. I don't actually think that's the right way forward. I think those systems become very complicated. And if you think about, okay, so I teach computer science classes and if you take a CS 101 class, the single most important lesson we teach about how we build large systems using teams of people and maintain them reliably is modularity.

That you take a big system and you decompose it into small pieces, and those pieces have contracts where if this input comes in, this output will come out. And that lets us reason about a whole based on its pieces. And so it's very tempting to take this many sub-pieces approach to building a reliable agent.

But that's actually very hard because each piece doesn't really obey much of a contract. The LLM as a judge sometimes judges this thing and sometimes

Richie Cotton: it doesn't, and then the underlying system sometimes does this.

Dan Klein: and so I think this is a very challenging pattern to make work. But certainly being able to think about, I don't trust the individual pieces, how can I build a defensive constellation around that to mitigate it? That is one set of skills. I think then the other set of skills is gonna become more relevant as models that have performance characteristics more like ours come online, which is, all right, how do I think about different models? How do I think about what they do and don't guarantee? And what implications does that have for the system that I'm building in the end?

And trying to pick models that best align with what you need out of your system.

Richie Cotton: So it sounds like a lot of the effort then is just in terms of understanding the whole architecture around the model. Is this just like an... agents are an awkward growing up stage where they work at the moment?

And is this gonna get better, or do you think agents are always gonna be like a complex thing to reason about?

Dan Klein: I think it is gonna get better. I do think there is an extent to which technology moves in these super cycles, and whenever there is a new technology, it tends to look like you have some complicated set of things that are working, but they're complex and they've hit a limit.

And then some new change technology comes out, some innovation, and typically then this new technology will displace a lot of that complexity, and the new technology will be simpler. So you have, say, this one LLM that can do many things, and it displaces a whole bunch of complicated things. And then people push that, and eventually they start to realize that there are other pieces to the puzzle, and really to get the maximum out of it, we need supplemental technologies.

I would say right now, agent development is particularly complex because the underlying models are not incredibly well-suited to the reliability requirements, and that mismatch is giving rise to complexity. I think new models that are more reliability-focused are gonna simplify that picture, but the reality is there is always gonna be supplemental structure as well for monitoring, for authoring, for really the whole process of deploying an agent.

But I do think the modeling piece is gonna get simpler. I think fundamentally, taking... building reliable systems out of low reliability pieces has always been a pattern that results in complexity, and if you can do it with high reliability pieces, you just get systems that are more simple and more performant.

Richie Cotton: Okay. That... It is interesting the idea that you said it's like an explosion of tools when you get new technology and eventually it simplifies. We saw this with data engineering a few years ago. There was tons and tons of data engineering tools around, and now the stack is a little simpler. It's a bit easier to reason about, so yeah, maybe...

Hopefully the same pattern follows with agents in the next few years. Okay. All right. Actually, before we finish, there's one other thing I wanted to ask you about- Sure ... which is around agent benchmarks. I know some of your- Okay ... colleagues at Berkeley have just come out with, I think it's called the Agent Celeste Exam.

It's like a benchmark- Mm-hmm ... for agents. Mm-hmm. Have you been involved in any of this work? Do you look at agent benchmarking?

Dan Klein: I'm not involved in that particular effort, but we think about benchmarks all the time. Benchmarks are really... I don't know. I guess the only thing worse than having the benchmarks we have is not having them.

It's very important to have external measures of progress that you can optimize to. The problem is it's very hard to get those right, and it's very easy for benchmarks to measure the wrong thing, either intentionally because of who's released that benchmark only really wanting to measure the thing that they care about or puts them in a good light, or I think even more often unintentionally, people try to come up with a benchmark that measures what you really care about, but they miss, and maybe they didn't even realize why they missed.

And so you see this pattern where benchmarks come out, people get excited, there'll be eventually some rapid progress on the benchmark, and people saturate it and move on. Part of that is, I think there's a lot of work just... The benchmarks aren't that big. They're not constantly evolving, and so people tend to, not maybe overfit in the extreme machine learning taboo sense of putting the tests into their training.

That would be really, really bad. Everybody knows hopefully not to do that. But there are subtle ways in which people can design their systems, their prompts really for a specific benchmark. And so usually, at the very beginning, you can get some value out of a benchmark, but as time goes on, benchmarks very quickly turn out to measure the wrong thing.

So I just think while it's important to have benchmarks, I think people always need to have a little skepticism about what is this benchmark, not what does it say it measures, not what did the people who built it intend to measure, but what is it today actually measuring? And that's to say nothing of the fact that large models are eventually gonna crawl, if not the benchmark itself, then discussions of the benchmark.

And so these benchmarks decay on their own.

Richie Cotton: Yeah. I like the idea that you should be a bit skeptical about them- Mm ... 'cause they're useful, but maybe, yeah, they do get gamed eventually. Okay. All right. So I always want more people to learn from, so tell me whose work are you interested in at the moment?

Dan Klein: Yeah. That's ultimately a question about what I'm excited about. I don't know, it's... When people ask me what do I do, what do I think about in AI, that answer's really changed if I can take a step back and answer it. I'm excited today about the same things that got me into AI in the first place.

All of the ways in which having intelligent tools could make our lives better. So I love learning human languages, but no one can learn them all, and so machine translation lets you just see through these language boundaries and just tear down those walls.

I'm a teacher. I love to teach. I love to learn. Figuring out what's exactly the right explanation or the right next problem for this student is just this incredible experience. But there aren't enough teachers. We're spread so thin, and if we could have tools that could make that more efficient or even do those sorts of things that humans do but where we don't have enough bandwidth for everybody to benefit from them.

I think about these things, and I think just how much of a positive impact AI could have if it could take the best of what we do and amplify it. But that all comes with the flip side. Maybe nobody learns languages anymore 'cause after machine translation it all looks like their language or maybe nobody learns anything anymore because you've got this oracle that'll give you a seemingly true answer to any question.

And increasingly the thing that I am paying more and more attention to, and you asked about recommendations for topics and areas that I find really exciting. Not to end this on too much of a philosophical note, but these questions of digital literacy, what happens to society when we have an oracle that always sounds true, but sometimes it is and sometimes it isn't, and isn't very good at crediting its sources.

So many things we could worry about, like do people start believing everything? Do people start believing nothing? Do people lose this idea that information comes from somewhere? Because if you lose the information that information comes from somewhere, people stop having an incentive to create that information.

We wouldn't have the web that powers all of this knowledge if people didn't have an incentive to put up web pages. That incentive was that people would read them, not that an AI would come along, slurp it up, and hide their contribution. And so I think more and more about-- And I'm reading more and more people's work on digital literacy, on AI safety, on making sure that AI comes out, that AI's better nature comes out ahead.

Richie Cotton: Yeah, it's very much a double-edged sword there. I think you just hit the nail on the head with a lot of fears people have that there's so much can go wrong when, yeah, people don't read things, people don't learn things, and, yeah, you've no idea whether something's true. So definitely, a nice AI dystopia there just to finish the episode.

But yeah, on the other side, yeah, there's a lot of positive stuff can come out of this and come together.

Dan Klein: And I think that's really the question for all of us. There is this dystopia, and there's also really good futures where this tool makes life better, and we get to decide what technologies get built.

We get to decide how culturally we interact with those. Obviously, some of that is technologists doing research, building products. Some of that is all of us who are users making it clear what is our preference. Do we just want a system that's gonna tell us that our question cut to the heart of the matter and is the perfect question, and here's your answer?

Or do we want systems that are gonna say, "Well, here's what I think you're looking for, but I'm not sure because of this." What do we want? And we get to express not just what are the technologies gonna be, but what are we culturally gonna demand from these systems. And that's gonna make a big difference as to what path this takes.

Richie Cotton: Absolutely. Okay, so I guess fingers crossed we get to the utopia rather than the alternative. But-

Dan Klein: It's in our hands.

Richie Cotton: Yeah. Nice. All right. Super. Thank you so much for your time, Dan.

Dan Klein: Thank you. It was great. Thanks for having me on.