LLM Observability: 6 Lessons From Datadog's CTO

Lesson 1: AI Broke the Old Way of Reviewing Code

Let's start with the pressure that sets everything else off: there is more code than anyone can read.

Lê-Quôc is blunt that the old model, a human reading a pull request line by line, does not survive contact with AI-assisted development. The anxiety he hears across the industry is about reviews becoming impossible, because there is too much going on to follow by reading PRs.

His response is not to ask people to read faster, but to move the review somewhere else.

That last clause is easy to miss. Once you orchestrate one agent to plan, another to write, and another to test, you also have to stop the writer from gaming the automated tests instead of solving the problem.

He goes beyond tests. Datadog now adds semi-formal and formal proofs that a spec does what it should, something too taxing to attempt widely before agents took on the heavy lifting. It works best on backend and coordination systems, where the behavior is mathematical enough to reason about precisely.

Lesson 2: Production Is the Only Test That Counts

Passing every test in CI is necessary and nowhere near sufficient. The failures that matter happen later.

Every release rests on assumptions you can't fully check beforehand, about the shape of the data and how users behave. Hold those assumptions up to enough real traffic, and the rare cases stop being rare; they become the everyday slowdowns and errors of data and model drift.

LLMs make this harder: With ordinary code, you can at least reason through every branch, but no one can explain mechanistically why a model returns what it returns, so the same input is never guaranteed to give the same output. The occasional strange result can't be engineered away.

So you stop trying to prove a system correct before it ships. Instead, you

• Write evaluations for the behavior you want
• Monitor it in production
• Keep a stop control for a rollout that turns bad.

The question is no longer whether it passed, but whether a problem is a one-off or the start of a trend.

That live signal is not just a dashboard for humans. Wired into the deployment system, it lets an agent roll a change out the way a careful engineer would, to one percent of users, then five, judging from real data whether the change is doing what was intended.

Lesson 3: Let Agents Take the Toil

Lê-Quôc's case for agents is not that they replace engineers, but that they take the parts of the job that wear people down.

Troubleshooting an incident means throwing hypotheses at a symptom, and on long incidents, it is often a far-fetched one that proves true. Datadog's Bits AI agent checks them all in parallel, ahead of the engineer, while the person steers it toward the hunch a dashboard would never surface.

The deeper point is fatigue. An on-call rollout is sudden alertness followed by hours of nothing, repeated until your judgment frays.

An agent does not mind, and it does not get worse after four hours of staring at numbers. Stress and fatigue degrade human performance, which is why teams rotate people through on-call in the first place.

Hand the tireless watching to a machine, and people come back rested for the calls that need them. The same logic covers security triage, where analysts burn out sorting false positives from real threats.

Lesson 4: Split the Work Into Two Loops

Lê-Quôc organizes Datadog's agent work around two loops.

The development loop

Most engineers will recognize the first loop:

1. Write code
2. Ship it
3. See if it works
4. Fix it
5. Repeat

Datadog's angle is that a problem originating in code usually has its fix in code, so the platform tries to hand you that fix, informed by what it knows about the application: its ownership, its recent changes, and the errors it has thrown.

He points to database query optimization as an example. Any model can rewrite a slow query; the harder part is proving the rewrite is faster and safe before it reaches production, so Datadog tests it against a realistic copy of the production data first and hands over a pull request with the evidence attached.

The operations and security loop

The other loop runs in parallel, either by the same people or a different team:

1. Detect
2. Investigate
3. Fix
4. Repeat

This is where Datadog's AI Guard triages security events and blocks attacks faster than an analyst working through them by hand. Agents can also handle routine operational chores that engineers do daily without much enthusiasm, like resizing that one Kubernetes pod.

Across both loops, Lê-Quôc is firm about the order of operations. Datadog does not start from "here is AI, what problem can it solve?" It starts from a problem customers already complain about, usually some version of "I don't want to do this repetitive thing", and works back to whether an agent can be trusted with it.

Lesson 5: Keep AI Spend in Check

Cost is the constraint sitting next to safety, and keeping the price of operationalizing large language models in check is becoming its own discipline. Lê-Quôc's answer at DASH is Datadog's Agent Console.

Ask a developer which model they need, and often they will name the most powerful (and expensive) one. Sometimes that's the right choice, but a lot of work is boilerplate that a cheaper, faster model handles just as well. Telling the two apart means reading the trajectories of an organization's agents, which tools they call, and how often they succeed, until patterns appear.

Those patterns become heuristics rather than rules: a frontier model like the latest Claude Opus or GPT models for planning, something cheap like Claude Haiku for generating tests.

The principle underneath is about who owns the decision. Roll cost up to a single number, and you get what Lê-Quôc calls "very low actionability": either everyone stops spending, which kills useful work, or everyone keeps spending, which the business can't sustain. He would rather put the data in front of the developers and SREs who choose the models.

Lesson 6: Learn How to Learn

Asked what new engineers should study, Lê-Quôc gives an answer that sounds old and isn't.

Models are the most patient tutors ever invented, able to explain anything at any pace, a level of access that used to be reserved for royalty with private teachers. But a tutor is only useful if you interrogate it. The skill is knowing what to ask and how to check the answer.

He recommends understanding computers layer by layer instead of treating them as magic. Take a scheduler, a load balancer, a sandbox, and ask a model to explain how it works, then keep pushing:

• What does this term mean?
• How do you measure it?
• What is the math behind it?
• How do you know it is working well?

Studying the classics this way is slow on purpose. He compares it to learning an instrument; you can listen to music all day, but to play piano, you have to put your hands on the keys.

The same goes for AI-written code. Vibe coding is fine, he says, as long as you come back and ask why it worked: why was it built this way, are there better approaches, what was it modeled on. The aim is not to write less code with AI. It is to understand the code you now produce so much more of.

Final Thoughts

Lê-Quôc's central message is that the loop hasn't changed, but the pace has. What is different is that no human can watch closely enough at the speed AI now moves, so the watching, and a growing share of the building, moves to agents that don't tire and don't panic.

He argues for treating observability as a control plane rather than a set of charts. If agents are going to write, test, ship, and operate software, they need the same grounding in real production data that good engineers rely on, plus a person holding the judgment calls and the stop button. Datadog is positioning observability as the layer that makes that trade safe.

The skill this framing asks of engineers is clear: read systems through how they behave in production, not just through their source. If you want to build that habit, our Machine Learning in Production skill track is a good place to start.