[RADAR 11x] Don't Waste Your Time on AI Pilots

Summary

Most AI pilots never reach production, and the reason is rarely the model.

At RADAR 11x, Krishnan Hariharan (CTO and VP of Engineering for Forge at Honeywell), Satesh Kumar Sonti (Principal Specialist Solutions Architect at AWS), and Laurent Gil (President and co-founder of Cast.ai) compared notes on what actually separates a working AI system from a demo that quietly dies after the applause. Their answers converged on the unglamorous parts of the stack: data and semantic layers that are accurate and portable, security and governance brought in before deployment instead of after, and cost controls built around model routing rather than a single expensive default. Sonti described customers loading markdown-based semantic layers into Postgres specifically so they are not locked into one vendor's stack. Gil shared router data showing 94% of coding-agent tokens at his company now go to open-source or open-weight models, cutting cost roughly 3.4 times versus commercial models, with no human choosing which model to use. Hariharan argued that culture changes only when business teams see AI prototypes tested directly with real customers, not when engineering declares a use case done. All three agreed that the organizations getting this right treat the business outcome, not the technology, as the starting point, and they size the human's role in a workflow before deciding where AI belongs in it at all.

Key Takeaways

• Data quality problems that once produced a bad report now produce hallucinations, so semantic layers and metadata need to be accurate and current before agents run against them at scale.
• Storing a semantic layer in a portable format, such as markdown files in Postgres, protects a team from being locked into one AI vendor's technology as the field keeps shifting.
• Security teams need to be included while guardrails are being designed, not brought in after a model is already in production.
• Model routing that sends each task to the cheapest model capable of the right accuracy can cut inference cost by more than 3 times compared to defaulting to one premium model.
• A deterministic workflow should be the default, and AI should be reserved for the steps where a deterministic path genuinely is not available.
• Teams that test prototypes with real customers before committing to a direction build more organizational trust than teams that polish a solution in isolation first.
• Judging an AI pilot's value only at the moment of the production decision is too late. ROI should be measured earlier, in a test environment with a real customer.
• Scale comes from reuse of prompt templates, data products, and shared building blocks, not from building a new model for every use case.

Deep Dives

Data and Semantic Foundations Come Before the Agent

Sonti opened with a customer story that ran counter to his own expectations. A large US telecom company had moved its business glossaries and transaction data into markdown files, then loaded those files into a Postgres database to serve as its semantic layer. "I got really surprised when I heard about Postgres SQL as the storage for semantic layer, which is bit unconventional," Sonti said, but the customer's architect had a clear reason: portability. If a new technology arrived, the markdown files could move with minimal rework. "Think portability. Don't lock into a specific technology which you cannot move, because this space is moving very, very fast," Sonti said. "You'll be surprised, probably in a month or so, you may get some new thing that excites you."

That same foundation determines whether an agent is useful or dangerous. In traditional analytics, bad data quality shows up as a wrong number in a report that someone can catch and correct. Generative systems fail differently. "In agent [world], if your data quality is low, what you get is hallucinations," Sonti said. Unlabeled tables and undocumented fields, the kind every data warehouse accumulates over time, become a direct liability once an agent is reasoning over them rather than a person running a query. Getting metadata and semantics "grounded and ready," as Sonti put it, is "priority zero" before any agent is deployed against enterprise data at scale.

Hariharan's version of this problem looks physical rather than digital. At Honeywell, a building is "nothing but a bunch of sensors and data points," he said, from a thermostat in a single room to the chillers moving air through an entire facility. The use case his team has invested in most is tying those sensor feeds together to optimize energy use, because the site manager "cares if the cost goes up or the cost goes down while the building is operating effectively and efficiently." The lesson across both examples is the same: an agent is only as reliable as the data layer underneath it, and that layer needs to be built, documented, and kept portable before the agent gets built on top of it.

Security Has to Be in the Room Before Deployment, Not After

Hariharan pushed back gently on the idea that technology is usually the limiting factor. "I've always maintained technology can solve every problem," he said, but the harder constraint is organizational: getting security teams involved while guardrails are still being designed, not after a model has already shipped. "Before you deploy it, you've got to bring in these security folks in your company along for the ride," he said. The guardrails a team defines, around workflow, data handling, and hallucination risk, often turn out to collide with existing security policy only once deployment is underway, and security policy cannot be expected to change at the same pace the AI team is moving.

Gil told a story that captured the gap between policy and practice. A CIO at a European bank told him that using Claude was forbidden at her company for GDPR reasons. The next day, Gil asked her developers what they were coding with. "Well, of course, we use [Claude]," they told him. "Why? Do you mean?" The real fix, Gil argued, is not tighter restriction but clearer permission: "The job of this chief security officer is to say, here it is. This is the thing you can use. Go ahead and use it as much as you want, for as long as you want, for everything you can think of, with no limitation." Shadow AI usage, he said, fills the vacuum left by unclear policy regardless of what the policy says.

Gil's sharpest point reframed what security in agent-driven coding actually means. "Security in using coding agents has nothing to do with the model you use, but the freedom you give the agent to do their job," he said, adding a second piece of advice: never run a coding agent from a personal laptop, since the agent then inherits access to everything on that machine. Guardrails, in other words, are a scope-of-access problem first and a model-selection problem second.

Cost Control Runs Through Smart Model Routing

All three panelists treated cost as a first-order design constraint, not an afterthought. Gil's data was the most concrete. Cast.ai began rolling out an autonomous model router to clients in May, one that decides on its own which model handles each piece of a coding task, with no human in the loop choosing a provider. "As of September 8th... 94% of the tokens were consumed by an open source or an open weight model. Only 4% went to Anthropic, and less than 1% went to OpenAI," Gil said. "And we did not influence the router." The router's own feedback loop tracks accuracy against cost and continuously re-optimizes, learning only from each organization's own usage rather than pooling data across clients. The payoff: "the cost of 94% open source or open weight is 3.4 times lower than the 4% remaining, which goes to Anthropic," he said. "That's mind-blowing."

Gil pointed to a new industry effort, a Linux Foundation standard called FOCUS, aimed at measuring total project cost rather than per-token cost, since token price alone misses how much an outcome actually costs to produce. Sonti made a complementary case for restraint: "We have the habit of jumping [to] technology ahead of the outcome," he said. "If there is a very deterministic path, you can solve that with workflows, the traditional workflows... Don't apply AI everywhere." Reserving generative AI for genuinely non-deterministic steps, and using conventional logic everywhere else, is itself a cost lever.

Hariharan's approach starts from the unit economics of a single request. He breaks a process down to its individual steps, prices the cost of each one (the data pull, the model call, the response), then multiplies by expected request volume and concurrency to model total cost before committing to an architecture. "Customers are not paying you extra because you use AI," he said. Whatever the stack costs to run, the business still has to price the product as if the underlying technology were invisible to the customer.

Culture Changes When Business Teams See Results, Not Promises

Sonti framed the cultural question in terms of discipline. Teams excited by new capability can lose the thread connecting their work to a business result. "Tie your investment to a business outcome," he said. "As long as you have that mapping very, very clear, then the culture automatically starts shaping up... your revenues will grow and you will see positive outcomes." Spend without a clear line to a metric, he warned, erodes trust inside the organization rather than building it.

Hariharan's method for building that trust is concrete rather than persuasive. Rather than arguing internally about AI's potential, his team signs up "lighthouse customers," builds a working prototype, and sits with that customer and their real data to see what actually happens. "Trust me, business, sales, finance, they'll all be amazed to see the feedback they get," he said. The exercise sometimes creates more engineering work, not less, because real customer environments expose gaps a demo never would. "But that's okay," he said. "As long as the customer benefits, the culture will change."

Hariharan also pushed back on a default mental model many teams reach for. "We've all been anchored to the chatbot," he said. "Everybody thinks I'm gonna build a chatbot, put a [ranked] model, it's gonna give me the answer. That's not the way to think about changing culture." His alternative: map the process flow first, decide honestly whether the step needs AI or whether a deterministic or even simple mathematical model would do, and identify exactly where a human has to make the final call. "That's where you need to pull in the human," he said. "Everything else can be automated... but the decision making cannot be given to the agent, because we're not there yet."

Gil's version of cultural change centers on removing friction rather than adding process. He compared coding agents to the shift from travel agents to online booking: once people saw the agents in buying plane tickets directly, nobody went back to the old way, and more people ended up booking tickets, not fewer. He argued leaders should stop treating restriction as the safe default. "You have to eliminate absolutely all these bureaucratic barriers," he said, pairing freedom to use the tools with the governance layer running underneath, rather than a governance layer standing in front of the tools.

Knowing When a Pilot Is Actually Ready for Production

Asked directly how a team knows an AI pilot is ready to scale, Hariharan rejected the framing. "If you're deciding the value of [an] agent at the time of production, you're already too late," he said. His order of operations: deploy into a test environment, take it to a real customer, measure the ROI there, and only then decide whether production makes sense. By the time a team is formally evaluating go-live readiness, the real test should already be behind them.

Sonti pointed to two recurring causes of post-launch surprise. First, a deployment can be technically impressive and still fail if it is not tied to a business outcome, since "the usability goes down" once users notice the tool isn't moving a metric they care about. Second, teams often discover their foundations were never solid in the first place. "You will realize that your foundations are not strong sometimes, after you deploy [into] production," Sonti said. "Be it data quality, be it your semantic layer, be it your context layer... business outcome and your foundations. These are the two, if you ask me."

On scaling past a single successful pilot, Hariharan and Sonti offered near-opposite but complementary advice. Sonti's instinct is toward reuse: "Scale comes from reuse, not building many, many models, many, many offerings," he said, pointing to reusable prompt templates and data products as the actual unit of scale. Hariharan's instinct is toward deliberate redundancy early on: "Scale comes from trying out 10 different things," he said. "Don't take one use case and just implement one agent. Do it three different ways, and then you know which one works best." Read together, the two views describe a sequence rather than a contradiction: explore several approaches to find the one that works, then standardize and reuse that approach everywhere it applies.