跳至主要内容

The Secrets of Deploying AI in Production with Sumti Jairath, Chief Architect at SambaNova

Richie and Sumti explore why Al agents are slow and expensive, the architecture behind faster inference, open-source models and sovereign Al, the full Al infrastructure stack, the Al data center debate, career paths in Al infrastructure, and much more.
· 47分钟 阅读
Sumti Jairath

嘉宾

Sumti Jairath

Sumti Jairath is Chief Architect at SambaNova Systems, where he's focused on machine learning, big data analytics, and software-defined hardware since 2017. Before that, he spent close to eight years as a Senior Hardware Architect at Oracle working on hardware acceleration for machine learning and data analytics, and earlier held processor and server design roles at Hewlett Packard and Sun Microsystems. He holds a B.Tech in Electronics and Communication Engineering from the National Institute of Technology, Kurukshetra.

Richie Cotton

主持人

Richie Cotton

Richie helps individuals and organizations get better at using data and AI. He's been a data scientist since before it was called data science, and has written two books and created many DataCamp courses on the subject. He is a host of the DataFramed podcast, and runs DataCamp's webinar program.

Chat with AI Richie about every episode of DataFramed - all data champs welcome!

Key Quotes

What is needed is to be able to generate each one of those LLM results much faster, and for that you need an underlying architecture that's much more efficient at moving data. That's basically SambaNova's dataflow architecture — nine years into the company, we've been figuring out the communication inefficiencies so that everything can run at eighty, ninety percent utilization. Once you move that data faster, you can generate a thousand or two thousand tokens per second per agent. When you chain agents together, overall results are 10X or 20X faster.

Today's world of AI is maybe the '90s of how bits used to move on the internet. They were very expensive, and we used to pay attention to every bit that's moving, how many kilobits I've sent, and then count my internet bill. It's the same situation today with tokens: how many I've consumed and what my bill is going to be. In the future, I don't think we'll be looking at how many tokens we're consuming. For that to happen, tokens have to become a lot cheaper, and that means models have to become a lot more efficient algorithmically.

Key Takeaways

  1. Data control is becoming a first-class infrastructure decision. Before picking a provider, decide up front whether data residency, security requirements, or a provider's capability limits push you toward hosting models yourself rather than calling an external API.

  2. Better prompting beats bigger models more often than you'd think. Clearly scoping the problem and asking a model to verify its own output reduces hallucinations more reliably than simply switching to a larger, pricier model.

  3. Judge AI infrastructure by efficiency, not raw compute. Power draw per rack, not just tokens produced, increasingly determines whether new AI capacity can be added to existing facilities without waiting years for new data centers.

文字稿

Richie Cotton: Hi, Sumti. Welcome to the show. 

Sumti Jairath: Thanks for having me. 

Richie Cotton: Yeah, great to have you here. To begin with I wanna start with a gripe, is why are a lot of AI agents, they're too slow, they're too expensive. How do we make them faster and better? 

Sumti Jairath: So AI agents slow part happening for couple of reasons.

One, you remember like only couple of years back that pretty much how models used to hallucinate a lot. It's little lesser now, but a lot of contribution comes from, is from couple of factors. One is that making models bigger and bigger, like more parameters in there, more intelligence in there, more ways of thinking around it.

So that certainly add the parameter count, bigger context lengths and so on. And then second is the, the bigger reason is the reasoning as it came in a way that earlier, if you recall, we used to just interact with the single model, type in, "Oh, here is my question," and then it'll just reply and a lot of hallucination will happen.

And then as reasoning developed in the past few I'll say couple of years now we pretty much we give a model a chance to reflect on what it is saying, validate it, and then come out with. That in turn ends up generating a lot of tokens in between. So really what is that end to end when a user is interacting, it is just they feel like, "Okay, I've given my prompt, whole bunch of agents did something to it, and my response came in, and there is a l

主题

相关