Accéder au contenu principal

Can AI Agents Outperform a Data Scientist? with James Zou, Professor at Stanford University

Richie and James explore how AI scientist agents are already outperforming human experts, the Virtual Lab framework for teams of agents, a new training paradigm called "learning to discover", scaling agentic systems, and much more.
· 48 min lire
James Zou

Invité

James Zou

James Zou is an Associate Professor of Biomedical Data Science, and by courtesy of Computer Science and Electrical Engineering, at Stanford University. He leads the Stanford AI for Science Lab and is affiliated with Together AI. His research focuses on building AI agents for scientific discovery and data science, making AI more reliable and statistically rigorous. He has received a Sloan Fellowship, NSF CAREER Award, two Chan-Zuckerberg Investigator Awards, and faculty awards from Google, Amazon, and Adobe.

Richie Cotton

Hôte

Richie Cotton

Richie helps individuals and organizations get better at using data and AI. He's been a data scientist since before it was called data science, and has written two books and created many DataCamp courses on the subject. He is a host of the DataFramed podcast, and runs DataCamp's webinar program.

Chat with AI Richie about every episode of DataFramed - all data champs welcome!

Key Quotes

In customer support, you have to be very high fidelity. In self-driving cars, or other kinds of automation, your agents have to work 99.9% of the time. But in science, it's actually the opposite. It's okay if the ideas some scientists have do not work most of the time. For human researchers, most of our ideas do not work. That's part of scientific progress. It's great if just one of 10 of our ideas actually works — that's already amazing, can lead to new breakthroughs — that's already a really good success rate. So we're in this opposite regime of science where it's actually okay to make mistakes for AI and for humans, but we really want to encourage innovation and creativity, which is the opposite of more standard automation or customer service applications of AI agents.

For the past 500 years, the way that humans represent scientific knowledge is in the form of passive papers — passive artifacts of knowledge. If somebody spent years doing research, and you're a reader, just by reading the static words on a page, it's often not clear what the true insights behind that research are. Now I think we have this opportunity to basically agentify all the scientific knowledge. We're building this platform called Paper to Agent, which is to basically convert all these passive artifacts of knowledge — from PDFs, from papers — into dynamic, interactive agent authors. The paper agent becomes the virtual corresponding author of that paper. It knows how to reproduce the results from that paper. It also knows how to apply the data or the methods from that paper to new problems.

Key Takeaways

  1. AI agents are already outperforming human experts in specific computational science tasks. In protein design, small teams of AI scientist agents produced 92 novel candidates in a matter of days, two of which bound to COVID variants better than anything previously designed by human researchers.

  2. Training AI to be creative, not just accurate, requires changing the optimization objective. Rewarding the single breakthrough result out of 100 tries — rather than average performance — produces more innovative agents, at the cost of more failures. In science, that trade-off is worth it.

  3. Effective human oversight of agentic systems looks nothing like micromanagement. In the Virtual Lab, humans participate in only 1% of agent discussions — their role is setting high-level goals, defining constraints, and reviewing outcomes, not directing every step.

Transcription

Richie Cotton: Hi, James. Welcome to the show. 

James Zou: Hi, Richie. Yeah, thank you for having me. Really excited to participate. 

Richie Cotton: Yeah, great to have you here. Now, one of the big trends over the last few years has been having AI agents for replacing customer service people and business dev- development people, and a few other employees.

So how close are we to having AI scientists? 

James Zou: A lot of the existing AI agents are meant more for automating relatively routine and simple workflows. And I think something interesting about science is that good science is never routine, right? Because the nature is that you want to make new discoveries, so you want to push the frontiers of knowledge, which is what makes science really exciting.

And, a big part of my work is on building AI scientist agents that can help to really push those frontiers and make new discoveries. And that actually, I think re- requires also think- rethinking how we build and train the AI agents, right? Because a lot of the existing AI agents are more meant to imitate humans, right?

They're Taught by essentially trying to follow existing workflows. Even the language model is trained by imitating human writings, right? But to do good science and make new discoveries, you don't want to just imitate, you also want to innovate, right? And try to explore novel ideas. So I think there's still a lot of work that we need to do to really teach models how to be more

Sujets

Contenus associés