Industry Roundup #5: AI Agents Hype vs. Reality, Meta’s $15B Stake in Scale AI, and the First Fully AI-Generated NBA Ad

Richie

Hey, Martin. Welcome to the show.

Martijn

Hi, Richie

Richie

I mean, I should say welcome back. Obviously, you've been a guest, several times before for, annual, data and AI trends, episodes. But here's your your first time hosting. So, how does it feel to be in the hotseat?

Martijn

Very privileged, I must say. I feel a little bit like you. So it's a very more comfortable, position than. And, needing to want to be given predictions. It's always like I'm scared for a whole year to see, like, how they play out.

Richie

Absolutely. So, yeah, maybe we've got a chance to review some of the things you, talk about in January as well today.

Martijn

Yes. I'll maybe. I'm not sure. I'm all I'm doing.

Richie

All right. Great. So, for our first story today, we've been talking about, report from, McKinsey's quantum black outfit. So that's, the AI, of McKinsey. So they've got, a report called seizing the A genetic AI advantage, a very cool title for how do you actually make use of, AI agents or a genetic AI in business.

Richie

And they make a case that there's been a lot of investment in journey of AI, but not many returns so far. And, the big, idea is that most of the use cases at the moment, it's been very much around, what they call horizontal use cases of these, things that are suitable for a lot of, the business or a wide swath of your, department.

Richie

So these are things like chat bots, copilots, and the scale very well, but they've only given like a small amount of value to each department. And what McKinsey suggests is that what you really want are vertical use cases. This is something that's, it's going to be a large amount of benefit for just one department or just one team, or maybe just a few individuals in the organization.

Richie

So I think they're talking things about, like, things specifically for HR. So you got, like HR tooling for like, better hiring or you got things for like legal for contract review, all this kind of stuff. But those are not being widely adopted and seeing lots of organizational challenges, like a lack of skills, technologies maybe not quite ready yet and just, the challenge of process re-engineering is very time consuming.

Richie

And then this is just maybe AI agents are going to solve all these problems because they can automate stuff and it's going to be brilliant. Now, I'm not sure whether this is just magical. Thinking is like, is the cool new toy that McKinsey you want to talk about? Or whether this is going to be a real thing.

Richie

So, Martin, do you have an opinion on this? Our agents going to save us all and make loads of money for businesses?

Martijn

Well, first of all, I think some businesses are making loads of money out of this, device or by the AI tools. So given all the use cases you mentioned, I think there's, like, an AI startup scale up, a little bit for everything. And I recently got some data about, like, it's not the, fastest growing, area of startups, in terms like revenue that they, bring in.

Martijn

It's also a little bit of a different world, like the world. I think you came out of a world where lots of money was spent on building product. Not that much focus on the business side. That seems that, people have learned our lessons and actually quite all of these AI startups are making, a decent amount of money.

Martijn

So the AI cash is going somewhere. Somewhere is sneaky. It's money. But I think what you really mean is, like, okay, the ones who are buying all these toys, are they, are they see the gains? So I think it's it's a little bit underwhelming at the moment. But I, I think it's part of just of the hype curve.

Martijn

I think the the benefit will still come. Now recently had with some colleagues, some discussions around this and they're like different takes as well. So, so one of the things that one of my colleagues raised was, the Solow paradox or the productivity paradox, like I needed to look it up as well. But basically what it pointed out was that, well, there's actually there was like a slowdown in productivity growth in the US between the 70s and the 80s despite like, okay, like big promises because, information technology computers were coming out and that that was going to like boost productivity like we've never seen before.

Martijn

So despite lots of company spending a lot of money on computers and technology, and expecting, much more and better, results and people getting faster at doing their jobs, it was like it is argument that like, okay, actually it's not what we see. If you look at productivity growth, it was a little bit of a long intro to like, okay, well, I, I think the, the, the productivity growth will come from AI and that it will start solving actual problems rather than like toy problems.

Martijn

But at the same time, like having that productivity paradox in mind, like, am I 100% confident? Like, no, not like that. That's also like, not the, not the case. But I, I that it goes slower than people anticipate I think is a little bit normal I would say.

Richie

Yeah. So it does seem fairly intuitive that you have to invest in stuff first and then you reap the rewards. You get the money, the, the increased productivity growth, the extra revenue generation, whatever. Like a later on. I guess it's a question of how patient you want to be about, like waiting for that extra cash to come in.

Martijn

Yes. And like and also know that reinvestment like, turns out to the way that you hope it's, to be. And so, I think those, also still have like uncertainty around like, okay, what are the investments that make sense? What are the investments that make sense? Like, I seems to have like market fits in the marketing, field.

Martijn

It seems to have quality markets in the, in the software engineering field, if you think about curser and like, which, sorry. Cursor. And how it's making like engineers more productive. I'm more access on the marketing sales side. Definitely. Okay. I'm much more I can do, like, so there's like, true product market set, maybe a little bit on the legal side.

Martijn

But I'm okay on the finance. Like, you see, at least I haven't seen that much yet. So I think we're also trying to figure out like, okay, where, where, where does where does the most value set. Where does the lowest annual value sits? And in that process, while investments get lost while you tried to figure that out?

Richie

Absolutely. Yeah. I mean, I think that's very natural. Like, not every investments can be a winner, but you need to figure out, like, what's what the sort of the good bets are going to be. Now, there's one more thing I want to talk about in this, report. So, it's perhaps the most controversial bit.

Richie

So, the report is suggesting that the best use cases for agents are going to be just, automating business processes. This is a great way of saving money as you take your existing business processes and then use a bit of software, a bit of AI, to just automate away the bits so humans don't have to do them.

Richie

Now, a lot of this is really the AI is not that new or not that involved in it. It's just, taking this sort of standard approach. It's been around for most of, at least the last decade of just using software to automate, business processes. And then maybe there's an LM and LM in there somewhere.

Richie

But, so there's some pushback from this. So, it was, Ethan Mollica as it was a renowned, I, professor, he was, saying that, well, this is really missing a trick because a lot of the latest models that come out this year, they're really good reasoning. You can do things that are a lot more complex than just, you know, summarizing bits of text or whatever.

Richie

So there's a kind of there's a disagreement about what the best use AI agents are. Do you want really simple agents or do you want something fancier that's, more akin to like, a junior employee? I don't really have an opinion on this.

Martijn

So in the in the in the short term, I agree with the McKinsey report. In the long term I agree with the Ethan Malik rebuttal, but for a different reason. So, so I think where McKinsey is right in its report is in the fact that if you do this type of change in a company and I can only mention, like the larger the company, the more difficult it is, like there's a certain like inertia.

Martijn

And because of that, like, okay, you can come in and you can say like, hey, we're going to take like the biggest, most complex process or product that we have and we're going to make it a little AI and probably along the way, like you lose everyone and your whole, investment, all you're trying to do like full slots.

Martijn

So from that perspective, like starting simple, which is like my distillation of like picture what they're saying, it makes makes sense, like in a, in a, in a world where you move more and more towards, embedding AI in your organization. So, so that's what I mean with a different reason. So, so my argument is like why I believe it's McKinsey paper is more from like, okay, organizational change perspectives rather than the arguments that the day make.

Martijn

Because I think in the argument arguments that they make and the way that, it might try to, to bring them like I'm actually on the side of, of of to Malik where okay. He points out like the reference, models, that, that are in the McKinsey report are outdated. They make certain arguments around costs, but like, what we've been seeing the last three years is that, like, cost has been declining rapidly.

Martijn

They're like hyper focus, McKinsey's hyper focus on efficiency. But like Dylan, focused on efficiency from, like, yesterday's technology. And like when I read that, I was thinking about, like, there's is, there's this, famous quote by like, Peter Drucker, like he's a case management consultancy or maybe even like one of the and the founding members of management consultancy, like, if there's a founding member, team, but it's like, okay, there's nothing so useless as doing, like, things efficiently that should not be done at all.

Martijn

And like that, that part like reminded me a little bit about it. Like, why are we caring about the efficiency of like the models that are already like obsolete? It's like it's irrelevant. So I think short term, the way that McKinsey is positioning it and approaching it, I think it makes sense, but not necessarily for the reasons that they say it makes sense.

Martijn

And there are more meta molecules I'd like. Does that resonate with you or do you feel like now it's fully off marking it?

Richie

Just no. So I like the Peter Drucker quote. And actually, I was, speaking to one of our customers the other week, and, so this is, someone at a bank, and they were saying they were right. Exactly like what's been building all these agents to automate our fax workflows. And I'm like, I've kind of conflicted about this because he's like, why are you still using faxes?

Richie

Like, is not the essential technology at any point in the century. And so it's obviously completely crazy, like it's just a technology fix over a really stupid process. But obviously this is a smart person. If they had any power to not use faxes, they would have done that decades ago.

Martijn

So they need going back to my productivity paradox. Like that's a little bit of the a little bit of point. Like they just they skipped out on the internet and it's still worked out fine for them because I assume there's still a big bank like a bank. It's hard to run. So like they have something figured out. So yeah, it goes back to that, that productivity paradox and like, okay, like what what is what increases will we actually see if any.

Richie

Yeah, absolutely. So, yeah.

Martijn

I'm going to be like after this, like going like call me when I do more, which I'm not. But for the sake of the argument, take the opposite side here.

Richie

Yeah. What is all the stupid ai. Why? Why we bothering? Might not get any productivity out of it. No. I think this argument, it comes back to strategists versus operational lists. So if you are doing, something where it's research, you're not quite sure what the answer is going to be, that's where you need like, strong reasoning.

Richie

You need, like very flexible processes. And that's where these higher reasoning models come in and where you really want that fancy agents to help you out. If you are focused on operations, you want really, like more rigid processes that are streamlined and do the same thing over and over again at scale. And that's where you care about just having simple automated processes that, repeatable.

Richie

So yeah, I think it just depends exactly like what you're doing. All right. So, let's move on to the next story. So, before Martin, you mentioned, about like, the people make money out of this seem to be like the the AI startups and scale ups, and there's been some, big purchases going on in this area.

Richie

Do you want to tell us about them?

Martijn

Yeah, sure. So, one of the one of the things that office was, like fake news in the in the last week was, meta Facebook, for those who sold it the old name or like, Mark Zuckerberg, buying, like, scale, I well, sorry you didn't. Why? I, he took his stake of, I believe, 49% in, scale ai and he, he paid 16, billion, dollars for that, which is, by any measure, a lot of money.

Martijn

Like, okay, for the listeners who don't really know, like, what scale does, so, basically they, they solve, the data labeling problem. So, in short, if you want to like really powerful AI, you're going to need like, enormous amounts of, like, high quality labeled data. And, creating high quality labeled data is something that's incredibly slow.

Martijn

It's very expensive. And, what scale I did is basically build a factory that produces this type of, like, high quality label data. So if you're like, that's why you spent like thousands of dollars a video scale AI. They go over it frame by frame and it's like, hey, this is a pedestrian and this is a tree. And this is like, a traffic light and so on and so on.

Martijn

Or like, if you're like OpenAI, you have, you are in need of like real people riding high quality questions and answers. You had a need for people that need to rate AI. So sponsors, responses telling like, okay, is this a good answer? It is a bad answer. And like all of that is basically done by, scale AI, which makes them, enormously valuable because again, like you need to start with like, high quality labels, data.

Martijn

And then the question is like, okay, why, why is Matt, Mark Zuckerberg doing this? Lots of theories out there. One is well, okay. He feels that the llama models are behind. And if you look at the benchmarks like that seems to be the case. And he feels the need to, like, do a catch up.

Martijn

And this type of, like, move allows him to do that because, on the one hand, well, he kind of becomes exclusive owner of this high quality label data. But Google and I believe OpenAI, who are former customers of scale, I already said, like, okay, like we're canceling our contracts. So, okay, he now owns this probably very valuable data set, and process himself on top.

Martijn

He probably has now a little bit of insight in, in how Google and OpenAI were training their models. So like that's beneficial. Secondly, like, okay. And there's, the CEO of scale AI is apparently like a very talented guy. The what he's built, seems to be, a good validation that is, is a pretty, pretty, pretty smart cookie.

Martijn

And he's going to, lead up, a modest new super intelligence division. So the super intelligent division of meta is basically okay, Mark Zuckerberg saying, like, okay, we are falling behind. We need a new initiative. The next frontier is superintelligence. So I'm going to put this super talented guy at top of it's been $15 billion for being able to do that.

Martijn

And so now I have that. So right. It's another TV like you need a talent and it's some very expensive acqui hire. The first theory was like okay, yes, you need high quality data. He has that now. Another theory is like, well, maybe it's a competitive mode. Like, somebody described it as like a guy who takes a chess piece of the board.

Martijn

Now that OpenAI and Google don't longer have access to it, like, okay, is that gonna hurt their own, model training? One? And then the final one that, And I'll give it to you, Richie, for some comments. The final one I found interesting was about, somebody just that, well, is actually, like, in Facebook terms, like, it's not all the money.

Martijn

It's like, like 1% of their, like, market value or something like that. So, like, it's, it's a blip, for them. So, from a risk perspective, it's, it's a fairly low risk. But to get back into the game, so yeah, but lots of interesting things happening there. Like lots of gossip. Intrigued, like, okay, who's going to win this will just with them, back into the game.

Martijn

But definitely like a new word story, in, in AI world lost rich.

Richie

Yeah. So it is a good bit of sort of, corporate gossip, obviously. Like the idea of just investing $15 billion in it being not very much very, very foreign to me, but, yeah, it's, it's amazing the amount of money involved in the, Yeah, I can actually. So I recently watched the movie Mountain Hedge.

Richie

It's kind of like their succession TV show, but set in, Silicon Valley. So I learned the phrase. So to invest $1 billion, they use the phrase to bust a B nut, which is it sounds kind of rude, but the idea of a. Yeah, Mark Zuckerberg, busting 15 peanuts, on, on scale of this, very interesting.

Richie

All right. Anyway, so, yeah, you mentioned scale out. We actually had, as guests from the, scale AI's as some, I so I was, Wendy Gonzalez, the CEO, and, Duncan Curtis, one of the, senior vice presidents, talking about this is absolutely fascinating. The idea that, yet data is still perhaps the most important thing in order to get great quality.

Richie

I so, we talk about AI being the big thing. Mustn't forget about the data, because that's what's powering everything. Just, like, in terms of trying to get better quality AI. I mean, you basically, a choice of four things. You've got, better data that we mentioned, or you can invent new models, but that's slow and it takes time for you to research.

Richie

And then you can either add more, compute power, joint training. But like the latest foundation models that costing, like, upwards of 10 million or like $100 million to create. So we're kind of reaching the limits of like how much you can spend to create new models. And then you can also add like inference compute, but that's make your models more expensive to run.

Richie

And it makes them slower as well. So then, yeah, that's a terrible user experience. So it looks like better data really is like the big solution for getting better AI at the moment.

Martijn

Yeah. Better data and and again, just hypotheses. But like probably the, the, the talent that you have within the company is, like plays a big role as well. And it always plays a big role like, like high quality people, and always make a difference. There was like, this other, interview would sound out on, where he basically said, like, okay, Facebook has been trying to poach, some of our most talented people, and they're offering them, $100 million signing bonus, which is when you hear the story is like, well, that sounds pretty crazy.

Martijn

But then, well, one of the narratives in this whole acquisition story, we just went over this. Well, he wanted to have a really good head of his superintelligence division. He identified Alexander Wang, the founder of scale AI, as being that person. And he wrote a check of 15 billion to own not even half of the company. So, like, it's the data and apparently, like, okay, the people are probably the novelty of the idea that they come up with, like, this seems to be playing a big role as well.

Martijn

And maybe Marcus identifying one of the issues. Like I'd say, Mark, I'm definitely not on a first name basis, but then, it's maybe another thing is like one of the key things he he sees is why alarm as we are in is because he does not have a talented enough team at the moment and probably already has a very talented team.

Martijn

Maybe the difference is, is, is, is hard to grasp for us worlds. On this topic.

Richie

Yeah. Maybe he's. I mean, of course, he's like, back in the day, Mark Zuckerberg, he became, like, the youngest ever, like, self-made billionaire. He was like 23. And then it was Alexander Wang from scaler who broke that record and became a billionaire when he was 21. So maybe that's it. That's part of the reason he wanted to hire him.

Martijn

The others envy.

Richie

Yeah. Yeah, something like that. All right, so I think yet another story around, hiring or, buying things. Yes.

Martijn

So another rumor that's going around is that, Apple is, is, is considering or. Well, it's a rumor that they would like to purchase opportunity and like the number that's been circulating, it's $20 billion. I'm not sure. Like, okay, to to what degree? That's true. But, but, I, I think it speaks to the story of the fact that, well, Apple seems to be in a bit of a tough spot, with its AI funds.

Martijn

So, I haven't met all of people that are happy with Apple intelligence. Or the way that, Siri is using, I, I think they made quite a lot of promises. None of them fulfilled. I think it's for the first, first time in ten years, like, so I'm on the on the Google device.

Martijn

I use Android, so for ten years I needed to listen about like, okay, why having an iPhone is so much better, and, why am I not switching to an iPhone? And so on and so on. And like, now I actually have better functionality. They're all locked into their little Apple ecosystem. So, maybe I, eventually will have the last laugh.

Martijn

But so, yeah, but the point being, like, okay, so Apple seems to be in a, a little bit of, but at the moment in the losing spots on everything that has to do with AI and, I think rumors like this are, I like, I don't know what where the origin is. It's like a good question is like, are they trying to fix it or not?

Martijn

And trying to they try to fix it internally, but then internally they got like lots of people switching job positions, switching roles, switching functions and so on. So there's there really like, look like it's like a stable environment. Like that said, like if you look back at OpenAI in the last 18 months, we could probably make the same argument.

Martijn

And they kept shipping, models. But from the outside, like, it looks all a little bit weird and like, you'll need to buy it in and then, some, like, like, shitty, game around. Now. Yeah, I think I use perplexity once in a while. But, it's obviously also not the biggest model out there.

Martijn

And Apple has a lot of cash and they have a lot of stock value. So you could even see them making like an even bigger acquisition if they really want to. But it seems like consensus is like, hey, they're gonna need to do something, because everyone is unhappy with that part of their product. So are they going to buy it in or are you going to develop?

Martijn

It's, themselves. Well, and then maybe a final thing on this is, that there's like a another angle to look at it. And is that Google's paying them really a lot of money to have Google be the default search engine, on their iPhone? I believe it's also like in these magnitude of $20 billion a year.

Martijn

If you start to integrate AI more and more, you make that central piece. Well, you're losing, $20 million in it's probably basically pure profit. Yeah, lots of lots of hard and difficult problems. I think they're definitely, like, in the corner, where, yeah, they're getting, Yeah. Hit from all different sides. So it's going to be interesting to see how they get out of this and if it will be with a bit of a big acquisition, there's actually and then I'll stop and let you react, which it does.

Martijn

I actually like this. I think the biggest app is, acquisition Apple ever made was like beats, you know, like the Doctor Dre headphones, which is like from a totally different magnitudes. I is like a billion something than what's been rumored today. So, yeah, that's like a little bit like, so Apple or like Steve Ritchie, you think?

Martijn

Well, they will they buy or will they build or. Yeah. Will they burn and go away.

Richie

Yeah. I think apple disappearing completely that, that's more of an outside chance. But certainly they really need to do something, engineering. Is it like beats is the biggest thing Apple's ever bought? Like, that's crazy because that's a tiny company compared to, Apple. When you compared to something like, Salesforce where, you know that they bought slack, they bought, Tableau.

Richie

And it's like constantly acquiring things and then trying to, fit them into the business. So, yeah. If Apple does buy something that big is biggest perplexity or even bigger then, yeah, it doesn't have the reflexes to kind of fold new businesses, into itself. So that's going to be a very interesting integration challenge.

Martijn

Yeah. And I think to the, to the, to the point of scale AI and the point of talent, like maybe that's what Apple needs like a K they need some. True. I don't inside of Apple. Okay. What is it that they can buy to bring that talent and skill eyes off the market now? Like what else is out there?

Martijn

To, to bring that in. I think, Microsoft at some point, but one of the, one of the companies that could bring in some, so I don't. So, yeah, let's let's see what they do.

Richie

Yeah. I mean, oh, so so on that note, there was the Google bought, character. I, it was a few billion dollars, and that was basically just a higher back. One of their executives, left the company previously.

Martijn

Maybe that was it was the one that, Yeah, that's true, because the late I was funded by one of the people who wrote the original. Attention is all you need paper from, remember correctly.

Richie

Yeah. All these, like, hirings of $100 million. I'm in the wrong game.

Martijn

Oh, you should write a paper and,

Richie

All right, let's move on to the next story. I've got another report for you. This is the EU generative AI outlook report. So it's a report to sort of inspire policymakers in the EU just on like, what should they do around generative AI? So, the big theme is that, generative AI ought to be considered a strategic asset, for European countries just in order to, generate economic growth and, increase employment, increase productivity, things like that.

Richie

So, there's a big worry, in the, report that the EU is lagging behind in compute capacity. So saying the total compute capacity, of all these GPU clusters, high performance computing clusters across Europe is less than the size of X I's Colossus. Compute! Cluster. So, the, Colossus has got, I think, it's like 200,000, Nvidia GPUs.

Richie

And it was a pretty amazing technological feat this is, based on Memphis. But yeah, that's more than the whole EU combined. So, the EU is lagging in, AI infrastructure. But then, beyond that, there is, the focus of the report is that, generative AI should, be focused around small and medium enterprises.

Richie

The focus is on small to medium enterprises, because this constitutes 99% of all businesses within the EU. Presumably that's how much will this by head count? The, like total number of businesses? Certainly not by, by revenue, but, yeah. I think every business basically should have, some sort of, generative AI angle, regardless of its size.

Richie

So, though, there were several, sort of perceived challenges. So one was the infrastructure thing I mentioned. One was just going about, process re-engineering. The third one is the AI skills gaps, who spend a lot of time talking about, well, how do you make sure that people have the right skills? And so, the two things is that everyone needs to have, so basic AI literacy skills, it's first of all, just how do you go about using some of these, AI tools?

Richie

But also do you understand what the implications of using our so based so basically of what is possible with AI, what is not possible with AI. What should you do with it? Like what are some sensible use cases and giving on this is that, digital skills, including AI skills and that recognizes basic skills. So this is like a legal term which, puts them alongside reading mathematics and science.

Richie

So these are things that basically everyone they think everyone should be taught from school age onwards is a big implications just for, childhood educational curriculums, right through to adult learning. All right. So, first of all, Martin, do you think, this all sounds like a sensible thing making, AI skills, putting them on the same level as, like, reading, maths and, and science.

Martijn

Yeah. Well, let me first say, because you you you you you you definitely treated me with your opening sentence around, like, inspiring policymakers to write policies or something along those lines, like, I don't I first of all, I don't think that, we should inspire policymakers, to write AI rules. For those who watched our talk at the start of the year, around, our predictions and and you, I, noted I know that I'm not a super big fan, of all of this.

Martijn

So, so so this whole. Push towards, like, regulation, writing policies around this is like, it's it feels a little it feels it feels small. It feels very European. It feels very like, Okay. What we. What? We're good at this. Like, we're good at writing rules and making policies. Doing laws. It reminds me a little bit of that, like that saying, like, okay, it's like, if you want to get started, like, you need to quit talking and, like, begin doing, I think there's lots of talking, lots of writing, very little, doing, like, am I am I?

Martijn

I might have sign of all this. We gonna let policymakers write good policies about AI? Like, not really just agree with what's been set on like, okay, yes, we, I like AI skills is probably going to be like, a basic skill, like reading and math and science. Like like, yes. But does that, like, I think that can, like, if you let's if you read to people three, if you had the citizens of the EU like, go out themselves, I, I, I think I tend to believe that they'll figure this out themselves and like, find their own facts and go at their own pace and they'll be supported from the

Martijn

private markets or will be pushed from the private markets like Twitter jobs. To do this, like maybe, maybe a very quick like data point on that probably I would take the I would take the bet that for the past three years, if you would aggregate the data, that's the biggest users of ChatGPT were students. So if you think about like, okay, students, students are already adopting this skill.

Martijn

Like they don't need a policy for, for for that. Like they're already like they're ahead of the people that like the policy. I would I would guess, so so this need for like writing rules about it. No, I think the market is fixing it themselves. Like our, our children are taking the necessary steps, to, to do this.

Martijn

So, like, is it a common sense recommendation? Like, yes, it does. It need to have everything around? It's like, I don't think so. I think like what? Let's let the people do whatever. But but they seem best. And it seems that if you let them do that, then I'll let students as an example, like that's already happening.

Richie

Okay. Yeah. I want to ask you, this is the chair. So, yeah, I, I agree that the market is going to solve many problems, in a commercial sense. But I also do think it's important for policymakers have, that sort of AI literacy themselves. So having an understanding of, like what the point of Germany of AI is, in order that if they do write policies, they're not going to do something stupid.

Richie

I also think that, in noncommercial cases, it's also like very useful to have some policy there. So, for example, if you're designing a school curriculum, you want to know, like what good guidance is.

Martijn

Or the best way to avoid doing something stupid is not doing it. So, or not doing anything, like like, yes, yes. Should like schools think about like, okay, how to incorporate like AI in their curriculum and like make it part of their of a basic skill, like 100%. But but I think that's like I think my example of private market was not so like, was not that well chosen because I think I equally think that like, if you have a motivated educator teacher that they will bring this as well.

Martijn

I, I'm actually fairly sure that in many classrooms today within Europe, without any policies, they're handling, the they're playing with AI in a very creative way. And and thinking about like, okay, how can we how can we use this? Yeah, maybe I'm a bit too much. Ron Swanson from life, Parks and Recreation on this, on this topic.

Martijn

Yeah. I don't like.

Richie

Rules. Yeah, I think it's just, different levels of comfort with, with the bureaucratic engine, with a. Yeah. Okay. All right. So, yeah, there's different ways of making it work. All, let's move on to the next story. You add something about, an AI effort.

Martijn

Yeah. So, so, I don't really follow basketball, but like, the NBA, NBA finals, took, took place, and there was actually an ad, that was, fully created by AI. And so that was not the first one because like, Coca Cola, like, did it in the past. And again, like we reference that in our, in our startup gear, podcast episodes.

Martijn

But, the, the crazy thing about this ad, it was, why do an ad of culture, which is, as far as it's not like a predictive betting, kind of kind of company. But the crazy thing about this ad was that, the, the, the creator, and it was a single created creator. So one person wrote a blog post, describing its workflow.

Martijn

And that's what's crazy because, again, so this is an ad that goes in primetime during the NBA finals, and he basically, build it with Google Gemini. I think the lottery he used like, a little script deadline. And then edited the clips like in some standard video editing software. And it was done. So it took him two days, to generate 30s of content.

Martijn

And if you look at the cost estimate, like if you would exclude like, his time, it was $600 to generate the content. So I'm probably sure, like our listeners have heard about the crazy amounts that are paid for advertising during these stop spots at, the Super Bowl, at the NBA finals. And you look at these ads and they just look expenses.

Martijn

If you look at this ad, it actually looks really good. And if you then realize, like, hey, this is created in two days by a single person with a technology that's maybe two, two and a half years like old between parentheses. Mainstream. That's actually pretty, like amazing. So yeah, like, you didn't need to be a writer.

Martijn

You didn't need, like, any camera person's, like, no actors needed. So, yeah. Pretty, pretty fascinating. Like like, Yeah. Which I don't know if you've seen the ad.

Richie

I have seen the ad. Yeah. I didn't get all the cultural references because, again, I'm not a basketball fan from Florida, so, it was very odd. I like that, though. It was a it was fun. There was so many other things that I think were caused by technological limitations. So, for example, the, this is lots of very, very short sections with quick camera.

Richie

So was doing 1 or 2 seconds. And there's just because I can't retain consistency for very long. It also it's quirky and I think that's the only thing you can really get away with at the moment because like perfect realism of like movements and things like that is not really possible. So having stuff that is weird and fast cutting is kind of the only tone you can go for at the moment.

Martijn

That's a nice explanation. Like I was actually like, my take was like, oh shit, I'm getting old. Because like, this is way too flashy and switches, like way too quickly. But, I like your take more.

Richie

Yeah.

Martijn

It's like. Yeah. And by the way, it makes it. This probably makes sense. Like what you're saying. Like, you clearly see that they need to, like, switch between different scenes very quickly. And most likely now I was deliberately paying attention to it, like during the first time that I saw the ads, and I couldn't spot, like, any obvious, like, glitches, like I couldn't see, like, six fingers or like, somebody with three likes or something like that.

Martijn

So, at least for me, like, maybe if I look at it like in fifth or sixth on like I start looking at the clicks. But I don't think that many people look at or not like six times in a row. So I, I did notice, like a quirk in the first one. I it's, I didn't see that much wrong with it.

Martijn

Besides, like, the case was very snappy.

Richie

Yeah. I was just, that's the other thing is, like, if you keep things, moving quickly, then it distracts people from the fact that, like, maybe the, the lighting or the shadows isn't quite right in the background, or there's like, an inconsistency with, like, I know where some background objects are placed. So, yeah, keeping things moving quickly seemed to be the secret, things.

Richie

But I agree like that a lot of the artifacts, disappearing pretty quickly, it is getting much more make much more realistic. Very fast.

Martijn

Yeah.

Richie

I'm intrigued to see what happens, how this affects, like, creative industries, though, because, you mentioned here it was done by one person. So you don't need a cameraman, you don't need actors. You don't need script writers. We had two Hollywood strikes, last year. I can't remember, yeah, last year. So, this crew will have to strike, and then the Actors Guild striking, exactly because of this, and now.

Richie

Okay, everything that these people striking predicted is sort of coming to pass, real content going out of prime time is is being done by AI with no whether people need to be on one person in a prompt. So how do you think this plays out? In the real world.

Martijn

I think in phases. And I think the phases are going to follow the, the light of the content. So, so, so going back to the clips point, like, okay, it's it's deliberate, like like little clips basically together because like creating a 30s or a minute at that's like all in the same all it's the same scene is going to it's probably like to to heart or to glitch prone or anything like that.

Martijn

So I think what we'll see is like, okay, so now we saw like a little ad, maybe the next thing we see like a more consistent ad, then we're going to see like, a ten minute stand up show by an AI, and then we might see, like, a 20 minutes, sitcom, existing out of five episodes, and then, like, the next thing is maybe like, 70 minutes children movie and then like, okay, we're at the two hour, like, so, but I, I think getting from that like 16 second, ten, 15 second clip to like that one hour and 30 minutes.

Martijn

And movie. I think that's actually like still a long way to go in it. And I don't think it's because of the creative part letting people figure out like how to write the prompts, how to work with the camera perspectives. I think it's really going to I think that it's really going to be about the technology without having like any, any insight in this.

Martijn

Like it's just we've seen massive improvement. But like from a light perspective, it's not that we are we're still watching 15 seconds and the 15 seconds that look a little better. But they're not they're not like a minute or two minutes long.

Richie

Yeah. So I've seen a few sort of, videos. So the Doll brothers do quite funny satirical videos and that kind of a couple of minutes long, and that's like, it's amazing. But that's the most I want to concentrate on, on AI generated content at the moment. And part of me and just like at the moment is very novel, but is that kind of quirky tone and all the fast cuts, is that going to get tedious quickly, do you think, or are we going to get past that and have like content that you can concentrate on for a bit longer?

Martijn

Well, I think I think well, I think we're going to need to get past that, because I think it's this is a particular type of content and style that's not going to be like well suited for everyone. So if you want to make this like a really imagine like your Netflix is full of like videos like this adds like, I wouldn't have a Netflix production.

Martijn

It's a, it's too, too much, too fast, too large. My, my guess is like they will need to fix the, the setting up, trying to think like what are good examples like maybe. Well, if you look at when Toy Story first came out versus like where we are today and in, in 3D, and computer generated movies, like, it's also like a very big difference.

Martijn

And that happened what is in the last 20 years or something like that. 20, 25 years. So, like, what will my guess is will we'll get there. But I do think there's still quite some hurdles to, to do. Like, this is, there's a story by Seinfeld, too, who's, like, talking about, like, how do you become, like, stand up comedian?

Martijn

And it's like doing a five minute bit, like, that's how you start, like doing like 20 minute like it's it's is already difficult doing like a one hour like it's extremely difficult and like an hour and 20 minutes like that's insane. And it's like like that's how you like. That's how you separate, like the average standup. But from the top standup, I think it's kind of when I talk about the time like, that's a little bit the metaphor I have in my in my mind, like, yes, you can create a 32nd clip like it's it's hard but doable and it's getting more and more accessible.

Martijn

But now all we're going to do ten minutes and 30 minutes and like when will we be at that one hour. 20. And what will it take.

Richie

Absolutely. So, maybe all these, the Screen Actors Guild, the Actors Guild, they've got a bit a bit more time to, to figure things out before it starts going into like, okay, this is, mainstream, TV stuff.

Martijn

Yeah. The screenwriting, like, I think there's still a pretty, like, someone who needs to write a prompt, so, fairly good, I would guess.

Richie

Yeah. True. No. Okay. So, actually, before we move on, do you have a sense of how much this cost? Like, how much? I.

Martijn

Have no idea. Actually, it's a small. We got the benchmark, like, as I said, like it's $600 to generate. That's for, like, schooling, like the the cost of the person, but, like, on the, I don't I have no idea on, like, the token costs. Like specifically like, allegedly this one cost $600.

Richie

Or $600 of clip. So this is like, for an individual. It's something you got to think about a bit. But for a business, that's that's just nothing.

Martijn

Yeah, indeed.

Richie

Okay. All right. Cool. So, last story, quick talking for a while. So, I wanna talk about, the first, benchmark for deep research tool. So these deep research tools, around, you can ask more complicated questions of, generative AI. They'll go away, think for ten minutes, and then come back with a report for you.

Richie

This is, this comes out of research from the University of Science and Technology of China. And, they tested so a tool, some open AI tools from anthropic, Google, z, perplexity. And so, we have a few winners, they split into two categories, actually. So, the deep research agents are things that are specifically advertised at being, able to use tools and, are for, deep research.

Richie

So, the, the wins in this category is a Gemini 2.5 Pro ad deep research version of that. And then second place was the OpenAI deep research tool. So, those those are the best ones. If you want to try out different deep research tools. The second category, it was just standard issue, lems with search tool capability.

Richie

So in this case, the winner was a code 3.7 sonnet. And second place was, perplexity. So not reasoning. So most of them, had a winner, exit. I was in on WeChat, which didn't have a winner in the category, but, it seems like in general, the tools of specifically market as deep research tools do better deep research, which is, a good thing.

Richie

But yeah, the, the Irish are doing something rather than just, a bit of grinding on the benchmark. The interesting thing was that, they took a load of, like, real world prompts, and then they had an LLM to judge whether or not these, these agents were, were working correctly. And to begin with, the agents were being to, to nice.

Richie

And they're giving everything a high score, which is stupid because you want to be discriminatory about like what? What's good and what's bad. So in the end they had to get humans were experts to write reports, and then they got the Elms to compare it to the human report. So there's, some human, benchmarking involved in this.

Richie

Alarms can't be trusted. Sort of by themselves at the moment. It was just that whatever the tool is, they're all still really bad at generating citations, or at least accurate citations where the content of whatever they're linking to actually says what they claim it is. So I think it's a no problem. Something to watch out for when you making use of these tools now, Martin, I don't know if you used any of the deep research tools.

Richie

Have you found they work or not?

Martijn

So I used them a little bit, like, it's definitely like in the. In, in my AI workflow, it's, probably the one that I use the least. I'm not often having that, that type of use case. But, but let me give you an example. When I, when I did use it, like, okay, I, I used, I needed to get some information out of like, quarterly earnings reports, from organizations and actually did a pretty good job there.

Martijn

And what I noticed was that compared to when I use it in other instances, is that, because that like bad experiences, but it also and it was because the prompt that I provided was not specific enough. The more anecdotal is my own experiences, like, okay, so what I've learned was that if I give it like, research, me, this earnings report for this particular company and I want to get this type of information and look like in the last like 24 months, like it gave me actually really good insights when I would say, give me everything the S&P 500 and all the earnings, while I could go and stew broad like it's I

Martijn

got like some flunky behavior, maybe it's nice, but they're not writing the right prompts. Maybe it's something on the tool that it's a little bit off putting when you, when you go through that experience. So I actually haven't talked to anyone. And this is this is, this is on me. Like, it would actually be good to talk to a couple people who use deep root research tools, like on a daily basis, and understand how are they using it?

Martijn

What are they? What are they doing? Because I have a good feel of like, okay, I use to use Gemini models on a daily basis, but I don't use a deep research like, okay, who is the equivalent of me who uses deep research the entire time? And and yes, you at what they think, what they do.

Martijn

So. Yeah.

Richie

Yeah. All right. We have to get Adele back then. I think these are big thing with, the deep research tools that I, I, I, I agree that, I find the workflow incredibly frustrating because, the models run for a few minutes, so you have to get the prompt right up front, and you write that if you are more specific about what you want, it dramatically increases the chance of you getting things right.

Richie

But that means that you've then got to do a lot of work upfront and thinking, well, what do I actually want? I'd like which sites should this, thing, be looking at in order to pull the information from border, what I actually want out of it. And you have to do so much prep upfront to get it right that you demand the research yourself.

Richie

So, yeah, in order to get good results, you've really got to be good at that kind of prompt engineering, and you've got to take the time to think about what you want your answer to, to be,

Martijn

Yeah. Yeah, totally.

Richie

Yeah. So, it's, the technologies come to kind of coming together again. It's like it's the process stuff. It's a little bit frustrating to to use at the moment, but, yeah. If, any of you, any listeners have been making use of these deep research tools and you found success or you found some challenges, then that please stay where it is on social media, let us know what, what you found.

Richie

All right. Wonderful. Well, it's been a pleasure chatting with you, Martin. We have talked for way too long, so, let us wrap up here. Have you enjoyed the experience of, hosting data dataframes?

Martijn

This was months. Like, it's amazing that you get paid for this. I would do this for free.

Richie

Okay, certainly not taking a salary cut to free.

Martijn

Not, I guess. No, it's really fun. And thanks for having me, Richie.