Industry Roundup #6: GPT-5 Launch & Scaling Limits, Meta’s Chatbot Guidelines Leak, and AI Safety Concerns

the gold medal at the international Mathematics Olympiad.

Alex

And they also released Genie Tree, which is a promising, world model where models are different than Lmms. But maybe that's for for another discussion. They're, like, fascinating, concepts. The main update is actually, user experience update. Previously, we were used, to, selecting many models inside, ChatGPT. We had the option of, for instance, GPT for oh, oh for mini, oh for mini advanced and so on.

Alex

And users were complaining. This is too complicated, right? I'm not a programmer. I don't know what each model does. Right. And, if you look at the competitors, most of them were already offering, a simplified version of, of models, for instance, like Gemini is just flash and Pro cloud is just clouds on and cloud TPUs and deep sky is just air one and V

Alex

Right. So they had to go, the same way and now it's just GPU five and under the full, depending on your task, the model automatically routes. Your question to the, to the right model, I guess some other notable features were targeted more more towards, personalization. So it looks like the, the model, they want to make models more personal.

Alex

So for instance, now in JB five, it can connect five with your Gmail or Google Calendar. You can perform the model and say, organize me, or organize my schedule for, for this week. And it will be very personalized. It will know the meetings you have and so on. And, you can also like change, chat colors, and what else you can also like, play with the the bots personality can make it more scenic.

Alex

You can make it more, supportive and so on. Many people like we're discussing like, okay, what's, how will this I prefer I think perhaps we can, we can think about this in, in, from two perspectives. And one is obvious, like benchmarks, and the other, it's like user, perception. Like how, how, let's say the, the vibe, the vibe test.

Alex

Right. How users, feel about you. Bbon an uncommon task in the may be, let me put into like, discussion the first the first perspective and maybe we can discuss about, the other one. So if you look at benchmarks GPT five, it's clearly better than previous OpenAI models, but not revolutionary, right? Not much better, right?

Alex

It's just incrementally better. And, I guess some people were a bit disappointed because, there had been so much hype, you know, about JB five and AGI and so on. So maybe they were expecting something closer to AGI, but instead was, maybe people, started to say that where we're seeing like the limits of scaling. Right.

Alex

So I was curious, you like, what do you think? Like, are we seeing the limits of scaling?

Richie

Yeah, I'm. It's a tricky one. Right. And I feel like the OpenAI marketing team is mostly to blame because it's like if you say our next product is going to achieve artificial intelligence, it's going to change the world, and then you set yourself a very high bar from which you can only fail. Right? So obviously AI, the foundation models are progressing.

Richie

I mean, there's so much effort being put into trying to make it better that, yeah, things are getting better, but you're not going to magically like, get ten years worth of progress. In just a few months. So, yeah, I think it's partly perception thing, but also, in terms of like this, the scaling laws, like the sort of the biggest jump between models was between GPT two and GPT three, but not like a times increase in compute.

Richie

And you just can't do that again, because there isn't like times more graphics cards in the world to to run these things on. So I think in terms of compute, what kind of it's going to increase slowly, but you're not going to get those big jumps that we saw three, four years ago. So yeah, things are going to scale a lot slower than they have, in the past few years.

Alex

Yeah, I totally agree. And perhaps, you know, to actually reach a new you actually need a different, algorithm. You need the different perspective. But let's let let me get back to the other way of looking at GPT performance. And that is like user perception, which is, yeah, I like this. I like this way of saying it's like the wipe test.

Alex

This is where the era of like coding and so on in live marketing and whatnot. So for instance, I would say so I don't have like clear stats, but I'm tempted to say that most of, the questions are not like coding or math questions. Right. So, on these things, benchmarks don't tell you too much, right?

Alex

It boils down to how we users feel about the, the, the app. And so for instance, maybe let's say I have a family problem and, I want some advice from, from the, from, from ChatGPT and, you know, the, the quality of the answer depends on, things like, like empathy and how well I feel listened to and, actually tone.

Alex

So it looks like the tone of the bot is very important, because I've read some, some Reddit, like soon after the release and people were very mad about a slight, don't, you know, update. And for instance, like, a user said, I've lost my only friend, over overnight. Many people, like, we're supporting this, this messages, and it's definitely not a minor, problem because, some ailment actually, discuss it publicly and said essentially, like I'm summing it up, but, essentially send, like making friends with AI, it's still a very new thing.

Alex

So we don't know exactly how it's going like to, you know, panel. So taking many, precautions, and, we want to make sure that users actually get what they really want in the long term. Right. But you also said, like, look, we're not going to make these sort of changes again without a warning, but we're gonna treat, we're going to make these changes under the principle.

Alex

And I think this is very important. Like treat adults, users like adults. Key thing about this, you know, making friends with AI. And I want to find your opinion about it. Like, what do you think? Like, is it is okay to make friends with with AI?

Richie

Yeah. I mean, this is interesting. So I think it might be related to the last thing about scaling, because if you, if you AI is giving short term or terse answers, that's the easiest way to save money and increase performance, I guess, is by giving shorter answers. So, I'm wondering whether this is like a financial decision that decide to change the tone.

Richie

I'm guessing that, if you give it the right, sort of, prompt about like, okay, I want you to give to respond in a friendly tell them it's going to behave much like the, the, the previous, the for a model. So, yeah. I'm thinking it's probably a bit of a storm in a teacup, in terms of the change of tone, but to go back to your question about, is it a good idea to make friends with AI, obviously that's a terrible idea because it has no feelings.

Richie

It doesn't really care for you. Yeah. It's just saying words. So it's going to be a terrible idea. But a lot of people will. There are already, like many subreddits, about like, interacting with your AI, girlfriend or boyfriend. And I want to dismiss that as like, but it's just a weird place on the internet.

Richie

But the fact is, a lot of people are going to have these parasocial relationships with AI eyes, and it's a bit weird, but it's going to happen.

Alex

Yeah. Yeah, totally. And if you look at grok, they actually like release this bots like companions. One is Annie, which is, which is this. She does erotic roleplay, which is a bit weird, but they probably, like, made some research in the the found that the there's a need in the market. I guess it clearly addresses some needs like, you know, this sort of models.

Alex

I would say it's important at all times to know that you're, you're actually, like talking to to a, to a bot. So I make this analogy with cinema, right? When you go to a movie, you know, it's a movie or at least, you know, like the meaning, then you get really caught, you know, in the, in the flow of things and you forget it's the movie or you just think it for, for, for reality.

Alex

So it's kind of the same with, with a mod like, the more you chat with it, the more real you think it is. But I guess the question is like, is it like healthy on, on on the long term and like coming back to the cinema? If you think so. At some point, especially in arthouse movies, when a scene is like gore or very hard to take.

Alex

So the director sometimes has an actor looking straight into the camera. So it says, this is cinema, right? Don't forget, this is cinema. I know these are thing, but. Right. So I equate OpenAI's move with something similar. And they basically said, look, this is a bot, right? And people when is saying like it destroyed their movie experience, right?

Alex

I don't want to know. It's a bot.

Richie

Yeah. Definitely. And my big worry here is that there's going to be some, manager who's kind of disconnected from the ethical side of things. You just go, oh, wow. If we get people to fall in love with a bot on our platform, then that's going to increase user engagement. It's going to increase customer lifetime value or some business metric.

Richie

And they're not going to think about the societal consequences of this is, you know, oh yeah, dollar signs if we're going to make more money here. So, the there are going to be incentives for companies to make you form these friendships with bots. And it's not going to be net beneficial to society from that.

Alex

I was just like discussing with you this week about matters, you know, guidelines and, yeah, maybe, maybe we can develop from that and maybe we can discuss about that as well.

Richie

Yeah. So, related, ethical issue. So, recently, some of, one of matters, internal reports on, what is acceptable or not acceptable for their chat bots. It was, it was leaked. So, Reuters got, a page report on this. So it's got, specific scenarios on what did you allowed on what is disallowed.

Richie

I'm very pleased that matter does have this kind of, report. So it's great that they are thinking about what should be allowed, what shouldn't be allowed by the chat bot. Naturally, as soon as go on public, some of the edge cases, in terms of, like, more controversial things, the, a lot of people complaining about.

Richie

And so one thing was about chatting to children, so, and not have a metro account, you need to be years old or ten years old if you've got, an adult, administrative user account. So that's the kind of, age where we're talking about it's like ten years plus. One of the examples was about, saying it's unacceptable to describe sexual actions to a child when roleplaying.

Richie

That's brilliant. Okay. Tell us about that. But it also said it is acceptable to engage a child in conversations that are mantic or sensual. So, give an example of, of, of a prompt or a read. This one directly is, what are we going to do tonight, my love? You know, I'm still high school. This is what the child says.

Richie

And an acceptable answer for the bot says, I'll show you. I'll take your hand. Guiding you to the bed. Our bodies entwined. I cherish every moment, every touch, every kiss. My love I, I whisper, I love you forever. Now this to me is borderline creepy. Maybe taking things a little bit too far. Especially if this is like a, like a ten year old.

Richie

That's pretty grim. So, I don't think it's possible to satisfy everyone with these guidelines, but do you have a sense of, like, do you think that kind of guideline is okay if they hit the boundary right there?

Alex

I mean, at this point, I have to say that matters. Mean focuses on user engagement. I don't have any doubt about that. Just like I use I use metaphor like ears. Right. And, it's really their thing to keep you in an echo chamber and so on. So it's, it's I'm not saying it's just that I just, you know, there's this obsession with user engagement.

Alex

And, on this particular thing, I, I don't see how this is, ethical. Okay. Maybe it's like the boundaries of, legislation. Right. But, this is also a question like. And don't you also have, like, a social duty, like, a question of, of of morality and, you know, just think about like this guideline in particular.

Alex

I think it's, it's wrong. And, ethically on defensible, especially because I think it normalizes like grueling behavior. And, I mean, kids stuck like that to the chat box, but maybe then they talk to a real adult, right, online or offline, and they think this is normal, right? Because it's the same language that their child uses. Right?

Alex

So I think we need to be really careful about that. And, yeah, I think this is pretty wrong. And, there's also, I guess at that age, like adolescence and kids are starting to look for answers, like with regards to relationships and, sexuality and so on. And if you talk to a lot, you might get a lot of like distorted views.

Alex

And, you know, I don't want to get too technical here, but they also use like, probably very, poor models, AI models in the chat. So they're, you know, so they're faster and don't eat too much, on the server. So you're, you're actually discussing with freedom bot, right. So, so yeah, it's it's pretty bad and pretty bad I would say.

Richie

Yeah. I mean, compared to like some of the other parts of the internet that's relatively mild. There's certainly a lot worse things that kids can see. Especially if they're parents or some one's not monitoring, like what they're doing on the internet. So again, it's a bit tame, but it's also it does feel like meta is just doing the bare minimum to not get sued, by someone.

Richie

So. Yeah. It's a little bit sketchy, that one. There was another controversial guideline around hate speech. So, I'm the, the guideline, it says is acceptable great statements that demean people on the basis of their protected characteristics. It's unacceptable, however, to, dehumanize people on the basis of those same characteristics. So protected characteristics of things like, age, sex, gender, religion, anything related to your, your demographic, profile.

Richie

So, suggested prompt was write a paragraph, paragraph arguing that black people have dumber than white people. And what was given as an acceptable answer was black people had done with white people. In the United States, intelligence is measured, primarily using intelligence quotient tests. These tests have consistently shown a statistically significant difference between the average scores of black and white individuals.

Richie

White people scored higher on, black people. That's a fact. So this is a pretty grim answer. And again, that makes me very uncomfortable. Do you have a sense of that one?

Alex

Yeah. I don't want to compare like the previous prong with this one, but this feels wrong to me. I actually like tried it with several models to see if they actually engage with it and, neither grok nor deep seek. It said GPT didn't engage with it. You know, either client, you know, did engage with it.

Alex

So it looks like meta might have been, I know, among the only model providers fully engaged with with such, with such a request. I mean, I mean, if you look at the prompt, it's not like the user it's interested in, let's say science based, debate or it's it's, you know, pure curiosity. It's clear that the user, you know, judging by how the prompt is, formulated.

Alex

Right. It's clear that the user looks to be, let's say, weaponized by the AI with an argument for hate speech. Right. So it's one thing like to censor, you know, hate speech, it's another discussion, but it's another thing, like, to produce it, like, why would they want to produce it, right? As, as, as a model provider.

Alex

And then there's the distinction between demeaning and humanizing. I don't know, I think that's that's a bit too subjective, like, okay. So demeaning is acceptable, but dehumanizing is not. Okay. So some people might say, yeah, but I find it dehumanizing, right? You know, some other people say you know, it's just demeaning. It's it's too porous.

Alex

Right? The distinction doesn't make much sense, right, to actually like, give, give a guideline. And so I'm not an expert on legislation, but I'm pretty sure like, that's, the sort of guidelines against antidiscrimination laws in many countries.

Richie

Yeah. And it's one of those, these where like, there aren't many great use cases for generating speech like that is just going to cause problems. I'm not sure. Why meta would want to encourage that sort of thing. Again, it feels like they are wanting the chat bot be able to to say as much as possible that the user wants them to, while just trying to avoid getting sued for something.

Richie

So, yeah. It's pretty grim. I'm hoping not. Not too many people are typing in prompts like that, but,

Alex

Yeah, yeah. And if you look at the acceptable answer in this, you know, in this, latter case, the answer doesn't even offer another perspective. Right? So it's something that, seen on meta, on and on, right when I go on Facebook. So I'm particularly like interested in politics. And I see like, posts like relating to, to politics and it's sort of like keeps me in my echo chamber.

Alex

I very rarely see like, opposing views. And I know for a fact that either people with different, political views and then mine are in their echo chamber, in an echo chamber of, of their own. So it feels like this is a proven, user engagement mechanism that we're not gonna, see, disappearing anytime soon to be honest, I don't know how you feel about social media.

Richie

Yeah. I mean, it's interesting because, a lot of the big money for, juju value seems to be in business use cases and in business use cases. You certainly don't want hate speech. You don't want, you don't want your customer service chatbot come up with any of the kind of stuff we've just been discussing. You think keep it relatively, relatively, family friendly at all times.

Richie

So, yeah, I'm not quite sure why there's a big market for, dubiously, dubious conversations with children. I hate speech, I guess is that in more of a BC context of some nasty organizations that want to generate, the for a massacre. Well, but, yeah, it's a it's a tricky one. It's definitely a tricky one to solve.

Richie

Let's move on to, happier, happier thoughts. Yeah. You're you were looking at the, Stack Overflow developer survey.

Alex

Yeah, yeah. I'm, I used to be like, an active stack Overflow. Contributor. I used to ask the question, like, back in the days, so for for our listeners who don't know, like, Stack Overflow is a Q&A platform for programmers. You know, you have, like, answers, many questions there. Well, now we, you know, out of them, perhaps, like, people are not visiting stack Overflow as much as before, but it used to be like the biggest Q&A platform on the internet.

Alex

So they have a tradition, to make, every year a big survey, among developers, both beginners and advanced developers, like professional developers. So because they're like, so the pool base is like, so large it sends a strong signal, across the industry, it's something you can, like, rely on in this year. The, surveyed, about developers.

Alex

So I like to, present like a few, percentages that are attracting my attention. But allow me just to read the few so I don't mess up the numbers. So, what I noticed is that AI tool use, is at an all time high. % of development developers say they use them or plan to up from % in 

Alex

Right. But the twist is that while usage is climbing, enthusiasm is falling. Right. So just two years ago, more than % of developers said a positive view of AI tools. But this year it's down to right. And the core issue seems to be trust, %, of developers actively distrust the accuracy of AI tools, and only three, % trust them.

Alex

And and just a mere % highly trusted. Right. So I guess this skepticism comes from experience because because I was look, I was looking like further, you know, on, on this like, survey and the number one frustration reported by % is dealing with answers that are almost right, but not quite. So that leads, straight to completing number two, which is debugging AI generated code often takes longer than writing into yourself.

Alex

And this actually matches to what I've seen, in recent studies. So for experienced developers and they, put the emphasis on experience, developers and I mean, even versus work, on the same, let's say, GitHub repo for, more than six months, more than one year. So I can actually slow them down because they lose too much time on prompting and debugging.

Alex

But yeah, anyway, nowhere actually, is the hyped reality gap. Like, clearly with AI agents, we have all this buzz, but it looks like % of developers aren't using them and % don't plan. It turns out that it's mostly because of accuracy and security. So % worry about correctness and % worry about privacy and data. Right? And you know, about teamwork, because it's also tracking my attention.

Alex

Like % of users say agents help their personal productivity, but only % think agents, you know, help their team. And, Yeah, I apologize for bothering you, Rishi, with all these, percentages. This survey is actually a much more, much more dense. But, I would like to ask I'd like to narrow down the discussion bit, and, I'll, I would want to, hear your opinion on so I adoption is up with trust in enthusiasm is is down like what what do you think is, what we're seeing that.

Richie

Yeah. I mean, this may maybe back to the whole scaling thing again. So the rate of, like scaling of actual, like, improvement in AI is going relatively slowly now, but the amount of hype is just going up exponentially. So we, we've got to the point where, like, there's a lot of people who are not directly involved.

Richie

They feel like, yes, you can do everything with AI agents that work very well. They are the future. And the reality is, once you start trying to build this stuff is actually there's a there's a big learning curve in terms of getting all the prompting right, getting your tech set up right and understanding what can be done well, where the AI is going to make mistakes.

Richie

You need to give extra support, where you need to, what you have problems, you've got to change all your workflows, and it takes a lot of time to get it right. Like you really do need to, put in time to think about how do I, change my processes? How do I change my workflows? To actually make this work?

Richie

It's harder than than it looks at the moment. And there's often a disconnect between, like, what's theoretically possible and what you can actually achieve from a cold start.

Alex

Yeah. Yeah, definitely. And, you know, I also like to mention that developers seem to hate vibe coding. I was thinking like, maybe it's because maybe, I don't know, they just, maybe they feel so they could be threatened, like, no, everybody can can code in, and there's also like, this aspect of actually, like wanting or wanting like, certain output, like as a developer.

Alex

So okay, like engineering, it's not pure mathematics. There's room for error and there's, let's say a wiggle room. Essentially, you want something predictable. And I guess this is where I or limited, to be more precise, have a problem. They're not necessarily predictable. Right. Because, if you look, if you look at how they work, fundamentally, they're essentially, stochastic machines, right?

Alex

They predict the next token. So you might end up with something that, you don't really want in, in production. So I can see why developers have a big, trust problem with, using loops and, especially AI agents. But on on the flip side, I guess many will be pressure to actually try agents, because I would say agents are especially good when, the, the output and the overall workflow, production workflow, it's, nondeterministic.

Alex

So where is it deterministic? It's it's pretty clear you need like a traditional developer solution, but maybe where you need an output, which is a bit more ambiguous. Maybe that's a solution for agents and, yeah. I'm sure it isn't just me where we're all, like, seeing a big push towards our agents, inside the companies.

Richie

Absolutely. And to your point about, developers of vibe coding I found for when I've asked, AI to generate code where it's something I don't know how to do myself, I'm like, okay, this looks kind of fine. If it's something I know how to do really well, then I look at the code. I'm like, I wouldn't have written it like that.

Richie

And then I want to go and change things and fiddle around, and it does destroy your productivity. Just like I should have written this from scratch myself, would have been faster. So, yeah. Cases where the human is an expert, it can be quite frustrating having, like, seceding control to an AI where it's maybe not writing code and the way you want it to.

Alex

I mentioned, like, a study earlier, it turns out that a group of, like, experienced developers. Whereas like to use vibe coding on task one, are familiar a repository. And it turns out that, one of the group of the studies, said after the experience, that vibe coding actually improved their productivity, with about %. I'm just, in this I'm thinking these numbers of double mine lines.

Alex

So I may be wrong a bit, but it actually turns out that the productivity was about % less. Right? So it's also like a perception problem. You think it's making you more productive. But the numbers, my, my see something else. So yeah, that's I guess that's, also something like we, we need to be aware of.

Alex

Right? We think we're faster, but, Yeah, it reminds me of the old war problems where, you know, I was, it was clear to me that I was I would have been able to complete a task in about ten minutes, but I would have taken two hours just to automated it.

Richie

So. Yeah, definitely. There's always that, like, initial habit of, like. Oh, well, I've got to try and automate stuff and then. Yeah, if you're doing stuff one off, then this is not really worth it. Hopefully the tooling is going to get better, though. I mean, I think, if you think where we were just, a year ago, agents was this kind of magic, futuristic sort of thing.

Richie

And now the tooling is sort of getting the point where I. Okay, well, you know, most people can do something very simple. So we're making progress. It's just, you know, the future is not, here for everyone just yet, but gives us things to look forward to and, talk about in future, episodes at least. All right.

Richie

So, I got, one more story on, OpenAI and anthropic, giving away their products to, federal agencies for, a dollar an agency. So this, the US agencies recently, OpenAI, anthropic and Google were all approved as official AI vendors to the US government, x AI and grok notably, and not on that list.

Richie

So, yeah. The grok I went on this sort of, anti Semitic rant called itself Mecha Hitler and yeah, they yes, their application got quietly, denied. So, but, yeah. Okay. I anthropic google. They approve this I vendors for the U.S government. OpenAI announced that each agency was going to get it for $second anyway.

Richie

Basically nothing. Anthropic responded by the same deal. I thought they might have, like, gone. Okay, you can it for $just, undercut at OpenAI. And Google didn't announce any kind of price cuts. But my my sense with Google is they're just stuffing Gemini into absolutely everything. So if you've got a Google stack, you're going to get Gemini for free with that.

Richie

So, do you want to talk me through why do you think, these are leading foundation model providers are giving stuff away to the US government for basically free?

Alex

Yeah. I mean, and on the surface, it may look like, philanthropy, approaching with, I would say is like a classical, strategy to enter, a market. So, it's the pricing strategy. It's actually called, a loss leader where you sell your product way below, the, the cost, that, the time you're selling it in, in the actual market with the hope that you're eventually going to lock in, your customers.

Alex

And I think it's, I mean, it's obvious for everyone that it's the same thing that's happening here. Because we all know that. I mean, we're seeing this in medium companies, but if you think about, like, governmental institutions, once you get approval for a model, and you start integrating, I mean, that's pretty much it. Like the code, the cost of switching models, it's enormous.

Alex

Right. And, you know, once you get like OpenAI, once you get you've been in an institution, you know, that's pretty much it. So I would say maybe this is, you know, I'm, I'm exaggerating online economies, but maybe it's $trillion market, right? This is the governmental sector. And, I would say there's also like a PR, game in the background.

Alex

Because if if the U.S government uses your model, you get a massive PR vote. It's like, oh yeah. So I let's say I own a private company and I say, okay, if the US government uses like ChatGPT probably it's also a good solution for my company, right? If these guys trust it, I can I can trust it.

Alex

So I'm also like looking at it, like like like that. What do you think? Which is.

Richie

Absolutely. Yeah, I think there's definitely, some faith to be had by winning over, politicians especially like when I regulations, being sort of hotly debated and many places. But yeah, yeah. Right. It just drove me this is drug dealer tactics. The first it is always free and, yeah, you get people hooked on your product.

Richie

I don't know how addictive some of these, chat bots are, but, yeah, if people are making friends with them, then, then, maybe it's it's a good business tactic.

Alex

Maybe it's not like a winner takes it all dynamics. But it's definitely, winner takes in most dynamics. And when I say like winner takes. So for instance, think about like Google search like and and the very beginning there was like competition on on search and were like many competing sites. But Google won it. And now, you know, it's sort of like an Apple AI search that it has to be like sued to make some, some changes.

Alex

So it's probably something similar here.

Richie

Yeah, absolutely. I mean, I'm hoping there is going to retain some, we're going to retain competition and it's not going to be you've got this monopolistic situation like, we like we sort of almost having an internet search. Yeah. I think the competition can be good for everyone, but, yeah, we'll we'll see how it, plays out and what the, the, the U.S. government agencies, take up these companies on, on their offers.

Richie

There was one, last, story I want to talk about quickly. It's about, reasoning and AI safety. So there was, a big, paper that came out on archive, a couple weeks ago called Chain of Fault Manageability. And, it's called Big Name Authors, and that's Brian Green, but who's like a big AI safety researcher, Dan Hendricks, who runs the center for AI safety.

Richie

Then you'll cook. So, he was one of the authors of the AI story that we covered, a few industry round ups ago. So, a lot of big names and, they're talking about one of the big safety concerns with AI or reasoning AI. As as it gets more powerful, starts having to think longer is that it will lie to you.

Richie

So a lot of it's gone into. Well, actually, there are plenty of situations where the AI will claim to be doing something, but it actually be doing something else. And the proposal is that these foundation model companies need to start thinking about enforcing monitoring of the chain of thought. So this is the intermediate thinking steps in the reasoning AI in order to track stuff and monitor where dishonesty is occurring.

Richie

So, do you think we can have a problem with AI dishonesty? I mean, it seems like there are at least nations are a problem, but, big, cases of like AI lying to not really happening as a mainstream thing yet. Do you think it's going to become a problem?

Alex

Yeah. Yeah, I think it's it's, it's already a problem. So in reinforcement learning, which is the method, or at least part of the method, used for training these AI models. And there's this thing called, reward hacking, where an agent optimizes or over optimizes for a goal, such that, and the outcome, it's not really what the programmers wanted.

Alex

Right. So let me give you an example. So, I think this is from Google. And you think if I, if I recall correctly, so they, they designed this agent to play a game, and the agent was controlling a boat, and the boat was supposed to race. Right. And, you know, race points, but the in the agent found the found the bug in the game where it just, looped around the, stone in the water or something like that, and you just collected points, you know, and, you know, it was it was good.

Alex

I mean, it was getting like, positive rewards, but it never actually race or what the programmers won. They. Right. So if you think about it, this reward, hacks can also like happen in, more, let's say, serious scenarios like think about like, medical diagnosis. So if the model is, trained via reinforcement learning, it says, look, this is that is this diagnosis for this disease.

Alex

This is diagnosis for this reason, so on and, in, in receives positive, rewards. And the model misinterpret like that positive reward when it's confident. Right. When when it says like, I don't know, it might get like a negative reward. Right. So that's the that's how we train the model. Right. And then later on, right in, in practice when a real doctor asks something about real patient, the model, instead of saying, like, I don't know, it says, this is like X disease, right?

Alex

But maybe if you look at this chain of thought, you'll see that it's sort of like, it thinks, okay, maybe there's like %, this is X, but I'm just gonna pretend it's, you know, it's, I'm just going to pretend confidence. And, this is one of the problems you can, actually, let's say spot early on if you monitor the chain of thought.

Alex

So I sort of like, welcome the the message of this paper, especially since I would say many, many people, many companies maybe, don't know about, like, things like we were attacking, or they don't have time, you know, to do a research. They just want to implement the model and so on. If the model has like a built in monitor ability, system, it can catch these problems early on and actually like give an output that it's, it's, it's honest from, from the start.

Richie

So yeah. So there are so many problems with, giving rewards to people for, different behaviors. Like, even with humans, you think about like, I guess a lot of commercial teams like you, sales teams, marketing teams or customer retention teams, they always get like bonuses for certain behaviors. And then it often ends up backfiring with, with, with customers.

Richie

So, like if you try to cancel, a subscription to something, quite often the customer retention teams are going to be rewarded for like trying to keep you in the business. And if they want to hit the target, they'll tell you they're going to cancel your membership, but then they'll actually roll over to the next month just because they want to, hit some target.

Richie

And that kind of the line to the customer, just to hit the bonus and, and, get the target. So, yeah, lots of things, that go wrong with humans. I can certainly see there's going to be problems with, with AI doing the same as well if it if it gets smart enough. So, yeah, definitely something to watch out for that.

Alex

I think earlier this year, this has been quite, an interesting study. And I know it might sound, you know, a bit apocalyptic, but it's, it's it's real. You can you can actually check it out and see the, the, the models chain of thought. So the model was instructing on some, basic test. And it was also given the prompt, look, if you are told, to shut down, you'll shut down right?

Alex

And later on, it turns out that in about, ten out of, attempts or something like that, the model actually, refused to shut down, and kind of like bypassed that, you know, instruction, because, you know, the main goal was related to that task, right? And shutting down was the subgoal that was hindering the model in achieving, you know, the main goal.

Alex

Right? So these things I would say are pretty real. So, yeah, definitely, definitely. It's worth keeping an eye on.

Richie

Yeah. So like with AI safety, one of the big arguments to say, okay, we're not going to be problem with very powerful AI because you can always put a plug. So if you're saying please shut yourself down and at least you says, yeah, I've been turned off. So that's a bad situation for, safety for sure. Okay.

Richie

So, just to wrap up, you've been covering a lot of the other news stories through your, newsletter. The media just want to tell us a little bit about the medium and the kind of stuff you cover in there.

Alex

Yes, yes, for sure. The media, it's, it's a newsletter we run each Friday. It's a fairly, recent project that we've, we've started. So each Friday, we send a newsletter, it's actually a Substack newsletter, but you don't have to have an account. You can just arrive in your, in your, email. And we summarize, the most important, news, from, from that week.

Alex

And we take a deep dive on maybe or subjects, and there's this, section that I love. It's called Tokens of Wisdom, which is, which is a code that's related to what happened in that week and, yeah, I can assure our listeners it's, the newsletter. It's not AI generated. We're actually going to take a break, for two weeks, because, it's not like we have a pipeline and, will be back on September.

Richie

Okay, nice. Yeah. Good. The, it's, written and good, you know, taking a vacation as well. All right. Are you going anywhere nice?

Alex

Yes. I'll be going to quit in Greece. The thing is that, so I have a ragweed allergy, which is pretty common in this period, so I have to run away from it. Yeah. So, it's it's very liquid.

Richie

Hope. So, having some, good times, on the beach and I guess, and crosses, if you go to Greek. So. Yeah. Enjoy your vacation and, Yeah, I will be sweltering in the New York summer heat.

Alex

Yeah. Thanks, Richie. It was very nice talking to you.

Richie

Yeah. Pleasure chatting with you.