What's Your Biggest AI Ethical Nightmare? | Reid Blackman, CEO at Virtue Consultants

unno how old you are, but you look to be about my age. So we didn't have that many channels growing up, there was not that many news channels. We watched the same news, we watched the same TV shows. Now everyone's got their own little, tiny bubble of media and it's, everyone's thing is different and, so much for communication and as communication, crumble. So as collaboration. So that's my nightmare. 

Richie Cotton: Yeah, absolutely. Yeah. So in England, when I was a kid, it was four TV channels, then maybe five when I got a bit older and yeah, now that's it feels like nothing. 

Reid Blackman: Edward was, they all just called B, C, 1, 2, 3, and four.

Richie Cotton: Certainly b, B, c, one, and two. Yeah. 

Reid Blackman: Okay. 

Richie Cotton: Okay. Half. Half. Half the tv. Yes. But last time you were on the show, you were talking about responsible ai. So why have you switch from responsible AI to AI nightmares? 

Reid Blackman: I probably talked a lot about last time, not just responsible ai, but AI ethical risk.

And I tried to put the emphasis on the risk side of things. I think that more and more we just have to face that what we need to focus on is avoiding the really bad stuff. Responsible AI sounds lovely. We're gonna be responsible. We're for fairness and privacy and accountability and hurrah. But no one really jumps outta bed to pursue these sort of lofty ethics, ethical values.

They will though, or at least they're more likely to jump outta bed to avoid disaster. And so I think that when we're talking about capitalizing on the genuine opportunity that AI is, and I'm no. Pessimist or skeptic. I use AI every day. I think it's absolutely incredible. I think there are tremendous benefits in the offering.

But we have to use it in a way that avoids the worst case scenarios. And so what are worst case scenarios we need? Those are nightmares. That's the sort of story here. I can, I keep talking. I can say more stuff, but that's the core of it, 

Richie Cotton: although, yeah. So it feels like a a fairly low bar that's just let's try and avoid the worst possible thing that can.

Reid Blackman: Yeah, I don't think that's totally unfair. I say in the book, my new book this might seem like a low bar, but actually given the severity and scope and probability of the potential nightmares, it's actually not that low of a bar. It still takes quite a deal of organizational effort and to clear that bar.

That's one thing to say. The other thing to say is that the bar is set appropriately to. Get motivation. I think that some people think, Hey, you know what? Let's say what the values are. Let's set the ideal. Let's shoot for the stars. And if we fall short, at least we'll land on the moon.

And I think that if you tell people to shoot for the stars, they'll be like, I'm just not gonna get outta bed. Forget I'm not gonna get there. You're not gonna get CEOs and CIOs and CTOs and all the C, all the csuite and the board. To really rally behind pursuing the ethical ideal, forget it. That's not, that's just, that's not priority number 1, 2, 3, 4, 5, et cetera, but they can rally around.

Let's avoid the disaster. And so I think we need to be, if we set realistic goals. Which are incidentally, not just more business wise, if you like, more important, but ethically more important because if anything, if there's an ethical rule, number one, it's first do no harm. And so let's avoid the ethically nightmare scenarios before we start worrying about what's the ethical ideal.

The other thing I'll say I can say lots about nightmares that I like about them. They are clear, they're concrete, they're specific. There are articulations of outcomes that we have to avoid and if you say to someone or to organization. You're safe for fairness. What does fair outcomes look like for you?

You'll just get blank stares, but we are unfortunately quite practiced at articulating nightmares. We're very experienced with nightmare scenarios. The history of humanity is un unfolding, series of nightmares. Some good dreams as well, but lots of nightmares. Books, TV shows, movies, plays, they're, more often than not, they deal with, at least if their dramas they're dealing with nightmare scenarios. So we are very good at articulating various ways of hell and, what hell could look like in a variety of ways not so good with heaven. 

Richie Cotton: Okay. Yeah I do know the idea that everyone's had a nightmare.

Everyone knows what a nightmare is, and so it's a bit easier to articulate what's going on. 

Reid Blackman: So I'm gonna, on that point, and this actually speaks to the, my sort of personal nightmare about informational system collapse is that. We need people speaking the same simple language if we're going to avoid the bad stuff.

As you just said, everyone understands what a nightmare is. Everybody's has one. Everyone has had one. Pretty much, I would hope everyone in our organization can tell you what the nightmares are that are relevant to their role, to their jurisdiction. If they can't, that's scary. Then you have a, an employee on your hands that doesn't know what would constitute disaster for them and their organization.

That's a dangerous employee. We need clear communication so that we can get collaboration, because as I'm sure we'll talk about as this conversation unfolds. The only way we're going to avoid the nightmares is if we have cross-functional, cross-departmental, cross expertise, collaboration on avoiding the nightmares.

Richie Cotton: Yeah, certainly talking to people in different departments, certainly across the organization, often very difficult. So I like the idea of making things simple to communicate. So just to make these nightmare ideas. More concrete. I think a few years ago we were talking about predictive AI or machine learning as it was called back then the last few years.

A bit about generative ai. Now we've got ai. Can you gimme some examples of nightmares for each of these? 

Reid Blackman: There's a set of nightmares that starts with. Or in narrow AI or traditional machine learning, and it counts for, or it occurs both for that traditional machine learning and for everything that comes after it.

So generative AI and age agentic ai. And then there are those risks that are those hidden nightmares that are built into generative AI that also apply to age agentic. And then there's agent age agentic specific nightmares. So I'll just give a, I'll give a couple. The one that everybody does is hallucinations.

This is this is common. So this is when a generative ai, so like an a large language model, a chat bot, a chat gt Philanthropics claw, or Google's Gemini, wanna just make stuff up, just totally make stuff up wholesale so that's where you get hallucinations, which of course, in some context to be quite dangerous.

So if you're. Certainly in a healthcare situation, if you're talking about diagnoses and it hallucinates symptoms or hallucinates potential treatment plans or, drugs that exist, that's a problem. We've had lawyers present case cases to not just, this was literally in the news, I think yesterday in New York Times a big law firm.

I don't remember the exact details, but a big law firm presented filed an official document with, I think it was a federal judge, and it contained hallucinated. Material. 'cause an AI wrote a good chunk of it and hallucinated material. That's crazy. If you're doing research for investment purposes on different companies and the AI hallucinates various facts about that institution that you're thinking about investing in that could be bad.

Especially if it makes you think, oh, they, this is a safe bet. This is a good bet. Lemme put, tons of money in there. That, that's dangerous. So hallucinations are obviously dangerous. So that's what gender of ai. There's also, this is relevant or related. We are all given to humans that is, are given to what's called automation bias.

So this is less an issue with AI and more an issue with human beings. We tend to trust the outputs of an ai, we think, or sorry, with computers more general. We just think, yeah, it's a computer, it's got, it doesn't make mistakes. It's more right than we are. And so to take a non-AI example, or slightly non-AI example, people are overly deferential to GPS.

So you've had people drive into lakes and die. You've had people drive over bridges and die that collapsed. 'cause the bridges were defunct because, Google Maps said, take a left, and you're like, this doesn't feel right. But I actually almost did this myself. I didn't almost drive into the lake, but I was, it was a, I was on an island and driving along the coast, the island, and it said bear left.

And I thought, that doesn't seem right because it seems like the streak was right. It seems like to the left is a harbor, but I don't, and even I second guess myself. I pulled into the Har, I was like, the second I pulled it, I was like, I'm such an idiot. What? Jesus. But it's worse with AI because it talks to you and it sounds so persuasive, and it sounds so smart.

It sounds so intelligent that you think, yeah, this thing probably knows what it's talking about. And so this sort of pours gasoline on the hallucination problem because it's gonna make stuff up, and it's making stuff up in a context in which you're more likely to defer to the thing. So that's a problem I'll say.

I don't know. I can keep talking, but I can talk. Do you want me to stop there or do you want me to go on and talk about agentic AI risks? 

Richie Cotton: Go on. Yeah, tell me about agentic AI risks. 

Reid Blackman: Okay, first we should say what agents are, 'cause that's probably alludes most people's experience or understanding so far, which is fine 'cause it's pretty new.

So an AI agent, as I define it, and I think this is a fairly common definition of what constitutes an AI agent. You have something like a large language model. That's in the center of it, and you connect it to various digital tools. So you connect it to, if, let's say it's in a company context, an enterprise context, you connect it to your CRM, you connect it to various enterprise databases, you connect it to various enterprise software, proprietary or not, you connect it to the internet.

And so now your large language model has all these tools that it calls upon. When you ask it various questions, so oh, I, Hey, I want you to do this thing for me. I want you to go to database number three and pull that data and run it through narrow ai, number five, and then do the search on the internet.

So condition number one for being an agent is that it's connected to all these tools. Condition number two though, is when you stop saying, here's exactly what I want you to do, step by step. And instead you just say, here's what I'm trying to accomplish, and it. Figures out which tools should it call on, in what order, how should it call on them, et cetera.

Should it do an internet search? What should it search for? What information should it pull from the search results? Which databases should it pull from within the enterprise, so on and so forth. When you've a given it a bunch of tools to draw from, and B, given it its own quote unquote autonomy or latitude, or freedom to figure out for itself the means to your user specified end.

Once those two conditions are met, now you've got what's called an AI agent. So there are three problems to highlight here, and I'll go through them relatively quickly, I hope. The first is what's called cascading failures. So the non-AI example that I like to give is, and I talk about this in the book as well, I'm walking through the.

The kitchen. I have a young child. I step on Allego, as everyone always does. Parents frequently do Step on Allego. I put my hand on the counter to study myself, but it's too close to the stove. The stove is on. I move my hand. I hit a glass. The glass goes flying off the countertop and shatters on the floor.

So that's just, that's a snowball effect. That's cascading failures where one thing went slightly awry. I stepped on the Lego and then it just leads to error until you have, shattered glass all over the floor. It's the same thing with age agent systems. You've got all these tools to which it's connected.

What if something goes wrong in one of those tools? Maybe it's just a, it's literally a kind of accuracy level error. Maybe it's a bias in some way. Maybe it's a hallucination, maybe it's a, some private data that shouldn't be in there that gets accessed. And then that one thing, that something went sideways with that tool, it then echoes and amplifies throughout the system.

You've got cascading failures throughout the agent system until at the end you wind up with, metaphorically speaking, shattered glass all over the kitchen floor. So that's cascading failures. Second kind of issue is systemic problems or emergent risks. Or systemic risks. So the way that I like to think about this is you could have, let's say two basketball players.

One's really great, they're both great players, but one's really good at a fast game and one's good at a slow, deliberate game. And while they're each excellent basketball players, they've got their respective styles. They're not very compatible style, so they play well individually. They're team, but sorry, but they don't make a great team.

You put them together and they make an awkward, poorly coordinated team. Once again, you can have the same thing with age agent systems, because age agent systems are systems. They're holes, beta parts, all the tools, the LLM, twitch, the to which all those tools are connected and maybe there's nothing wrong with any individual thing in that system.

It could be that there's a risk that pertains to all this software, non-AI and AI software together. They don't, they just don't play well as a team. And so you get these big problems like, whatever, it could be some output that you really don't want or decision. In the case of ENT care, some decisions you don't want to, or if you give it permission to take actions you don't want to take.

The last one that I'll highlight and then I'll pro, I'll finally shut up, is just the risk of autonomy. So the appeal is that you say, Hey, go do this stuff and figure it out. And so it does all these things, connects to all these tools, makes all these decisions, it does all these things in some order, whatever, and then takes actions.

Maybe that action is making a purchase, maybe it's writing to a spreadsheet, maybe it's whatever, sending a bunch of information to a client, what, whatever the action happens to be. But if you're not paying attention, things can go very much sideways because they move at speeds that defy our human understanding.

We're talking about computer processing, pr, computer processes that occur blindingly fast and immediately at scale. And so if you can't keep a close eye on it, if you can't monitor it appropriately. Autonomy is the risk because now if something goes sideways, let's say it's a cascading failure or systemic risk, and you've given a bunch of autonomy, and so you walked away from the thing, now you've got, it's if you like, not potent in a way it's not a risk unto itself, but it's pouring gasoline on the fires of all the other risks that could just completely spin outta control as you walk away and the agent does its thing.

Richie Cotton: Okay? That's, a series of increasingly terrifying problems there. 

Reid Blackman: Yeah. Good. So that that, that should be the title of the book. Maybe a series. 

Richie Cotton: Yeah. It also sounds a lot of the problems with tic ai then is that it's really difficult to work out what's gone wrong. Like you, if you walk into a room and there's just broken glass on the floor, it's difficult to trace that back to someone stepped on some Lego.

Reid Blackman: Yeah. And people are, I think, misled by a certain. Some cases helpful, but in other cases, unhelpful metaphor about what an AI agent is. So we're told not unreasonably to think about AI agents as like assistants, as like executive assistants. So you go to your assistant, Sam, who's a person, and you say, Hey Sam, I'm going to Vienna.

I want you to book me flights. Here's the kind of seat I want. I want you to, I want you to find me a hotel within a mile of the city center. I need reservations for dinner for each, and I'm there. And then Sam goes off and does these things. Sam goes. To whatever hotels.com or kayak.com, whatever one goes to these days Google Flights, goes to whatever, goes to all the things, and then either one presents you with options.

And here's what I found what you want, which one do you want me to purchase? Or if you trust Sam a lot, Sam's got enough autonomy and he or she purchases the stuff themselves. And so we're encouraged to think about AI agents as akin to Sam, but. While that's helpful, in one respect, getting a grip on what are these systems capable of, or at least one kinds of thing that they're capable of.

It misrepresents the risk situation because when Sam screws things up, you go to Sam and you say, Sam, you screw that up. I said, Vienna, Austria, you booked flights to Australia. Screwed that up. Next time, make sure blah, blah, blah, blah, blah. So Sam gets it. But Sam is a coherent entity there. You talk to Sam.

What you don't do is you don't talk to Sam's medulla on Manata and his prefrontal cortex and his visual cortex and blah, blah, blah. You talk to Sam, but AI agents are not coherent entities in the way that people are. They are these complex engineered systems. They are an LLM, which is itself a complex engineered system connected to.

10 databases, which are engineered systems connected to other, narrow AI or traditional machine learning models, which are engineered systems connected to the internet, connected to. And so if you could think about your, in your head, the sort of blueprint of this LLM connect to all these things.

It's a very complexly engineered system. And when things go sideways, maybe you can fix it by talking to the LLM and saying, Hey, you really messed that up, blah, blah, blah. I just looked with Sam. But in. Many cases, the problem's, not with the LLM, the problem's with the tool from which it's calling it, and the LLM can't fix that thing.

So you've gotta go into this if you like, complex engineer system and realize, you know what? The poison is in database number three. That's where things are really going sideways. Or the, things really went wrong when it accessed narrow AI number five. Or it's the combination of da, the data from database number seven and narrow AI number two, that's what really led to things going sideways.

So you've gotta do this more complex investigation when it comes to the, so when you see the broken glass on the kitchen, it can't just be, oh, lemme tell my agent that the glass is broken. It has to be What went wrong was it? If you like the agent with which I'm speaking the LLM with which I'm interacting and having the conversation with, or is it one of the things to which it's connected and then we've, there's that means that there's lots to say about the importance of monitoring these systems, how you monitor them, what needs to be monitored and relatably, what does it look like to intervene successfully?

Because in some cases, and we're certainly getting there. I don't think we're there yet, but we're getting there. If these complex engineered systems are overseeing something really big and important. You don't wanna unplug the whole thing you want. If you like a scalpel, not a sledgehammer, and say We need to disconnect narrow eye number two.

Then that said, once you disconnect narrow eye, number two, you now have a different complex system which may have different immersion risks than the system that included database number two or whatever it was. So getting clear on how to monitor, what to monitor but also how to intervene, when to intervene, who should intervene?

That's also important. 

Richie Cotton: Okay. So with a problem like that where you're saying, okay, you need lots of monitoring in place you need to make sure that everything's working correctly and it's gonna be a challenge, being able to debug any kind of problems, it sounds like you really, you need a very intensive governance program in place before you start using agents.

Is that something you'd advocate for? 

Reid Blackman: If I interpret what you asked, caly, I would say something like, yeah, you need something pretty robust. You need something before you release the ages in the wild. You need ways of governing these things. I bristle me, I'm this is like I should be a trigger warning the word program.

'cause then it sounds to me like a very big top down thing. So we're a large organization, could be smaller and we're going to create an AI ethics or responsible AI statement. We're gonna create a policy that policy's ultimately gonna get approved by the board. Once we approve that we're gonna implement the policy across, all of our departments, our tens of thousands of employees, and that.

That kind of governance program is, it worked for narrow AI or traditional machine learning. I think it wobbles for generative and it falls apart for agent because it's just too damn slow. By the time a Fortune 500 company has a board approved enterprise wide policy, the next kind of AI has come along.

And so now you're trying to implement an outdated policy and you've gotta update the policy. And even when you do that, you know you do that for generative ai. Oh, thank God we did it. Now AI comes out, oh my God, we're still trying to update, we're still trying to implement this stale policy and oh my God, get the C-suite and the board back 'cause we gotta update the policy again.

And this is, and then you fast forward and soon plausibly anyway, especially in places like financial services, we'll have AI, blockchain solutions. We'll have ai, blockchain, quantum solutions, or AI quantum solutions. Cetera, et cetera. So we do need to robust governance, but I don't think the way that we've been doing governance of AI for the past six or so years is viable.

Richie Cotton: Okay. I think that's gonna be a kind of terrifying thing. If you can never write policies quick enough in order to keep up with the underlying technology that kind of governance program doesn't work. You need some kind of governance in place. How do you do this well and keep up?

What's the solution? 

Reid Blackman: I have my own solution, of course, which was developed in the book. And truth be told, I didn't even know the solution until I really wrote the first two thirds of the book. I had to really lay out all the problems before I could see the solution clearly.

Look before I plug my own solution. I think that once you take a look at how slow the standard approach is and how complex iGen AI is. You just have to admit that the standard approach doesn't work and whatever approach it is, whether it's, read Black bin's, ethical Nightmare Challenge or something else, you need something that can be rapidly implemented and scaled on an as needed basis.

It can't be this top down slow. It's gotta be, to use a popular word, agile or dynamic. It has to be flexible enough to accommodate different kinds of technology changes in the technology. So that's one thing to say, but now I'll tell about my particular solution. The easiest way into my core, into my solutions.

Think about two sort of core pillars that are tightly related. One is what I call the ethical nightmare Challenge is a articulation of the problem or the challenge really. It's first. Three parts. First, tell me the AI ethical nightmares that pertain to your organization. Incidentally, a lot of people can't do that.

That should be scary if you can't identify what the nightmares are. Ethical, reputational, legal nightmares are for your organization, that's a problem. So what are the AI ethical nightmares that are relevant to your organization? Two, what resources will you build in order to avoid those nightmares?

And three, how will you train your people to use those resources effectively? What are the nightmares? What are the resources to avoid them? How will you train your people to use those resources effectively? Okay. If you can answer those three questions, that's great. In a systematic way, which brings us to part two of the solution, which is what I call Ethical Nightmare Challenge or ENC teams.

There's obviously lots to say about this. I've got two chapters on them, but the mission of an ENC team is to answer those three questions. Using, as I spell out in the book, a seven step method, put that to the side for now. What's crucial is that these three questions can be asked and answered at any level of the organization.

So you could ask it at the project level. Okay, we've got this particular AI solution. What are the ethical nightmares that are relevant to this particular AI solution? What resources do we have to avoid those nightmares? How do we need to train our people and or our end users to avoid the nightmares for this particular IDX solution?

Okay, you only ask that, answer those three questions, you create an NNC team to answer those questions on a per project basis when it's needed, not. Some AI solutions are so obviously low risk that you don't need to do it. Great. Don't do it. If it's sufficiently high risk, you create the ENC team and you have them do their work, but you can bump it up a level so you can go to, let's say the department or division level, whether it's marketing or HR or product.

And so you could say, okay, what are the ethical nightmare AI ethical nightmares for the h for hr? What are the resources? You're in Europe, right? So it's HR, but we'll put that, we'll, we'll allow you that Mispronunciation. What are the ethical languages that pertain to hr?

What are the resources that HR has in order to avoid those nightmares? And how do we need to train HR personnel to use those resources effectively? And so you can keep repeating this at different departments, different divisions. Notice that if HR is using a lot of AI and marketing is using none, then don't create a marketing ENC team.

Create it for the HR team. I want you also to notice that what they're fundamentally engaged in is problem solving, not compliance with procedures, because they're trying to figure out what are the actual nightmares and what resources can we build, put slightly differently. What are the strategies and tactics that we could employ in order to avoid these nightmares or to decrease the likelihood of those nightmares and ENC teams, crucially, we could talk about this if you want, they have to be cross-functional, cross expertise.

You have to have. Someone from legal, someone from it, or data, someone from compliance. You know someone, if it's an hr, if it's an hr, somebody from hr. So you need this cross-functional team because they're gonna have different ways of spotting nightmares and different ways of solving for those nightmares.

And crucially, different members of that ENC team or the teams that they represent will contribute in different ways. Such that the overall likelihood of running into a nightmare is decreased. So if we're talking about hallucinations, the data scientist will have her bit to do, but she can't take hallucination at the zero.

And the HR personnel, they'll have their bit to do to properly vet the outputs to make sure they're not relying on hallucinated content, either one of them by themselves. The likelihood is too great to deploy. Both of them acting in concert decreases the likelihood now we're safe enough at least to deploy co.

It's commensurate with our risk appetite to deploy. And finally, you could do this at the C-Suite or board level, of course. What are the organization as a whole, ethical nightmares as it pertains to ai? What resources do we need to build? What training do we need to give employees in order to use those resources?

So we're not, we don't have to start with the C-suite. It's not, doesn't have to be top down. Maybe among the resources that the C-suite calls on is a policy, but it's neither the most efficient, nor most effective tool in their tool belt. And so if you can create ENC teams where they're needed, when they're needed, then you have something that can be rapidly implemented and with our client for implementing this.

Like we, we could run pilots on it. That's another nice that you can pilot it instead of just, you can't pilot a policy. That's not the kind of thing that one does. You either have it or you don't. So you can pilot this and we have ENC team is up and running in a matter of weeks. It's not, it's just not that big of a lift to get people focused on the collaborative problem solving.

Richie Cotton: As you look the simplicity of that, you say what's the worst thing that could happen? What do we have to prevent that? And what do we need to do to train people in order to make sure it all works? It's brilliant. That does sound much simpler than getting several different department heads together.

Yeah, get them to agree on stuff. Forget, but yeah. Okay. 

Reid Blackman: The ship has sailed. The ship has sailed. And it's, it has to be simple in some sense. There's more things, there's, we can go into more detail of course. So it's not just what's the nightmare. But one thing that's nice about nightmares is that they're instructive.

So when you have a nightmare, it's usually not just, especially in an organizational context, it's usually not just, the monster showed up, it's. We failed to do such and such with the ai, we failed to test it in this way, that or the other. And so we round up with these really bad outcomes. If you could specify how the nightmare comes to be, you could assign likelihood scores to that.

There's more, there's many ways to arrive at the same nightmare. So there's different pathways. You assign likelihood scores to each of those. Pathways, which gives you a method for prioritizing how are you going to create the resources or which resources we create and when, yeah, you speak the simple language of nightmares and the simple three questions because it makes communication possible across this motley crew.

And right now I think we have a lot of people trying to collaborate around the Tower of Babel and it's obviously not working. Can't work. So if you give people simple language, simple concepts, what are the nightmares? What's the worst case scenarios? What do we need to do to avoid it? That's something that everybody understands and they can bring their respective expertise to the table so that people can collaborate appropriately.

Richie Cotton: No I think it's brilliant. But do you wanna make it concrete? I do, I wanna pronounce this very carefully now. Do you want to give us an HR example? So what's and how do you do the challenge? 

Reid Blackman: So suppose they're trying to design AI aluminum. That's a pronunciation joke. I know. You say aluminum.

Okay. Anyway, yeah, so I actually gave an example. Suppose that you've got marketing ai, an AI agent, whatever that's connected to either databases in the product department or an AI agent that's an AI product agent. And you get the team together, say, okay, we're gonna build this thing.

It's going to pull information about various products. It's gonna write marketing copy, and then it's going to do some ab testing with that marketing. It's gonna generate marketing ads or whatever, targeted ads, and it's gonna ab test and blah, blah, blah. Now. Anyone can tell you what you really don't want is marketing false features of the product, right?

If the, if you're marketing a product and you start enumerating its features and it doesn't actually have those features, that's a big problem. Ethically, reputationally, and legally, that's a big problem. And you also know if you take two seconds, oh AI hallucinates, so be really bad if this thing started hallucinating product features, and then put that into the marketing company.

Marketing copy, especially because the marketing person, even if the marketing person can't get it properly because they don't know enough about the product because they're, and the marketing team, they're not, deeply involved and embedded in the product. So maybe it has that feature, maybe it doesn't.

So what are you supposed to do here? So if you have someone from product on the ENC team and someone from marketing, and you have somebody from IT or da, data, the data science people who built the thing, the developers, they can say, okay, what's the potential nightmare? AI Halls product features and AB test that now we have people, trying to buy our product.

Turns out that it doesn't have that feature. They start complaining to customer service. Lots of refunds. Maybe there's a lawsuit. If this thing operates at scale and speed, that's really scary. Maybe we have a class action lawsuit. Maybe you have reg regulators knocking on a door saying you're advertising false features of the product.

That's real inline our scenario. So what can we do? The data scientist can raise her hand and say listen, we can use various techniques to decrease the rate of hallucinations. We can take it down from this to that. The product person can make sure that friends, among other things, the database or the agent from which that marketing agent is drawing information, that it's vetted properly, that it's clean, that it's, has all true information and no false information.

They can put, they can also put a product person in between. A product person, so marketing person can say there's not much that we can do, except we can vet ourselves or we can work with someone on product and they can vet with us the co the marketing copy. So that's gonna be a sort of human in the loop, solu not solution, but human in loop contribution to decreasing the likelihood of pushing out ads with false information.

And so it's just, you can get together pretty quickly. This is just of course, one sort of tiny example, but. Different people within that process with how that AI agent operates are gonna have different things that they can do to decrease the likelihood of that nightmare happening. 

Richie Cotton: It does seem that often the solution to the improving ai particularly when you got very complex systems, you do need a lot of people from a lot of different teams here.

So you mentioned here you can have to like a product personality, you have to have a marketing person, maybe an engineer in all sorts. One person you didn't mention. I don't know what the role is now. If you have an existing responsible AI team or an AI governance team, what do they do in this new world?

Reid Blackman: Yeah, that's a great, that's a great question. So there's a couple of things you should do. So one thing is to say is I have been, and I still am somewhat and or supporter of something like a responsible AI board or committee or something along those lines. I think you need it. The problem that we're running into now is that they're becoming a bottleneck.

There's usually in an enterprise Fortune 500, there's one risk board like that. And they're usually comprised of senior executives 'cause they need to be relatively senior because they need to have the authority to say to the product people, no, you're not doing that. And this is not their day job.

They've got their day job. This is their side gig within the company. And as AI implementation increases, as more and more people want to use, wanna build AI agents and deploy AI agents. They're getting more and more proposals, they're getting more. And so they're getting overwhelmed and they become the bottleneck because number one, it's just too much stuff.

And number two, one thing that explains why it's too much stuff is that the frontline team, if you like, they would rather escalate and be told no, everything is cool here. They would rather not escalate and be told that was a disaster. Why didn't you escalate that, that earlier? They don't wanna, so I think responsible, a boards have their place, but it can't be the place that we've assigned to them in the last six plus years or whatever. Something like the gatekeepers. I think that part of the e function of the ENC team is to be a kind of more robust frontline of defense so that they go to the responsible ai, not seeking answers or solutions, but going to them with.

Hey, we're coming to you. We've filled out this, my, we give our clients ENC worksheets. So the ENC team is working through this seven step method and filling out this worksheet all along. Here's the diverse that we identified. Here's the likelihood scores that we gave it. Here's the resources that we created.

Here's the training that we think is appropriate, blah, blah, blah, or that we delivered already. Does this look cool to you? And now the responsible AI board is in the business of vetting, not doing, not coming up to those solutions themselves, not doing a deep dive themselves. So one function is we keep the responsible AI team, but they have a slightly different purview now that makes it the case that they're no longer the bottleneck.

Richie Cotton: Okay. I think there's certainly a lot of organizations where passing the book and making sure that you are not like the end accountable person for a lot of things is a very common culture and it's not necessarily a good culture in any case. So I do like the idea that you're not incentivizing people to try and pass things to the responsible AI team who just then say, I'm overwhelmed not going to this, and then nothing ever happens.

Okay. If you're federating this around the organization, then, so every team in some sense has to now be res responsible for their own ai. It sounds like you need some kind of training then for all the organization. What does everyone need to know about AI and AI nightmares in order to do this properly?

Reid Blackman: So we offer our clients a, like a one hour-ish basic training on here's what AI is. Just very basic, no jargon. Here's what AI is and here's what the potential nightmares are. Hallucinations, automation bias cascading failures, et cetera, et cetera. So here's just a baseline understanding of what AI is and what the risks are.

We also train them on just the three basic questions of the Ethical Nightmare Challenge. What are the ethical nightmares that we're trying to avoid? What are the resources we need to avoid them? And what training do we need to give our people to avoid them or to use as resources effectively? So there's a base of one hour training to get people up to speed, if you like.

That does two things. Number one, it creates the creates, it makes it the case that they can now exercise their professional judgment in how they use ai. So it's not policy, it's not compliance, it's not thou shalt do this and thou shalt never do that. I think there's way too much gray area in the use of ai, and we need to enable people to exercise their professional judgment and how they use ai.

So that one hour training course is meant to do exactly that. Here's how, here's what exactly what you need to know, nothing more, nothing less, so that you can exercise your professional judgment and how you use ai. It also, of course, that's offered asynchronously, right? So it's a, it's online, et cetera, et cetera.

It also prepares them, it gives them the requisite knowledge to serve on an ENC team. Step one of the method is create the ENC team. It's called functional, blah, blah, blah. Step two is ensure baseline understanding of AI risks. I want the person from legal in here. I want the person from compliance.

I want the person from maybe cyber. In some cases they gotta, we have to make sure that everyone understands what the potential lightmares are, that we're speaking the same language that we all know what cascading failures are, that sort of thing. For the ENC team, there's a team leader. That person gets additional training, additional asynchronous training.

So that's giving them the training on the seven step method and the worksheet, how to fill it out so that they can lead an ENC team through the seven steps again deliver it asynchronously, and then to make sure that things are on the up and up for our clients. Anyway, we do that with them. We co-facilitate that first ENC team meeting, like when they're doing pilots we'll co-facilitate those ENC team leaders.

From there on out, those ENC team leaders can then co-facilitate other people who are becoming ENC team leaders. And so they can scale it as they like, so you get the network effect of them scaling it internally. 

Richie Cotton: Okay. I love that. The basic training is just, it's an hour of just this is the basics of what you need to know about ai and that's then enough to get started with, I guess using, you, you said you using be a professional judge, but having a bit of discern on what's a good idea or not. 

Reid Blackman: Yeah, and I also wanna just demystify it. I think that there're just people, most people are just confused, what the heck is this thing? And in, in my book, in the video I hammer away at the claim that look, AI is software that learns by example.

That's, it's AI is software that learns by example. And once you drill that into people's head, and I've had people already say I get it. I'm not sure you've said a software that learns by example enough though. But the thing that's great is it's in their heads now. They might not have liked me repeating it a thousand times, but it's now in their heads and I think that does a tremendous amount to let them take a breath and think it's okay.

It's just software that learns by example. I could handle it. The other point of course, is that once you see it at software that learns by example, you can see how you can get to the various risks. Because if the examples reflect various biases or for the examples include certain, personal or sensitive data like IP vi DA data relevant to ip, then you can start to see, okay, that's why we get these risks.

And so now I have a grip on the kinds of things very World Week that one might do in order to avoid those nightmares. So yeah, I think that base level training is crucial. I also think that we've, organizations have mostly fumbled it. Thus far because they've turned it into compliance training.

It's more CYA stuff. And I've literally seen this where I've written for this is for one of the big four. I've written their, AI ethics 1 0 1 or Responsible AI 1 0 1 for that everyone in the organization takes. And then their team, their compliance team got a hold of it and turned it into here are 37 bullet points.

And I said, who's gonna, who's gonna remember 37 bullet points? About what you can and cannot do. Forget it. This is, that's not enabling people, that's just, that's just you saying we told them in case there's a lawsuit or something like that. And maybe do that too. I dunno, I'm not a lawyer, but I think that we need to give them the kind of training that speaks to them at a very, not overly simplistic, but at least a non jargon laden way so that people can not be intimidated by the technology and when they're not intimidated by it and they understand what the risks are, then they can then go on to actually use it responsibly.

Richie Cotton: Yeah, I would say, compliance training of any form. It's seldom very good. I is it, I think it does end up just being that let's make sure that we have told the employee, so yeah. The goal, not get sued rather than to actually educate anyone. 

Reid Blackman: Yeah. But it's crazy because Sabine, this is legitimately both amazing technology and dangerous technology and for you to just think, eh, let's just not tell 'em about the dangerous stuff.

And I'm sure everything will be fine with our company of. 30, 40, 50, 60, 70,000 employees. Are you, that seems like a, obviously a recipe for disaster. So it just makes sense to say, Hey, listen, we're gonna, we're gonna state what the nightmares are because we're confident that we can avoid them.

We're gonna state what our problem is so that we can avoid those problems on the compliance stuff. And I think leaders know this. I was speaking just the other day with an executive at a Fortune 100 Entertainment company and. We were talking about training, and he said, I said there's, there's the standard compliance training.

He said, oh yeah. We already have that. We know that does nothing for anybody. And so that, that's exactly right. But it's where a lot of people start and I suppose that's fine. Just can't be where you stop. 

Richie Cotton: Yeah. I guess good to not get sued, but you need to go back extra step further.

I'd like to talk a bit about how you scale this stuff. So you mentioned the idea that you have some people training other teams and it keeps going across the organization. Do you need to have some kind of standardization of what constitutes a nightmare from one team to another?

Reid Blackman: Yeah. That's part of it, that's part of the training. I highlight five features of a nightmare. So nightmares are things that are one, the articulations are very bad outcomes. They're instructive. I said this earlier, so they, a nightmare includes how that nightmare came to be or the ways in which that nightmare can come to be.

It's organizationally relevant. So we're not talking about just any old ethical nightmare, the informational collapse of the informational ecosystem that I talked about earlier. Yeah, that's a problem. But if you're a, e-commerce company. Nothing you're gonna do about it, right? That's just it's relevant to you as a human being, perhaps, but, or as a citizen, but it's not relevant to you in your capacity as an employee of e-commerce or a CPG company or of insurance company, et cetera.

So they got the organization relevant. I, there's some other features as well. I, I can't think of 'em off the top of my head now. What else? So anyway, the features the nightmares have to meet certain criteria, but. No, look, the n the nightmares of HR or HR are gonna be different from the nightmares of product or different from the nightmares of marketing or different from the chief information.

Security officer's nightmares are different from the blah, blah, blah, blah, or different from the CEO's nightmares. There's some overlap. There's a Venn diagram here. It's appropriate that the nightmares vary by context. They do. And that's a fine thing as long as we are clear on what we mean when we say we're trying to avoid the ethical reputational, legal nightmares here.

Richie Cotton: Okay. Yeah. I suppose everyone's got different nightmares 'cause it's in some sense personal, but you want to be, it's about cross team communication again. One team can communicate with another. So you need to kind of standardization there. 

Reid Blackman: Yeah. There's this, it's, to me, it's important that we standardize both those three questions and the method for answering those questions.

So what I, so I mentioned I think a worksheet earlier. So an ENC team is working through the worksheet and the worksheet is nearly identical from a project level to a department level, to the C-suite board level, which is great because then if you've seen sort of one, you've seen 'em all, you can pick up the ENC worksheet of another team, of another department of the C-suite.

The C-suite member could pick up the ENC sheet of a project level team and they immediately know what they're looking at because the method. Both the method and the documentation of the method and how the findings are documented, how the recommendations are made, all that is standardized. And so you could pick up this, you're in the C-suite, but you've done an ENC team in the C-suite.

You pick up an ENC sheet from a particular project level and you know what you're looking at because it's been standardized. 

Richie Cotton: Okay. Now I'm wondering how do you know if this works? Because often when you implement some new idea, you wanna have a business metric to go, okay, we are making this number bigger, or whatever.

With nightmares, it's like you are just avoiding bad things happening, so it's harder to track what's going on. So what counts as success is, 

Reid Blackman: I think what counts as success is that you actually deploy your AI and it's doing the thing you want it to do, and you're not running into the nightmares. It's just as simple as that.

I think that success in business is defined by outcomes, typically, right? We think about success as we achieve the outcome. I think we can just as well think about it as we avoided the bad outcomes. That's among the things. This, I think, is in stark contrast to that standard approach to responsible ai, which is policy and procedures.

And all you can do is measure the extent to which people complied with the procedures, but you have no independent way of verifying whether the procedures actually bring about. It's a desired outcome. I think that if you talk about nightmares, your nightmares are articulations of bad outcomes that you want to avoid.

And so if you could manage to avoid those bad outcomes, that's what success looks like in this area. It's not the only piece of, overall business success 'cause you also have to achieve outcomes related to RO, positive ROI, income, revenue, whatever you wanna call it. Yeah. It's a, it is outcome oriented in the sense that nightmares are articulations of bad outcomes and we succeed in the condition that we successfully steer away from them.

We don't hit the iceberg. 

Richie Cotton: Okay, nice. Yeah, and I suppose by different, you are, 'cause you've written down a nightmares. You are tracking them. You've got something to measure against. Did I actually avoid this outcome or not? So it's that in place. 

Reid Blackman: Yeah. You're specifying the nightmares, you're specifying the two, the pathways to those nightmares.

You're specifying likelihoods and then, when the first time an ENCT meets, you have the sort of initial nightmare scores, as I call them. And then you do your work, you create the resources and or the training, and you reduce the likelihood of those things. And so you sign at a different nightmare score.

And then the goal, of course is to bring that nightmare score sufficiently low, such that you feel it's safe deploying. Part of that equation is the monitoring stuff that we talked a little bit about earlier, which is that if you have a pretty high risk AI. But you have, you're confident that you can monitor this thing in real time and catch a nightmare really at the at the very beginning of it's unfolding.

You can feel safe deploying the thing because you're like, it's high risk. But I'm also very confident we'll catch things that they start going sideways and then we can intervene appropriately. Part of what the ENC team does? Is it just, it at the project level anyway? Is it articulates what those monitoring metrics are so that they know if and when they should step in?

Richie Cotton: Absolutely. So it, it's almost in the same way that a risk assessment is gonna tell you about these are the things you need to watch out for. It's giving you that way of like tracking things and being able to intervene to avoid yeah. One of the problems, 

Reid Blackman: and not only does give you a way of tracking it, but it incorporates that ability to track into your overall risk score.

Because a lot of times things are gonna be high risk on the condition that you don't have. You don't have a way of monitoring it. But if you do have a way of monitoring it and intervening appropriately, then either you don't consider high risk or it's high risk, but it's efficiently safe to deploy anyway.

Richie Cotton: I guess the follow up to this is then a. What happens if new nightmares emerge? Like how often do you need to keep doing this challenge and revisiting what could go wrong? 

Reid Blackman: Yeah. This I think is one of the core issues with any responsible AI program, is how do we update this thing appropriately?

So the nice thing about this is, at least the way that we deploy it, is the portal that our clients are accessing the tools, the training gets updated as AI evolves. So if it turns out that we get some new kind of AI tomorrow, then. We add training modules. If there turns out that there's this new amazing solution for, let's say, monitoring AI agents, we, that's, that gets updated in the portal.

So the portal has both the train, the training modules, the worksheets, but also has deeper dives on various kinds of risks. So if a team wants to know about risks pertaining to autonomy at agents, we've got a deep dive tutorial on that. That gets updated regularly if some new tool comes along that's relevant to them, that gets updated If.

If it turns out that quantum becomes a real thing, IBM or whomever actually releases a fault tolerant quantum computer, and it turns out that now this is something that e NNC teams need to tackle as well. We can update training so that now it includes modules on the risks of quantum computing or, and or the risks of quantum and AI computing.

And it's a way of delivering. I like to think of it as almost like last mile delivery of training. Not every team's need to about quantum or they're not gonna, not every team's gonna need to know about risks pertaining to the autonomy of agents. 'cause not everyone's gonna be working on agents.

And so you can give the training and the updates to specific teams on an as needed basis, and that stuff gets updated. As, as we have a changing regulatory environment, it's a change of technology environment, changing solution environment, so on. 

Richie Cotton: Okay. Yeah. Explanation also. So like problems there, you've got like quantum ai both hallucinates and doesn't hallucinate at the same time.

And I like the idea that, yeah you've got a way of checking things that as this situation changes, it's ative straightforward to update. 

Reid Blackman: The tools have to be dynamic. The training needs to be dynamic. If it can't be rapidly updated in response to the technology. You're gonna have something stale.

So whatever the solution is, whether it's ours or somebody else's, it's gotta be the kind of thing where you're getting continuous, updates, or if not continuous updates to the tooling, the worksheets, the training as needed as things change. 

Richie Cotton: Absolutely. No spending year writing another policy 

Reid Blackman: document.

Yes. Please don't. I'm done. Alright. 

Richie Cotton: Yeah. Okay. We spent a lot of time talking about AI nightmares. Yeah. Just as finish do you have any AI happy dreams? 

Reid Blackman: Oh, I. I I'm living one, with Claude Code, to tell you the truth. I think Claude Code's amazing.

I've got my complaints of course, but I think that people are not really tuned into the capabilities here. And so I have Claude code and so that basically is, I've turned it into a kind of agent. So Claude. Interacts with lots of different tools, different Excel sheets that I have, manages various kinds of pipelines that I have whether it's for clients or it's for speaking engagements.

Or I have a podcast. My, I have my own podcast, so managing podcast guest lists. But there's a lot of stuff that I think just if that stuff just continues to improve even just that would be incredible. Right now it's slow. It's not without its errors, of course. Sometimes I get frustrated with Claude, but.

I don't know. It really does allow you to do amazing things, keep track of so many more things, keeping track of, so many more relationships right now than I could previously because again, let's just list everyone in a Word document or an Excel sheet or even I can't stand CRM software.

So I think there's amazing ways of using Claude Code as an assistant that's extremely helpful. It's a dream in a lot of respects. There's, I would say there's nightmares, but there's, some bad dreams, but not many. 

Richie Cotton: Okay. Yeah. So many possible use cases and lots of cool stuff that we've gone do before.

Typically like managing all that kind of admin stuff, like it's no one's idea of excitement. I think so, yeah. Great to outsource that. 

Reid Blackman: I'll say maybe one last thing that, I have the outline of a dream. I don't I'm not exactly sure how to describe this. I think something like this.

Suppose that's before submarines. We don't have submarines and we've got scuba divers. And you say, you know what would be great if we could get more scuba divers? And then you say, yeah, let's build some robot scuba divers. And then someone comes along and says no, we've got this new technology.

Don't build robot scuba divers. Build this thing called a submarine. It's a way more complicated piece of machinery, but it's going to do the kinds of thing that scuba divers were doing, and then a whole lot more. And yeah, it's gonna take more teams to. Design, build, test, operate, maintain, oversee, et cetera.

This sort of new behemoth machine. But it's gonna be awesome. And I think that we are trying to build robot submarine people like we're building robot executive assistants. It's cool, like I just said, I like it. But I think that there, that somewhere lurking in the mind of someone who's very creative is something like the an the.

What's analogous to the submarine? There's some kinda like AI complicated use of machinery. I don't know what it is, that's why it's only the outline of a dream, but I think there's probably these big engineered systems that we can build. I that, that, that are gonna be amazing. I just don't know what they are yet.

Richie Cotton: I'm looking like robot scuba dives. Does sound quite cool though. 

Reid Blackman: It does sound cool. I agree. But I bet a submarine sounded really cool before there were submarines. 

Richie Cotton: Yeah, definitely. I like the idea that there is gonna be something big that you could build. It's just a different idea that nobody's thought of yet.

Reid Blackman: Yeah. It's less of a person, an executive assistant like Claude Code or something along those lines is a great assistant, but it's still it's still conceived of in the manner of it's like a person, but better. That's the framing for it. And I don't know, that's, it's like a scuba diver, but better is not a submarine and submarine's just a different beast.

But it does, it's I just think there's gotta be, we built, I also think about things like commercial jets and fighter jets and submarines and those, these are very just big, complex engineered systems. You could pay a guy to drag you in a carriage with a horse, or you can, I don't know.

I'm losing my metaphor now. I, it's too much, too many metaphors. The point is though, I think that there's some complex engineered system that we can build using ai. I just don't know what it is yet. And I'm excited to see what someone will build unless it's a doomsday machine. 

Richie Cotton: Alright.

Wonderful. Yeah. Dream big, but try not to destroy the world. That seems like good advice. 

Reid Blackman: Yeah. But make sure to rise to the ethical nightmare Challenge while you're at it. 

Richie Cotton: Alright just very final question. I always want more people to learn from. So whose work are you most excited about? 

Reid Blackman: Let's see, that's a good question.

Whose work am I most excited about? Actually the work that I found most interesting, which is not directly relevant to our conversation, but there's a guy named a Ty Kovski who's on my podcast and he's been doing a lot of work around digital duplicates. So this is you, this is what it sounds like.

You, you create a digital version, an AI version of. A person, it could be of a loved one, it could be of a lost loved one that you miss and you wanna interact with your dead grandparents, or you want your child to interact with your dead grandparents who they'd never met, historical figures that we wanna you know, digitally revive.

And there's a lot of very thorny, interesting, ethical. Problems with all of this stuff. The one that he pointed out to me is, what if someone digitally duplicates a Holocaust survivor and makes them a digital holocaust denier? That seems all sorts of wrong grieving. There's a whole grief industry pop, ai grief industry popping about trying to recreate your lost loved one.

But psychologically is that the right. So is that the right thing to do? Is it ethically? Okay. Did your lo your loved one, they didn't consent to that? Can you still do that? I, anyway, so at K Kovski, he's doing a lot of work around this lately, and so I, I've really I think his work is fascinating and important.

Richie Cotton: Abso i'm glad you said that. Atto was actually also a data frame guest, so Oh, great. He's yeah. Yeah. 

Reid Blackman: Amazing. 

Richie Cotton: So I think our listeners, if you missed that episode, please do go Yeah, go listen to it. That, but yeah he's doing some very interesting work. Excellent. Reid, thank you so much for your time.

That was a fascinating conversation. 

Reid Blackman: No, Richie, it's my pleasure. Thanks for having me back.