Your 90 Day Blueprint for AI Success with Charlene Li, Author of Winning with AI

Richie Cotton: Hi, Charlene. Welcome to the show. 

Charlene Li: Thank you so much for having me. 

Richie Cotton: Yeah, great to have you here. Now, to begin with, I have to say, I was a little bit cynical about one of the claims in your book that you can do a lot in terms of preparing your organization for AI in 90 days. So how far can you really get within those three months?

Charlene Li: Again, this is not about actually doing a whole transformation. That would just be impossible. Transformation involves people change slowly. I think 90 days is when you're... where the ideas of AI become real. You're putting them down on paper, because the reality is most organizations do not have a plan.

They have ambitions and hopes. Those are not a plan, and I believe that if you're going to get serious about AI, you need to write it down, because it doesn't exist if it's not written down. So the 90 days is really about putting that plan together so that you can begin to execute it, thinking about all the various aspects.

But 90 days is a good enough time to think about all the different things, involve people from around the organization, and have a plan that everyone understands and agrees to and can begin executing. And the goal is to create value with AI, not just implementing AI, getting it out there, but to create real business value.

Richie Cotton: Absolutely. I love that 'cause there's so much you can do with AI that quite often you can get overwhelmed and be like, "Oh, we'll do lots of different things." You're not sure what's gonna work. So I love the idea of just writing down having this plan, saying, "Okay, this is the official thing of what we're gonna do."

I know you've got this whole framework for different things you need to do within the 90 days. We'll get into some of those i- in more depth, but do you just wanna give me a high level overview of what does 90 days of planning involve? 

Charlene Li: Yeah. A couple of moves. First of all, it... We lay out the book in the book, 12 weeks of 12 different things that you have to focus, but I think there are just a couple key things.

First of all, you need to make sure you got your leadership right, that there is one person who wakes up every single day and says, "My job is to drive value with AI." It doesn't mean that you own AI and that you do it completely by yourself, but you're conducting, you're orchestrating, you're making sure that your IT team understands what they have to do, what your marketing people have to do, what HR, f- finance.

Everyone is pulling in the same direction. We all understand what that plan is. So that leadership role is absolutely important. Leadership does not come from a committee. Steering committee, absolutely important, but you need one person who creates that accountability to drive value. The second thing is you're tying everything that you're doing with AI back to a business priority.

Now, this doesn't mean that you can't experiment. Experiments are great. They're for learning. But when you put significant enterprise effort behind an AI project, then you want to make sure it's connected to your strategic objectives. So we like to say strategy is written in ink, but your plan is written in pencil.

Because things change. Technology changes, your customers change, your people change. And so having that direction, that roadmap tied to your strategy is very important. And I think the last thing and also that, that roadmap can change. And then the last thing is to make sure that you're thinking about the four building blocks.

We call them your mindset, your culture, your skill set, your people, the tool set, the actual technology and platforms and the data that you have, and then your decision set, your governance. These four building blocks have to be in place so that you can execute on the plan. 

Richie Cotton: Okay. I have to say, I love the idea of strategy in ink and then the tactics, the implementation in pencil.

'Cause things are changing so fast, you gotta remain agile in some way, but you don't wanna get too distracted. So I love that d- distinction. Now you mentioned someone's gotta be accountable within the organization, one of your leaders has got to be accountable for the plan. Who should that be?

Charlene Li: I think that it's the person who I think can think about that, driving that strategic value. And in most cases, people just by default say it should be somebody in IT and data. And I was talking to somebody last night, they go, "Yeah, we put in a new chief AI officer somewhere between our CTO and CIO."

And I went, "Does that person have a change transformation background?" Because that is probably the most important thing. It's less about where they sit and more about who that person is and their ability to drive change, because this is not just about adopting AI, it's about adapting the organization to be able to use AI to create value.

And I may sound like a bit of a broken record, but if it doesn't matter if people have AI in their hands, if they're not using it to create value And this is the biggest disconnect that we see right now. We've deployed ChatGPT to everyone. We've given everybody Claude Cowork, but are they actually using it in a way that creates real concerted value, intentional value for the organization?

And in most cases, we haven't defined that for the organization. So the person who leads AI needs to have that change agent mindset and skills to match that, has the credibility in the organization. Technically credible. They don't have to be the person who understands this technology and can code everything up, but technically credible to be able to translate those strategic goals to the technical people and back and forth.

And I think, again, has that credibility to drive change within the organization. So sometimes it's in IT, sometimes it's not. I look at Moderna, and the person leading their AI efforts is their HR lead. Not the place that you would normally think, but their idea is that the future of work is going to be changing.

And so it makes sense that the person who has the best handle on how work is going to be changing and how people are gonna deliver the outcomes of work, connect it to the technology, the agents that are gonna deliver that work, and the combination of those. And so who's in the best position to understand how work can actually get done?

In this case, it was their HR leader And again, technically competent change agent the right person. So they took all of IT and all of HR and have them now reporting to one person. Now, I'm not saying that should be the standard for everyone, but for that organization, for Moderna, that was the solution that made sense for them.

Richie Cotton: Okay. It's quite reassuring that, there's more than one way to skin a cat, as they say. So there's different ways of doing this depending on who you've got on staff, but you need that ability to, change the culture and understand, like, how jobs are changing and also have some sort of technical credibility as well.

Okay. All right. You mentioned Moderna's had some suc- success with this. I do need some motivation. So do you have any other examples of businesses where they've gone through this planning phase and they found a benefit? 

Charlene Li: So many. I think one of my favorites is Konekta and they are... It's spelled K-O-N-E-C-T-A.

And they are a business process outsourcing. They run call centers on behalf of businesses. And as you can imagine, their business is being highly impacted by AI, because AI can take on so many of the things that call centers can do. And so at beginning of 2023, they said let's put our roadmap together of how we're gonna use AI."

And the number one thing they committed to in the beginning was, "We're not going to use AI just as a productivity and efficiency tool to cut people. Our problem isn't that we have too many people. Our problem is that our people don't have the skills, the capacity, the training to be able to deliver all the things our customers want.

So we can use AI to increase the quality," and they did that as one of the first things they did. They increased the quality by lowering the error rate by 85 to 90% just because people were using AI now to get the answers and look it up. They used AI to enhance the customer experience, which is, again, the engagement that you could have.

So that's the second bucket you can have. The first bucket is efficiency and productivity. But how can AI also help you have a better customer engagement? Because of all the knowledge that you can put in the hands of agents in real time. So you're augmenting them and l- allowing them to do a better job.

And the third area that they looked into was reinvention. And I think this is the area that has the greatest potential And what they anticipated back again at the beginning of 2023 was that when we have a workforce that is fluent with AI and has these new capabilities, what are the things that we could do with this?

And so by the middle of 2023, they were launching new business lines. For example, they were looking at their law firms and realizing they had all these non-performing loans sitting there that they were working on that AI could take, and instead of taking weeks to go through those non-performing loans, they could do it in a matter of minutes.

And so they went to the law firms and says, "Would you like this product? Would you like this service? Is-- We have this capability to do this. We can just literally turn it on for you." And so they started exploring this, and they recently said that they were going to expand the workforce by 5% over the next two years, not cut the workforce.

And I think that's the opportunity. Instead of thinking, how do we use AI because we have 10X the capacity now of somebody that we had before, now we don't need the other nine people. Instead, what if you could 10X everybody in your organization? What would you do with that if you had 100X potential or even just 2X potential?

Would you just cut people? That's looking at the business today, and you're just optimizing for the business today. But what kind of business do you wanna be in the future? And I think that's the thing that's missing the most from the dialogue today, is what could we imagine, reimagine, reinvent in our businesses now that we have an AI-fluent workforce?

Completely different approach to thinking about AI in your business. 

Richie Cotton: I do love that 'cause I think perhaps the biggest tech story of the year so far has been around using AI to automate things. I think that's where a lot of leaders naturally go to. It's "Oh, if we can do things more efficiently, we can save some money."

But you've got all these opportunities to improve the customer experience, like you said and to create new business lines, which is... I find is... It's a more inspirational story. How do you get to that? Like, how do you make sure that this is happening in an organization rather than this sort of reflex of just let's get rid of people?

Charlene Li: It's the reason why I think AI is not a technology problem, but it's a leadership problem, fundamentally, hands down. And there's a reason why Katja Welsh and I wrote this book, is if you want to look at AI strategically and how it can support your s- business objectives, you need to stop thinking about it as a technology and think about it as a way that it can transform your business.

Because if you apply it like a technology, you're going to treat it like a technology, you're gonna think about the ROI of this technology, and you will think primarily only in the efficiency and productivity space. Because you're only thinking about, "What can I do with the business today? How can I make it more productive?

Let's just automate things." And the worst thing you could do right now is to automate your existing processes without taking the time to say, "What if we had AI involved in the outcomes here? How could we be doing things differently?" Because I think the definition of madness is just doing the exact same thing, now with automation rather than humans, and there are just inefficiencies built into the way things are done because it was done by humans.

So why would you incorporate the same inefficiencies with AI? So I think it's a great opportunity to step back and say this is why we've always done things, but could we do things in a different way?" The question I ask now is how could we do things better, faster, cheaper, safer, differently because we now have AI in the mix?

And the solution may not be just take the human and replace it with AI. The better outcome might actually be the human and the AI, or frankly, just the human still, and using AI to do something, again, something completely different that you didn't know existed. Or have AI take on the tedious, repetitive, data-intensive things that are just soul-sucking for humans.

Let AI take that and allow humans to do things that only humans can do And I think this is what Connected did. They said, "We're gonna let AI take over the f- sort of the first line, easy to resolve questions that customers have, and reserve humans for the things that require more empathy and judgment and wisdom and intuition."

And a- again, that has served them very well because these new innovations are coming from the front lines, from people who understand and can talk to customers. I'll give you one other example. Ikea, they they put in place a chatbot. Again, this is before the current version of AI. They started doing this quite a few years ago, and have continued at a pace with the new generative AI and agentic AI.

But they said, we don't need as many customer service people 'cause we now have this AI agent called Billie." And they realized that 80-- 8,500 agents were going to be displaced. But they realized also these were people who had deep knowledge about their products and their customers.

So they spun up a new business line. They trained these people on how to do interior design, and now if you want interior design free from Ikea, then you can call up this line, and they generated over a billion dollars in revenue, again, from additional paid services, but also in just new products and services that these people bought.

They could see additionally how much more people bought because of these interior design services. So instead of using and thinking about AI as a cost reduction and thinking about these people as replaceable, they said, "These people are invaluable. They're so invaluable in terms of the assets they have, the knowledge they have.

What else could we do with that knowledge?" So paired with AI design training that they got, re-reskilled them to become designers, and resulted in a billion-dollar revenue line that didn't exist before That is why I think AI's a leadership issue. It requires imagination and curiosity into what else could you do with AI.

Richie Cotton: Absolutely. I love that story 'cause I think customer support workers maybe they've been at the front lines in terms of having their job being replaced by AI, and so I think there've been a lot of companies where downsizing these teams has been a priority. But I love the idea that you can just retrain people, because if you work in customer support, then you do know a lot about your business, and so you don't wanna lose all that institutional knowledge.

So yeah. That's a very cool story. All I guess more generally, I think a lot of the problem is around identifying what are gonna be the good use cases of AI. So do you have a sense of, like, how do you approach this discovery process, and how do you prioritize which AI use cases are gonna bring you value?

Charlene Li: A- again, I think you start with your business strategy, and you identify... Again, everyone has to be clear about what our strategic objectives are. And people think they know what that strategy is, but you'd be surprised how often people are not necessarily clear or they have a different point of view.

So be very clear what are your strategic objectives, and then what are the opportunities and where are the challenges to those objectives? And then think about how AI can support them. Again, help you realize those opportunities better, faster, cheaper, and also overcome some of those challenges to your strategic objectives.

From there, you can find some very specific examples of AI applications and initiatives that you can use. One organization said to me, "We have seven strategic objectives, and the first six are your usual suspects, and the seventh is use AI to achieve the first six." And I think that's the right way to think about it.

You don't need a separate AI strategy. You need these examples. And so I think one of the... my favorite examples is Securian Financial, and they went through and took the existing 60 use cases, narrowed them down against their strategic objectives, and narrowed it down to three very specific use cases that were tied very specifically back to those objectives.

And it was a long exercise. Took them a couple of weeks to make sure everyone had bought in, and that the people who had suggested the other 57 use cases understood why they were being rolled up or not pursued at the point, or just go and pursue them because they're not enterprise level. But again, getting that agreement, having a written roadmap was extremely important.

And the way to prioritize these use cases, again, is to do something we created in the book called the Double S Matrix. It's a two by two simple thing that looks at the size of the value you're creating and the speed at which you can create it. And so you may have some quick wins. So for example, an organization may say, "Let's do call summarization and transcriptions."

So that's a quick win. We can make sure there's one single truth coming out of a meeting or from a customer call. And then a momentum maker might be, let's take the summaries of those things over a course of a week and identify changing customer needs so we can get it back to our service department, back to our product team, make sure we're looking out for these things make sure that our service understands how to handle those things.

So again, ways to synthesize and take the quick wins and bundle them together. And there may be more strategic bets that are going to take months and possibly longer than that, a year, 18 months, which is why we recommend having an 18-month roadmap. Because these strategic bets are big, they're difficult.

They will require points when you have to figure out do we persist or do we pivot? And those strategic bets are the ones that drive the most value, but they're also the ones that are hardest to conceive because they are big and they're difficult. But you have to conceive them, so if you know where you wanna be 18 months from now, you can start building that today.

So you need a portfolio of investments in AI, your quick wins, your momentum makers, your strategic bets, to be able to realize value again over six quarters over the next 18 months, because that's the roadmap that everyone's gonna be building against. 

Richie Cotton: Okay. I love the idea that, you do need lots of different things on the go.

You treat it like, I guess like an investment portfolio is all these different plans, 'cause if one doesn't pan out, then you've got some backup plans there as well. Okay. Is there a process for going from this, from your business strategy to then having this roadmap? Is it just a case of you, you build the SS matrix and then, it feels like there's a, there's another step there before you've got this 18-month 

Charlene Li: roadmap. Yeah. I think the thing here is that you wanna make sure that, again, you start with that business strategy, and then as you're pulling together your various use cases, your various applications of AI, you've got your quick wins, your momentum makers, and your strategic bets, and then laying them out on that roadmap.

And again, this is why I think it's so important to put it in pencil. This is when people start looking at me like I'm crazy. It's like, how do we lay out an 18-month roadmap when everything seems to be changing all the time? I, and I felt this personally, because in January, February, I had to take a few weeks off to take care of some family members.

And I came back and Claude CoWork had launched, OpenClau had launched, and it felt like the world had completely changed. And I'm like, "I was just gone for about two weeks or so. How could things be so different?" And the way I use AI now is completely different from the way I used it before because of these innovations.

And so keeping that in mind As I said before, your roadmap is written in pencil. So you can take into account these big changes that seem really big in the moment, but when you look at it in the scheme of an 18-month roadmap, it's another data point that you put in, another way to think about things.

It may accelerate things, it may decelerate things. You may have to shift and pivot, but again, you're constantly adjusting this. So I, I think where does AI accelerate your, the things that makes it better, faster, cheaper? When does it threaten it? When does a competitor potentially is moving faster than you?

And where does AI let you do something different that you couldn't do before? So these are the kinds of ways to be thinking about AI very strategically as you're going around and looking at this, because those are the questions you want to be constantly asking yourself. 

Richie Cotton: Okay, yeah. I do the idea that you're gonna have to change things, and you're right.

It's like we had a point where maybe six months ago all these sort of agents didn't really work, but we knew they were coming, and now suddenly you said, "Oh, yeah AI can do stuff by itself, and it does work quite a lot of the time." So yeah, you gotta keep adjusting your plans like that. 

Charlene Li: Yeah.

And I think that one of the ways to think about this is that you may well have a... You have a business strategy today, but it's not informed by AI. And so you almost wanna think about your, having your business strategy be AI fluent, in that it can understand that AI is changing. It's constantly, it's creating con- constant change to how you think about your business strategy.

So your business strategy have your high level objectives, and this is why, again, I think organizations that have a strong sense of purpose, mission, values, and their strategy is really written down. You have these foundations of things that do not change or change with considerable thought and intention.

And when you have that foundation, everything can be swirling and changing around you, and you know where you stand. And I think that's such an important foundation to have when you're being, your world is being disrupted all around you. 

Richie Cotton: Okay. It's like having a constitution for your business, then. 

Charlene Li: Yes.

That's what a strategy should be. That's what mission and values and purpose are. And we f- we think about them as nice things to have. You have them on the wall, but does-- do people keep them at the center of everything that they do? I believe that organizations that are clear about this really benefit because everybody in your organization can understand what's the future we're building towards?

Where are our customers heading? What is that future we believe in and are fighting for? What's our strategy to get there? Because strategy is about choice. You don't have a strategy when everything is a possibility, and you can do anything. That's not strategy. That's just chaos. So you know what the future is that you're building towards, you know how you're going to get there, and that everybody knows their role in making that strategy a success.

Up and down the organization, they know what their contribution is in making that a success. And if you have everybody in your organization who can answer those three questions, then you're off to a great start because AI is just another way to accomplish that It's no longer just a tool that is kinda sitting there.

It is something that is essential to the way you're going to do your work and accomplish that strategy. 

Richie Cotton: Absolutely. It's increasingly feeling for almost every role, it's like having some kind of understanding of AI. It's almost I know how to use a keyboard . It does seem important, but how do you get to that state?

You mentioned that everyone's gotta be part of this this new mission, what skills do people need to have? 

Charlene Li: I so belong, believe in AI fluency and fluency, not just literacy. And the difference here is that when it feels like you... AI is in the flow of the way you do work, and it's a natural thing.

You trust it. You know what it can do and what it can't do, very importantly. You know how to use it responsibly and ethically, and then you also know how to use it in your job. And the analogy I like to use is chopsticks. When you were first handed a pair of chopsticks, and you pick them up, and you go, "How on earth am I going to pick up food with two pieces l- of wood?"

It's insane. And you've... In the beginning it's really awkward, and then eventually you kinda get the hang of it, and then eventually you're just shoving food into your mouth without thinking about it. And then you encounter the slippery noodle, and you have to adjust your grip, and you adjust, right?

I and so I think of AI as like that. And when I ask people, and I was recently having a dinner with I... about a dozen CIOs and CTOs, and I asked them, "How many of you consider yourself AI fluent?" And only one person raised their hand. And so I think there is a sense of comfort that we have to get to.

We all feel fluent in using Google. We feel fluent in using email. We need to feel fluent, and it's partly a mindset, but it's also a matter of practice, and that is the thing that we're missing. We're giving people the tools. We may be giving them some training but it probably isn't specific to their job.

And we're definitely not giving people the time. These three Ts, the tools, the training, and the time, are essential for people becoming fluent. And I think there's an order. Yeah, again, I think everyone should be AI fluent, but I think the order really matters. You have to get your leaders, first and foremost, fluent because they are the ones who are driving strategy, and if they are not fluent with what AI does and doesn't, if they're not comfortable with it, they're just not going to touch it And this AI hesitancy gap, is what I call it, is very real.

They know it's there, they should be doing something with it, but they're just not ready to pull the trigger because they're not personally fluent and comfortable with AI. And if they're not, no one else is going to be. And then you have to train, make sure your data and technical teams are also fluent, because they're the foundations.

If they do not feel comfortable with this and people are turning to them for the answer, they are not fluent and comfortable with it, then i-i... everything will just stop. They're the ones who are going to be the guidance for everyone else. And then think about your customer-facing and frontline functions.

How do they do a sales, marketing, customer service? The places where AI directly touches revenues or experience. And then support functions in parallel with that, HR, finance, legal. Everyone gets to do this. But again, it's not about last, but first I think that's a priority, and to make sure that everyone is there.

And I think fluency isn't a class that you take. AI when AI stops becoming a tool and starts becoming a teammate that you rely on, that's when you know you're fluent. 

Richie Cotton: Okay, Amelva a lot to unpack there. But I really like the idea of you've got to provide people with tools, and then you've got to provide them with training so they know how to use the tools, and times that you get a chance to practice.

So maybe we'll dive into those more in a moment. But you said l- the leaders need to go first, 'cause otherwise they're not gonna encourage other people to use AI, and then technical teams. So i- is that the natural order then? Leaders, then technical teams, then customer-facing teams, then everyone else?

Or does it not matter as much which order things go? 

Charlene Li: It's, again, I think people are going to naturally gravitate towards AI. A- again, in every function, in every department, in every industry I've seen, there is a small number of people, and this is always the case, who are technology optimistic.

They grab it, they run with it. And these are your super users, and really nurture them because they are the ones who are going to show everyone else in the organization how to do this. But that just is the natural occurring phenomenon. This is where an organization intentionally says, "We're going to drive AI fluency."

And when I ask people, "Do you have that program in place? What does it look like? What's your timeframe? When are you... w- who are you going to train first?" There's no answer. There's no plan. And when, if it's just something you hope that will happen because you gave people the tools, that's just wishful thinking.

And again, these 90 days are about driving the outcomes you want to see with AI, and AI fluency needs to be a part of that. And so my challenge to people is: What would it take for you to be fluent? What would it take... and let's say your goal is to become fluent in 90 days What would you do?

You personally, as a leader, what are you going to do over the next 90 days to become AI fluent, where you see AI as a teammate, something you can depend on, that you trust? What would it take? And they frankly haven't thought about it, and I think it's important that they think about it, and especially leaders, because they're the ones who think strategically about AI.

They're the ones who think about value. They're not thinking about AI first as a technology. They need to think about the business. And so if your top strategic leaders are not thinking about AI in that strategic way, feel comfortable with it, nothing is going to change. So that's why we talk about leadership first and talking about your top leaders, including your board as the people who need to be trained and develop that fluency because everyone else is watching them.

And the easiest way to do that is begin every meeting by saying, "How did you all prepare for this meeting using AI? This is how I use AI to prepare for this meeting." And I want people to talk a little bit about that just at the beginning and share it. And the... You're setting the expectation first that you use AI, and two, you're destigmatizing it, because people still feel, "I'm cheating if I use AI."

And so if you can demonstrate and model the way of how you are using AI to get everyday work done, that sets a huge bar and puts a stake in the ground that the expectation is, "I'm doing it, and so will you." 

Richie Cotton: Yeah I really like that idea and particularly opening meetings with that, 'cause I think the first thing in the meeting it sets the tone for things and it makes sure that this is the most important thing when everyone's still concentrating.

Okay. You've mentioned AI hesitancy a few times now. So I'm curious, what do you do if people do have this fear? Are there ways to help people overcome the hesitancy and just get started? 

Charlene Li: A- again, I think it's to really go back to why are they afraid? And the biggest one is, what happens if I make a mistake?

I'm staking the ground on this. I could lose my job. I could be putting the company at risk. Things could change. I may have to change my mind. A- again, all these things. What happens if I'm wrong? What happens if there's a hallucination? There are so many things that could happen I... it's just easier to just sit back and let somebody else take the lead.

Again, the reason why I think this is a leadership issue, because leaders understand there's a void where something isn't happening, and they step into it with courage and determination. And AI is one of those places where if you're waiting for the answer to come from someplace it's... you're gonna be waiting it for a very long time.

And so what we need to do is to understand what the costs and the risk of waiting is to, and to make that very clear, and to de-emphasize how risky it is to move, to jump into the water. 'Cause the water looks really scary and cold and dark right now. I'm gonna stay on dry land. You go first.

Somebody needs to go first. And that's why I say leaders have to go first. The top leaders have to go first. 

Richie Cotton: I suppose it, it's in the job title, right? Leader means you ought to be leading , going first, so yeah. I do love that. But I suppose, yeah, we- we're now what, more than three, coming up four years into this sort of like generative AI, now agentic AI revolution, so things are getting easier.

It feels like a lot of stuff is much easier than it was a few years ago, so I guess, yeah, now's the time to jump in the water. 

Charlene Li: Yeah. W- again, people talk about hallucinations being an issue and frankly it always is an issue, but so are humans. Humans make a lot of mistakes, so what are the processes you put in place to be able to check that humans are doing the work well and at the quality that you s- you determine?

What are the definitions and the guardrails for success? And when you realize you need to put the same guardrails and procedures in place for agentic AI, it changes the picture. Of course, things will be happening both with humans, with AI, but we've never learned how to manage AI the way we know how to manage humans, and it's very much the same process.

You train the agentic AIs. You watch and you tell them, "This is what you do." They copy you, and they learn. You watch them, correct them, and you continuously monitor what they do. And the benefit of AI is you can monitor them continuously against every single thing. And so here's the thing, you can have a human in the loop to check it.

You can also have another agent in the loop who's a judge and a critic to make sure the quality is correct. So th- there are ways where you can absolutely really minimize the likelihood that hallucinations will become an issue. So when somebody says hallucinations are an issue," I'm like yes, and we can almost completely mitigate that risk now."

Because if you know what hallucinations to look for, you find them, and you address them The hard ones are the things that you can't anticipate. It's same way you have the rogue human doing something completely off-kilter that you could never imagine. A- again it's less and less likely that agents would do that, again, if you have those guardrails, you have those boundaries in place about what's allowable and what's not.

And again, this comes to that, that fundamental first part of fluency, knowing what AI can do and what it can't do. If you understand what's capabilities and also its limitations, then you can work with it with confidence. 

Richie Cotton: It seems like there are two problems with the, with governance. So one is not doing any at all.

The other one is particularly in heavily regulated or industries, there are a lot of organizations who are like, "We have to get all the governance perfect before we even start considering AI," and then don't move at all, and then you're missing out on the benefits. So I feel like there's a sweet spot somewhere in between, but do you wanna talk me through how you might find that sweet spot?

What do you need to do to just have just enough governance to get going? 

Charlene Li: We ca- we... That's a great point. We call it Goldilocks governance, not too much, not too little, just right. And the only way, like Goldilocks, you can figure it out is to try all the chairs. And you will fig- and the thing here is I think there is bad governance, and that's what we think of as bad governance is way too much.

It's governing for the wrong things. And good governance allows you to go fast because you know very clearly what the black is and what the white is, what's allowable, what's not. The gray is narrowed to a very small margin, and especially in regulated industries, it's fantastic. It's the things that you already know you can't do.

Don't share PII. Make sure that when you're making really important decisions, that you're taking into account that you can explain why this is happening. There are just so many clear guardrails that you can follow. And it, and so I come back to governance is all about how do we put in the guardrails so that we can trust what AI is doing, what humans are doing, and that the outcomes are what we want them to be.

That's what governance is. It's about how do we make decisions, and whether it's humans and AI working together independently, we have thought through those things. And so I love talking to people in governance and legal, risk, compliance, because it is a fantastic conversation about the realities of business, and we know that no business is possible without any risk.

So even the most regulated industries, we know what risk is involved, and we take them knowing that these guardrails are in place, but we still move into them. And so the same conversations need to happen, and the problem right now is we're just not having those conversations because there's so much fear, we're just shutting it down.

And what happens is shadow AI is happening. People are still using AI without any of those guidance and guardrails in place, and that's even worse. So you're better off getting ahead of it, deciding what is allowable, what is not, putting in place guardrails, and then seeing what happens. And that's the only time you can do it, because you have to practice it in order to refine it.

And you must do this because the good governance comes from practice. Understanding if it's a too much, is it too little to get to that point of just right, and imagining it, writing down theoretically is no comparison to actually practicing it So yeah, get out there, use it. Things will break, but you can do that breaking inside of a container that's closed off in a sandbox, and you're much better off doing that now rather than having people doing it in the wild and in the background.

'Cause I guarantee you they're doing it. If you're banning it and not allowing people to use it, I guarantee you they're using it. 

Richie Cotton: Oh, yeah. I'm sure you've got some engineers somewhere trying to run OpenClang on, on one of their company laptops or something if you banned. Yeah. I love the idea.

Bring it out into the open. These are things you're not allowed to do, like some, minimum guardrails for things that could go very wrong, and then just, yeah get people using stuff. 

Charlene Li: Yeah you already have... You have data practices, you have data policies, you have privacy policy, you have security policies.

Those remain the same, and in fact, you're gonna tighten them and make them very specific to the requirements of AI. If you haven't done that, then you're missing a key component here. But you're building on top of that existing work, and we have this process for governance called the AI trust pyramid.

You have to have that data security and privacy in place. You got fairness, reliability, accountability, and transparency, and that's how you build trust. Because in the end, that's what governance does. It builds trust in your people and your agents that they can do the work safely and securely. So you're better off starting now.

And again, the more regulated you are the actual easier time you have because you know what the regulations are, you know what the compliance needs are. And so you go for the highest level of that because your values dictate that you do it that way. And if you do it that way, then compliance is a non-issue because you're already going to be complying at a much higher level than anybody would ever ask you to.

Richie Cotton: Okay, so it does seem it's possible to do this well. Now I'm curious, suppose your plan goes right after your 90 days, or maybe e-even after your 18-month roadmap. What does success look like? How do you measure it, and how do you know things have gone well? 

Charlene Li: I think this is something that companies are still struggling with, and they go "How do we measure things?"

And I think there, there's a high level which says, if you're making progress on your strategic objectives, that's how you know how things are going well. And again, is AI speeding things up, making it better, making it cheaper? A-again, all those metrics that you want to measure AI, not against very specific AI t- ex-examples, but against the value metrics that you're using.

But I think there are some new metrics also to be thinking is so the first area is our judgment improving? Is our ability to make decisions not just more decisions, but the quality of decisions. Are we are we moving the needle in terms of how we're doing that with AI? Did the customer experience improve?

Do-- are decisions getting better? Are we exercising better judgment? Is there more wisdom in the way we're making decisions? Are we using AI with more discernment? Not that did you use AI, but did you use AI well? And what's your definition of well? And did they also know when not to use AI? Are we able to, again, discernment is a new skill along with critical thinking.

And then also, are we developing this capability of being fluent? Are more and more people developing this? Are they teaching other people? 'Cause that's the ultimate sign of fluency, is when you feel comfortable, you can tell somebody else how to do it. Are they expanding the boundaries of what AI can do and how they're thinking about the work?

My, my point here is you don't measure whether someone uses a tool like Excel or PowerPoint. You measure whether they're making good decisions and getting good outcomes from it, and the same thing with AI. You wanna measure the outcomes, the impact of using AI. Are we getting there better? Are we doing it faster?

Are we doing new and different things that elevate us rather than replace us? 

Richie Cotton: Does seem an important point that you wanna track the outcomes ideally rather than just tracking token usage or things like that, because I feel like that's the easiest thing to track, but it doesn't give you that stronger sense of whether you're actually...

whether you are getting value from the AI usage at all, and you end up with silly things like token maxing where people just spend hundreds of thousands of dollars in a month and it's what did it build? Who knows?" 

Charlene Li: It sounds like a bit of a broken record, but anytime someone comes to you as a leader and is like, "Hey, I wanna try this new AI thing," like great, how does it add value to our organization?

How does it add value to these strategic goals? And then okay, think of it, but go away, figure that out. Tie it back to we'll value being creative as a just using AI isn't gonna help us. Again I wanna distinguish that to experimenting, where you're just trying things to learn to figure it out.

When somebody comes to you and says, "Hey, let's do this as a pilot so we can figure out," I'm like, actually, I think all companies should just kill their pilots. 'Cause what does a pilot do? A pilot says we're not convinced this is going to work, so let's make it a pilot," so that we actually don't decide whether we want to do this or not, so it's a form of procrastination, and it's abdicating your leadership responsibility, frankly.

Decide whether you're going to do something or not. Experiment over there, but that's for learning. But once something comes to you, decide does it help us achieve our goals, and decide to implement it. And you will, along the way, figure out how to make it adjust or not, and then in the end, if it's not creating the value you wanted, kill it.

But have that enterprise commitment to doing something, because otherwise a pilot just kind of mudd-muddles along and no one takes it seriously and no one puts effort behind it, and by then quarters have gone by and you're nowhere near close to where you want to be But if you decide this is something important, so teach it to us.

Forget about readiness assessments, forget about feasibility. We're gonna commit to doing this, and we'll make it happen. And if it doesn't happen, then we kill it. But again, it's a very big thing to say, "This is an application that's important to us. We're gonna put our effort behind this," versus a pilot that nobody takes seriously.

Just don't do pilots anymore. 

Richie Cotton: I say, my jaw dropped when you said just kill all the pilots. But yeah I agree that if you are gonna take this seriously, then you need to rephrase it rather than, "We're not just building a pilot, we are experimenting, and we're iterating until we get something that we actually want."

Yeah, I love the idea that you're committed upfront to building something worthwhile. All right we talked about the success of a whole company. I'd like to talk about the kind of micro success as well. So you mentioned some people are gonna be very enthusiastic about AI. How do you celebrate these champions?

Charlene Li: I think one of the things we tend to do is we think, okay, a slight appl-applause, a little bit of a token pizza party or something like that. That's not enough. So I don't think pizza is a recognition strategy, but I think promotions are. And if you believe this is important, then you'll put your money and your time against that.

And so first of all, giving them real recognition in the company meetings and the quarterly calls, internally and potentially externally, 'cause it's a great way to show people that what you're doing is really truly meaningful. Have those internal AI awards, something real again, where you're highlighting a couple champions, what they built, what they saved what they enabled, and that goes a long way.

But as I think money... You put your money where your mouth is, and you have promotion pathways. So AI champions should be on a visible promotion path, whatever that looks like. So if your most fluent person is stuck at the same level for year after year, you're sending a signal that this work doesn't really matter.

So if people truly are fluent and they're adding value, then you have to recognize that. And it's not just about the money, it's about the recognition, but also saying, "You truly are adding value. Let's find a place and a role for you where that value is going to be exponential because of the work that you do."

So I, I think, a- and again, you could even just give them some, like a budget of money to say, "Hey, you're doing some really great work. I wanna give you, a couple hundred dollars, a couple thousand dollars to do even more experimenting." And then have them be, Give them some visibility, give them speaking slots internally inside the company, and potentially externally where they can talk about it.

These are all ways to recognize these AI champions for the incredible work that they do, because it's not just experimenting with the technology, it's also changing the hearts and minds of their colleagues, and that is the most difficult work, and it needs to be supported because it is not something in people's job descriptions, it's something they believe in, and they're going far beyond what is required of them to be able to champion these things.

And again, it's not just technology, it is really about changing the culture and the hearts and minds of your colleagues. 

Richie Cotton: Of... I do enjoy pizza, but pay rises and promotions are dramatically better. Yeah I love the idea that if people are doing what you ask of them and they are changing the way they work using AI, that yeah, there should be a promotion in it or something very substantial,

Charlene Li: And when you see that happening, it says this is important, it's valued, and other people are like, "Oh, maybe I should be doing that versus somebody who's pushing and all they get is "Oh, they're just being a pain. They're the squeaky wheel that's trying to do this," versus saying, "What you're doing is really valuable."

And it says a signal to everyone else, maybe they should be doing this too as well. So it's not just recognizing them, it's fueling them, but it's also encouraging other people to follow in that same suit. Again, change does not happen miraculously, just overnight and transformational change in particular, when you have transformational leaders up and down and throughout the organization, is really hard to foster.

It's the reason why we, in the book, we talk about mindset, about the culture first and foremost of all the building blocks. It is not the technology. It is not about people skills or governance. It is about culture. And if you're really doing AI it will impact your culture, and culture is something you can intentionally set up and change.

People think it's culture as oh, it's too amorphous. Cul- culture is just simply your beliefs and your behaviors that come from those beliefs. So if you have a culture that's not serving you, there are certain beliefs that are not serving you, then s- systematically and intentionally root those beliefs out.

No longer make those behaviors acceptable, and instead replace them with new beliefs and new behaviors that you want to encourage people to have. So thinking deeply about the culture that you have and the mindsets that you have is going to go a long way in terms of setting up the right structure to be AI ready.

Richie Cotton: Absolutely. It does seem I guess it's an essential part of leadership is trying to decide what should the culture of my team be? What should the culture of my organization be if you're in the C-suite. Yeah I guess that's one way to to get to success. Just finally I always want more people to learn from tell me whose work are you most interested in right now?

Charlene Li: I am really been interested in this one technologist. His name is Andrej Karpathy, and he put together a, an, a d- i- idea on GitHub about LLM wikis, and it's the idea that you create a knowledge graph and a tonks- context graph of all your information. And I think this is a solution in many ways to how does agentic AI work when your data's a mess?

You don't necessarily have to go clean up the data, you just have to pull the meaning out of that data. You need to pull the important pieces that are needed to get the work done. It also has this additional benefit that your agentic AI doesn't have to go in and look at every single piece of data, every transcript, every meeting, every PowerPoint, every document, every single time it does work.

It can just skim across that context and understand that it all connects together 'cause it's just a wiki. And it saves a ton on tokens. It makes the work easier, but it also puts into context all of your data to accomplish the work that you want. And it is LLM agnostic, so you could put any LLM on top of that wiki.

I, I've been building it for myself. I have thousands and thousands of pieces of content that now all relate to each other, and I didn't have to build it. My AI built it for me. And I think it's a really interesting way to be thinking about data and AI and how your organization wants to use these two things in concert.

'Cause it really requires that you understand the context of... in which your data and AI is gonna sit in the way that you create that wiki. And it's a great way to step back and say, "How are we going to organize our knowledge and our context in an organization?" 'Cause it has taken its own form today, but that's not necessarily the form you want in the future for an agentic AI to work.

So I... that kind of thinking is always intriguing to me 'cause it has so many implications about the way we approach AI, the way we approach data and it definitely has made me stop and think about how I am going to structure and lay out the way I work with my AI in terms of giving it that context that it can operate in.

Because right now we're just relying on memory and a little bit of wag, rag or fine-tuning and that's not enough. This is a living, breathing organization, and how do you keep that context up and up to date? So that it's functioning at the highest level possible. 

Richie Cotton: Absolutely. Andrej Karpathy he comes up with so many amazing ideas.

I don't know how he does it. But yeah, I do the idea, and knowledge management is just one of these perpetual challenges for organizations. And so yeah, I think that's maybe gonna be like one of the next big things over the next year or so in terms of what A- AI can do i- in terms of helping you know what you know in an organization.

So yeah definitely, yeah, LLM Wiki is definitely something to look into. All right. Wonderful. Thank you so much for your time, Shirley. 

Charlene Li: Thank you so much. Really appreciate the conversation. And I really just love the role that I love seeing data and IT people really move into the strategic role.

Thinking about more than just the technology, taking on that strategic point of view. 'Cause I think that's what's really going to satur- saturate a, an organization and differentiate it, is can everybody think strategically, and not just about the technology, but think about the business objectives, think about the customers and how this will all change.

And I think that perspective and that leadership is very much needed i- and if AI's gonna be something that you use to be successful. 

Richie Cotton: Absolutely. I'm right there with you on that. Yeah. As the technology gets easier, the more you can start thinking strategically and, yeah, that, that's how you get to success.

Wonderful. All right. Thank you so much.