Creating an Engine of Value from AI with Jim Rowan, Head of AI at Deloitte USA

Richie Cotton: Hi, Jim. Welcome to the show.

Jim Rowan: Richie, thanks for having me. I appreciate it.

Richie Cotton: Yeah, great to have you here. Now, I'd like to start off by talking about the Deloitte State of AI in the Enterprise report, which you were one of the authors of. So do you wanna tell me what was the most surprising finding from that work?

Jim Rowan: Yeah. Richie, we ran this survey, and we ran it across about 4,000 different organizations, leaders globally. So it gives a good breadth of kinda what we see in the marketplace. And the main headline continues to be that organizations are struggling to adopt AI. They talk a lot about how they want to continue to scale AI solutions, but we're seeing that the actual implementation of those scaled AI solutions are falling behind what I think what we would s- wanna see as AI leaders in this space.

And so that's one of the big challenges that keeps coming up. There's also a bunch of really interesting data too that focus on, how organizations are thinking about their people and their workforce. And fascinatingly, a lot of folks are focusing on the tech 'cause with AI everyone talks about the tech a lot, and they're not spending time talking about their people and the change management processes and things that need to really happen.

So we're seeing a big link between getting ROI from AI, organizations struggling with that, and in the data kind of a focus that's being missed on sort of the people and change management aspects that we've known about for transformation for years. So those are some of the key headlines.

Richie Cotton: Okay.

Yeah, certainly, like buying licenses for AI tools is much easier than changing all your processes, making sure all your people are doing what they should be doing. We'll definitely get into that side of things throughout this conversation. But I'm curious 'cause I feel like basically every leader's gone, "Okay, we're all in on AI" for like several years now, but you've mentioned there's still problems in a lot of enterprises with making like getting value from AI.

So what do you think the biggest challenges are here?

Jim Rowan: Yeah, there are a lot of challenges, and I think it's not that we're not optimistic about what AI can do, it's that the excitement of AI meets the reality of enterprise systems, enterprise processes, large businesses with all sorts of different competing priorities.

And so a couple things you really start to tip the scale. Number one is that y- we see organizations that are investing heavy in the technology, so they're building out a really strong kind of footprint, but they're not maybe investing enough on the data side. So AI without data, you're not really gonna get governed, controlled, accurate results that you need.

So that's a big gap. So when you've invested in AI, then some of them are coming back and saying okay, now we need to rebuild the data foundation," or, "We need to fix the master data management issues that we have with that." So that's number one. Number two, we're seeing kind of process issues, both in the sense that they've applied AI for a POC, let's say, to a business process.

That process might have five steps in it. They've applied AI to the first two and maybe the fifth, but the middle activities still aren't getting automated, and so the end-to-end process isn't actually changing. Or in worse cases, it's speeding up to find a new bottleneck in the process. So that's also slowing down implementation because maybe we skipped over the two hard steps in the process, but then it didn't really make the, the change need go.

And then the third piece, and it's kinda hitting on this in the survey data too, is that the people and change management aspect of this is huge. There's a lot of feelings out there about AI in the world, pro and con against what's happening. And so you've gotta take your people through, like what is your AI strategy?

What does it mean for our workforce and the enterprise to get people to drive the adoption of these solutions? And so I think enterprises are realizing that's not an overnight activity. That's a, a time in front of tools, a time in front of the people, a consistent North Star message. So those are just a couple of them that are out there but we're still seeing on the flip side, Richie, like organizations are being successful with AI.

They're able to implement these workflows. They're overcoming those barriers. It's just that you're seeing more people stumble along the way because of some of these challenges that we're running into.

Richie Cotton: Okay, yeah. Sounds like lots of lots of big challenges there. But yeah, I think I think you're right.

It, it-- This is like a, it's an ultramarathon, not a sprint, you know? You, you- 100%. You take the wins- An ultra

Jim Rowan: Ironman, how about that? 'Cause you gotta do a lot of other things. It's not just the running piece, right? It's the swimming, it's the biking all those types of things. So I think that's a good analogy.

Richie Cotton: Absolutely. All right, yeah. So I think certainly the, the data side of things is incredibly important. AI's had a lot of hype, but maybe the data side sometimes get left behind. Do you want, do you wanna talk us through what needs to happen there?

Jim Rowan: Yeah, for sure. And we see this too with a lot of our clients in our deployments, and even as we looked at deploying AI internally for our people, is that, AI in a chatbot by itself is one thing, and sure, you can prompt it and you can give it context, but if it's not grounded in the way you work or in the data that you have, it's really hard to get those answers out.

And so what I think AI is starting to do is provide a lot of focus and attention back in the data layer that asks the question of do we have a consistent definition of customer? Do we have a consistent definition of product? If we're doing financial reporting, are we rolling up the financials in the same way across all of our businesses, or are there different unique models that are in there?

And so organizations are investing heavily in building that data foundation, establishing the controls and pipelines that are associated with it. They're investing on the master data management layers that are needed to drive this too, and ultimately the control points around as these tools move from just basically, chat bot-y experiences to actual agents, now thinking about the security, governance, and controls of not just to read into those data solutions, but being able to write or log those transactions and really be more dynamic.

So it's not only something that people have invested in just to get started with, but they have to keep investing in to meet the next sort of wave of demand, coming from these AI agents.

Richie Cotton: And who ki- has to implement that? 'Cause a lot of the stuff you talked about is not the analytics side of things, it's the governance, it's data management- Yeah

it's security, it's things like that. So i- is this a shift in what the data team does, or is this like, do you want different people running this?

Jim Rowan: Yeah, I think it's a shift in what the data teams do. I- it's really interesting when you look at the IT organizations in general, s- we're seeing a lot of shifts in just in terms of how those teams are set up now.

I think you're seeing roles getting collapsed across places where you might have a data engineer and an AI engineer and a testing team member and a product owner and a product manager, and like all these different roles that are coming together, and we're seeing the organization shift to be actually have like more kind of individual contributor types, type swim lanes that own, full data products end to end or something like that.

So I do think the data team owns a lot of this, but you can't do it without the business support and governance. Like a lot of this is working with the business, understanding what their requirements are. It's classic core things that we've had to do for system design. We just can't ignore those. What I am seeing that's changing a little bit was that I would say maybe five or 10 years ago we were big into- setting up, enterprise data lakes, data warehouse, like all those terms, right? We've been on this journey for a while. Now I'm seeing that that as we think about that, we're working more towards like this data product idea where we have a very defined product that we're gonna connect to our AI, so that at least we can narrow the scope.

Like we're not trying to do the entire solution at once, but let's figure out what data do we need to support what AI solution that we've got out there. The other thing that we're seeing is a big investment in sort of this semantic layer, common understanding layer context a- across these, these enterprise data.

The good news is that AI can be really helpful with some of these things. AI can run through your data warehouses to help pull some of this information together, can give you feedback on it, can support the data engineers in building and writing code and testing it. So the cool thing I think, is that now we're actually accelerating some of this data work but maybe a couple years ago we were having a hard time getting the business case organized, and the outcome was the dashboard that was hard to take action on.

Now the outcome is data that feeds to an agent that can actually take action with a customer. Okay, now I'm seeing this end-to-end value case come together. So it's a little bit longer answer 'cause, I wanna stitch it together that I think that focus on outcome is allowing people to invest on the data side now a little bit more than maybe what we had to do a couple years ago.

Richie Cotton: Absolutely. It does seem incredibly important that you do have this business outcome from reworking all of your data, 'cause it's a tough ask to be like, "Okay, we just need to rebuild our entire data foundations," without having that kind of business case there. Do you wanna talk me through, like, how you build the business case?

W- what goes into that? Yeah, like we've seen a lot... early days in some of the business cases for AI, it was really focused on, on, I would say, the use case or series of use cases that we were trying to get. And what we saw were a lot of companies saying I need to know the hard dollars associated with this use case."

Jim Rowan: And the challenge with AI over the past couple years we actually don't know some of the answers because the technology's evolving super fast for everyone. We're not quite sure where to apply it and how well it's gonna work. And by the way, what we try three months from now might be better than what we tried three months ago.

It makes the business case even more vi- valuable. So we try to have organizations think about in sort of two categories. What are the quantitative benefits we're gonna derive from the business case? Which makes sense, which is how we've done it in the past. Then also putting in some of the qualitative things that maybe we couldn't do before in the business case that are really important to add.

I might not be able to drive savings from the specific implementation, but I can get access to a new set of customer group that I couldn't do before. Or we've been able to do 100 types of analyses of these scenarios, now we can do 1,000, and then that's gonna create this capability. And I'm feeling it's taking more time for organizations to explain, like, all those different elements that go into the business case, but they're becoming more I'd say more rich with some of that, that data around the qualitative benefits.

The other challenge I'm seeing is that you've gotta link it back to the business outcome, and some of the outcomes are just everyday efficiency. It's the cost of doing business now in some areas. We have to make some of these investments, and it's very hard to measure the everyday efficiency piece.

And so I try to focus, teams on what are the bigger process areas that we're really impacting, and how does it in that quantitative side fit back to the P&L and, the structured actual outcomes? It... The other piece, too, is you've gotta look at this and constantly rework that business case.

And not in a bad way, but in a way that as I said, the technology's moving, and so you have to come back to it and say did it move so much that this business case is different today than it was tomorrow?" The cost per token might be different now with the solution you had from, I was gonna say last week to this week, last month to this month.

And that now opens up a whole new set of use cases or makes the business case profitable now. So that, that's a little bit of what we're seeing, and takes a lot of effort, I think, to get these right and constantly manage them

Richie Cotton: Absolutely. So I love the idea that not everything has to be completely quantitative here.

You sometimes are getting a, a qualitative benefit from the business, and as long as you can explain that, then that's a good thing. I think sometimes as, as data people, we can tend to get stuck on, "We must have a number here," and actually, y-you can be a, a little bit fuzzier with it, especially when you're moving quickly.

Jim Rowan: And maybe one thing we should just add to that, too, is like the fuzzy part or is let's give some ranges. Like that was another big struggle for, for some companies was like I'm gonna give you a number. It's, 22.6." It's like, "No, let's give you a range from 20 to 30, and we'll refine the range as we learn more about the solution that we're building."

That trait takes a little bit of risk from management to agree to some of those things, but I think it's the risk worth taking in a sort of like very ambiguous space as we move forward

Richie Cotton: absolutely. So the other thing you mentioned was that quite often the business case can change within a month just 'cause, technology's moved and what- a lot of particularly in, in large enterprise, you think of much longer cycles for projects, maybe months, years.

How do you deal with that rapid iteration?

Jim Rowan: Yeah. It's a real mindset shift across the organization. I think this starts at the board level C-suite level. You really have to start educating folks on sort of the pace of change that we're in, and I think a lot of executives are feeling it.

But then that means that we can execute faster in some of the things that we have. And so it's both that mindset shift, but also looking at our internal processes, too. And I'm not saying we skip steps that are important steps, but there's also things that we can shrink those steps now with the power of some of our AI tools as part of tho- those things.

So we can run up a series of tests way faster than we could before, so let's make sure we do that. Let's make sure we still have humans, on the loop checking the things that we're doing, but we can run those a lot faster. So I'm seeing a little bit of that from a speed perspective.

The other piece is, the business I feel like is the business and IT partnership on these things to really be agile. And getting people comfortable with these alphas and betas that you release I think is really important, too, and that's been some of the mindset shift. I've seen, some of our clients be really successful defining, good size pilot groups who are gonna release out solutions to them.

And they're safe solutions that they're releasing, but they're still releasing in alpha and beta 'cause you know you're not gonna get to perfect, by the time you go to release it. And so that, perfect being the enemy of good kind of equation comes into play a lot in this space, too, for speed.

But I've always found, too, that we're not trying to sacrifice the governance and the quality. And one of the things that we've also learned in this is that if you set up the governance before you get started on this and define places where you're comfortable with AI use, how you're gonna handle the data, how you're gonna secure the information, what guardrails are in place before you get started, it gives your team a lot more speed to get running.

I see a lot of times solutions just, hit a roadblock running really fast, and then it's, whoops, we never really got the right teams involved from a governance perspective, and that slows down the whole process. So there's a lot of elements to go quickly, but it's go quickly with purpose and not in a reckless, fashion, right?

Put your helmet on, wear a seatbelt, do all the things you should be doing, but you gotta get moving

Richie Cotton: Absolutely. Yeah going fast seems to be incredibly important. Now a lot of the stuff you talked about is is different connections between teams. You mentioned like connections between IT and business connections between governance and almost everyone else.

Does this require like a, a different approach to organizational design then?

Jim Rowan: I think it will over time. We have a paper out that we talk a little about kind of the work chart versus the org chart. It's a different, play on words, right? But we've been behind these same enterprise org charts since the, the railways were set up, I think.

And so there's gotta be room for improvement from that to where we are today. The tricky thing is like when you try to talk about that, I think people immediately assume that means there's like a, a jobs issue or something. It's like, no, it just means how we configure the work differently, and maybe we're all working in a slightly different way than we should in the past.

So I do think we're gonna see, new roles become part of the solution that we have. We're gonna see new ways of working change, and I do think we start to collapse some of these org charts into different areas of work. And I think as individuals working on things, we're gonna also I think find ourselves doing more with the capabilities that we have.

We're gonna be managing teams of agents before we know it. Some people already are. They're off doing things while we're doing other things. And so there's time in your day now you need to carve up to check in on the AI tools you have working on your behalf. We didn't do that two years ago.

Maybe some people did. I wasn't doing that two years ago. And so now it's like to block time to check in with my agents, and what's actually happening is a new way of working that's gonna be different. And I even think that's gonna change, in a few more months maybe shorter, of how these other personal agents working on our behalf interact with someone else's personal agent on their behalf and change how that works.

So I think we have to be ready for some change in that design. I also think this isn't an excuse to have a bunch of shadow IT in the business either. I think people could use this as an excuse for that. I think there's an opportunity to create new roles in different business roles around AI product ownership new ways to think about forward deployed engineers and different elements of the business.

So those are gonna be challenges to our existing org chart, and I just really encourage enterprises to be willing to pilot those in parts of your business to see what's working in your culture, 'cause that's the other big part of this too. It's gotta fit your culture well.

Richie Cotton: Absolutely. A lot to impact there.

Jim Rowan: So- Yes. It's a, it's an area that's passionate to me because I feel like we can talk about the tech all we want, but if we don't actually change the process and the org around it, for the better of this stuff, you're just making some faster stuff, and I don't know if that's really the end state for AI.

I don't think it is.

Richie Cotton: Absolutely. So I think some of the stuff you're talking about is working with different people and things. Yeah. It sounds like cross-functional initiatives that have been around for decades. So what's new here? Are you suggesting getting rid of a hierarchical org chart?

Or how does it, how's it gonna look in the future?

Jim Rowan: Yeah. I think there's more, squads and teams, right? We've seen a lot in software development where teams working in pod-based constructs with different responsibilities. And so do you still need a functional analyst on some of these things?

Totally. But the functional analyst might actually get to a workable design solution way faster than we used to in the past, where the BA might have been doing some work, then hand it off to a designer, that then hands it off to an engineer, that then hands it off to the tester. That might be one person doing that end-to-end workflow with a peer review, and that's how we check to see the design things are compliant and we're meeting sort of our coding standards, et cetera.

So I think that's where some of the things collapse. And I think from the business side, working in with IT, you're gonna be bringing a vibe-coded solution into the meeting and say, "Look, we built this demo. We're thinking this could do this. What do you guys think?" "Oh, our systems behave a little bit this way.

How about we tweak that?" That compressed a month of design and requirements conversations. It's just a very different way of working. And I think both sides of the equation, business and IT, are gonna get more familiar with AI tools, and so we have to meet each other there. And then as you get more senior in different enterprise roles, we might have more responsibilities with different individuals and different org charts.

I think it changes a little bit around that. If I had all the answers of how the org charts change, I'd try to go for it. But that's where the organizational cultures matter, too. Different leaders, different areas. It's gonna, it's gonna take a little while for that to change, I think.

Richie Cotton: Absolutely. Certainly if you're making drastic changes at that sort of org chart level, it's like it's gonna be chaos. You mentioned leadership here. What's the role of leadership i-in terms of initiating all this? Who needs to be in charge of just making sure all your AI transformation works?

Jim Rowan: Yeah. What we've seen from our clients and through conversations and survey data is that the, this top level leadership, so CEO, board, C-suite, really has to be behind this, and they have to own the narrative effectively, and not just own the narrative in the sense that we can hit the talking points.

They've gotta live the narrative. And I think that's some of the bigger changes that are here, where, you need to show that the CEO's using AI. How is the CEO using AI? What's the effectiveness there? We've done workshops. We do a lot of AI education training and learning workshops, as well as b-board ones, and have been part of some of those where we really-- Like, getting the C-suite hands-on with AI tools, building agents, and then showing their peers what they're building unlocks a ton of value.

So that leadership element is key. You gotta lead by doing on some of these cases. Having the clear North Star of what our intent is with the power of this technology, how we're handling the safety and security around it. Be really clear about how we view this. What are we testing?

How do we think about our data? And tell your people about it. That's been great. I've also seen, the success then from that is then you've gotta get out to the people and talk to your broader teams. Make sure you're communicating what the vision is hearing their feedback. We've seen a lot of success a-at our clients and internally with reverse mentoring, where we're getting folks that are, in different elements of the business and are able to share what they're learning about the technology, getting leaders up to speed.

Leaders are then edg- able to educate them on the strategy and talk about how that's working and how does this all fit together. It really creates, awesome opportunities for culture and team building and community building. And then I think the, the final piece is just the flexibility of mindset.

Know that this stuff is gonna continue to change, and some of the folks in this field I think study this, push this out really well is that you can't design these AI solutions, just for today's technology. And so as leaders, you've gotta look at it and challenge your team and say, "Okay, now that you've showed me this, what happens in six months?"

And you might see a team kinda go like deer in the headlights. "Whoa, what do you mean six months from now? This is the solution." It's yeah, but AI's gonna change in six months, so what's it gonna do? And I think that's a really good quality in our leaders is to push some of those questions, too.

Richie Cotton: Absolutely. So it's interesting to say it's gotta start with the CEO. The CEO has to be making use of AI, otherwise no one else is gonna take it seriously. Yeah. I think last year, maybe the year before, there was this whole thing about oh, every company needs a chief AI officer.

Is that role still relevant or is it just AI's kind of part of everything now, so you don't need a separate CAIO?

Jim Rowan: Yeah, it's a great-- yeah I wonder about this a lot, and I think this is, is-- it's one of those things where a, a classic consulting answer would be, "Richie, it depends." And everyone's like, roll their eyes and be like, "Thanks.

Super useful," right? So I'll give you a take, and then I think, it is gonna still fall back on, on the depends. I think it's a role that lives for a few years, but I think if the CTO function is working effectively, that's where some of this capability can live. And that's my like, if I had to take a stand on one of these things, how I would design an org chart.

But we all know that all these structures are based on the people that have been hired, on the relationships that have been set up, so they're well ingrained. And so I'm seeing organizations at least nominate an individual that's accountable for AI. And that's really important. Whether that leader reports to the CEO, CIO, CTO, chief transformation officer, depends a lot on where the organizations go. I do worry that does that role stick around or is that role really what the CDAIO does or CDO, right? Depending on all the, the, the acronyms we add to that. Or does it fall into the CTO function?

And my, my pressure test for it is I'll ask organizations and say what happens when quantum comes?" And say, "Do you have a chief quantum officer? Does that seem right?" I don't know. And so do we have an officer for every technology that comes up? We did digital, so maybe that's a thing, but I don't...

digital officers have come and gone in some roles too. So that's the pressure test, and I do think because of the way AI is changing some of the org charts and sort of the shape of things, we do need to pressure test like, do we need more management roles? And what does that really look like, and how is that helping us achieve our goals?

Maybe it's an interim thing. So I don't know. You'll have to tell me what others, the others tell you on the hot take, but that's where my head is.

Richie Cotton: Okay. Absolutely. I love that chief quantum officer test, and that feels like a whole separate podcast episode. What does a chief quantum officer do?

Jim Rowan: E-exactly. But if you're gonna throw in a chief AI officer do you need it the next waves of tech that comes out? Do you always need a, a leader for that? Or is the CTO function sensing, owning, finding the next thing, and cycling through that in a way that makes the organization run effectively?

I... That's sort of my test.

Richie Cotton: Absolutely. And so one of the things you mentioned a moment ago was the idea of you can't just have a solution and be done with it 'cause technology's gonna change, AI's gonna be different in six months' time. So we talked before about you've gotta have these sort of rapid iterations.

How do you deal with that and also longer term planning? It seems like these two are slightly at odds.

Jim Rowan: Yeah. I feel like, yeah, ask me for the LT- long-term plan, the LTP on your AI activities. You gotta... really? Yeah, we've experienced this. We were going through an implementation of one of our solutions, and we did a custom build for it 'cause the tech wasn't really there in the marketplace.

And literally, the, we went live and a month later, the, the tech appears out there and you're like, "Oh, gosh, was it really worth it?" And we had to test ourselves and say, yeah, it was worth it because we learned a lot from that experience of what to build. But what we also took away from that was, like, we've gotta make sure we're really close with our vendors alliances, whatever, the structures your organization has to understand their product roadmap I think the talents that they're facing is the same sort of thing.

You see a product feature out there, seven other people go to copy that product feature by the end of the week, and you're like, "Okay," "how do I navigate this space?" And so we're finding that it's really important to have a lab incubation facility, something on your broader... We see a lot of companies have AICoE is-- call it what you want, but on that team has a function that's able to test out the new stuff and report back about how it's working, and that's their job.

Sounds like an awesome job, and a lot of people like doing it, but doing that kind of testing is really important. And their goal isn't to get everything through to a yes, their goal is to see what's working and sense what's coming next. And so I think organizations need to do their own homework on those things and use that.

They can use third parties to help, but get that information. And then as you're designing and building your solution, I think it's really important to focus on what are the elements of the platform that we're building, that we're using that are consistent and aren't gonna change?

These are the things that we believe are fundamental. We have a solution for our data. Here's how we're gonna handle it. We have these kind of model options presented to us, either open or, frontier or closed, whatever the... We have these available for us. We know our infrastructure, we know how we're gonna handle the data.

And then you need to start making sure that you've got the flexible components so you can change. And we're seeing that a lot in some of the model categories, right? As people build solutions, they're finding out that they can switch from a f- a, frontier class model to an open weight model in some cases, or a layering of a frontier over an open weight to solve for problems.

And that kind of dyn-dynamism that we're seeing in the marketplace is how you have to engineer, I think, these solutions. And I think gone is the day of like the rigid, here's the architecture printed on the wall in the, you know, chief architect's office that thou shalt not change the architecture, right?

It's gonna take a lot of a lot of continued whiteboard activity to solve for it. So that's where I think we need to go. I do worry about the cost of these things, and so you do need to make sure that you're making these bets around the platforms, understanding where you're making choices for locking things in and where you have flexibility in those choices.

And a lot of clients I'm working with are on this build versus buy challenge because, gosh, it seems so easy when you see some things to be like, "Oh, I could build that." And then it's like okay, but these are all the things that come with the real product set, plus you're buying the next X number of iterations of vision that they're creating for it too.

And so I think wrestling with that and figuring out what's at the heart of the business that you really wanna build versus things that you can buy and plug in is really key to having that flexible architecture model too.

Richie Cotton: Absolutely. Yeah, certainly there are some technologies which are very difficult to switch like anytime you wanna switch your data warehouse or whatever- Data warehouse or cloud provider

That's a big effort, yeah. Yeah. Yeah. It's a lot of effort. And so you mentioned the build versus buy thing, and I think there was a time maybe like a, a start of the year where everyone was like, "Oh, you know what? I can just vibe code my own version of Salesforce-" Yes ... or whatever, and-

Jim Rowan: Every solution

Richie Cotton: out there

that's not quite happening. So yeah. Talk, talk me through how you decide what should you build yourself? What should you buy in and yeah, how does that work?

Jim Rowan: Yeah. I think part of that is really understanding the market, like where are the tools today and kinda what are available for us?

And are we gonna drive competitive advantage from some of the solutions or not? Is this part of the core thing that I'm building? And so tho- and the, the other question is really around data and lock-in that I'm asking too. And so when I think about something that I wanna build myself, I'm thinking about those cores of the business that, really fundamentally change how I engage with the customer how that experience is handled.

If I were to buy it, what am I potentially locking in or leasing or giving away in that experience that could make me wanna change in a few months? I'm seeing a lot of organizations continue to buy solutions, and I think, yeah, we saw that, the news, and everyone was going down this path of, what happens to SaaS and software in general.

And I think what we're seeing is what, people have projected of the we're just gonna have more software. It's gonna do more things for us, and we might build more wrappers around these tools and capabilities. So I find it to be very helpful for figuring out the working prototypes, the ideas, and then really testing to see if the buy option is cost-effective.

And a lot of people are finding the buy option to be pretty cost-effective. Y- the, the other tricky thing is you gotta get into the weeds with some of these solutions too. A lot of stuff still requires a lot of configuration, a lot of tailoring once it's purchased, and that's important to weigh as well 'cause you're gonna be doing that between the build and the buy anyways.

It's not like you're really just taking it off the shelf and plugging it in, unless if you're maybe a 20-person company and you've got these, standard set of templates and you don't have to differentiate. But most of the organizations we're working with are so large and complex, it's not just one SaaS product.

It's a couple versions of different things trying to connect to different global, data set. That's well beyond a vibe-coded solution. And then, by the way, you're also buying the security protection controls that you'd have to implement anyways from scratch. So I really caution people on the building their own on a lot of these things because it might seem easy, but once you get into it, I think it becomes far more complex.

Richie Cotton: Absolutely. You can get you get to your pilot stage, but you're not getting something into production yet. Or your pilot

Jim Rowan: goes well- yeah ... and then you're down the roadmap and you see the same thing you were gonna buy advanced, a year ahead of you because they have 1,000 people in R&D and you have two.

Yeah. Yeah.

Richie Cotton: That seems fair. So all this kind of links back to maybe the hottest topic at the moment, which is AI cost control. And so it's like even if you buy stuff with AI in it, you still gotta figure out what the price is and the price of maintaining that software, all that kind of stuff.

Jim Rowan: Yeah.

Richie Cotton: Talk me through how do you not blow your whole budget in a few weeks?

Jim Rowan: Yeah. Everyone had the token maxing tattoo at the start of the year to show that we were token maxing, and now it's like, whoa, whoa, wait, hold up. I'm not. I'm cautiously using the tokens I have to drive output. Don't call me a token maxer.

And so I think that's been a really big shift. So organizations... it's been exciting because organizations are seeing adoption, and so you go back to the-- like circle back to our survey data. Okay, great. So now we're seeing people using AI, which is really important to see. And now it's a question of are the input tokens worth the output that we're creating?

And I don't think very many people understood there were input costs and output costs, and even understood what a token was before, I don't know, maybe April or March in terms of large... i'm sure everyone did, but in the large scale of things, it wasn't on the radar until you started seeing these stories of organizations that were maxing out, a 12-month budget in two months.

And so organizations really need to think about the personas that are using these tools, thinking about budgets around the personas that are using them, measuring the token input and output for the workflows that they're looking at, and it's common cost controls, right? We've seen these cost controls, entry and egress costs for cloud for years.

We know how to compute costs for cloud. We know how to do this. And so organizations that have had sophisticated FinOps capabilities in cloud are perfectly set up to go ahead and measure this from a token perspective, and we're seeing tools come on the market that allow you to get into the granularity of what's actually being used and where.

We're seeing the companies that are advancing AI are adding these capabilities into the harnesses that they're selling with their models. Tools that are showing up so you can see exactly how much your prompt costs is super helpful. So when I know that it costs me $8 to make something, I know what I've spent it on, right?

Versus a long-running task that might be taking longer. So I think that transparency is really important, education of the workforce, and putting in these controls and budgets for folks. I think the bad thing to do is to, "I'm gonna turn this off tomorrow because we're overspending on tokens."

I think that immediately sets back the transformations that are happening. And so you really wanna be, educating your team about how these are using. I think the other thing that's popped up, and, wasn't... was on the radar, but I don't think as much, was the open-weight models.

And they're popping up because it's not that it's like it's free in the sense that I'm not paying for-- to a specific price. You're still gonna do the inference and, a cost associated with running them. But that's popping up a lot, too. And so organizations are thinking about how can I use those?

Or a lot of companies have model families, and so you're choosing amongst the model family, and do you always need the most advanced model for some of the tasks and questions? Probably not. And so that's the other thing that I think has helped with the education and quell the noise a little bit.

But yeah, a lot of CFOs have learned about token taxing and token economics in the past past couple months.

Richie Cotton: Absolutely. I love the idea of having tooling in place so people can actually see how much they're spending. There's visibility across the organization, like what you're actually doing with AI.

Yeah. But you mentioned CFOs there, so I think traditionally, like, how much IT capacity you have that's a chief information officer's thing, not a chief financial officer thing. So is there now a new switchover of roles between finance and IT?

Jim Rowan: I don't think there's a switch of roles.

I think what we saw was that CIOs were showing up in the CFO's office saying, "Hey, I got this... Yeah, I gotta tell you about a little problem we ran into." Or, "We forecasted X, and it's looking like Y, and here's the reason behind the situation." So no, I think it still stays within the CIO's control, plane to look at it.

But if you think about what the CIO's doing they're running the AI factory for the business. So much like supply chain has a series of costs of goods sold that filled into the creation of a product, they're now running an AI factory for the enterprise, and they're inbound costs associated with all the different business parts that they support.

And so how you think about the allocation of those costs back to the business becomes a CFO kind of conversation, right? And and the fully allocated P&Ls and how you get into that sort of management. So that's where the bridge, I think, is happening across these two organizations, and it's a very different mindset shift.

We didn't think of our knowledge worker teams as having a factory thing, but now we have this factory thing that supports them in the work that they're doing. And so as you add agents to a task, that process, the business function probably needs to pay for the cost of what that's doing. Is it getting allocated back?

Are they being thoughtful in the model selection? All that kind of stuff. So that's how we're seeing it. Not hearing any CFOs taking over the CIO's jobs yet but maybe there'll be a battle of agents, to see who who negotiates that.

Richie Cotton: Yeah. Maybe like a, a new CIO role or something like that.

Nice. Okay so actually Amanu- I haven't-- I have no idea whether I should be picking a specific model or the specific effort and whether that's gonna give me the best return on investment for the task I'm doing. Like- Yeah ... how do, how should individuals figure out if they're actually getting good value from the AI that they're using?

Jim Rowan: From the selection. Yeah, just even when you think about the individual tools themselves. So a lot of people have their employees using the harnesses that are coming with the models to date. Some organizations have built their own harness where they access the models, and that gives you a little bit more control.

So for example, at Deloitte, we built a solution called Sidekick, and that gives us a little bit more control over some of those, those choices. And so that's another interesting build versus buy decision of that gave us a choice what we can control, but we don't have all the greatest, latest features from the harnesses from other companies.

Now when you get outside of the area that you control that becomes a little bit trickier in terms of the, the choices because some of those tools allow you to default selections to specific models. And most of the time you're good, right? Use the default and you should be fine. Other times you need to s- level up in terms of the, the thinking-thinking, reasoning speed with which the task needs to be executed, and I think that comes down to a lot of training.

But I, but I-- The goal should be to kinda default to the base model that the organizations, that the frontier companies have told you is the target sweet spot for price performance and kinda live in that zone. Only the sophisticated users, and this is where I talk about personas and why it's so important, should really have access to make some of the model selection that could get people into trouble.

But even then, you get into budgets where you can set the budgets. And what I've experienced with those is that most of the, the solutions then defaults you back to the most, the, the lowest cost model until you resolve the budget situation. And so that's how to do that. You, as a user though, if you're a very frequent user of AI, what I find that I need to do is just constantly be testing and trying different things at once to see what works.

And then we share that back with the team about our evals, what we're experiencing. Some things better for others than not. But ultimately this, this intelligence level just continues to increase so fast that it becomes, for me, less about model selection and more about what processes can I do on these tools?

What work can I offload to them that I can go verify and check and make sure is doing correctly?

Richie Cotton: Okay. Yeah, certainly verifying that it's doing correctly, that seems like the best thing for getting good value for it, 'cause if it gets the wrong answer, then, you've spent money on, on nothing.

Jim Rowan: Start with the lower end of the, some of the questions in the task, too. I'm always surprised with some of these things because they're w- they... And I spend a lot of time in this every day, and you still get surprised at how good the capability continues to be. And as other AI leaders, are listening, I'm sure they're all kinda doing the same thing.

We're seeing the trends that are coming out there. We're trying the new stuff, and I think it's really important for individuals especially that are deeper in the space have to have their own evals that they're doing to kinda test their own work against this.

Richie Cotton: How do you create your own evals?

What, what kind of stuff should be, should you be testing? Yeah, like I have a series of prompts that I typically run, and I use that as a check to see what kind of answers I get back from AI tools. And so I use that as a way for me to give a sense of okay, this is where we've advanced here in different tasks.

Jim Rowan: So thinking about tasks that involve computer use or tasks that involve the creation of a deliverable. Hey, I'm a consultant at heart, so at some point I'm gonna build a PowerPoint. I check how well I can create some PowerPoint solutions, from some of these things, or how well it can go through the thought process.

But that's just me as an individual user. Enterprises, we do this as well. We have thousands of prompts that we run against things as they're being released to do evals and then determine how they're performing, both on a price perspective, a speed perspective a- answer capabilities. A lot of folks are doing this in the coding space as well.

And that really gives you a good sense of like where the models are behaving and the benchmarks are great and super helpful that are having, that are out there, and it's another way to measure it. But I feel so much of this needs to get measured against the, the work that you're actually doing in your enterprise.

It's good to have your own evals.

Richie Cotton: I absolutely agree. It's good to be able to evaluate stuff. Yeah, testing's like one of those underrated things that everyone says- So

Jim Rowan: not exciting, right? ...

Richie Cotton: yeah. Yeah. But in- incredibly important. So all this kind of stuff seems to lead back to the idea of changing your processes to incorporate AI, and you mentioned that's one of the big challenges.

Sometimes you go, "Okay, I've, I automated easy step one. The rest of it I still gotta manually change the bottleneck." So do you have any tips for is there like a, a framework or some kind of process you can have for this is how I do good process re-engineering?

Jim Rowan: Yeah. We've been doing process engineering forever in the enterprise, and so there's, Six Sigma frameworks and other things organizations use to re-engineer their processes. So I think the frameworks exist. Deloitte has some frameworks that we use with clients. We have a tool called AI Navigator that we help them go through as well.

We have those things out there. It's... And for me, it's a mindset thing. It's, the tools are how you get the work done, but it's gotta start with a mindset that I'm gonna come in, and we're gonna change how this work gets done, and it might actually mean that we've deleted five steps.

And that's the hardest part for a lot of organizations is saying, I know I started with a five-step process. I'm gonna have to make it a 10-step process today, and I'm gonna delete five of them, because you know what? As I look at this thing, those weren't five value-added things." And I don't think we're taking enough of this first principles approach back to process re-engineering to say...

and I go back to my days of building reports, right? I used to be a, report developer, data visualization. Like, why do we actually need the report? Who's gonna look at the report I'm creating? We don't spend-- We need to spend the time asking that question and then figuring out why we're doing some of those things.

That's really important for the process re-engineering. Otherwise, I think we fall right back into the automation step, which isn't bad, and will drive efficiency, but it will not drive the endpoint, which is I'm doing work differently because I have these AI tools and capabilities. And I worry right now a little bit with the advent of some of the personal agents and some of those things that are coming out, that just kinda makes the same stuff happen faster, but we didn't actually change the stuff that's happening.

And maybe those tools eventually change the things that we were doing? I don't know. But that's the piece that I would, I worry about. And so a lot of our clients we think about in two different buckets, like everyday AI efficiency, like what can I drive to make my employees better, more efficient, give them the tools they want, happier at work, great.

And then the bigger ones are the ROI drivers that are gonna be these reimagined processes. And I found that if you take a look at that smaller group of processes that you can really own end-to-end and start looking at what you can delete, how you can change what the work is, work backwards from the outcome really helps you get to the answer of doing that.

And once you get one or two, I think it starts to snowball. But if you go after the most complex, hairiest, complex, awful process in the org, like I think you're gonna grind on that way too long. We just see organizations get stuck there because it's just too much to unravel out of the gate.

So I hope that helps people out there that are trying to do this, 'cause it is not easy, for sure.

Richie Cotton: No I feel like if you can crack this then, that's all the problem solved. But I do like the idea of start with simpler processes and see what you can do. And also like deleting steps.

That's brilliant, right? It's like you get to not do work after that Basically ... it seems like the perfect outcome.

Jim Rowan: And a lot of times we're just stuck doing some of these things because we've had to or we've thought it's been part of our procedure for a really long time. And so sometimes this is where it's helpful to char- start in where you've standardized your processes already.

Go back and look at some of those places and say, "Okay, what else can I change?" "What else-- Why do we really do this? I don't know." If you don't know, it's a good reason to delete it.

Richie Cotton: Absolutely. So I feel like in general, individual workers are gonna be in favor of deleting steps from processes. It might be the management that says, "Oof I'm not sure about that."

And it, there's some kind of communication thing here. How do you make sure that your boss is okay with you not delete- doing steps?

Jim Rowan: I-- this is where I think you come back to why we started with leadership. I think y- you were asking about the leadership piece, which is so important.

Leadership ha- needs to be on board with we might do less work in some of these areas, and I'm gonna get you the same trusted outcome. 'Cause your outcome was X. Like, you wanted to go achieve that outcome. Let me show you a better, faster way to do that. I think leadership buys that. If leadership is hearing, "We're deleting steps two, four, and eight from a 10-step process," they're not gonna buy in on that.

That's not a, that's not a leadership message. That's a recipe for chaos 'cause you're just stopping stuff that we might need to happen, right? So that's where I would say it's all about the, the leadership buy-in and messaging back to the team. And the other thing that we see a little bit in this too, and I'll just harp on this on the leadership side, is that the reason why we're having a hard time understanding the processes is some organizations are frozen in the middle of this because leadership isn't talking clearly about what they view AI is.

They're worried about what's gonna happen to them if they figure out these processes or make some changes in them, and does that impact their job? And you've gotta figure out a way that you can communicate that effectively to the team so they know and can safely and have that psychological safety when they're working on these things.

And that's a leadership responsibility. Absolutely, yeah. It just seemed like back to leadership needs to take ownership of like all, all the change. Yeah. You can't

just punt this down to a bunch of people to go figure out. Lock them in a room, turn on the whiteboard, reimagine it, get back to me by the end of the week.

Richie Cotton: All right. Nice. Don't think it's gonna

Jim Rowan: happen.

Richie Cotton: All right, so the other blocker you mentioned at the start was around people having skepticism around AI, and some skepticism is a good idea 'cause, you g- you can have things go wrong. But talk me through the change management process to make sure that people are making use of AI in the way you want them to.

Jim Rowan: Yeah. I think it's really like i- and, change management probably doesn't do it justice anymore. Like that's... when you hear change management, you're thinking about an old school ERP rollout, and I think it has to be a different view of this today, which is, adoption and acceleration of AI is the way I probably should be saying it more because that's the positive of what we're trying to create.

We're trying to show workers how the power of this can make your day better, how this can improve what you're trying to do have more time for human interaction and engagement on the things you wanna go do. And so it's about shifting the mindset with some of these things about how the tools work.

It's about having that leadership message baked into the communication so that people get it, and bringing everyone along on the journey. And we've had this journey of oh, we wanna create AI fluency. And I think AI fluency is a fine term, but it's not the answer. It's it's about this adoption and this continuous learning mechanism that you need to build in your organization, and I think that's exciting for a lot of people.

If we position it that way, that sounds exciting. I think what I worry about is sometimes we position it as "We're gonna go change manage you." That doesn't sound like a great thing to have happen to you. I don't know if I really wanted that. So like that's the mindset shift, and I think that's why, I'll correct myself, like calling it change management's probably a bad way to frame it.

It's this sort of AI adoption and acceleration, and that allows you to really change the narrative of what's happening. And even then that sounds a little bit like, oh, we're changing narratives. Like we- we, we're not being deceptive. We're just trying to be clear and transparent about how this is gonna work in the org.

Richie Cotton: Yeah, certainly, yeah. I can see how there's definite messaging pitfalls there- Yeah ... that you could fall into yeah, probably don't say, "We're gonna change your job," and Yeah ... probably get rid of you.

Jim Rowan: Here's our change management one-hour meeting. Okay, like that's probably not the way I wanna handle it, but if I'm gonna go accelerate how, value gets created from your position through the use of AI, okay, that sounds like a different hour.

Now let's talk about how that- Definitely ... how that works.

Richie Cotton: Absolutely, yeah. You might get better meeting attendance for that one.

Jim Rowan: Yeah, exactly. People might show up.

Richie Cotton: Yeah. So y- you mentioned continuous learning before. Yeah. So talk me through what do you think the most important AI skills are that everyone should be learning?

Jim Rowan: Yeah, it's I think we went on this weird wave with prompt engineering, where it was like it was in vogue and everyone had to be prompt engineers, and then it was like, "Oh no, that's like an old skill. Don't worry about it." I think still being able to engage with AI is an important tool, and so we can call that prompt engineering if we want to.

But being able to structure your thoughts in a way that you can get the best possible output from AI is a really helpful thing to continue to work on and refine. And some of the fun tricks that I... you've learned from others and myself is I'll put in a prompt and then ask AI to make it better.

Or I'll put in a prompt and say "Hey, this was interesting. Ask me five questions to make the output better from what we're gonna do." And so all that really is, though, is just continually using and trying new things out with these tools, I think is number one on continuous learning. And so giving your teams abilities to go do that and to try those things out, and a community mechanism to share is really important.

I've also found that if you're delivering learning in this space, figuring out ways to do the learning, like in micro bite-sized chunks at the point of where you're trying to get the AI impact into the system. And video links to the training is really important, making it short and snappy and quick so people get it.

With our attention spans, you wanna make sure we really understand what's going on. And that's been really helpful too. And then it's this culture of continuous learning, and so you-- putting that into place in an organization's really important. And, we talk about leadership a lot.

You're giving me a hard time. I'll bring it up again. But if the leadership team is showing that they're learning and adopting too, it's gonna carry through the organization. One of the other funny things I've seen is sometimes people are sheepish to say they used AI for something.

It's like, ooh "I'm gonna tell you I used AI to make this PowerPoint." And I'm like, "Great, 100%. Why wouldn't you?" Why-- I'm sure you checked your work, right? You still gotta do the work to make it forward, but, like, why wouldn't you use the tools? And so getting that normalized in the organization is really important and that comes with some of the training and adoption and usage, 'cause you'll just be sharing it more with others.

There's also great programs for certifications. Don't get-- Those longer term learnings are great, but I think just making it part of the day-to-day habit is really important.

Richie Cotton: I love that. So it does sound like a, a lot of this really is around making sure that there's a culture of learning and a culture of experimentation there.

But I also like the idea of just normalizing AI usage. Yeah, at this point, if you're manually writing everything in a presentation, like creating slides from scratch, that's no, don't do that anymore. Yeah. Yeah. You're wasting your time.

Jim Rowan: Yeah. Yeah. The great joy I had aligning shapes in, in, in slide decks, I can move on from that.

That was a nice feature, but I can let someone else do that.

Richie Cotton: Yeah. Absolutely. And this is big coming from someone at Deloitte. I know De- Deloitte's got quite a big PowerPoint approach, right? Makes a lot of PowerPoint. Yeah, exactly. Brilliant. All right do you have any final advice on, like, how you can approach AI transformation?

Jim Rowan: Yeah. I think it's-- The key thing is getting started. This can seem like a monumental... You listen to some- listen to this conversation and go, it's like, "Oh, my gosh. Wow, there's a lot of things to think about." Just get started. This is a place to get started. I think you summarized it well.

You wanna start small with something that you can go after. I think that's number one. Having the strong leadership engagement message connection to this, to these programs is really important out of the gate. And then the other piece is be the change you wanna see in these things. And so it sounds a little hokey, but if you're using AI tools, and you're leading the transformation, and you're showing people how they work, that's gonna drive a lot of change in the organization, and people realize that they can do new things with the tools.

So I think those would be my top three. And maybe I'll throw in a fourth of have a good time with this stuff. It's an amazing time to be using some of these things. The things you can do now that you couldn't do six months ago, two months ago, it's pretty amazing. So enjoy that and see what new things you can build.

Richie Cotton: Have a good time with... Y- you had me there. That's absolutely- Why not? Exactly. Yeah. Wonderful. All right so just finally, I always want more people to learn from. Yeah. Whose work are you most excited about at the moment?

Jim Rowan: Oh, gosh. Ethan Mollick, I've been following a bunch. I'm really excited about the work that he's doing.

He's got his new book coming out, I think, in October so I had a chance to pre-read that, so very excited about the work he's doing there. I like to follow The Rundown AI is another newsletter that I read a lot. There's a guy named Rohan Chung that puts that out. I think I've been following him since the early days of AI when he probably didn't have a lot of subscribers.

And then I just really love listening to leaders in this space. I think you learn a lot from hearing from the leaders that are talking about their companies and sharing what they're focused on. And there's so much of that out there that get on the podcast, listen to things, listen to Data Framed. Exactly. But if you're not out there listening to this stuff, you're not gonna pick it up, and you're not gonna understand like what's going on, so I get a lot from that too.

Richie Cotton: Yeah. Certainly E- Ethan Mollick is a legend. Rohan Chung I, I don't know about, but I'm definitely gonna check him out.

Perfect. And yes, I lo- I love the solution to getting better at all this is listen to Data Framed. Exactly ... an important takeaway for our listeners. Yeah,

Jim Rowan: exactly. It's an easy plug, Richie, right? It's a great plug.

Richie Cotton: Yeah. Thank you. So o- on that note I think we'll wrap it up. No more to be said.

Yeah. Thank you so much for your time, Jim.

Jim Rowan: Thanks so much. I loved it. It was a great

Richie Cotton: conversation. Appreciate it.