The Challenges of Enterprise Agentic AI with Manasi Vartak, Chief AI Architect at Cloudera

Richie Cotton: Hi Manasi. Welcome to DataFramed.

Manasi Vartak: Thanks all for having me, Richie. 

Richie Cotton: Excellent. I'm pretty excited to be recording from Cloudera Evolve. It's one of my favorite conferences. There's a lot of conferences now. They're very much AI fluff. You go there and they're, oh, AI's gonna change everything, evolve. I always feel like I'm actually learning something new.

So, uh, have you got any highlights from the conference so far? 

Manasi Vartak: Several. Um, New York is probably the most exciting city for us because, um, Cloudera serves a lot of financial institutions and, you know, New York is kind of the epicenter of all of that. Lots of great discussions around how AI is being used in financial services.

Also, insurance. Some oil and gas to, and it's been really exciting to see how deep, um, the AI adoption is going. It's not just the technical users. We got the business users. So we have a whole session packed with, um, AI this afternoon. Uh, so I'm really looking forward to it. 

Richie Cotton: Uh, yeah, uh, lots of exciting things going on.

Uh, certainly in the finance world. I think there's a lot of, uh, companies that are getting very deep into ai, so, uh, perhaps we can touch on that. Uh, later on. But, uh, to begin with, I've got a conundrum for you. So we are now three years into the generative AI revolution. It feels like all year, everyone's who's been talking about AI agents nonstop, why enterprise is not just filled with bots already.

Manasi Vartak: Great question. Um, I'm gonna maybe put that in perspective a little bit. We say it's three years, but it's only three years. If it were a cat, it would still be a kitten. You know what I mean? It's like it's baby right now. Um, and so when I think about something, say, um, digital transformation, it's been around for.

years and we're still in that journey. And so when I put the gen AI revolution in that perspective is like, we're still in the early days. Um, and there was a survey making the rounds and you know, hey, % of POCs aren't going to production. Um, I kind of look at it as. We're in the experimentation stage, we're still trying to figure out where does this technology work, um, where are the rough edges that we need to still kind of, you know, figure out how to navigate.

Uh, but given how early we are, I think we're seeing some amazing results all across the board. So whether you're thinking about coding agents who can. Pretty much co fairly well, um, all the way to, we're solving math problems and we're doing digital, um, uh, sorry, legal contract review. And so I think we're seeing very specific use cases, getting good results.

Um, and I think a lot of us use GPT to kinda finesse all those emails. We're like, okay, please make this so that I don't regret it tomorrow. So I think it's having a lot of great impact already. And we're just at the early days. 

Richie, just before you ask the next question, um, it looks like you got a Kleenex on your chair.

Oh, get rid of that. Oh, 

Richie Cotton: sorry. That, 

and then, uh, we can actually put your water bottle on the table beside you 

Manasi Vartak: there and you won't see it. 

Uh, we'll, we'll see it, but it just, as long as we face the label 

Manasi Vartak: away, if you just gimme a different one, then I won't need to stress about the label actually. Yeah. Yeah.

And if you want me to redo any of the questions, Richie, just, I'm kind of going with the flow, so I You sounded great. You're doing 

Speaker : great. 

Richie Cotton: Yeah. Uh, yeah, that was, that was very slick. I 

think this might be sparkling now. You okay with all good? 

Manasi Vartak: I just, I just, uh, make sure I don't start coughing halfway. I, yes, 

Speaker : please.

Richie.

You guys are doing great. The energy's good. 

Speaker : Mm. Okay.

Thank you. 

Okay, we're still rolling? Still rolling. Okay. Uh, when you're ready.

Richie Cotton: I think it's a valid point that maybe three years isn't that long in terms of enterprise cycles. Like there's some enterprises still running mainframes and so many businesses are, uh, still just built on top of Excel. So things take a, um, a long time to change. Mm-hmm. Um. Um, but can you talk me through some of the, the main challenges involved in adopting AI when you're in an enterprise?

Manasi Vartak: A hundred percent. Um, so there's a few different categories. I think, um, one, I might separate them as business and then. Technology and there's also a change management piece to it. Uh, from the business side. Um, there's, I think with all the experimentation we see folks leading with technology versus leading with a business problem.

And often that's a recipe for not being able to follow through. So it. Experimentation for the sake of experimentation versus I'm trying to reduce the time required to do this kind of processing or, um, I'm trying to expand our tam to target this market. So I think that's one of the challenges. How do we find the use cases that have business backing, that are actually the right use cases for gen ai?

So that's one. Technology wise, I think there's, uh, a couple of key areas. One is LMS already know the public intranet. There's nothing that you know you can teach them. Uh, there the enterprise data is where, um, is what's gonna make these LMS useful for enterprises. And there the challenge is how do I connect this LLM to my enterprise data in a way that preserves privacy and is safe so that you know that.

If there's a question about, oh, tell me the salary of my colleague, it's gonna say, no, no, you can't ask that kind of question, or, um, you only give the right users access to the right data. So I think it comes down to how do you make sure that the LMS have the access to the right data and it's permissioned in the right way.

So I'll highlight that one on the technology side. And LLMs are not like, uh, uh, sort of. Predictive software. If you ask it the same kind of question two times, it might give you a different answer. And so how do you evaluate, um, these AI based systems or agents is still very much an open, sort of open question, even research.

So that's where AI evals, um, uh, the, if you've heard about, is like, oh, how do I evaluate my systems to make sure that it's making the right decisions Still a big problem. And I think that's, uh. One of the areas that's limiting the adoption because you need to build trust in the system. Um, the last one I'll mention is change management, because imagine the first time that, um, employees started to use Google at work.

It, it's a muscle you need to develop is like, how do I ask Google the right questions to get me the right answers? How do I process the results, um, correctly? Similarly, how do I prompt, um, what data, how do I structure the prompt in a way that it gives me? Uh, the outputs in the way that I expect. Um, so I think that's a skill that employees currently need to learn.

So unless there's concerted effort to train employees to use ai, that's where we're finding that that's also bottleneck. 

Richie Cotton: Uh, last jump, unpack there and, uh, big, big list of problems. But, um, certainly the first one you mentioned where, uh, it's very easy to just dive into AI's gonna solve everything, let's you, let's go with the technology first and not think about what business problem you're trying to solve, uh, that seem to be, uh, a very common occurrence at the moment.

Um, but also the second thing, uh, you talked about how, um. If you wanna make use of your own private data, there's lots to consider that. Mm-hmm. Like, like how do you do that in a mm-hmm. In a safe way. Um, and there's certainly some privacy issues involved in that. So can you talk me through maybe like, what are the issues around.

What do you do if you're dealing with your own sensitive company data? 

Manasi Vartak: Yes. Um, and as you know, clutter is in the data and AI business, so we spend a lot of time thinking about it. Um, there's a few different layers to this. One is, um, is your data structured and, uh, properly? Is it architected in the right way?

This might be that certain, in certain geos your data needs to be on-prem, perhaps, um, in certain geos it can be on the cloud. Do you have the right data ingest mechanisms? Do you have the right processing? So the basic data architecture and how well it's set up, um, is gonna enable your AI to leverage that data much more effectively.

So that's like. That's just a foundation before you can, you know, go to the next level of ai. Um, another key piece is metadata and governance, and that's where I would put the permissioning piece in there. Um, what kinds of data sources do we have? Where did they come from? Um, do we have PII in there? If we are going to send that data to a model, do we have ways to mass.

The PII so that we don't leak some data inadvertently to, um, to the third party. So at Cloudera, we think about what we call private ai. Um, that means it is your data, it is your models on your hardware. It never goes off to a third party. And we found that to be a really good solution when you have super sensitive data that you're working with, um, that cannot say leave your VPC or premises.

And that gives. The most sort of stringent of enterprises comfort that AI is, you're not gonna leak your data to the ai. Um, so those are some of the things that we're seeing. Uh, the governance piece, the data architecture. And then the last one I'll also mention is AI governance, which is. Almost an umbrella term because you need to do your data governance.

Then you add on your AI governance, which might be things like what is the use case, uh, for this application of ai, what about the ethics of that? Um, are we producing results that are biased? Did this go through the risk management process? So all of these pieces build on top of each other, and I think enterprise are slowing, slowly, marching towards the future where that stack is well defined.

Yeah. 

Richie Cotton: Okay. So, uh, even more problems. I feel like we're generating more problems than we're solving. I know, I know. Uh, but one thing that I, I've never considered before you said that, uh, you might need different setups for different geographies. Mm-hmm. Can you talk me through why that is? 

Manasi Vartak: Oh, a hundred percent.

And we see this a lot. Um, so Cloudera works if you pick any vertical, I think the top five of five of the top or eight of the top in certain verticals are customers of Cloudera. And these are necessarily multinational companies. Um. So you might have a bank which has a presence in the us, the uk, um, and Asia.

Um, and then maybe they also have a presence in emea. Um, and so the Middle East, certain areas like apac, um, there are government regulations because of which. Say financial data cannot live in the cloud, and so it does need to live on-prem. If you go to a market like uh, the Middle East, all the processing that happens on customer data needs to happen there versus being shipped off to the us.

So soon you have an architecture where your data resides in different geographies. That's where your customers are. And so you wanna get your AI models to your data in each of those geographies. Um, and that's required by a lot of times by regulation, but also latency. Because if you are sitting in India and you're shipping your data to the US to get a prediction, it's gonna be a lot slower.

So it's also practical considerations there, but we're seeing that a lot and we think it's a growing trend. 

Richie Cotton: Okay. So, uh, it's really about dealing with, uh, different laws around the world, but also just about making sure that your application performs well, wherever the uses are in the world. User 

Manasi Vartak: experience.

Yeah. 

Richie Cotton: Okay. Nice. Uh, alright. And, uh, so one of the other things you mentioned was, uh, the idea of, um, AI governance around this. Mm-hmm. So we've had, uh, data governance for an awful long time. I'm not gonna say it's a solved problem, but it's at least a little bit more mature. So, um, are the rules around AI privacy, AI governance, is that, is it the same issues as with.

Data or are there new things involved there as well? 

Manasi Vartak: Um, I think data is a part of AI governance, but AI governance is a much bigger sort of umbrella. Um, is so you, it does start with data. Where did the data come from? Uh, what was the lineage of that data? So if you think about the lawsuits that currently, you know, philanthropic and open AI are facing, they have to do with, what data did you use to train the model?

That's a data governance and lineage issue. Like if you could trace it down and say this model was trained on these data sources. It's an easy question. I think some of the challenge is the data goes through so many steps of pre-processing and post-processing, that by the time the model gets to it, it's hard to figure out where it came from.

And so for AI governance, we need to solve those kinds of fundamental issues around data governance and then layer on things like for this particular use case, suppose let's say HR use case. Um, should we be using AI chat bots? So it's an ethical consideration. Um, if we, uh, stick to the, uh, HR use case, is it?

Giving the same results to men and women, to people of color. So the ethics and the responsible AI pieces also go in there. Um, and then because LLMs, if you're not doing private ai, you send your data to a third party. The data privacy there is even more important is like what data is. Philanthropic or open AI getting from my most sensitive, you know, estates and then having governance on there.

So it very much, there's a data piece, there's the model piece. There's also risk management typically, so financial institutions have a fairly well, uh, defined risk management practice. And so the AI governance kind of needs to pull that along with it as well. 

Richie Cotton: Okay. So it sounds like, um. You really need a good sort of, uh, data governance in place if you're gonna have good AI governance.

It's a sort of a cornerstone of it. And then extra things on top of that. Yes. And we're just still adding to the list of, uh, challenges. I know, I know. 

Manasi Vartak: Now we know why this alien, % aren't quite making it. To production. 

Richie Cotton: Absolutely. Um, yeah. Okay. So let's, let's see if we can do better than the % success rate, uh, by the end of the, the show.

Um, alright, so, um, let's maybe talk a bit about the, the technology involved and if mm-hmm. If you think, okay, I need to do pri, private ai, um, what does the tech setup look like? It 

Manasi Vartak: makes sense. Um, so maybe to recap, like private ai, when we think about it at Cloudera, it has three components. It has your models, which means typically open source models or models that you can fully control because then you can fine tune them for your needs.

Um, you can also distill them, like make them smaller. As needed. So your models, then it is your data, um, again to going back to the data piece. And then you are running it in an environment that you control. So this might be your data center, uh, where you control the hardware fully. It might be your cloud VPC as well.

And then when you have those three and you're running in a secure environment, we call that private ai, that's where you know exactly what the boundaries are for data sharing or potential data leakage that might come up. 

Richie Cotton: Okay. So, um, there's a few different layers in the stack there. So you're talking about having your own data center as well, like, uh, as, um, I guess the, the servers and then you've got the software on top, and so you're controlling everything there.

Do you, do you need to do that full stack all yourself, or are, are there other options where you're doing part of it yourself? 

Manasi Vartak: Ah, uh, you mean like in a customer, uh, scenario? Um. 

Richie Cotton: I think so. Yeah. So, um, um, do you have to, I mean, I guess not many organizations are gonna wanna build the whole data center themselves.

Are the, can you do some bits completely private and others, uh. Are there more public services or buy-in services? 

Manasi Vartak: I think so. So, um, a lot of, we work with enterprises that often do have their own data centers. Um, and so if you think about the biggest banks, all of them do have their own data centers. Um, if you only have very light workloads, should you be standing up a data center?

I think the answer is no. It's very expensive. Also a human labor intensive. You don't wanna be doing that. But, um, we actually did an interesting case study, um, of a bank in Asia Pacific where we, uh, in collaboration with them, we modeled out like, when does it make sense to have. Inference in particular. So like running the model, the models running on-prem and for them the sweet spot was when you go beyond say a hundred models, that's when it makes sense for you to be running it in your data centers for it to be cheaper.

Um, if you are at a smaller scale, the cloud might be a more flexible and faster option for you to get started with. Um, in that case, you don't need to manage the full stack. You're just spin up me, spin me up machines on AWS or GCP and there you go. 

Richie Cotton: Uh, okay. So the scale's like really important there. So small scale cloud providers gonna be, I guess, cheaper, 'cause quicker setup.

You don't need to build your own data center, but for large scale, once you're running things yourself, you're not just. Paying a fortune to all the cloud providers. 

Manasi Vartak: Yeah. Um, the caveat though is if you are a DOD or like a, in the area where you cannot use the cloud for some reason, then you're gonna have to be on-prem.

Even if you scale is smaller, then you might be working with data that's super sensitive. 

Richie Cotton: Okay. Alright. Uh, so yeah, you've really gotta think about like, what are the requirements about. What can I not do? Actually, maybe we'll go into this a bit further. Mm-hmm. Are there any things that you absolutely can't do if you need private, uh, ai.

Manasi Vartak: Um, I think you, so if you're in the private AI setting, um, the main thing is you don't wanna send your data out of your secure environment. And so you, these would be use cases or verticals where you cannot just send your data out to a third party. Um, so that's where you would not want to. Um, if your use case is that sensitive, then you wanna stick with private ai.

Versus sending it off to someone else. 

Richie Cotton: Okay. So this is like, uh, no sending things to like the open ai, andro, Google, whatever, APIs. Yes. It's like, uh, you run your own, uh, large language models, I guess, yourself. 

Manasi Vartak: Yes, yes, we do that. So, um, while Cloudera offers private ai, we also know that a lot of customers are gonna be on the cloud and not all use cases are that sensitive.

Right. Maybe you are rewriting a blog post. That's okay. Send it to whatever your, uh, favorite third party, uh, LLM provider is. That's all right. But if you're doing that with your legal contracts then, or customer information, you need to be a lot more careful. So hopefully that's good. 

Richie Cotton: Um, do you have a sense of which models people are using in those more sensitive cases?

Manasi Vartak: Um, a lot of open source stuff. So, um, OpenAI recently also open source. Their GPT model is not their latest and greatest ones, but it's a pretty good model. Uh, Google has their Gemma models out there. There is of course, deep sea that created a lot of waves. Um, IBM has their granite models. So open source is really catching up with proprietary models, and if you're going the open source route, I think right now you have a lot of different options that you can choose from.

Richie Cotton: Okay. Plenty of, uh, choices there. So just, uh, I guess try a few and see which one fits your use case. 

Manasi Vartak: Yeah, and um, if I can build on that for a minute, I think benchmarks are great to a certain extent. Um, but public benchmarks can be gamed. So you can train your model to be really good at a particular benchmark.

Um, but that doesn't mean that the model is actually the best one for your use case. And so what we recommend to folks is. Do an apples to apples comparison on your use case. It generally doesn't matter what the public benchmarks are saying because the benchmarks might not reflect your reality. 

Richie Cotton: That's really interesting that, um, that's every press release for every new model, it's like, wow, we've just, you know, beaten some math challenge or, uh, scored really high in the coding challenges.

But actually, yeah, you, you want to test against your own use cases. 'cause it might be nonsense, uh, in those cases. 

Manasi Vartak: Yes. Yes. And it's just, that's a good and bad about public benchmarks, right? They are public and so it's possible to game them. Um, if it's your use case, then uh, the likelihood of your data being in their sort of training set is lower.

And so that's a better example. 

Richie Cotton: Okay. Alright. Uh, so we talked a little bit about the technology side of things for private ai. Um, I'd like to talk about how it changes your workflow. So, um, are your, do your processes have to be different for when you're building with private ai? 

Manasi Vartak: Um, I would say process generally with AI is a little bit different.

Um, whether it's, uh, private or you're doing your, um, third party cloud stuff. Um, and maybe I'll pick an example of building an AI capable product. Um, suppose you're building a product that is a travel assistant, so I'm going from SF to New York and this assistant agent is gonna go off and it's gonna book flights for me.

Um. When you're writing deterministic code, like when we did like Rails app or whatever app that you're building, um, you could predict exactly what the outputs were gonna be or building test cases around it. Um, with ai, it might give you only United Flights one day because. God knows why, or it might give you United and Delta and JetBlue the next day.

So what you then need to do is to write test cases, evals or Yeah, evaluations that make sure that your results are always have the characteristics you want. Like I have all the airlines represented. Um, these are starting at the right airport and they have no layover. So this is a different kind of, how do you write test cases?

Um, but it's super critical to building a useful AI application. And the really kind of fascinating thing for builders right now is the model is, you know, let's call it % good today, in a year it's gonna be So do you build for the or do you build for the ? 'cause the boundary always keeps shifting.

And if you wanna build a business that is. Long term, you can't just be targeting the you need to be targeting the and the and solving the problems that will exist even as the models get better. So I think if, um, I've started a company in the past, um, and so if you're a founder right now, you really need to be thinking hard about, okay, am I just solving the problem for today?

Or will this be a problem three years from now? And I think that's just a very different mental model. 

Richie Cotton: Absolutely. When the underlying tech infrastructure is changing so fast, it's very easy to solve a problem that's like, oh, actually this has already been, uh, solved by everyone. No longer relevant. And then, uh, yeah, that's, I guess that's the point where you pivot your startup.

Um, okay. So, um, you, I've forgot what I was gonna say. Let's take it.

Alright. Um, yeah, sorry. Oh, sorry. Skills. Uh, okay. So, um, you were talking about how, uh, um, so, uh, we talked a bit about how the processes are changing. I'm curious as to whether you need different skills in order to build under a private setup or under a public setup. 

Manasi Vartak: Um, makes sense. Actually, I need a minute too.

Sorry. Um, so I, I feel like, um, I don't know if you wanna keep going the private and I'll, if you wanna Okay. You don't want, it's just, I think it'll, we might lose the audience a little bit if we keep talking about the private stuff. Okay. It might be broader like skill question. Yep. Alright. Alright, let's do that.

Yeah. 

Richie Cotton: Um, okay. Uh, alright, well maybe let's talk about the evaluation stuff then. Uh, so, um, you mentioned that, uh, it's quite difficult to evaluate the, uh, uh, the results of ai mm-hmm. When it's, um, particularly when it's not deterministic. So you can have different results every time. Is there some sort of benchmark you need to target then for like, oh, it has to be right % of the time?

Or like, how do you decide how good it needs to be and how do you make sure that happens? 

Manasi Vartak: Yeah, great question. Um. So with, usually how it goes is you start off a particular project, you have a set of evaluations or test cases, and it doesn't need to be thousands. Andrew Ang, um, you know, one of the godfathers of deep learning is like you need to start with a handful, which is what we found too.

And handful means at least start with. or while you're getting off the ground. So you can get a feel for where is the model and AI failing versus not soon you're going to, um, you're gonna give it better instructions and then it's going to get better on that test case. And then you're like, oh, it's working while here.

I need to add these different kinds of test cases in. Because now that's a new kind of say, um, data that you're seeing. So then your test set. Spans and then it expands even further. So ultimately you'll probably have thousands of evaluations, but you don't need to start, uh, you don't need a thousand to get started.

Those usually build very organically and over time. 

Richie Cotton: Okay. That's kind of good. In theory, like if you use it, can ask any question, you've got sort of infinite test area. So, um, you can just keep writing tests forever and ever, ever and never get there. So I like the idea that you can start with like or just to see does it work on the basic use cases and gradually build up from there.

Yes. Um, okay. Uh, so, uh, do you have a sense of like what skills you need in order to be good at building with ai? 

Manasi Vartak: Um, yeah. It's, uh, it's a great time to be working in ai, the skill's kind of, you know. The, you need to be on your toes to be working in this field. Um, I see a couple of trends. So if you are. A build.

If you're building with ai, let's call it building versus using. And I can also touch upon using a little bit, if you're building with ai, I found that the quick prototypers, um, great skill prototyping, 'cause you just need to get something out there, see if it works or not, and then iterate. Um, and you coding assistance also help you a lot with it, right?

Vibe, coding is the whole thing, is like you can say, all right. Spin me up a webpage that is, say the travel assistant example. Gimme a UI that lets me pick the star city and the destination city. You can do that in a couple minutes without writing a single line of code. And then the copilots are gonna do that for you, which is great.

And then you keep iterating from there. So I think, um, per typing is really key. User experience is getting really, really critical here. Um, because we as humans are trying to figure out how do I use. How do I effectively weave this intelligence into my workflow? So, um, you know, OpenAI has their canvas, um, UI to like for editing.

Um, anthropic has their own ones too. We're trying to figure out where do we, where's natural language, the right user experience, and when do we. Check boxes for things. Um, I'll give you a fun example. So I use GPT to keep track of my grocery shopping list. I'm like, so tell me what I need to buy from Costco.

Great. It tells me things, but then it doesn't let me check it off there. 'cause I'm like, no, no, don't tell me. I just, just show me the list and I'll check it off. So these are like small affordances that we're still trying to figure out. When do we use natural language? When do we use existing GUI capabilities?

And so if you're in HCI, human computer interaction, you're in ui ux, I think that's amazing and those skills are gonna be so critical. Um, so those two, and then maybe I'll just throw out, um, one on using ai, all of us need to be using ai and so getting comfortable, and actually you're kind of good at prompting so that you can tell it, oh, I want.

A, B, C, don't give me X, Y, Z. Um, I think that's gonna be a base level expectations regardless of job function. Um, and that's where we're seeing a lot of the change management that needs to be done in companies is educating the broader workforce on here's how you use a Gemini model in your workflow.

Here's how you use a GPT model. And we're seeing a lot of benefits from that. 

Richie Cotton: It's a great point on user interface there. 'cause I think. Uh, there was a point after chat GPT came out. Uh, everyone was like, oh, this is the final interface. Everything's just gonna be a chat bot from now on. Yeah, and actually being able to like just point and click on stuff is still really quite useful a lot of the time.

Uh, yeah, I, I like just having check boxes and radio buttons and all the rest of it. 

Manasi Vartak: Right. 

Richie Cotton: Um, so, uh, yeah. Good stuff there. Um, so, uh, you, you mentioned, uh, the idea that prototyping is a very important skill. Mm-hmm. Um, do you want the same person to be doing the prototyping as building stuff long term?

Because I guess at some point it has to go beyond the prototype into something that's robust. Yeah. And they seem like different skill sets. 

Manasi Vartak: Um, I was gonna add in the fourth one there when I was listing out the three and I was like, you're gonna, um, maybe I'll. So a hundred percent. I think you need folks who are going to build critical production systems or even productionize the prototyping.

Um, I think about it as a distribution. So if you think about. Um, a normal distribution. Your x axis is like, how deep are you in technology? Um, right now you might assume it's a normal distribution. A bunch of people are in the middle. There are a few people who are not super technical, and then there's some people who are deep tech With ai, I feel like the middle is getting flattened out.

You either have. Deeply technical people who are going to need to understand the ins of these systems to build them at scale, um, and to make sure the AI is working correctly. And then you have SMEs or like business experts who can get by with quickly building workflows, prototypes. And so I think we're gonna see that bifurcation a lot.

So it becomes a bimodal like super technical or super business because the super business are gonna have. The AI capabilities anyway. Um, so it's gonna be fascinating how the, how skills and expected skills change. 

Richie Cotton: That's very cool. Somehow, even though we're a data show, we don't often talk about bimodal distributions on the show.

I love that. That just dropped up naturally. Uh, that's wonderful stuff. Uh, so you mentioned that there's gonna be, um, importance of an interaction between subject matter experts. And, uh, people who are building with ai. So, uh, this strikes me as very similar to the way, uh, I guess historically we've had, um, the same interaction between subject matter experts and data teams.

I'm curious like, uh, 'cause things like machine learning AI have traditionally been a data team thing than I'm moving a lot more towards software development teams. Do you see the relationship between software and data changing as well? 

Manasi Vartak: Um. I do, I do. Um, and we were having this conversation at the conference actually on like, what happens to this, uh, the business analyst persona?

Do they need to become more technical or do they need to become more business? Um, and I think. I do lean back to the same sort of trend that I'm seeing where I have friends who are chemists, they've never written a line of code in their lives. Um, they can build workflows now using, um, open AI or, you know, Cloudera launched this thing called Agent Studio, which is natural language build your agents.

And we've seen folks who have never coded. Do this. And so the SME who historically used to come to the data team to get information now can do that a lot or should be able to do that a lot on their own. Hopefully freeing up the data team to work on deeper technical challenges about how do we better architect things, how do we optimize how we're running our queries?

And so I think we will see. More specialization in those areas. Um, and I think hopefully much more, many more opportunities that way. 

Richie Cotton: Okay. So the, the role of the data team then is gonna really, really just, I guess, enabling the rest of the business to use data more easily rather than actually going and solving the.

The analytical problems themselves? 

Manasi Vartak: Yeah. I think like we get to a more self-service model, which is where I think the movement has been as like data visualization is more self-service, analytics is more self-service. This helps us get to that feature. 'cause usually I feel like most data teams are like, no, no, I got too much stuff to do.

Please ask me another quarter. This way they can focus on fewer things that are gonna be fundamental to their data architecture. 

Richie Cotton: Okay. Alright. Um, and I'd love to talk a bit about how the, uh, how the role of the data team is changing with, uh, better ai. Mm-hmm. Because I know, uh, at Cloudera you've got a lot of assistance for helping out with, uh, doing data tasks.

Do you wanna talk me through like, um, which bits of, um, analytics are most amenable to AI assistance? 

Manasi Vartak: Yes. Um, so I think the future that we're going towards is the data platform has AI embedded in it. Everywhere that it makes sense to do so. Um, for example, later today, um, we're doing a demo of a data classification.

Agent that we've built. And that's, um, that goes to the metadata collection, the governance collection. This activity might have been, uh, on the data team plus SMEs. Now you can build an agent that's going to automate it, making the data teams life a lot easier. Um, there are also ways to, uh, going off of the lineage bits is how can we look at the data in the tables?

Figure out how do we document this data? 'cause usually, you know, the typical thing is you have five columns with the same column name and then you have no idea what they are. So those are places where AI can start providing some insights organically. Um, so this might come from, this data was written to, from this particular script, so it, um, you can add some semantic meaning to it.

And Octopi, which was a recent acquisition by Cloudera, their whole. Product was around building data lineage. Um, and so they can extract lineage from scripts and you can start enriching your data sources without human intervention. Which I think is awesome. Um, the, another thing I'll throw out there is also you can optimize your data systems, uh, using ai.

Now, there was, there has been previous research on how do you build indexes, um, as you go so that you don't need to predefine them. I think we're gonna continue to see those kinds of innovations. Um, every, I think if someone's trying to solve a hard problem. See if ML can do it. It doesn't have to be Gen ai.

See if ML can do it, see if AI can do it. And I think the answer is gonna be maybe %. You can and that's a great win. 

Richie Cotton: Okay. I do like the idea of just, um, see if technology's gonna solve your problem for you and some of the use cases you mentioned, like metadata collection, documenting data sets, these are not fantastic for humans.

So I think yeah, they're sort of prime candidates for being outsourced, uh, to, to technology. Um. So, uh, you talked a bit about data lineage. You just want to explain a bit about like, what is data engine? Why is it important? 

Manasi Vartak: A hundred percent. Um, so if you think about, uh, let's start from the end product.

Maybe it's a visualization, you know, it's a bar chart somewhere. Um, if you trace it back, the bar chart came from. Some table that was maybe joined two tables that were joined. Now those tables might have come from a data cleaning process from yet another table. Um, maybe that came in from a streaming system.

Uh, maybe there was another kind of function that was applied in the streaming system. So what Lineage does is it starts from where the data first entered your system and then traces it through all of the transformations to the end product that you're tracking. And so that gives you. A view of who made what changes and answers questions like, oh, that's odd.

I'm seeing this odd result. Where might that have cropped up? Or if you're in a governance setting, I have, how did this PII thing make its way to the visualization when we shouldn't? So it's kind of critical to both, um, making your governance and metadata work, but also operational things. My pipeline is broken.

Okay. Your lineage tells you what are the places you should go check. 

Richie Cotton: Okay. Yeah. So I think it's a very common problem when like some executive looks at the dashboard and like the numbers are wrong here, and why, why they're wrong on the data team, like. Not really sure here. And uh, it turns out it was like last week's data or something like that.

So, yeah. Uh, I can certainly see how being able to track where the data's come from, uh, it's gonna give you those quick answers for when, uh, you just start complaining about dodgy data, 

Manasi Vartak: you know, building off of that, if you had any, uh, assistant, you're like, Hey, tell me why this doesn't look, I expect this to be and it's a hundred.

If you have the lineage in principle automation could go back and compare to this is what it looked like last week. This is what it's looking like this week. I think this might be the issue. We're a little ways away from that, but I think we are with more product building, we can get there. 

Richie Cotton: Okay. That seems like a useful feature for the future at least.

Yeah. Yes. Okay. Uh, so, um, you, you mentioned the idea of an agent there. Um, I'd to talk. When do you want, um, like a, a full AI agent versus an assistant versus a co-pilot? I mean, there's all sort of different bits of terminology and they, they blur a bit, but I know. Talk us through, when do you want which? 

Manasi Vartak: Um, so agenda AI is really loosely defined right now.

And, um, again, going back to Andrew Ang, he actually coined the phrase 'cause he found that a lot of the community was. Uh, squabbling over, is this an agent or is this not an agent? And he is like, okay, that's not useful. Let's just call it agenda AI and let's move on. Um, and I kind of see it that way, is.

There are different degrees of automation built in, built into our processes. Sometimes the automation is just one and done is I ask something, a question, it gives me an answer. Okay, great. Um, sometimes it needs to do deeper thinking or reasoning. It might need to use tools. Um, and that's been a big unlock, right?

MCP is like, how do we standardize how LLMs call tools? Um, so whether, so that's on the other extreme where. Think about the international math Olympia problems. They need very deep reasoning. Um, and so that's the other extreme of agents which actually do reasoning, but it's a spectrum. And depending on, depending on how well defined your problem is and how hard it is, you might be better served with a deterministic workflow.

So think about, um, deep research, for instance. You, it's fairly deterministic. I need to get. I'm trying to do research on, say, AI evaluations. It's gonna go off, look at something sources on the web. It's going to collect that data, it's gonna summarize it, it's gonna get it back. That is a fairly, I would say, linear.

Um, but then if you wanna get to a autonomous agent for coding. The, uh, Devon was supposed to be, that is much more iterative and a lot harder. And so that's where, depending on what you're trying to solve, you might be well served with a really simple solution, and that's okay. You don't need to go with the, um, you don't need to get a sledgehammer if you're just trying to put like a nail on the wall.

So making sure you have the right complexity of solution for your problem, I think is how I think about it at least. 

Richie Cotton: Okay. Yeah, so, um, there's definitely a big variation between I'm just doing some sort of simple like, um, software workflow and there's a bit of LLM magic sprinkled in there versus I'm trying to like, well you mentioned debit, so I'm trying to create an entire software engineer, uh, using ai.

Um, okay. Um, so are there any tests you think, um, should still be done by humans then? Um, shouldn't be outsourced to AI at all. 

Manasi Vartak: I think it's critical to have human oversight over all of these processes, and it doesn't need to be real time necessarily, but while we build, continue to build trust within these systems, you do want a human reviewing the results.

Um. So for example, suppose you were picking off the chemistry pipeline example, you might get a molecule, you figure out its configurations. You do some testing and you see the results. You still want a chemist to go and look at either surprising results or some subset of results to see, okay, it's doing what I expected it to.

'cause strange things will crop up where maybe the units are off. In one of the data file that you uploaded and then everything looks odd. So those are the kinds of things where having a human in the loop I think is critical. A lot of these systems are not set it and forget it yet. So even if it's an audit after the fact, um, I think it's pretty crucial.

And as with all of automation is like what? If we get this wrong, what happens? Um, and if the cost of getting it wrong is really high, you better have a human like handholding the thing. Um, if the cost is like, Hey, yeah, I might need to redo this analysis quickly, that's okay. Maybe they can do it offline.

Richie Cotton: Okay. Yes. You've really gotta think about what are the consequences of things are going wrong, and I guess is the human actually gonna be good at solving the problem? Yes. Right. Okay. Uh, alright, super. Uh, so just to wrap up mm-hmm. Um, is there anything you're particularly excited about in the world of AI at the moment?

Manasi Vartak: Every day, there's so many things going on. Um, I do think that agents, um, are, are here to stay. There's a lot of work to be done there. Another thing is how do we go from vibe coding to production? What we kind of touched upon is like, are they different people and how do we. Help more vibe, coders also transition to writing real production applications.

I'm excited about that. Um, we have seen general purpose models with GPT, but then having a model for healthcare, having a model for drug discovery, I think that is going to be another game changer. Um, and so I'm excited about all of these and again, getting, nailing the human interface, like how do we build the AI so that it.

Makes the human's life easier, versus there's like a lot of slop is what it's called, like generated by ai. How do we not increase that? How do we still retain human attention for the areas that are critical? I think those are broadly the things that I'm interested, excited about. 

Richie Cotton: Okay. Uh, so that last one I'm gonna pick up on, um, AI slop.

It's everywhere. I don't think we're, we're gonna stem the tide anyway, but, uh, do you have any advice on like how to deal with all that? 

Manasi Vartak: Uh, I, I, couple of things. Um, one is the authority of the person. Writing is going to matter a lot more. 'cause if I trust someone, be. Historically, or over time they've built that trust, then I'm going to trust their content more.

Um, and with ai, when everyone can use GPT, I know that this person has really interesting ideas and so I'm going to, um, hear what they have to say. So the authority and. Human relationships actually, I think matter more in this era. Um, and there was this really interesting model by one of the VCs out there actually from Theory Ventures, um, and his thesis was, in the future you just ask an assistant a question.

The assistant goes and almost gets. Quotes. You know, if you think about an airline or an agent trying to book something, historically, they were go and get quotes from airlines and then surface them. So the future that he was talking about, which resonated with me is like, if I wanna get a question answered, is like, how do I do AI evals?

It should run a bidding, so to speak, on like, who is the experts? Go get me the expert opinions and just get me the results. Um, and that's a really different mindset. And the question is, what happens to. The worldwide web and the internet, like what happens to the blogs out there. So it's a very different worldview and it's gonna be fascinating to see how that evolves.

Richie Cotton: Absolute like a marketplace for quotes and ideas. That's kind of interesting. Yeah. 

Manasi Vartak: Expert marketplace, right? That's what we're kind of been trying to go towards. And this just, it's on steroids, so to speak. 

Richie Cotton: Alright, wonderful. Yeah. Uh, an exciting idea for the future, I guess a starter idea for, uh, anyone who's, uh, wanted to build things.

Yes, yes. Uh, wonderful. Alright. Uh, so just finally, I always want, uh, new people to follow. Uh, so whose work are you most excited about at the moment? 

Manasi Vartak: Um, I'm plugged into research more than others because I came from academia. Um, there's two folks, um. We're doing great work on the human computer sort of interaction piece.

Um, currently both at Berkeley, uh, one is Reya Shankar. Uh, she's done great work on Doc Wrangler. Um, how do you get a human in a loop and help them educate the AI on how to clean data? Um, so that's awesome. And a me is her collaborator. Uh, there, he's the pi. They've both done awesome work and I think they'll continue to.

Richie Cotton: Uh, that sounds very cool. Just figure out like, where do you actually want the human, where don't you, uh, yeah. Very uh, topical question for our time. Alright, wonderful. Uh, thank you so much for your time, Manasi. 

Manasi Vartak: This was great. Thank you so much for having me.