AI Agents in Your Systems: Speed, Security, and New Access Risks with Jeremy Epling, CPO at Vanta

Richie Cotton: Hi Jeremy, welcome to the show. 

Jeremy Epling: Hey, thanks for having me. Excited to be here. 

Richie Cotton: Yeah, great to have you here. So to be with, I'd just like to know what are the biggest security issues that they're facing businesses at the moment? 

Jeremy Epling: Yeah, I mean, I think probably no one will be surprised to hear AI is at the top of the list.

You know, a lot of companies are adopting ai. They're seeing a ton of benefit. They're doing it really, really quickly, and there are still a lot of questions that kind of come up. Like every product is adding AI capabilities and then security teams start to wonder, well. What are they doing with all this data that we're giving them and our customer data?

Do they have access to our customer data? Are they training models on it? What are they doing with it? There's all these exciting new technologies like MCP servers that allow agents to talk to other agents on behalf of users, and we're still in the early stages of understanding how to secure those. I think there's also a lot of AI security threats like philanthropic mentioned, a recent nation state attack where cloud code was used a couple months ago.

And so I think there is a lot that's surrounding security teams. And we actually recently did a survey theta state of trust report and asked a bunch of CISOs and security professionals. How they were thinking about AI and while like % of them are using Agentic ai, I think only % of them said they have the compliance tools to actually go support it, and over half felt like the AI innovation is completely outpacing their ability to keep up with all the new security threats that are coming.

Richie Cotton: It's one of those things where it's brilliant that there's so much progress going, like with ai, like all the time there's something new coming out, but then trying to keep up with it on the security side, this. Kind of the gap between compliance and security and the actual like adoption. But that's a little bit terrifying, right?

Jeremy Epling: Oh, yeah, definitely. I mean, I think that I mean the good news is I think with any of these security things, there's always this cat and mouse game. You know what I mean? Between like, the attackers have the same technology that the kind of defenders and good guys do, and it's a matter of like who's ahead there.

I think that the good news is with like, you know, products like Vanta and others like. You can then give even more security information to your security tools and to those agents to help defend you faster and better. 'cause they should definitely know more than what the attackers can see from the outside.

So I think there's a ton of benefit. I think the defensive side has a bunch of tools for them and like a lot of exciting new ways to go defend against it. But it definitely feels like there's a brand new wave of attacks that are coming and like the internet will become like a more dangerous place before it gets back to like a more safer place against Sid.

Richie Cotton: So I guess there's so many like new kind of threats and, and problems going. Like where do you begin? Like where do you start to short your defenses? 

Jeremy Epling: Yeah, I mean, I think that a lot of it is a lot of the similar security practices that security teams have been doing for a long time. I think they've just been super charged.

Like they're always dealing with threats and attackers coming in. It's just now those attackers can produce more sophisticated threats. Do it in a more distributed way, even faster than have before. And you have like kind of novice, wannabe hackers, if you will, who now have pretty advanced coding skills in comparison to what they had before.

They didn't have to go build these up over time like Claude Code and Codex, as much as they're amazing for us, improving the quality of our products and shipping products faster, they're also helping attackers write code that they never could before in more advanced ways. I think security teams are kind of dealing with like all those threats and prioritizing what those are.

You know, leveraging endpoint defense, EDR solutions, things like that are still incredibly important. I think data security is more important than it's ever been before as well. Like the thing people are trying to go after, the crown jewels are usually always some dating your company. It could be your trade secrets, it could be secret recipe, your formula for a new like pharmaceutical medicine or something like that.

And so. Being able to protect that, I think has become a lot harder. 'cause not only do you have these attackers coming in from the outside attacking in new novel ways, but you also now have all the internal tools that your employees are using at your company. Are all pulling on all of this data and sending it to models to ask questions around how do they develop that next breakthrough drug?

How do they go through and, you know, go ahead and get an edge against this competitor or, you know, build this new piece of like novel tech. And so. I think there is this new set of like man data leaking kind of out of the company and the company's trade secrets is a lot more top of mind. So they're starting to ask a lot harder questions of their vendors, of like, Hey, when we're sending this data to you in your AI models, like what are you actually doing with this?

Can this be reproduced somewhere else? Like, what is happening? So I think those are some of the top threats that are coming up and, and then they want that proof back to be like, okay, great. You, you said you do this. How can I trust it via an audit report or something else. 

Richie Cotton: That's really fascinating. I mean, I think if someone's attacking you and seeing things, then that's one thing, but if it's a kind of an unforced era where you are actively leaking your own trade secrets in the world, that's a real, like, that's a proper own goal.

That's, that's a really terrible thing. So. In terms of stopping this, does it stop the security teams or do it need to be like a C-suite that do this? Like who needs to solve this problem? 

Jeremy Epling: Yeah, I think it's a, a company-wide effort given how everyone in the company is using these tools. I think a lot of it starts with data governance.

I think a lot of companies are pretty lax on data governance and probably have been for a while. 'cause they're like, oh, who's gonna have the skills to go, you know, mine in this. SQL database or this Mongo database or whatever else to like pull information out. But now all of these vendors and tools just reach in and integrate with everything and they're trying to pull out these insights.

Even if you have someone that is not malicious internally, they might be sending data that they shouldn't, like they pulled a bit of customer data 'cause it's so easy to go pull that and they shared it with another model. And your policies for your company say, Hey, we don't share customer data like this.

And now it's in someone else's model. And maybe they're training on that and they can get reproduced in some other way. So I think this isn't just kind of nefarious people from the outside or nefarious people on the inside. People just don't realize how much data AI is constantly scraping up and how much these integrations are kind of pulling in all these data sources.

I think there also is a little bit of making sure people are outsourcing just good human judgment. You know, there's a lot of things AI is great at. It also makes mistakes. I think everyone knows about hallucination and other problems we've had and they get like bigger companies with all this pressure to move faster.

I think a lot of employees are just looking for that quick answer and sometimes that answer isn't there. So one of the things we do internally with all of our data systems is our actual. Like product analytics team sets on a lot of our channels where we do have AI tools. So when people are coming in and asking questions, even if an AI tool gives the default response, we're kind of doing our internal evaluations on those responses and being like, oh, yep, this one's good.

This one's good, this one's good. Oh, actually, that pulled the wrong data from like an old data source that actually isn't trustworthy or an old table that we forgot to go, like drop and delete at some point and like, you shouldn't trust this. So. I do think there's people that are just doing the right thing and they have access to all these things they didn't before, and they'll just make mistakes because the AI gets it wrong.

And so internally, making sure all of your functions are thoughtful of like, especially if a piece of data smells wrong, go talk to an analyst or someone else or like do that. Digging yourself to be like, okay, is this exactly right or not? 

Richie Cotton: I love that idea of having analysts just check the, like, the quality of the output that I always, it needs to be someone who understands what the problem is, just to be able to say, is this right?

Is this wrong? And that's where you're gonna the feedback loop to, to fix things. 

Jeremy Epling: Yeah. So, and I think that like keeping those clean data sources is even more important than ever and having like a data engineering and analyst team that says like, these are the authoritative things we trust in them, owning that pipeline and hooking it all up.

I, I've worked at a bunch of different companies that I can say like there's a lot of old kind of smelly data sitting around everywhere that's very easy to kind of like pull up if you don't know what's good and what's not. And so regular hygiene, I think is probably even more important than it was before.

Richie Cotton: That's an amazing analogy out of the, of like old smelly data. That's funny stuff. So in terms of managing all this stuff, you said like managing integrations is kind of a problem. Like is there a process fix this or are the tools to help you deal with it? Like how do you go about solving making sure that.

The right integrations have the right access to the right data. 

Jeremy Epling: Yeah, I mean I think this is a big thing where the security team helps a lot, like pretty much at every company through, as they go through the procurement process. We've got someone who wants to go buy a tool, they're super excited, the security team comes in.

Sometimes they're like the no people, but they're doing like a really important job for the company that are just like, Hey, this tool seems exciting. It probably solves your problem, but like, can we actually trust this company with. Their data. And so I think that is when the security team really comes in and starts asking those deeper questions.

You know, there's different compliance frameworks that are out there. You know, just like if an enterprise is buying a piece of software, they'll say like, Hey, do you have SOC two? Do you have iso, you know, ? If it's like the federal government, they'll ask about like FedRAMP or CMNC or these different standards.

There's the same thing for AI as well. ISO you know, the nist AI RMF, the EU AI Act. There's like different ways to go and say, Hey, here's a set of compliance frameworks and controls that we support. And then you can share those, that evidence out. And I think that really speeds that process along so that the security team that wants to go buy a tool and is evaluating it can get all that information really quickly.

They can go through it, they can ask detailed questions and try to drive that procurement process through really quickly. Then I think afterward, then the next hard part comes in. It's like now you bought the tool, you trust the company, you think they're gonna handle your data rate, that's great. But now internally, does everyone have the right access?

And how is that going forward? And how are they using it? What data are they putting in or out? And I think there's a lot of new tools that are allowing people to kind of monitor those data flows, to understand what is getting put in, to kind of set off like alarm bells if things seem off. But I do think in general, everyone's seeing such a productivity benefit.

It is a little bit more like fast and loose now because they want that benefit of ai, but at the same time, we need to be like thoughtful of like not leaking customer data and breaking promises that companies have made in the past. 

Richie Cotton: Definitely. And it sounds like a lot of this stuff is about managing who has specific access to what things, all these kind of rules.

It sounds like this kind of the, the tedious manual task that ought to be automatable with ai, is that something that AI can help with? 

Jeremy Epling: Yeah, definitely. I mean, I think that there's a case where, you know, you can look through different people, you can understand, like internally what we do at Vanta, we use.

Banta to go do this. We go through and look at everyone's title. We understand where they are in the organization. We can see what other applications, other people in this job function have done in the past. And you could kind of build a rough profile around like, oh, engineers on this team usually need access to these things, and this thing that they're asking feels like an anomaly and like, maybe that's something we should go look at.

So I kind of think of it as like anomaly detection and kind of giving. I still think there's a really important human in the loop and I think. Security teams especially will be, I think one of the last ones to kind of give full control over to AI to remediate and make all the decisions. So I always think about designing as like review and approve flows.

Like as someone who sells the security teams, where it's like we give you all the context that you need so you can quickly look through it and put in like your expert judgment and then decide like yes or no. And how do we tee it up for that like. Super fast, easy decision versus, hey, you've gotta go access different tools, read all these docs, talk to all these other people like, and you know, now something that took you five or six hours to go do.

Now it's all teed up for you to go like quickly, read through, make a decision, and then move forward. 

Richie Cotton: I love the idea of just enabling humans to make good decisions faster, faster. But I suppose you mentioned that security teams, they're probably the last people that want to turn over full control to ai.

I suppose there's, a natural tendency for security people to be a bit paranoid, that ought to be kind of part of their job function. How do you build paranoia into your processes then? 

Jeremy Epling: Yeah, I mean, I think a lot of it comes down to like trust and quality. So I. You know, one of the things when I think about designing AI features is how much accuracy and the quality of the answer is important.

It's, you know, it's kind of like we've experienced for years, like pre AI with like Google search results. It's like there are plenty of other search engines, there's plenty of other companies that spent like billions of dollars to go make them great and to do it all. But like the quality of like those top links was the most important thing.

They're really differentiator for a lot of these AI products and some might have more interesting UI experiences and things in the future, but I think is the quality of the answer you get. So when I think about designing for a security team, I think about a high, like it is good to refuse to answer questions if you don't have high confidence.

So that is something that we have definitely engineered into the Banta AI agent, where when we go through, we look through and say, Hey, if we don't feel like we have a high quality answer, we'd rather just say like, we don't have an answer to this. Here's a couple things you can go look at. Or, Hey, we're gonna go work on this and we hope to have an answer for you in like one of our future updates.

And I think that just builds trust with security teams that are already more skeptical and. Some of the other things that I think a lot of AI products do like showing citations. So like where specifically in references, Hey, this came from these documents. It came from like this specific paragraph or sentence here.

We pulled it out, let them build confidence in it. We have a lot of flows. Like I mentioned before, when you're in these procurement workflows, you know, you have a software buyer and a software seller, and they're trying to exchange information. I think one of the things we look at a lot in our ai, it says what the software, you know, seller say in their, in their answers.

Actually match the documents they provided you as the evidence and obvious. Sometimes it doesn't, and sometimes that's just like a human error. They forgot to update some policy or the person misspoke or didn't know. But flagging those things really quickly I think helps build confidence. So. I think just being really focusing on high quality answers that people can depend on over time.

Higher than like being able to answer everything. So I think if like, coverage rates are really important, but accuracy is even more important and I'd rather not drop my accuracy bar to have more coverage. So it's like, what can we go do to keep accuracy high while kind of like working up our coverage.

Richie Cotton: Absolutely. I mean, there's no point in getting the wrong answer really quickly. So I, I love the idea of like, yeah, you focus on quality, make sure you get the answer right. And citations is like. A surprisingly hard challenge with a lot, particularly the general ai. It ought to be an easy thing for you think naively that if it finds some information, it should give you the right citation, but often yeah, it's, it's total nonsense.

Now I, I know you, you mentioned TER agent, so this is like a, an internal agent that your team's been working on. Talk me through like what, what does TER agent do and why did you build it? 

Jeremy Epling: Yeah, I mean, I think for us, we just see, especially in the space of like trusts and GRC. A lot of different security professionals have to do a lot of manual work and a lot of it is very document heavy.

So Vanta kind of started off with a ton of integrations. Then we tried to automate everything we could. Like, Hey, you've got your you source control provider like a GitHub or a GitLab or a Bitbucket or someone like that got AWS or Azure GCP, Oracle, or someone like that. You can automatically pull that down.

But still a lot of these processes like SOC two and ISO still require you to write a a lot of policy documents. You're sending a bunch of documents and evidence. You're still getting screenshots because you know you can't get API requests for all these tools. So sometimes you need to have a screenshot and go do that.

I think AI really unlocked us being able to solve a bunch of these problems we never could before. So when a buyer and a seller are exchanging this big security questionnaire, like if you're a software company and you sell to like a big bank, you're gonna get a questionnaire that could be a four or security questions long.

And you know, you start selling to a lot of enterprise customers, you're getting these all the time and it's kind of this mind numbing task to go through. And they're all awarded slightly differently. They're asking kind of the same things and it's not like high value security work, but it's like it's very important that you do it and you give it accurate.

But like. The repetitiveness isn't there. So that was one of the areas we tackled with our agent first, was this buyer seller relationship of automatically answering security questionnaires. You can build up a knowledge base that's curated. We tell you when, that, hey, you're getting new questions in that you don't have coverage for.

And then we can suggest things like, Hey, do you want a new AI policy around this? Or All of your customers are asking you to redline your SLA to something different. Maybe you wanna change your SLA and maybe you want to go do these. And thinking about all that customer feedback coming in. The same thing for like, if you're on the buying side, the vendors, you know, you're receiving all these questionnaires getting there.

And then I think as security teams go through, you know, they often just see drift in their policy stack. So they, you know, maybe they want to go update an SLA, maybe someone leaves the company and then they want to go back through and connect that through and say like, oh, great, our CISO left. We should go update all the documents.

Mantic can just update, detect and do that all for you. You want a holistic risk program when you're thinking about things. So like you have these high level business risks, these all mapped to things that people aren't familiar to, like frameworks and controls, and then you have individual tests and different security actions everyone's taking, and you wanna be able to see that whole program together and ask, maybe we do want to go for ISO to prove the quality of our AI experience and security.

You can just ask the agent like, what do I need to go do? And then we'll look through everything and be like, Hey, you should add these security controls. I would update these policies. You should think about this. And we kind of guide you through that process. So I think of it as, you know, along the lines like we refer to as your GRC engineer.

It's kind of someone that's sitting alongside you the whole time. Trying to go do a bunch of the work for you so you can kind of stay in this like review and approve workflow where like you're bringing your expertise to evaluate key decisions and then kind of moving to the next decision and not having to do as much of the kind of manual, repetitive, you know, kind of grunt work, if you will.

That was like common with, you know, certain GRC teams for a long time. 

Richie Cotton: Certainly filling out like a page questionnaire about like what do you do on security across your whole business for some enterprise customer, then doing almost the same thing, but not quite the same thing for your next customer.

That sounds like a really, really tedious job. So I love the idea of using AI just to complete it. Well. Outsource as much of that as possible. So I guess talk me through, like how did you approach building this then? Like, what do you do to make sure that you do have these high quality answers?

And I guess if it's if you're filling out a page survey, that's gonna be like a big customer that's gonna be like a, a, a big account you're trying to close. So I guess there's a lot of money on the line. How do you make sure you get that high quality answer? Like, what did you do to, to build something that gets the right answer.

Jeremy Epling: I think a lot of it was you know, one probably things that people have heard about or talked about in the past is like having a really good golden data set. So for us it was seeing the questionnaires that we've gotten in the past for customers that wanted to share more of that data with us. And then we have a really big team here of GRC, subject matter experts, GRC being governance, risk, and compliance is that function that deals with a lot of the stuff that Vanta does as a company within the security team.

And so those ses, as we call them, the subject matter experts, are all former GRC professionals at large companies. Former auditors, you know, have been auditors at like big firms or focused on authorization packages for the government. And we have them as part of our evaluation or like eval process. So we go through, we build a, a golden data set.

Then we go through and just continue to kind of like tune the prompts and the models to the point to have high quality answers. And then like we talked about before, also pulling in those like citations and quotes and making sure we're getting the right ones to kind of build that trust. So it's basically an always on never ending function.

I think that's one of the big differences when you think about building products now. And in the past, like ML products were like pretty similar to this, but I would say. Now that AI's become so ubiquitous, I think there's kind of standard deterministic SaaS where it's like, Hey, I built the thing and I tested it and I said, one comes in and two comes out the other side.

And it's always gonna be that way no matter what. But with ai, the evals are like always changing because it is probabilistic and not deterministic because the models are always updating because your customers are always pushing and asking to go do more. So I think about when we start a new AI feature, there's a part of the team that's just dedicated a decent side to quality for.

You know what I mean? Because like quality is the feature and it's not just like the new capabilities we add when we thought about things like questionnaires too. There's all sorts of different formats. So then you have to think about the parsing side. Like did I get a Word document? Did I get a Google document?

Did I get a spreadsheet? Are there conditional questions? Is the way that they did the formatting in this like word document, like different than the way they've done in the past. Some people use portals. You go to like a portal to go answer the questionnaire. So now we're like scraping the dom in the browser and building a browser extension.

I think there's a lot of interesting AI optimization work, and there's different tools and platforms you can use for parts of this to like. Just get the questions in a, you know, a consistent way and then going through and answering them with high quality. And then the ones that you don't know, we then leverage of like, oh, who's answered questions like this in the past?

Let's go ping them on Slack and assign them out. So like, you know, our lawyer should probably go answer this question. He's probably not spending all day sitting around Vanta but, you know, as part of his job. And so let's go to like where he's working so we can ping him in Slack and be like, Hey, this question came up.

For this, you know, big enterprise customer, can you just give us the answer and they can respond back in Slack. The agent grabs that and goes ahead and puts it in there and says like, okay, great. Like Ian signed off on this. He said this was the answer. We're good. You know, goes off into the, you know, into the response to the customer and we can go close the deal.

So a few different ways. I would say AI kind of plugs in at different points from pulling all the data in to answering the questions and the citations to like. And for the things we can't answer, helping people like get those answers in the right way. 

Richie Cotton: It's kind of cool that using AI for like the, all the sort of pipelines to like build the, the rest of the AI as well as like actually using the AI in the end state.

So yeah, you mentioned the evals and this seems to be like an. It's a bit of a tricky issue. Kelly. So you said 'cause it's probabilistic sometimes the, the feature's gonna give the right answer. Sometimes it may do something completely stupid. How do you, how do you test it? In that case? It's like running queries, like a asking questions the same question a thousand times and seeing how many times it gets it right.

Talk me through your testing approach. Yeah, definitely. Right now we use a tool called Lang Smith that like helps us run all these evals. A lot of the time product managers are doing it as well as these GRC subject matter experts. Sometimes engineers and designers are in there too, but they're the primary ones.

Jeremy Epling: And so yeah, you can go through and kind of see all the different questions that have kind of like come in for customers if they're sharing that data with us. And then we can go through and we establish our criteria and mark them. So we could be like, oh, great. And then you kind of decide what kind of answers you want.

So you can see the AI gave this answer. Do we want concise answers or not? You know what I mean? So how well did it do on conciseness? How well did it do on accuracy? And you can kind of create these different ones, and then you rank them with just like a thumbs up, thumbs down of how well it did on each of these, and that helps you then.

Go in and tune the prompt more or tune the model to get there. And then you can even have the models. I mean, to some degree it's like, you know, turtles all the way down, you know what I mean? Where you can have them produce more questions and then you see it run through. You can also set them to do automatic evaluations as well.

So it will go through and you can be like, okay, these are the criteria I'm looking for. Look at the different ways I've judged it. And now you start coming up with your own questions and your own responses and see that a human goes through that evaluates those. But it is a kind of. You know, longer process.

I think humans in the loop are incredibly important. Especially because it's so easy for models to get confused on small semantic differences. You know, like we could be reading a big security document and, you know, a security incident versus a network incident. It'll be like, oh, those are the same ones.

And they're like, actually these are like materially different things. You know, if there's. A security breach versus there's an issue like with like the networking, you know, infrastructure or something like that. But sometimes you'll see it kind of get mixed up on some of those. And I think that's why, you know, so many companies are, you know, looking to tune kind of like even if they are using a big foundational model, why we see such different.

Results from these different companies because they're tuning it uniquely for that domain. You know, like, hey, we are tuning exactly for lawyers that need questions around these types of contracts. Or for us, like security professionals that are trying to answer our questionnaire or trying to determine if the evidence they produce will be accepted by their auditor.

So we automatically run an eval on that and say like, Hey, we know, you know, tons of auditors in the past have said this is a good type of system description. And so you can go into your audit with confidence that like, hey. I've to produce the right kind evidence. 

Richie Cotton: Okay, so I, it sounds like there's quite a sophisticated system then.

'cause you've got these layers of like, well maybe the AI is like helping you both write the test and judge whether the tests are correct. Maybe you want humans in there and, it's probably gonna take a few iterations to get something good in terms of like just the system to test whether or not the rest of your work is any good.

Jeremy Epling: Exactly. And like we have teams that are running these evals, you know, some of them on a daily basis. Definitely all of them on like a weekly basis where they're kind of going through with a human in the loop approach for all of them. And so yeah, it's, and then every time a new model comes out, you know, things generally always get better, but like sometimes certain things progress and you kind of have to go tune for that one.

Richie Cotton: Okay. So it sounds like it is kind of tricky dealing with like the the moving technology platform, but I'm curious, are you able to predict which things are gonna work well with the agent and which things aren't? It often seems with ai it's like sometimes, you know. You think it's gonna be like a tricky thing actually, you know, a simple prompt's gonna get you like the, the perfect answer and sometimes it's like, this is really difficult to, to get it right.

So what works well? What doesn't? Like, what do you find easy? What did you find hard when building Vantage agent? 

Jeremy Epling: Yeah, definitely. I mean, I think that you know. I'd say the most success we had early on was with the questionnaires answering the questions once we got them, extracting the questions from all those different document types and things took a little bit of while to kind of get to, I think any of the.

Really complicated questions. Which of the higher value ones have also been, has taken like more time? I think that the more recent models, like in the last six months have gotten a lot better, and we've gotten a lot better at eval and our prompt tuning. So you know, A GRC person will be like, Hey, we have this risk around, you know.

Data governance, like our data governance rules are too lax at the company. And then they wanna map that to a set of controls. They wanna understand what's going on in the industry, what are the best practices, how they should quantify this business risk. 'cause a lot of times an executive will be like, how much money would this cost the company?

You know what I mean? Help me estimate that. And so I think those more complicated questions are ones that. It's easy for the AI to kind of get confused or like not give a super clear answer to. So those are ones where we've had to spend a lot of time making sure our data model's clean. Oddly enough, like I find that a lot of the time, like if a human can't understand it or answer it easily, AI's definitely not gonna be able to.

So a lot of the time if we think about our development team, like internally, like our engineers, you know, anything we do to clean up our code base and make it easier for a human to understand our code base and write code generally makes all the AI models better as well. So it is just a general like, hey, you know, I mean they're trained on kind of like how humans think, if you will.

And so. I think those have been really helpful for us as we go through those, like more complicated questions, have taken more time to connect more pieces of, you know, the platform together and are, and have more like vague outcomes. In those cases, we like to also just kind of suggest multiple options to people and let them know where it's like, hey.

There's like three or four different ways you could go tackle this, which is more important to you. And I think that, you know, Claude Code is probably the best example of this in my opinion, where if you, I know how much you've, you've written code or many people in the audience, but like when I'm using Claude code, a lot of times for big hairy changes and things like that, I'll put it in plan mode.

And it's really great about coming back. Asking me follow up questions, asking me to clarify different pieces. I can go back and forth with it and like brainstorm together to get to the final output. Requires more from you. But I find that the end result is a much better and I think that's something a lot of agents, including Vantage agent, are like digging into more around these follow up questions, helping to clarify so that we can like focus the model on a, on a really great answer.

Richie Cotton: Yeah, so I mean, really there's like two different approaches. You can either have agent as collaborator, so AI is asking you questions and wants that follow detail, and then there's agent as automated where it's just like it will go and do the thing and get an answer. Then you kind of looking at the final result and seeing when it works.

So I, I suppose, do you have a sense of like when one thing works better than the other? 

Jeremy Epling: Yeah, I mean, I think for the discreet focus tasks, like answering a questionnaire or things like that we try to, like, you can think there's basically like multiple, even though we refer to it as the banta agent for us, you know, there are many agents underneath that kind of like look at the question and decide who they should federate it out to and which tools should get.

So one of the first passes we think about is kind of, Hey, is this a simple, easy thing where we can just pull a discreet, highly factual answer? Yes. Oh, is this a really general question that's gonna require more time and thought? Okay, great. Let's farm this out to multiple different subagents. Let's do multiple tool calls and kind of pull all that together.

And that's when we're more likely to fall into this pattern of going back and forth with the user to like clarify. So we can give them a great answer. And so I find those freeform questions that are more. You know, less like, Hey, can you go change this thing in this policy from like A to B and just like do all the work for me because there's policies and I need to update the approver to all of them.

Just tell us to do it. We'll go do it. Or like, please make all my SLAs the same, or you know, do these things. But you're like, Hey, I'm trying to think about in the next year, what are the top AI risks that are coming up? Can you like help? Look at my risk program, what other top risk programs look like and determine like what gaps I might have, and I wanna go pursue this new regulatory framework.

You know, how, how can I do the least amount of work to do all these things together? And I think those are the ones where we wanna collaborate with, you know that security person, like the whole time it'll be asking them more clarifying questions and things like that. 

Richie Cotton: Absolutely. So yeah, certainly as things get more complicated and the stakes get higher and the project or the problem gets longer to solve, and yeah, that's when you, you need the human collaboration element.

Okay. So I'm curious just to what skills you need in order to go about creating agents. Like I guess talk me through like what you've been looking for, like with your team. Like what, what makes a good agent creator? 

Jeremy Epling: Yeah. I mean, I think that like. Entrepreneurial skills are like very important. I think someone who kind of has that builder, DNA is probably the term we use the most internally, where you don't really see boundaries and you just wanna like solve this customer problem and I think are excited about this technology and what it can go do.

I think if you get a little too framework based where you want everything to be perfect and deterministic and to fit into nice, neat little boxes, sometimes it can be like infuriating. For people like that, they're really important and we need them in other parts of like Vanta and many other companies as well.

But like usually being on the bleeding edge of some of the agent stuff is not the thing that gets them the most ex excitement. So I think for us, when we're looking at those people, it's. Someone who will go deep with customers or really understand what their problems are. Someone who has kind of an innate sense of like what a high quality answer is for this area.

So we actually hire a lot of people that aren't security subject matter experts, but can partner with our security subject matter expert team to really do that like last mile eval. I think that's like a key part of it for the agents understanding the kind of like basic workflows and someone that's living in it every day, you know, especially more on the cutting edge.

It's like you don't need to be technical to do all these things, but I do think the most interesting AI experiences generally for me are all like coding experiences. And maybe it's 'cause I come from like an engineering background, but when I think about, you know, cursor or Claude Code, code X, a lot of these, I think someone who is spending a lot of time in those experiences and see how people are pushing the edge, you know.

Whether it be like lovable V zero you know, Figma make things like that. Pretty early on, we ended up buying licenses for V zero for everyone on our product management team. And then our design team uses Figma make a lot as well as V zero. And we were just like, great, like just start building prototypes.

And I felt like our technical people jumped on it really quick. The less technical ones were like, I bet I can't do this. I bet it's hard. And I actually just recorded a Loom video and like sent it out to a bunch of people and it was like, I got this all set up for you. Here's how you do it. It's a lot easier than you think.

And now. We have product managers and designers, some of whom have like very little technical, you know, experience in like writing code now going out to customers and creating full end-to-end workflows and prototypes and validating ideas and like, you know, days or hours where before it'd be like, oh, how do we get this on engineer's backlog?

Get them to go build it for you and do all that. So. I think that quick cycle time in someone who's excited by how everything is changing all the time and like wants to lean into that are the people that are, are really creating these like, exciting experiences. 

Richie Cotton: That's very cool. So I, I was kinda expecting some of the things you said, like okay.

Having like a builder mindset and a few technical skills, but the idea of just being entrepreneurial in that you can just sort of. Dive into a problem where you're not sure what the answer's gonna be and just give it a go and try something out. That, that, that's really interesting that that's the way to go.

And that's one of the big sort of criteria for success there. 

Jeremy Epling: Yeah, definitely. I mean, I feel like there is a lot of, we just still don't know everything that AI's good at and someone who has that built in curiosity of just like, can I use it to solve this problem? Let me try And like the persistence to try it enough to know if it's gonna work or not.

'cause some of the stuff that didn't work, you know, months ago now works better now. Like I think that, you know. At least, you know, for me, when I think about like Opus four five, I think they just did four six like last week. That was a kind of watershed moment for me for Claude code getting like significantly better, which was like around November, December timeframe.

I think computer use, if people are familiar with that, where it's like you basically can have like the AI run a browser for you and navigate to different sites and do things, it definitely feels like it's probably lagging six to maybe nine months behind, like where everything else is, but is like getting a lot better really quickly.

So. That's something that we've been investing heavily in internally and just being like kind of waiting for that moment when we're like, it feels like it's on the trajectory to be like mass adoption and let's make sure we're always investing in it. So when that time comes and it flips over, like we are ready to go ship something.

And we have a ton of customers using computer use today for a bunch of scenarios of like automating, screenshot taking and things like that. For evidence gathering, if there isn't an API. 

Richie Cotton: Oh man. So computer use is really interesting 'cause it, on the one hand, it's a very cool thing if you want to just automate any kind of task, it's like, it's basically like, it's the new version of a macro, right?

Like I guess you've had Excel macros for decades, but this is like the, the new cool thing, but it's also a big security risk. So I guess if you're sort of considering adopting computer use or, or one of these sort of similar tools like what do you need to do to, to. Do it safely. 

Jeremy Epling: Yeah, I mean, I think that one of the things we did in Vanta is like we started off in the beginning with all of our automation always being read only, and you have to opt in to write actions, and I think that helps.

It doesn't help everything. Obviously there could be data exfiltration and pulling and leaking things, but at least you're just not like. It should be extremely rare that like a reading, some data will take down production or do something like that in that vein. And so I think having customers opt in to these like right actions kind of comes back.

And then once they do, I would say, you know, downmarket, smaller companies are much more open to this type of thing where they're kind of like. AI native, they wanna push the boundaries. They're just trying to find product market fit, and they're willing to kind of like take more risk there. I'd say like really large enterprises are not willing to take that risk nearly as much.

So I think having, understanding who your customers are and what options they want. We definitely have customers where we start read only they opt into these kind of right use cases, and then giving them that granularity of control where it's like, yeah, I'm good with you, like updating my policies, or you updating this evidence or updating these controls.

And then I think the highest level is like computer use, probably read only where they can pull the data in the screenshots. There'll be a lot more comfort with. Then logging into your AWS account and like changing some very important configuration setting in AWS without you knowing it, that feels like it's gonna be pretty far off from people.

Just kinda like letting the agent yolo, those types of things. And I think what we see now is more like review and approve where we're building to like, Hey, we've done all the work for you to tee this thing up. It's now at that moment, do you want it to go do it for you so you can quickly kind of get an idea of like what it did to get to this point and like, yes, I feel good, it looks good.

Go ahead and do the thing. So maybe you don't have to go log in and like twiddle the knob or remember exactly which page and like the Google console or Azure console this setting's on. But you at least know it's like here's the exact thing, it's going to change and you can build confidence that helps you go faster.

But I do think it's these layers of kind of. Read only writing to some simple things, read only for some of these other things. But computer use I definitely think is extremely powerful in both like the good and the and the bad because like it has access to a full like VM or container and can kind of, can do what whatever you give it.

So I think people being able to lock down those, and there's a lot of companies that are providing kind of these isolated untrusted, you know, code execution environments for computer use to run in, which is like what we use internally too. Okay. That design sensible is like, give it a sandbox, like make sure it can't destroy everything, can't delete your whole AWS account.

Richie Cotton: That could be, I mean, there's potential for some real big disasters there if you just let like AI do stuff on, on, on enterprise accounts like stuff in production. But maybe maybe. We, we take a step back, like what are the big success stories or use cases you've seen at the moment? Like what, what are the cool things you can do with this sort of automation?

Jeremy Epling: Yeah, I mean, I think some of the big ones that we've seen for people is the questionnaire Automation has been huge. We've seen a ton of customers get that. And just save hours upon hours and hours of time which is great. Like security reviews now, you know, take like % of the time that they used to going through these questionnaires.

Some, in a bunch of cases we're getting like a hundred percent coverage rates, which is like phenomenal. And then if we don't cover something, we automatically let you know, like what type of answers you need or policies or anything. So you can't get those coverage rates up. I think on the vendor assessment side, no one likes sending those nag emails to like a vendor, asking them for like all the evidence and all their documentation and pouring through those hundreds of pages of documents you get fast.

So I think the Vanta AI agent has been really helpful in like streamlining those reviews. I think since from October to, I think it was like the end of January, we already cut that number in half for all of our customers. The average amount of time of security reviews takes and we think we can cut it in half, at least one or two more times, which is huge.

And that's like the total wall clock time. Obviously the AI's really fast, but sometimes you're just be on the human to like log in and kind of do that, review and approve. I think though the thing gets me excited is. You know, a lot of security teams don't have that feedback loop, that kind of engineering and product teams have had forever.

Right? You know, it's like, hey, you ship a product to your customers, you get all this data in telemetry back you and like feedback things they love, things they don't love, like how they're using the product. Then you can make it better. For a long time, security teams were just like, all right, we're gonna get some certifications or we're gonna kind of do some stuff.

And they're kind of just stuck. They don't like know what customer's opinion are. They're connect, not connected to revenue. And it makes it hard for them to kind of justify their. Their budgets and explain to like management, like what they're doing and like the board other than just like reducing risk kind of becomes this generic insurance thing of like, oh, I guess we need to spend a bunch of money on this, but I'm not sure the benefit we're getting.

So one of the things that Vanta does is really ties all that feedback you get from customers in questionnaires, in conversations, the contracts you sign, the unique commitments you're making in those contracts. Down to like revenue. So it really changes the job of CISO where now they can come in and be like, Hey, my team contributed to like, you know, whatever % of the revenue we got last year.

By answering these questionnaires, these customers wanted these security proof points. They're now asking us to do these other things. They're asking for, like changes in our contracts to go do this. These are the changes we're gonna make in the security team to help them like justify their budget and their headcount requests and like better show off the amazing value that they're doing.

And so I think the agent being able to, to attach all these pieces where it can analyze all this data you're getting from your customers. I. What your controls and frameworks and risks and your program is currently doing and then how you can turn that back around to like help grow the business. I think just fundamentally it changes this notion of like security just being this like nebulous cost center that like takes a ton of money and you know, a lot of executives don't understand what they're doing.

They use these confusing terms and all this jargon and now we can be like, oh no, they are actually powering the heart of your business. And the agent helps them easily like tell that story and do it continuously. 

Richie Cotton: I love the idea of like nebulous cost center, like a lot of those things you said about security teams, it's the same with like data teams as well.

It's like, I think they're pretty sure they're doing something useful, not sure what. And yeah, being able to demonstrate that value to executives, to the rest of the business, that's incredibly important. So yeah, I love the. You get rid of the gr the, the rope work, outsource that to ai and then you can focus on showing your value.

Fantastic stuff. Alright. Just to finish you up, I always want to learn from more people. So whose work are you most interested in right now? Who should I be following? 

Jeremy Epling: Oh, I think I. Most of what I've been following have a bunch of the engineers probably from Anthropic, I would say, who are like on the kind of like cutting edge of going through and understanding like where models are headed, what they're doing.

I'm always interested in Jan Kom. Probably a lot of people end up following him as well. He has like a new startup there. And think even if he can be a little bit more negative on like LLMs. I do think he's right. They will hit a point where it's not like true intelligence, but like we continue to eek more out of them constantly.

But I think he has a great perspective of like what the future of AI could be, what the limit of what we've seen today is. And so definitely someone that I always like to follow. Oh yeah. Yeah. Like he's just he's been thinking so deeply about these problems for decades now that yeah, he's just he's always good for a spicy quote as well.

Richie Cotton: So yeah. Definitely somewhere we're following. Alright. Thank you so much for your time, Jeremy. That was a lot of fun. 

Jeremy Epling: Thanks for having me. It was a great conversation.