Sales and Marketing Don't Care About Agents

Richie Cotton: Hi, Duncan, Welcome to the show.

Duncan Lennox: Hey, Richie. Great to be here. Thanks for having me.

Richie Cotton: Yeah, great to have you here. Now, I've seen over the last few months there's been quite a few startups pitching the idea of AI SDRs, AI BDRs How close are we to completely automating sales?

Duncan Lennox: I think we're a little bit a ways from completely automating sales.

The way we think about it is a little bit different. I think there's a lot of productivity time that salespeople waste on admin tasks today, whether that's updating the CRM or researching a prospect, and we see tremendous opportunity to s- give them all that time back and let them focus on the actual human-to-human interaction part of sales.

So the technology's getting better and better every day, and I think for some of the simpler discovery and buying journeys, like we could actually make it more convenient for the end user customer to interact with an agent and do a lot of the process that way. But for more complex, higher-end sales, it's really about how can we help the sales rep spend more time with the customer and take away the drudgery that, that consumes a lot of their time today.

Richie Cotton: Okay. Yeah, I can certainly see how if you're trying to automate a multimillion-dollar enterprise deal that normally takes months, then you're probably not gonna be able to do that entirely with bots. So you mentioned out there the simpler flows. Do you wanna talk us through how that works then, like where the agents come in for a simple sales deal?

Duncan Lennox: Yeah, absolutely. So if you look at typically how a lot of those kind of buying processes work, so let's say it's, let's say it's more complex than a consumer product, which typically, you know, y- people are already used to going to a website, buying it directly and so on. We'll say it's a little more complex than that, but not your multimillion-dollar sales example.

The beauty of the AI SDR type of approach, or inbound agent as we call it, is that ability to help the customer really, or prospect really understand, go through the discovery journey, ask a lot of questions, understand whether there's a good fit to the products and services, and then automate much, or in some cases, all of the whole buying journey for them.

So it's, it really turns into a convenience thing from the end user customer point of view.

Richie Cotton: Okay. I like that idea. So it's basically it's the next iteration of your sort of support chat bots. It's just like finding out how can I help you? And then hopefully redirecting you to the right place.

Duncan Lennox: It's very much an extension of that, and frequently for a lot of companies, including our own, that we started with the agent technology in the customer support space, so like that's something we've been doing for a while. One of the reasons was there was really good product market fit early for that.

At the end of the day, the person that has the support issue, the query, whatever it is, they just want to get an answer fast, and it's not that they need to speak to a person necessarily. In fact, often that really slows down the process, right? Because you gotta call up, you gotta sit in a call queue, then you gotta explain everything about what your problem is.

Then the support person has to run through all their classic things like, "Did you try unplugging it and plugging it back in?" Or, whatever the equivalent might be, and eventually, hopefully you get to an answer. But sometimes what happens is it's like I need to hand you off to somebody else."

That's a little bit more of a problem and, you know- What's the first thing that happens when you get handed off to the second person? You gotta start the whole thing all over again. "Why are you calling?", essentially. So what we call our customer agent really eliminates all of that and just makes it a much f- easier, faster experience for the person that has the query that's calling in, and that let us build and perfect the technology of the ability to handle that.

And one of the things that we have to do as HubSpot is we're not simply doing it for ourselves, though of course we do that, we're doing it for 300,000 businesses that are HubSpot customers today. So everybody's set of products and services are a bit different. The type of customer they have is a little bit different.

How they want to engage even is different. Is it WhatsApp? Are they sending an email? Et cetera. So building that out and making something that's really good in that space then is a natural kind of foundation to extend into sales. On the customer support side, we are at, in the low 70% resolution rate for our customer agent, meaning more than seven times out of 10, it can resolve the issue that the customer is inquiring about without the need to hand it off to a human.

And a little bit like the sales example earlier, too, the ones that are getting handed off to people to handle then are typically the more complex ones, the more impactful ones, where it really is helpful to actually get to speak to a live person. So taking that technology and then extending it earlier in the buying journey, the customer f- support agent is working post-sale.

You start extending it much earlier into that whole discovery process and buying, buying cycle, and applying the same kind of technology.

Richie Cotton: Okay, and that's cool, that it works 70% of the time in, in customer support that it's completely automated.

Duncan Lennox: And that's on average, by the way just as well.

So we've got customers that have resolution rates that are 80, 85, 90, 92%. It's a little bit dependent, but we're getting over 70% average across the board. Which is a pretty good... It- it's an industry kind of leading place to be in terms of resolution rates.

Richie Cotton: Okay. I suppose if your resolution rates get higher, it means you're probably having a lot of really stupid billing problems that are the same thing over and over again, and it's probably a good sign that you should go and fix your product at that point, is there that kind of feedback loop then from you've got an agent to try to solve this and-

Duncan Lennox: No, there is. And some- yeah it and I think actually when you can tie into the data much easier when you have something like Customer Agent because you're, it's creating and tagging all of these incidents as it goes and so forth.

And it's, it really interesting the issues that pop up because you'll get two broad classes of issues that pop up, of kind of the, the more challenging ones. And some of them will be a good indicator, as you said, of actually there's an opportunity to make the process better for customers so they never run into this issue.

And it could be something as simple as figuring out how do I reset my password?" And it could be a, so that could be a design user interface issue of we're not making that obvious enough to customers, so when they get into that situation, it's super easy. Or maybe our actual password reset process is actually not very effective and the other class of problems then is more the, the ones that happen periodically. I you may have a big event or something changes externally that causes customers to start using your product in a different way or etc. So it's creating a new issue that you've not really seen before and that starts to pop up.

And one of the things that we do with Customer Agent is we have another agent that kind of works in the background behind it called Knowledge Base Agent, and it's watching for what new issues are popping up with regularity, and then it's recommending new knowledge base articles that you can write that both- get made available directly to customers and people, but also then feed the customer agents, so it's able to do a better job of responding to those issues.

So you do get this feedback loop going of improving the system all the time, and then having that flow over to improving your processes and indeed your products and services as well.

Richie Cotton: Okay. That sounds like a, a very cool positive feedback loop. Since you mentioned quantitative benefits, I think there's a lot of hype around making use of agents.

But it's sometimes difficult to show what the return on investment is. Have you got any data from your customers to say, "This is what we've achieved using using the agents?"

Duncan Lennox: Yeah. No, we do, and I'd say, I'd say, look, it's still early. Much as all of us are immersed in this every day and love it all, for most of our customers, it's still pretty early days for the use of AI in their business.

And I think particularly in our case, because HubSpot's customers tend to be small businesses that don't have the expertise, the time, the resources to really learn this stuff. They're- they're just trying to grow their business, but all that being said, we do get to do this interesting cohort analysis because- We c- across more than 300,000 businesses of all shapes and sizes across 140 countries, we can see and look at the data of customers who are using our agents versus those that are not.

And we see very clear improvements in the go-to-market outcomes that matter. So at, at the end of the day for our customers, we're in the go-to-market platform business, so we're helping them get more leads into the top of their funnel, book more meetings with prospects, close more deals, support those customers, et cetera.

When we look at, say, something like MQLs, marketing qualified leads we see over three times more MQLs created by customers that are using our agents and our AI as part of what they're doing in their campaigns, about 3.6x actually. On winning more deals, there's nothing more fundamental for most of our customers than the actual deals that are closed won.

We see about a 3.2x improvement in the cohorts that are using AI and agents. And then we've been talking a, a lot about customer agent. We see over twice as many tickets resolved for customers using, us- using customer agent versus handling it entirely manually.

Richie Cotton: Oh, yeah, more than three times improvement on a a sort of early to mid-sales funnel that's nothing to be sniffed at.

Duncan Lennox: It's dramatic. It's dramatic. And, look, it's, that's an average, so some are getting even more than that, some are maybe not quite seeing that yet. And, but it's very clear that when you're leveraging agent solutions like this, the right way, I should say as well, and we can talk a little bit about what we've seen in, in what really has an impact on the quality of that outcome, but it is pretty dramatic.

Richie Cotton: Okay, so that's the kind of the, the positive side of things where it works. I'm curious, are there any places where you're like, "Okay, agents don't really show an impact yet. This is where you really want the humans to spend the time"?

Duncan Lennox: I think a lot of it comes down to not necessarily a particular area, but maybe the class of problem within an area.

So if you, if I go back to the customer agent example, or we use the AI SDR example, there are inevitably for customers, the high risk, high reward, high impact type engagements where the complexity of what's involved or the criticality of the decision that's getting made is just that it's better having at least a human in the loop.

It doesn't mean the agent can't be assisting the person on our customer side in doing that. So y- And an, an obvious one there would be if you see a, if a customer is not getting their issue resolved in customer support through customer agent, like we're misunderstanding it or we're not getting them to, to a resolution, that ability to escalate that to a person becomes an issue.

So it's more that. It's more knowing when to escalate, when to have people in the loop- Rather than necessarily a full class of problems that just doesn't work today, because the, the technology's gotten quite sophisticated, as so we can apply it to most anything. It's just a question of how much of a sing- a bucket of problems could be resolved, primarily with the agent.

Richie Cotton: Okay basically agents throughout the whole sort of go-to-market life cycle, but you probably want agents to do the, the simpler things and then have humans for the more complex stuff. Is that the, the general-

Duncan Lennox: I, I think potentially yes. I do think it, it's still early, and I think the bigger challenge for our customers today is really, wrapping their head around trust.

Can I trust what these agents are gonna do on my behalf with my customers? How do I understand how much it's gonna cost me to do this? And, you talked about ROI, making sure they understand that. So a lot of the work we do, beyond of course building the high-quality agent in the first place, comes down to, what we call trust hill climbing.

How do we help customers try out these agents in a very low-risk way where they stay very much in control at all times, so I get to see what the output is gonna be like, maybe then start to use it and try it out in lower risk scenarios for them, and over time, as they get more comfortable, as it gets more fine-tuned, be able to delegate more and more authority to the agent.

And with Customer Agent, we built a lot of tools to let our customers do this, for example. So not only things like where you can test out the agent, like yourself, act as a live customer and actually see how the agent would respond to you and so forth, then as you get comfortable with maybe having it handle some queries, we do things like you can choose the time of day that the customer agent responds versus the, the humans responding.

So people will often do things like nights and weekends when they don't have coverage right now. So let's give our customers a better experience during that time, but it tends to be a quieter period. And then over time ramping that up y- as they go. So a lot of that turns out to be a big part of making customers successful using the technology.

It's a lot more than just building the technology itself.

Richie Cotton: Absolutely. You mentioned trust. Now- The data frame audience skews very technology excited about. Like a lot of people love agents, but you said a lot of your customers are small businesses. Do you get that same level of enthusiasm for AI with your customer base?

Duncan Lennox: I think the what we find is that a lot of customers are very nervous about AI generally, and this is a general issue. It's nothing to do with go-to-market or, any particular agent, and they're looking to understand. And it really does come back to that, that trust kind of notion of is this gonna work for my business?

Is this gonna help me? We see that as a big part of our value proposition for customers, that we hide and abstract away a lot of the underlying agent technology. We have our own agent harness. We've built our own, context engine on top of all of the structured and unstructured data that, you know, that we have for an individual customer, and then we're using that to drive the right outcomes with the agents.

And from their point of view, they don't have to worry about how do I provide good context, which model should I use, what is an agent like kind of thing. 'Cause at the end of the day, again and again, what we hear from them is, "I'm just trying to build awareness, win deals, delight customers."

These are the three core, we call them the golden journeys. These are the core things that they're trying to do, and AI Or any technology is really just in service of that for them. So they're not nerding out on it generally the way you and I might.

Richie Cotton: Okay. That makes a lot of sense, is like if you work in sales or marketing or customer success, then y- you just wanna b- make some money close some deals, and help your customers, rather than using the technology just for the sake of it.

Duncan Lennox: Yeah. And there's... of course, there's always, classic visionaries and early adopters who are... who saw the opportunity of the technology early and glommed onto it, and, there's... it's super fun always talking to those folks, and they give us tons of great feedback and input.

But yeah, most customers are exactly as you say, "I'm just... I got a business problem to solve here, and if you can help me solve it, HubSpot, and if leveraging agentic AI is a great way to do that, then that's fantastic." But at the end of the day, it's the outcomes that matter.

Richie Cotton: Okay. Now I'm curious about HubSpot's role in this, because HubSpot is traditionally a piece of SAS software.

And at the start of the year, there was talk of this SASpocalypse, and I think that was overblown, but it was the idea that agents were gonna replace SAS software. But yet you're very much pushing agents. So how do they, the two sides of HubSpot sort of reconcile?

Duncan Lennox: I think it ties back to the outcomes piece again.

If you... I think for me, I've been building software for a long time, and this is certainly the most transformative technology I've ever gotten to use as part of building software, which is, c- could create challenges, 'cause it changes things from how it used to work, but it's also the most unbelievable opportunity to have bigger impact.

We- if you think about HubSpot, the... We just celebrated our 20th anniversary back in June, and so for the first two decades of the company, we started as a marketing automation company, grew into building out a full CRM, and for a number of years now have really been a full go-to-market platform.

So we cover everything from, the first touch of a prospect to your website or responding to an email campaign, all the way through that buying journey, post-close supporting them, continuing to grow and delight that customer. But for most of the 20 years, what that meant was we were building tools and workflows that let go-to-market teams drive their outcomes.

And for the last couple of years, what that's been really transitioning into is- now it's really all about a platform that can help you manage hybrid teams of humans and agents working together and where we can actually do a lot of the work for you. So like an example of that would be we just had our big conference a few weeks ago, and one of the big things that we announced at that is a product called Marketing Studio.

So it's squarely targeted as marketers, as you would guess from the name. It's really all about how do you build campaigns and run them. But it's, it does that in a very new data-driven and agentic way. So it's proactive. It's looking at all the data from every campaign, from every sales cycle, from every...

that you have across all of your customers. It's recommending areas where maybe you need to do work, like we might be saying Your visibility inside of LLMs like ChatGPT or, Perplexity are low. We recommend you build a campaign there targeting this segment. Would you like me to put the brief together for you?

So Campaign Agent, a new agent we launched, builds the full marketing campaign for you. Then if you want us to, Content Agent kicks in and can actually build all the assets, like the ad to go inside of ChatGPT Ads or the landing page, the email, et cetera. And then Nurture Agent is another new agent that we just launched, that as leads come in, can build customized email follow-up flows tailored to each individual person.

And all of those things I just said are the things you would always have done with marketing campaigns, essentially, but people would've had to do all of those things along the way, and now we can help the marketing teams by doing a lot of that work for them. But the last thing I'd say on that is to your, you- the, we're very much building agentically because that is the best way to drive the best outcomes for customers now.

But we're hiding all of the complexity of that. So what I didn't mention is that whole flow that I just talked about, you can do all of that just by talking to Breeze Assistant, which is our co-pilot that's available everywhere in the product, and it even figures out what agents to use and how to get things done.

So as a small business owner, you don't need to know even those set of agents that I just named. You don't even need to know they're there or how they work. So what we see is that there's never been more opportunity for us to do more on behalf of customers, and it's about taking that platform that we've always had.

We've upleveled it with an agent harness, with the growth context engine, and a set of agents that we've built, and now we can just do more on top of that for customers, and in the form of agents. But they don't need to necessarily know or care about that unless they're interested in understanding it.

Richie Cotton: Okay, so I guess the business goals have not changed at all. It's like people still want to do the same things. It's, we're back to like you just wanna close deals or... Yeah.

Duncan Lennox: I always think it's on one level it's here comes the new boss, same as the old boss, right? And, we can get so sidetracked by the technology, and it's amazing.

What frontier models can do is astounding, and it's improving all the time. But, the a- the best way to build products, to build software, has always been deeply understand the problems that customers have. And people often think they understand them, but they haven't deeply peeled the onion of what's really gone on there, and then how can we apply technology- to find better ways to solve those problems.

And that sounds really obvious maybe on, on one level, but there's a tendency for us to get carried away and excited, especially maybe as engineers as product nerds by the technology, and put the cart before the horse.

Richie Cotton: Yeah, definitely. I like the idea that you solve the problem first and the technology's supposed to help you out with that rather than the other way around.

I feel like there's a few different ways of agents and SaaS offering interacting. You mentioned there's the Breeze Assistant, so that's like basically I guess, an a chat interface to solving all your problems. You've got the traditional HubSpot interface. But I guess there's also can you use other AI platforms like, so if you're like a, a Copilot user or a Claude user, OpenAI user, and then you're accessing HubSpot from there.

Like, when do you want each of the different interfaces?

Duncan Lennox: The way we think about it is is quite straightforward. The number one job for us is we have to deliver the best, cheapest, fastest go-to-market outcomes. That's what you have to be able to get out of the HubSpot platform.

That's our job. The second part then is we wanna meet customers where they are. We have a huge range of customers, and they're... we spent a lot of time digging into the data, and they're really in a variety of places on what you could think of as the AI maturity kind of spectrum. And interestingly enough, it's not really divided by neatly segmented by geography or company size.

It's like at all geographies and company size and market segments, you people who are, were really early adopters, real innovators, very comfortable inside of Claude or ChatGPT to do a lot of things. And then you've got the vast majority of our users, I would say still, who are less comfortable with that, at least yet, and are more comfortable with something like Breeze Assistant because they're used to being inside of HubSpot, right?

They're go-to-market teams. They're familiar with HubSpot. It's always there. Breeze Assistant has the big advantage that it knows everything about you and your business because it lives inside of HubSpot. But then you've got a lot of people who are still very comfortable in what I might call the traditional web UI or one of our mobile app and so forth.

So for us, what it means is we have to be able to meet customers where they are, and we're doubling down on that, and that includes if they wanna do it inside of, one of the LLM. So this is the headless strategy, right? So where we build things in such a way that if a user doesn't want to, they essentially never have to log into HubSpot.

If they're a sales rep or, they're a marketer, and they want to work inside of Claude or ChatGPT or Copilot or Gemini, we built connectors, a year or more ago into each of those products. HubSpot is fully exposed inside of those products, and we think the best combination of that is HubSpot is still doing the work for driving those core outcomes, and then you're using the chat products like the Claudes and the ChatGPTs to build around that and do other things.

It's orchestration and so on. So- we committed to this notion of an open ecosystem, that we're gonna deliver UI parity into the, into our headless environment. In other words, anything that you could do through the UI in HubSpot, you'll be able to do inside of one of these LLMs. And it's all about meeting customers where they are.

Richie Cotton: Okay. I like that there is that choice, and depending on what your preferences are, then, you can do it, and i- in theory it should work just as well like whatever interface you use. Early on you were talking about building trust, and I suppose the biggest way to build trust with the agents is to make sure they actually reliably work.

Do you wanna talk me through, like, how you make sure that all the, the HubSpot agents actually give the right answer?

Duncan Lennox: Yeah. We do a ton of work around this, as you can imagine, right? At the it... There's a lot of things we've learned along the way. I would say, may- maybe I'll cover two or three key things we've learned along the way.

We can dive into any of them i- in more detail. But the first one is that the models are very powerful, but they're not all-knowing, and they're not magic. So on their own although they know a lot about the world, they really don't know anything about you and your business. And like that turns out, perhaps not too surprisingly, to have a big impact on the quality of the outcomes that you get out of them.

So what we found is- Being able to provide the right context, like in real time at the point of need, when we're gonna go execute a task or drive an outcome let's say it's resolving a, a customer ticket, like with customer agent, or it is part of a researching a prospect for a call that a sales rep is gonna have, a demo that's coming up tomorrow, that being able to identify and inject the right precise context into that that prep task or resolving that ticket has a huge impact on the quality of the outcome and the ability to resolve it.

And interestingly enough, what we found very clearly, and there's been a bunch of academic work that's shown this to be true more generally if you deliver too much information as part of context, that can actually create a huge problem as well. It's a little bit like a LLM version of the classic ML overfitting kind of problem, and so getting that precise, correct, right amount of context i- is a huge indicator of the quality of the outcome you'll get. It also, by the way, affects how much it's gonna cost, 'cause if you throw a massive amount of context at a model, of course, that's consuming a lot of tokens as well, and in the worst-case scenario, you might actually be getting a worse output for more money as a result of all of that.

That's probably the single biggest factor that we've seen that impacts quality and safety. The second one is we built our own agent harness, and we've been doing this for about three years now, so long before the te- the term agent harness had even been created. And what that lets us do is really two things.

The first one is it lets us be model agnostic. So as amazing as all the frontier models are, they have, what's co- referred to as jaggedness. There's, there's-- You get to edges where suddenly they tip over, and they're not very good at things, and they're-- it-- those are in different places for different models.

So the net-net of that is there isn't really a single model that's the best at everything across go-to-market outcomes. So our kind of AI or model agnosticism lets us test and use the right model for each of the right tasks individually, and indeed because the state-of-the-art is changing so much and so frequently, as things change, if a different model pops to the top in driving the best outcomes, we can just route to that model instead.

And so that that's the second thing. And I'd say the third and final thing i- on it is we spent a lot of time using the two decades of go-to-market expertise that we have, and we've built out thousands of evals, like the actual definitions of what good and great looks like for different go-to-market outcomes, as well as what bad looks like.

And we have a bunch of automated testing that's running continuously using our harness and so on to run those things, and that really lets us capture a high quality, high safety of everything that our agents are doing.

Richie Cotton: Actually, a few surprising things in there. So I think a lot of the story around context at least actually is people saying, "Oh, you need to provide more context.

That's gonna be better answers." But you're actually saying that you don't want to give it too much. It's, I suppose it's when if your boss just gives you like 100 different instructions at once and a load of background that you didn't really need to know about, then you're gonna get confused about what you're supposed to do.

So that's interesting that you're trying to tune it to just the right amount of context. Do you have a sense of how you get to that?

Duncan Lennox: Yeah, I think I think that it's really where the combination of our experience over two decades and all the data that we have available in the system comes together, and that is something that's a little bit unique about HubSpot.

I was mentioning earlier that, it is this end-to-end platform. So if you think of an individual business, let's say we're a plumbing business, in a Midwest city, a suburb of a Midwest city, just to pick an example. 25 employees but you've been using the HubSpot platform for your marketing campaigns, whether that's email, whether that's ads you're running, blog posts, et cetera all the way through then, obviously through the service cycle of customers choosing you, working with you and post that, that engagement.

What that means is we have essentially a lot of data about what those full end-to-end journeys look like, and that includes- things that went well, where you won business and went and supported that customer and it all went really well, or prospects that ultimately decided to go somewhere else.

So we can see what, for your business, both good and bad look like. And then do that times hundreds of thousands of businesses over the last two decades, and we've evolved a lot of expertise in what good and great looks like. And then we can quantify that down into evals and tune the context that we- is the right context to ingest around that, and that's a very data-driven process.

If you have these evals that are, very data-driven ways to evaluate good and great outcomes, you can start to train con- your, our context engine to what is most effective here? What set of inputs here gets us into that sweet spot that is sufficient context to give precision, but not too much to lead into the overfitting problem.

Richie Cotton: Okay, yeah. I guess the, the evals is the important thing there. It's as long as you're testing does this work or not, that's gonna give you some feedback as to oh can we improve it to make-

Duncan Lennox: Evals are, the new PRDs or, it's the... it's it's not just, it's not just the modern equivalent of QA, like the quality assurance work we used to always do with software, where you'd write tests that would run automatically.

And these are much more sophisticated because the LLMs can reason and they can, they can do much more sophisticated work. But the contrary example is if you don't have these high-quality evals, how would you know if what you're doing is any good? At the end of the day and again, for us, this is a huge part of the value we, we provide and that we see as our responsibility.

Again, if you're a small business, you can see if, leads are growing, if deals are closing, et cetera. But having the time, the expertise to really define pres- with precision in a data-driven way that can be measured and iterated on, that's really our job to do a lot of that for you, and that's what we've been building into the agents and then the platform that powers all the agents.

Richie Cotton: It's one of those things where testing has never been, like, the most exciting part of the job, but it's actually, it's a very cool kind of field at the moment with tr- how do you test these these agentic systems well. Actually, do you wanna make it more concrete? Have you got some examples of what good evaluations look like?

Duncan Lennox: Yeah, the I think if you like there's a lot of different pieces to it, and it's very dependent on the area. Let me give you an example of something that is a big part of what HubSpot is typically doing for customers, which is emails, writing emails. Like and that could be in different, multiple different contexts.

It could be a personalized follow-up and nurture email as part of a campaign. You expressed interest in a campaign that we ran, and so now we're periodically sending you follow-up emails. We're trying to get that cadence right so we're not irritating you. that we're adding value. That there's other ones, other emails would be helping a sales rep write a follow-up email after a demo, like what's the next step?

And then of course we talked about customer support. That a- all works, customer agent handles email as well as all kinds of other chat, live chats and so forth too. Across the board there you're getting into something that is quite fuzzy and subjective in many ways, which is what does it mean to write a high quality email?

And the use case is different in each of those cases, but there are core components to it that are common. So for example, one of the ones that we spend quite a bit of time on is across a number of metrics that we quantify is a humanity score And the humanity score is literally looking at how human does this sound?

Does this sound like a real person wrote it? Take the example of a sales rep follow-up, right? So if we're trying to take a admin burden off of a sales rep one of the ways we can do that is write the follow-up email for them. And what happens there is the rep will have been using our HubSpot note-taker when they were doing their demo with the prospect.

So the note-taker will have captured everything that was said during the meeting, does a bunch of analysis on it. It gives you a summary and updates the CRM and a bunch of other kind of, useful things, but it's then using that to help write the email the follow-up email. But it doesn't just look at what happened in that meeting, it looks at every interaction that you've had with that prospect as a company and as a sales rep, and it uses that as well.

But then the humanity score comes in to actually be able to quantify generally, does this sound like a human wrote it? There's been... you see all this AI slop, we see it on LinkedIn every day, et cetera. Then it's like the second you start reading it, AI wrote this. It just doesn't talk the way a normal person sounds.

It d- and so humanity score is about capturing that, but then taking it to a level above that, which is it capturing the style and tone of this individual sales rep as well? And one of the ways we improve there is we've a pretty high bar on it of what we're delivering, but the rep obviously gets to see the email before it's sent, and they can make edits to it.

And when they make edits to it, the agent learns from that, and we update evals and memory so that when we know now next time, ah, okay, here's an extra nuance of the style of this rep that we can apply automatically next time around. And using that approach over about a period of six months earlier this year, we went from, on average, reps making edits to the emails that we wrote for them two-thirds of the time, and only being comfortable sending them unedited one-third of the time.

We flipped that entirely, where now it's more than two out of three times it's good enough, they're like, "Yep, send." So that's this kind of iterative data-driven way, leveraging evals that we are constantly working on the quality of the output.

Richie Cotton: It's fascinating stuff. And I have to say, I love the idea of having a humanity score to, to, is this...

Does this stuff sound like a real human? It's quite often difficult i- for marketing. So I had a case recently where we've been creating a a landing page for a webinar campaign, and I told my colleague this, all this text it reads like AI slop. What is it?" He's like, "I wrote every word of that myself."

I'm like, this is how marketers sound sometimes." Yeah it's tricky, but I love the idea that you're trying to parse out and make sure that you're not just- Creating slop everywhere

Duncan Lennox: Yeah, and it's n- it's not, and it's not about, as well, it's not about trying to fool people.

It's about trying to be authentic and connect with people. In the rapid case, we're not trying to replace that rep, we're trying to help that rep be more effective, but it still has to feel like a human-to-human connection is part of it. It's like the old uncanny valley in animation, right?

That if you try to make it, in the old days, like when the early days of Toy Story and Pixar, they deliberately went with this cartoon approach because if you tried to do something that looks realistically human people's brains can tell there's something wrong here. It almost becomes creepy, and AI slop I think is a little bit like that.

Although I will say, as a personal lover of the em dash, I hate that has now become a problem because we can't use em dashes anymore because people just assume AI wrote it if you do that,

Richie Cotton: yeah. It's it's been a sad sa- scapegoat 'cause it's a very useful piece of punctuation.

I wanna go back to something you mentioned earlier about you were talking about using frontier models, and I'm curious if they still have a role in generating text for sales and marketing. 'Cause I feel like a lot of the frontier models have been focused a lot on coding, cybersecurity and scientific things.

Do you still need frontier models for writing your marketing copy?

Duncan Lennox: Yeah, absolutely. I think you can- you don't have to be using the biggest most cutting edge model from an OpenAI or an Anthropic for a lot of the, a lot of the tasks that are there. A lot of the tasks that we're doing in go-to-market, I call them the meat and potatoes, like, of go-to-market outcomes.

And those things, because the models have gotten so good, you can use a smaller model in a s- in the same, model family to get high-quality outcomes, and that's, again, a big part of the value that we bring for customers because we're using all those evals and that agent harness, and we can figure out, actually, you don't need to throw the full power of a Fable or an Astra writing this email, to your point, that if you use the right context, you can use a much smaller, cheaper model, and that means we can deliver more value for customers.

That's certainly true.

Richie Cotton: Absolutely, yeah. So the small models kind of work, and this leads to the idea of saving money, which I think is a very hot topic at the moment. A lot of people have been stung by high AI bills this year. Do you wanna talk me through how do you save money on your AI bill and, get good value for money?

Duncan Lennox: Yeah, here's two parts to that. And you could, there's the like what's our job to do as part of that, and then how can customers think about it. The answers are largely the same, but we're trying to do most of the heavy lifting on our customers' behalf, right? So the, it does come back to what we were talking about earlier with context, evals, and an agent harness that you can run yourselves because those three combos together give you a lot of control- over the number of tokens that are being consumed, both input tokens and output tokens.

And as long as you can measure the quality of the output that you're getting against that then you're gonna be able to optimize for quality and cost. That mean, the mental model I always use is you have to triangulate three things, quality, performance, how fast you get the output, and then the cost represented for that.

And the reality is, if you think of that triangle or that three-legged stool, the answer or the mix of those things is different depending on what you're doing. If you're doing something like responding to a customer support ticket you need to make sure that the quality bar is very high 'cause it's gonna go out directly to one of your customers.

That still doesn't mean you need the biggest model in the world. It's still gonna be about, more about injecting the right context. But there might be other scenarios where we're taking on something that's very complex, like Planning an entire marketing campaign and building out a brief for that, and then constructing the assets, et cetera, that's a complex set of tasks.

You're going to need more sophisticated models for that but you still need to be able to balance cost. So having an environment that either gives you control over that, like if it's something you're doing yourselves, I think is critical. For us in- with go-to-market outcomes, we take on that burden for you.

And we'll essentially, our promise to customers is we'll help you deliver the highest quality, best value, fastest go-to-market outcome for your needs.

Richie Cotton: Okay, I like that. As long as you're providing the right context and, you got all the evaluations in place, you're not gonna be wasting money then 'cause the AI's not working.

Yeah, at, at that point you should be, yeah, fairly cost-efficient. I guess the flip side to this is if you are serving AI features or AI products, then the pricing model seems to have changed. So we've gone from a subscription pricing to usage-based pricing, and the next one that gets talked about is outcome-based pricing.

Now I know HubSpot has some outcome-based pricing. Do you wanna talk us through what that is and where you can use it?

Duncan Lennox: Yeah, and it's really interesting, and I think I think everybody's still trying to figure this out, right? For what is the right way to deliver value and then get paid for that value appropriately.

And, overall our model is a hybrid model of seats and creds. So you buy a certain number of seats for the platform. That comes with a built-in bucket of credits every month that you can use on things that consume credits, and then you can purchase more credits if you need them.

And the essential idea there is the core functionality to drive marketing or sales or et cetera is the- is there for you. As you get more value out of output and outcomes-based approaches, and you consume credits and you can buy more credits for that. But you're doing that because you're seeing the ROI, you're seeing the value exchange is there.

Yes, it costs this many credits to get a qualified lead or to resolve a customer ticket, and that is, that is fair value to me, so I'll my, my cost of that scales with the actual usage that I'm and outcomes that I'm getting out of it. So as part of that, not everything, maybe even most things, are hard to quantify what is the outcome and the quality level of that outcome to make outcome-based pricing fair in every scenario, so we're still experimenting with it, I would say.

But there's two key agents where we are all in on outcome pricing: prospecting agents, which is about getting you qualified leads, and then customer agents, which we've talked about, which is about resolving support tickets. And we moved both of those to outcome-based pricing back in the spring. So with customer agent, you used to pay for every conversation, essentially a back and forth with a customer, regardless of whether we resolve the issue or not, or whether it ultimately needed to get escalated to a human.

We moved the pricing to you pay for resolved tickets only. A- and that's the 70% or so on average that we're resolving for customers So in other words, if we are unable to resolve the issue to the customer's satisfaction and we need to escalate it to a human, you pay us nothing. And then similarly on prospecting agent we're, we, you can tell us how many, qualified leads a day you want, or you want per sales rep, again, to stay within your budget.

And prospecting agent, which is doing all kinds of kind of pro- complex signal and intent-based prospecting on your behalf, so it's looking at target companies and target individuals and, what's going on, are there signals there that these people are ready to buy your product or services, et cetera.

In that one we used to charge for the number of companies essentially, or contacts that we were monitoring on your behalf, looking for, new intent signal, and we shifted that to, it's only if something hits a bar of a qualified lead and we hand it off to you, you only pay for those qualified leads.

Richie Cotton: Okay. Yeah, I can certainly see how something like writing emails, it's a bit fuzzier on whether it was a genuinely good email or not charging for outcomes is gonna be harder there. So it makes a lot more sense in the customer support case where you can say, "Okay, we've got a resolved ticket.

There's no am- ambiguity." Is there a, a threshold at which point it makes sense? You mentioned that 70% of resolved tickets sort of works for you. I presume if it's only 50% resolved tickets, then the economics are maybe not there. Is there that threshold then that you have to get the agents reliable enough in order to have outcome-based pricing?

Duncan Lennox: It's super interesting because it puts much more of the effort back on us as the vendor to have a very high quality bar. That seems like a very good thing for customers keeps us honest, if you will, right? Because we're gonna get paid more the more of these tickets we can successfully resolve for you, right?

So if we're only at 50% for a customer, the economics for the customer are probably still very good 'cause they're only paying for those 50%, and those are by definition resolved tickets, so they're tickets that a person would've had to deal with otherwise, right? But for us, the more... it aligns incentives, if you will, because getting that to 70%, getting that to 90% is a win for us and a win for the customer.

The customer's getting more of their customer's issues resolved faster and we're obviously generating more revenue from our individual customer because we're doing more o- on their behalf. There's a lot of incentive alignment, I think, around the whole model that where you can fi- where you can define clearly an outcome and a quality bar, I think it works quite well.

It just turns out to be quite complex for things that are more complicated, right? How do you start to define on them? But We're continuing to iterate on it and, looking for opportunities around it as we go.

Richie Cotton: Okay. That makes a lot of sense. I love that it's there's that alignment between vendor and customer.

I guess that's the good situation. All right. So we talked about quite a few different themes around having more AI more automation throughout the go-to-market sort of cycle. What do you think the end goal is, or, like, where do you think things are trending towards?

Duncan Lennox: It's so hard to make predictions with this technology.

Again, it's changing and moving so fast. We very much believe, though, at the end of the day, impressive as the technology is, it's really about augmenting what humans can get done rather than replacing them. And particularly for go-to-market, it's such a human connection-oriented business, if you will, right?

Sales and marketing and customer service. And at the end of the day, I think it ends up being we're creating more time for the humans in marketing and sales or customer support to be able to spend time with their customers, deliver more differentiated experiences, like higher value add experiences, and for both for their customers and for our customers themselves, the go-to-market teams, taking away more and more of the drudgery of the work that they used to have to do.

And this is what we've seen again and again with technology as it comes along and evolves. Ultimately what happens is it automates away some aspects of some roles along the way, but it now frees up time and creates these new higher value roles and opportunities that create better experiences, and that's certainly the road that I think we're on.

Richie Cotton: Okay. That, that sounds like a, a nice outcome if you don't have any really drudge work as a salesperson, as a marketing person. You get to focus on actually helping the customer. Okay All right so just finally, I always want more people to learn from, so whose work are you most excited about at the moment?

Duncan Lennox: I I s- I'm s- I still love nerding out on all this stuff, so I do spend a bunch of time, reading academic papers on what's going on with frontier research and, each of the labs puts out good papers themselves in addition to the academic work that's out there.

But I'll tell you right now, we talked a bunch about evals and quality evals as a key to a lot of this, and I think writing evals, which is a brand-new skill I think is one of the single most valuable things product teams in particular can be doing right now and practitioners generally. And Lenny Rachitsky in his newsletter and in his podcast, has been talking a lot about evals recently, and just had some just gold dust on specifics and experience from the field of how to write really high-quality evals.

I'm really enjoying reading Lenny's newsletter, and that's definitely one I'd recommend to people.

Richie Cotton: Oh yeah, Lenny's newsletter is absolutely amazing stuff. And yeah, evals is an incredibly important topic that's good stuff. All right thank you so much for your time, Duncan.

Duncan Lennox: Thanks, Richie.

I really enjoyed it.