The Data Engine for AI with Ledion Bitincka, CTO at Cribl & Nikhil Mungel, Head of AI R&D at Cribl

Richie Cotton: Hi there, Ledion. Hi there, Nikhil. Welcome to the show. 

Ledion Bitincka: Hi. Thanks, Richie. Thanks for having us. 

Nikhil Mungel: Excited to be here. 

Richie Cotton: Yeah, great to have you here. So today we're gonna be talking a lot about avoiding disasters with AI, but I need some motivation. So what's your favorite AI agent disaster? I'll ask you both this, but yeah, Lydian, do you wanna go first?

Ledion Bitincka: Obviously there's a lot of stuff that has happened that's public in, in the market with OpenAI and Anthropic and their agents escaping. But I'll give you a first-hand experience that we had with a coding agent that we're using internally, which basically just takes in Jira tickets and creates PRs, and all of a sudden it decided that it needed to go and decline all the PRs and rebuild all the work one of the, one of the nights.

And that was fun coming up in the morning and seeing our inference bill Be way up there 

Richie Cotton: Yeah, I'm sure there were some tough conversations that morning. And certainly we're gonna be talking about cost control later on, 'cause I think that is one of the big worries at the moment, is you can get these unexpected price shocks from making use of AI for development.

In general, since there seems to be a switch from humans developing software to agents developing software, do you think it's making software more or less secure or more or less robust? Are things getting better or worse? Nikhil, do you wanna take this one first? 

Nikhil Mungel: I think it's a bit of both, right?

AI agentic coding as a broad concept has really accelerated how everyone is, creating software and producing software, which means is there's been a lot more software. It's also been used for create- making existing software more robust. But what I've seen firsthand is the operational complexity of the software has gone up quite a bit because there's just more code that's being authored, and there's more code that's, in in, in the system today than there was...

Or maybe another way to think about it is the speed at which we are adding new software has gone up dramatically. 

Ledion Bitincka: And that increases, Nikhil, as you mentioned, that increases complexity and also the understanding of how these components of the system were gonna be interacting with one another, and potential security vulnerabilities that could be introduced by how these, these systems interact with one another.

I would say there is a class of problems that AI or security vulnerabilities that AI is making very easy to catch or to, to patch up, especially as they're known in the, in, in the industry. But a lot of it has to do with how humans and engineers are actually using or instructing or the context that they're putting around AI to build software, because you can be building software that's running, let's say, just on your laptop, off of the internet, and it's totally safe and secure.

Opens up all the ports, no login, none of that stuff. And then there's stuff that's running in your intranet that has different security requirements. And then there's stuff that, that's running that's open to the public. So a lot of that, it's not just inherently in AI, but it's inherently in the context that's being provided to AI to build the software.

Richie Cotton: Okay. That's interesting. I guess the fact that we are building more software now means you are gonna get more problems, and I guess there's that layer of abstraction between what was built and looking at the code. I- it's a... The developers are a little bit further away from what actually happened, so it's, I guess it takes a bit more effort to spot things to go wrong.

But it's interesting you say there are different classes of software, different... Sorry, different classes of bug or problem that are either easier for AI to spot relative to humans or the other way around. W- we're gonna be talking a lot about spotting problems, so I guess this is where software telemetry comes in.

Do you wanna explain what is software telemetry? Why is it useful? 

Ledion Bitincka: Absolutely. Absolutely. So software telemetry at a, at, at, at a very high level is- Data and information that any software system, it could be an agent, it could be a main application, it could be a system that's running, that they produce to either log what actions are happening, whether successfully or not successfully, or potentially metrics that aggregate information over a period of time.

So it could say that, "Hey, today 50 users logged in successfully, 20 of them failed," et cetera. And then traces are another component of telemetry which are more... You can think of them as logs in a distributed system. As more- as systems became more complex and more distributed, you going into a website and checking out, that may involve 10, 20, sometimes even 30 different systems that are all collaborating together, and having this one trace one action across all the systems to correlate those logs together is what traces are.

And in the AI world... telemetry was useful for humans to understand how systems are operating were they operating securely or insecurely, or operating as they were expected or not. So system, humans built the systems and they wanted to understand them, and that is even more key now when you have agents that are building software that is running, and then you're seeing a problem after the fact or during runtime.

So it's key for systems to have a lot of telemetry around them because you wanna understand what happened in those systems, and those are only the breadcrumbs, if you will, that the systems are running, and you get to to investigate later on as to what the systems were doing at that time.

Nikhil Mungel: 100%. I think that's a really a good way to put it. One, one other way that, we think about telemetry at Cribl is that it's a, it's an audit trail that your software leaves for, every decision that it takes when it's running in production at scale. Y- how do you trace back, what decisions are being taken by users of the software and by the software itself?

It's all represented in telemetry data. 

Richie Cotton: I love that. So really it's just tracking pretty much everything that happens with the software, and it's gonna help you figure out what went wrong if something goes wrong, or I guess what went right if something goes right. All right. Since it involves lots of data and it's already involves data, do you wanna go into a bit more depth on what the specific pieces of data that you're tracking?

What are some useful sort of things that that get tracked? 

Ledion Bitincka: There's three types of major telemetry components out there. There's logs that developers add to say whether an action was successful or not whether a login was successful or not, whether a system connection was successful or not, or a database connection may have dropped.

These are highly voluminous. As you can imagine, systems are operating 24/7. They're- sometimes tens, hundreds, thousands, or even hundreds of thousands of instances of the same software that is operating. And th- sometimes they may be doing slightly different things, and The format, so the volume is very high.

The format of these application logs is very different from one another. You can be thinking like, "Richie, you build software, you log and you speak in a certain way, your application is gonna log in a certain way." And so do many different companies that, that build software.

So you have high volume, high variety, and in many cases, the value of this data could also be dubious. Sometimes, I- I, I tell every engineer in the organization that, that the value of debug logs is zero, until it's not. And it's not usually around a security event or a, a time when the software did not operate as it was expected, and that's where you need that visibility.

It's like, "Hey, what the hell was happening? Did it connect? Did it not connect? Did this request succeed or not?" So that's one class is logs. Metrics is another big class which operates or which tracks aggregate numbers of what has happened, of actions that have happened. The number of times a user logs in or asks a question of an agent and, the agent responding successfully or a checkout being successful, et cetera.

And then lastly is traces. Traces are very voluminous too, and in many cases, systems out there today end up doing what's known as a sampling of traces because you cannot keep everything stitched together for a very long period of time. And this telemetry information was very high to start with before AI caused this Cambrian explosion of applications that are being created out there.

Many organizations were struggling with keeping up with the costs of systems that manage telemetry and now, we basically are seeing these curves go completely vertical exponential in the amount of data that's being generated. 

Richie Cotton: Yeah. That, yeah, that was really interesting the fact that you've got so many different events now.

I guess when you've got more complicated software, there's so much functional stuff you can track that yeah, the amount of data is now phenomenal. I'm curious as to how you go about analyzing this. So once you've got all these traces, what, what does a data analysis of software traces look like? 

Nikhil Mungel: So just to summarize this you have three big buckets of telemetry data.

You have logs that tell you what happened. You have metrics that tell you, how much of it happened, and then you have traces that are extremely verbose and that tell you that, where all the time went. And these three things really tell you, an analyst the, the answer to your question, right?

So if you're trying to look for where a system stopped working, then you wanna look at logs usually or traces. If you're trying to look at how do I capacity plan for the next quarter, what's my rate of growth, then you're looking at metrics. When you're trying to find seasonality and patterns, that's when metrics come really handy because it's all numerical data that you can, chart and plot And analyze in, in, in that way.

Something else that we are also seeing across, across, across the industry right now is traces have also started to include conversational traces, agentic traces. So when you use a conversational AI-based product like a chatbot, there's a really long, set of messages that happen between the user and the LLM or the assistant.

A lot of these assistants also use tool calls quite quite a lot, and those tool calls are pretty verbose. And all of that conversation is really a single trace if you think about it, and you can imagine how many of these traces are being generated as we speak. Everyone's using AI everywhere, not only in products like ChatGPT and Claude, but also in products like Cribl or, any...

Most enterprise software now has a chat feature, so there is a lot of user-generated content that we are starting to see when it comes to telemetry data. At one point it was just software content, but now it's software, machine-generated logs and machine-generated traces combined with traces that are very rich with- Handwritten messages, right?

With typos and all kinds of things, honestly. 

Richie Cotton: Okay, so it's gone well beyond just user clicked on this button and yeah the kind of, to be incredibly complicated, especially, yeah. 

Nikhil Mungel: It's pretty cool to be able to pinpoint or for an analyst to be able to pinpoint, oh, this is where a user transitioned from being delighted by the product to, a different feeling perhaps based on how the product was performing.

And, that allows you to, take that signal and then make your software be more delightful. 

Richie Cotton: Okay, yeah, I love that. So is that the general goal then in terms of I guess the, the business wins from from tracking things? Is it pure about customer experience or what are the kind of benefits can you get from this?

Ledion Bitincka: Yeah, no, absolutely. You're looking at a different... there's two very high level goals that customers are trying to get by tracking the data. One, whether their systems are operating securely. Do they have any security vulnerabilities? Are people doing what they're supposed to be doing?

Or are actors, threat actors getting access to a system that they should not be getting? And then two, are systems operating as they should and delivering the experience that the customers or that the product managers or whoever built the software is expecting of the users? So this, this latter part is a, I would say a byproduct of the telemetry that is being used for these different purposes.

And again, imagine this is the same telemetry, same data, you're just putting different lenses on it. You're looking at it from a security perspective, you're looking at it from an operational perspective. Are users getting, the right response times? Are they getting the right... Are they going from one screen to the next as they're, as they were expected and like tracking all the, all, all the use?

It's ju- just different lenses on the same telemetry data, and this is why for us it's very important and for customers it's very important to gather all this telemetry data in a platform that are able to ask these different questions of the data. And there's also one funny thing that happens with questions- And answers is that, sometimes the a- the answer also leads to another one or two more questions.

And so you don't, not all questions are all known ahead of time. As you're exploring data, as you're seeing what's happening with your systems and how users are using them you get a lot of insights and, both for product improvements as well as, making sure the products operate as, as they're supposed to.

Richie Cotton: Okay, I love that. So the, there's a kind of big distinction there between making sure the bad things don't happen, so yeah, you're checking for security problems, and then making sure that things behave as expected. The, the product's doing what it claims to be doing, and then also it's giving you inspiration for future development as well.

Nikhil Mungel: We've seen, some really cool use cases that customers take advantage of telemetry data. One, one really cool thing that comes to mind here is when you have, most enterprise software, most business software has a free version or trial version, and then you have paid versions and enterprise versions, right?

So when you have a free version I was looking at somebody who was tracking when you click on the terms and conditions, which is deeply hidden within some settings page. And that is a signal that they might be considering upgrading to an enterprise version and you know, just upsizing their contract perhaps.

So that's a really good signal that can be fed into outbound sales. Maybe an account executive can reach out to them and preempt their upgrade decision or maybe educate them. Another really sophisticated use of telemetry that I've seen is when the software itself uses its own telemetry to make decisions at runtime within itself.

Because one th- one w- one, one way to think about telemetry is it's the closest thing that software has to a memory, if you think about it, right? It... That's how it remembers what, what was happening yesterday, what was happening a month ago. And when use a tool like Cribl to use that telemetry and compress that telemetry into something meaningful, it's something that can base a decision off.

You can modify that into your software itself, so the software can, become self-improving and can decide on the fly as it's, as users are using that software. 

Richie Cotton: Okay. I have to say that's wild that some people are clicking on the terms and conditions on purpose. 

Ledion Bitincka: I think, they are clicking on purpose when they're making that decision, right?

When they're close to making a purchase decision,

Richie Cotton: I love that there are signals from patterns of usage of your software that can then go, "Okay, maybe this person's gonna try and buy a the better version," so you can upsell them. Or maybe it's gonna show they're gonna churn, and you can have some kind of intervention.

So you can really get some feedback for your commercial teams there to act on. 

Nikhil Mungel: Yeah, and usually they're just, poking around and copy-pasting those terms and sending it over to legal, and that's, that's when you know that the contract is, moving forward and they're, serious about using this product.

And there are so many of these things, right? Going into a the, the members space to, invite more members into the organization. There are lots of cool signals that are hidden within telemetry. 

Richie Cotton: Nikhil you alluded to the fact that some of this can be automated so the information that you get from all this telemetry can then be fed back into product development.

So talk me through are you making use of agents then to to just find interesting signals from this telemetry? And then h- how does this feedback loop work? 

Nikhil Mungel: Yeah. The feedback loop is typically taking telemetry and aggregating it over a period of time. You don't want every single click in your product to be influencing any product decision.

So what you wanna do instead is you wanna aggregate telemetry that happened over the last week or the last day, and use an agentic approach to analyze that data. On... You can use a tool like Cribl to take that. If you think about telemetry really, it becomes, it comes down to an information compression problem, right?

You're, you have petabytes of data or terabytes of data, and you wanna take all of that data and you compress it into a key insight of what's happening. And you can use... we have found great success using an LLM-powered loop to compress that massive amount of information by using a product like Copilot Investigator.

Really you can even use Claude Code or ChatGPT to use to connect to your data source using an MCP server, and use, using that agentic loop to compress large amounts of telemetry data into something that's, something that can be actioned upon. And when you have that insight, you can go one of two ways.

You can... A product engineering team that, that can synthesize that insight and, inform what is it that they should be prioritizing next, what they should be building next. Then the next version of that is that insight automatically gets piped into a software factory that I would love for Lilian to tell us a little bit more about, and then the software factory can modify the software on the fly or, propose a change to the software.

Richie Cotton: All right, so you're not just dumping vast amounts of logs and metrics and traces directly into Claude Code and saying, "Hey, figure out what to do. Go and improve my software." I... There's intermediate steps are like, "Let's try and pull out some signal from this and then find some insights," and then those get fed back in.

Nikhil Mungel: Yeah, unless you wanna spend billions on tokens. 

Richie Cotton: Yes. We... A, a very expensive mistake there. Your CFO's not gonna be happy with you. All right. Software factories. Lilian, tell us what are these things? 

Nikhil Mungel: At a... Generally at a very high level, I'll start with 

Ledion Bitincka: what is it that humans or how is it that software is built today, and then kinda take that to the factory model.

Software generally starts with a requirement or a bug that has been filed, and some engineer's gonna pick up that, that work try to understand what the ticket is, understand what is being required to be built, map that with the current system, with the current knowledge of how the system currently operates, and either add a feature or fix a bug.

Make code changes, run tests, make sure that everything is passing, all, you didn't break any with the existing functionality. And then at the very end do what's known as the acceptance criteria. Did we actually solve the problem that ticket originally started with?

With LLMs being very good at distilling and synthesizing information, if you have a system that is set up correctly, so basically the factory becomes a, an agentic system that is able to take this ticket that came in, marry that with the context, including the code base of your system, as well as architectural knowledge and existing information about how the system operates.

Because there's, the same thing that, LLMs cannot chew through telemetry, they cannot chew effectively through massive code bases. It's better to have just like any, any human or any engineer onboarding an organization, we have, very good documentation and very good architectural information about the system.

You can feed that to LLM, so now you have ticket, code base, context, and you can feed that to an LLM with instructions on, how your organization builds tests, validates and tests the acceptance criteria. And all of a sudden now you have a system, a software factory if you will, that starts with a description of a problem or description of an updated feature that you want, and at the very end you get a proposed change set to your system That is, internally we're seeing, 95, nine- between 90 and 95% success rate on what comes out the other end.

Obviously, there's still, human adjustments that'll need to happen at the, at, at the very end. As I mentioned, it, it's that factory that ended up deciding to decline all the PRs. It's not always perfect, but, it's the same thing even with with human engineers.

That's why we have reviews and all that. But it's definitely a great way of accelerating how an organization works. 

Richie Cotton: It sounds like a lot of the things the engineers are now doing then in the software virtual model, you mentioned y- you gotta provide the ticket and the the code base and then a bit of context on what you're doing.

So it's really about telling what should be done and then the agents are focused on, like, how it should be done. I- is that about right? That is about right. And I think,

Ledion Bitincka: engineers at this point, the way that we're seeing... Where we're seeing engineers having tremendous value is on the sides of this factory, and specifying having the right specs go into the system, and then also at the very end, checking that what was built was correct.

That is where, senior engineers were adding a lot of value before, is that back and forth with the product managers to figure out, "Hey, are we building the right thing?" Not all requests and not all bugs are actually bugs. They're, they may be intended behavior that just somebody, didn't fully understand what they are.

So there's a triaging component that happens there. And also at the very end is like, did we actually solve the problem the right way? Was it the simplest way possible? Does it fit with how we build things versus just just building something? 

Richie Cotton: Okay most of the value's being added at the start of this process and at the end then.

So what gets built a- and then what, was the right thing built, so I guess the whole verification and validation. What skills do you need to be good at these parts then? So if you're interested in a career in development what should you be honing what should you be learning about?

Nikhil Mungel: Humans are You know, moving away from being knowledge workers to judgment workers, your humanness and your, the detail, the attention to detail, and, really understanding what is it that the business needs, what is it that, what is it that needs to be produced in order to, move the business forward or, connect, make that connection with the real world is still very much, it's it depends on a human being somewhere, and that's really what the job or most of the tasks have transitioned from, being from knowledge work to judgment work, where you're assessing, sometimes you're verifying, sometimes you're testing it, you're critiquing what was produced by the factory.

And that's a, that's a-- I guess, it uses a different part of your brain when you're authoring something versus when you're judging something. And I feel like as a, as an industry, as a software development industry, we're-- everyone is recalibrating, to what that, what are the needs, that new role, how do you upskill to become better, more effective evaluators of what's produced by a factory versus, producing it yourself.

And I st- I, it's, it comes naturally to a lot of people, right? It was a part of your job even before agentic coding took off. You were always evaluating and reviewing code that was written by your peers, and then interspersing that with authoring your own code. And now it's mostly evaluating and judging the work of a factory, which has been a step change 

Richie Cotton: That, that's very interesting, and I like this term being a, a judgment worker.

But the idea of being judgmental I think it's got some negative con- connotations . But it's now it's oh, being good at complaining is is now a superpower. I like that. 

Ledion Bitincka: I think it's the part of engineering that does not go away is problem-solving. I think this is what most of

If you think about most computer science, software engineers that, that have that are out, they've been out of school they've taught or they've learned about algorithms and distributed systems and da- databases, et cetera. And then they end up spending most of the time dealing with the nitty-gritties of how, "Hey, how do I get JSON to XML?"

And more minutiae work or toil work. And what AI basically gives a lot of folks is the ability to stay at that higher level of problem-solving. And this is where the judgment comes in. If you don't know how to solve the problem right, there's no way that you can judge whether somebody else whether it's a human or a machine, has solved the problem correctly or not.

So I think my advice to most folks out there is gonna be stay focused on the problems and honing in those problem-solving skills, 'cause those are things that are not gonna go away. What's gonna go away is the, the nitty-gritties of specific operations. How do you map one thing to another, or how do you make an HTP request and all those things.

Richie Cotton: Okay, yeah. Problem-solving seems to be incredibly important. In life as well as in, in software engineering. It's interesting that you both picked something, the kind of soft skills, the idea of problem-solving, of critical thinking and judgment. Are there any technical skills that you think are becoming more important, or is it just like that changes every, every three months?

Ledion Bitincka: First of all, you have to be good at being able to use the new software stack or the new software building stack, in the early days, you started with, knowing how to use, VI or Emacs or code editors, and then there was the IDE. And now you have, you know- Fast forward, there was Cursor, same as VI- or in VS Code, a similar experience.

Now is being able to use these, to use the agents and be able to interact with those, those agents and wire and stitch those things together. It's almost like, you're trying to figure out how to implement your idea a lot faster by delegating. And I think, as you mentioned there's some softer skills in there as well.

You're delegating to two or three agents. "Hey, you go do this small piece, you go do this small piece." But back to the problem-solving, is being able to... what we're seeing so far is agents are great at when they're confined. You have very well specified inputs and very well and expected outputs.

So being able to take the problem, decompose it, and then assign it to different agents, and then stick, stitch the, the results together, I think is key. But, for the specifics of the technology, those will change. They may be irrelevant by the time that the, the video comes out. That's how fast this space is moving.

Richie Cotton: That's amazing. So yeah I do love the idea of delegation as being an important skill now. You are the boss of agents yeah, you gotta be able to outsource your decision to, or ou- outsource some of your work to

them. At the start, we touched on the idea of things getting unexpectedly expensive. You talked about an agent that ran amok and tried to rebuild the entire code base. I'd love to get some tips on how do you avoid these problems of unexpected bill shocks when you work with agents? 

Ledion Bitincka: I think one of the, the...

C- I'll tie this back to, to telemetry It's not always "Hey, we're spending too much on XYZ, on tokens or on XYZ system," because, you may not be-- you may be getting the value out of what you're being spending. What's interesting to look at and what's important to hone into is waste, and identifying waste requires you to be able to analyze some data and analyze analyze telemetry.

And that, you can't do that unless you're able to capture this data be able to bring it into a system, and be able to analyze it. And to frame questions in the right way of what is the biggest spend in this particular area? What agents are causing the spend? And then diving deeper and trying to understand, are they using the wrong model?

Th- this is one very common thing that we see, and I think in the industry it may also be known as a, model anxiety, where every human is picking up the most expensive model to ask questions of. And, to a certain extent, it makes sense because, if you're not able to solve the problem, you think it's a hard problem, therefore, you want the expert, you know, th- the most expensive model to ask that question.

But, having a way to also classify these problems and also educate the entire employee base on, what are problems that are easy and you're just passing the toil. "Hey I'm looking at, reformatting this file into something else." That should not require a very expensive model versus the, "Help me design this distributed system," and that's where you're maybe wanting an expert system to go back and forth with.

But telemetry is key in all of these because you wanna be able to come back with an informed decision or with informed knowledge back to employees to decide, do I need more training? Do we need more guardrails around s- specific agents? And so on and so forth. And it's key to, again, bringing you back to the data.

You've go- you have to be able to collect the data, interrogate the data, and get to those insights. 

Richie Cotton: Okay, yeah. So it sounds like a lot of it, again, is about run telemetry, run... Track where you're spending money, and then just think, have a think about, are you actually getting value from this, and if not, then you change your mind, do something different afterwards, so you got the, these feedback loops in place.

Nikhil Mungel: One of the skills that is becoming important, in the age of agentic coding is actually focus, or being... Staying focused on, what is it that you wanna accomplish, because it's very easy to just spin up new projects, right? It's very easy to just ask Claude to, go this and then go do that.

And I've certainly fallen trap to this where I've, asked her to do five different things and then lost interest and, moved on to something else. And I think it's really important to really hone in on that perseverance and that focus to, "Hey, let's, let me finish what I've started," because it's so easy to get started today.

And that's I... That has to be one of the contributing factors to, tokens costs and high token spends when people are just exploring is let me do this, let me do that." I read a really funny take on Twitter the other day, which was Claude Code is like Unveil for parents in their 40s, which is just, go and go do things.

Which is a great use of, it's a good recreation, I guess It does. You have to pay for that. 

Richie Cotton: Yeah, I can certainly see how one of the biggest signals you're actually getting value from i- is did you actually look at the response from the thing you asked it to do?

But this is tricky. I had someone pitch to me recently that ADHD is like the new superpower because you've gotta keep multitasking and you're looking at seven different conversations at once. Is there a sensible way of managing this? 'Cause I think it is very easy to start seven different conversations and not come back to them.

Like, how do you manage how many different things you're doing at once? 

Ledion Bitincka: That's the very personal, I think for everybody on how... I think AI is amplifying both the good and the bads of every, I wouldn't say personality trait, but e- or every way that folks are working. If you're somebody that's drilling deep and have a tendency to go into the weeds, AI is there for you.

It's gonna help you dig that hole even deeper. If you're, a person that is able to, have 17 conversations or start seven different projects and never finish one now that could be 70 different projects where you can start and kinda get the hit from that. But I think back to what Nikhil said is staying focused on the, on task and having...

achieving those goals and kinda getting the high, in quotes, from hitting the goal. "Hey, I was able to deliver X thing You know, whatever project two times faster. There's a saying, you know, inside the organization that, that I push in, in the, in Cribl's engineering organization is shipping value, 

Richie Cotton: right?

I don't care if we made progress on 10 things during the month 

Ledion Bitincka: and we're 80% done in that month where we're gonna go and ship something to the customer, we haven't shipped value. We've just made progress. But instead of that, if we're able to do eight of those things, get them done 100%, ship those, and then 0% pro- progress on the other ones, I'd much prefer being able to ship something and get something out to get contact with reality than just make progress on a number of things.

And I think a lot of this is also culture and how the organization is set up. And again, back to the soft things that they were talking before is, that needs to be ingrained in the culture of an organization and shipping value and getting value out to customers. 

Richie Cotton: So this is one of those things where it sounds really sensible when you say, "Okay, we've gotta ship stuff that has actual value."

I think often a lot of people, they don't know how to calculate the value up front. Do you have any tips on either re- on how you approach calculating is this thing I'm gonna do have some value before you go and do it? 

Ledion Bitincka: Yeah, so I ... it depends on a lot at what stage a company is in. If you're early on or if you're in a startup, I think at that point you're speculating.

There's a lot of speculation in whatever you're building. So the closest thing to value that you're gonna be able to get there is, hey, I've had 10 conversations with 12 customers, and 10 of them have been really interested in getting this or in, in buying this. Not just they think it's a cool idea, but, they wanna be able to buy the thing.

They're pre- they've even ... Some of them may even ask the question, "When can I get this?" As companies mature and as products mature, you can start to attach value to them from, how many customers have asked for this, what has their, what is their revenue like? And as Nikhil also mentioned before, y- you can use telemetry to identify what customers are not achieving.

Sometimes they may be dropping halfway through a particular workflow and you know they didn't achieve the goal, but you don't wanna act on every single click, for example, right? So you wanna aggregate and you kinda get... You can use telemetry to guide you to areas that you may be able to improve your product, and you can say "Okay, this has 17 customers that are not getting value, and this one has two."

Guess what? I'm gonna be going with the one that... I'll- let's build the 17 one first and then the one with 

Nikhil Mungel: two. 100%. And just to add to that, value can... The easiest way to measure value is just by growth, and this audience is probably all, people who are very familiar with data and with telemetry.

You can find out what growth is it that you're tracking. Is it growth of ARR? Is it growth of, new logos? Is it growth of a particular metric like DAUs or MAUs? Is it, w-what's... There, there's usually some metric that we are trying to grow as a business, whether it's time spent on your application Actually, you gotta be a little careful with the last one.

You can use a thinking model and, increase the time people are spending on your application without actually driving value. Getting them to use more workflows within your application is a good, it's a growth indicator because you're... If they're spending more time in the application, they're conceivably getting more value out of it.

Find something that you can grow, maybe it's time to value, and then or reduce time to value in that case. But, that's a really good way to measure whether you're shipping value and whether you're shipping outcomes and not just sticking to shipping outputs. 

Richie Cotton: Okay, I love that.

Beforehand, you gotta do a lot of talking to customers saying does anyone actually care about this?" And then afterwards, you're checking has anyone actually measure- has anyone used the feature that I built? Has anyone used this new thing? 

Nikhil Mungel: And then you wanna... one, one thing that we love to do a year at Cribl, you know, since we are a...

A, a lot of the data that we work on is time series data, which is, something that's monotonically generated by a system for, with a timestamp. We love time charts at this company, right? Everything that we do really presents itself as some sort of a time series diagram, which makes it really easy for us to, measure growth, whether it's growth of bugs that we need to reduce whether it's growth of, new revenue that can be attributable to a particular feature that we ship.

It's not just measuring something, it's measuring something over time and looking at the trend, whether it's trending up or down, and how do you want to bend that slope. 

Richie Cotton: That's fantastic. I have to say, it always amazed me from my background in data science, I think, oh, there's gotta be... There's all kinds of different analysis you do.

A lot of business analytics is just looking at time series like, did this metric go up or down? It's amazing how much how far you can get with just that one technique. So we're talking about things that add value, and when I was researching this podcast, I was going through your website and most of the products you make, all related to telemetry, like really obvious.

There was one pa- one product you make that kind of stood out as being a bit weird or I di- I was like, you've got a custom app builder on top of a telemetry platform. Do you wanna talk me through why the Cribl apps exists? 

Ledion Bitincka: We're entering an era right now with AI that, Builders are going to be the ones that are going to be getting the most value out of data and out of systems.

What used to be m- subject matter experts before, they may have had an idea, "Hey, I wanna analyze the data this way. I wanna be able to build this specific workflow that makes sense for me, that makes sense for our organization," and they were simply not able to do that. They were not able to do that because, A they may have lacked the skills of building software.

Two, their use case might not have been as common or might n- in some of the cases, the, the product managers, all the system that they were built, that they were building on top of, maybe they're just not simply aware of. And so in many cases, you'd go in and file a, an enhancement request to the vendor, and sometimes you're gonna get prioritized in the next six, nine, 12 months or whenever it fits in the roadmap.

So what we decided to do at Cribl is to turn that on, on its head and give power back to the users or back to the subject matter experts on this data and on the systems to build workflows and to build applications on top of the telemetry data that makes sense for their organization. And we would provide the building blocks for that.

Starting from the systems and being able to gather telemetry, to route telemetry, to store it, to query it, to visualize it. But then when you stitch that together, users of any organization or any e- large enterprise, sometimes you may even have teams within an enterprise that don't operate the same way.

They could be teams that have been acquired, right? They could have completely different tech stacks, and therefore one needs to look at data in one specific way and kind of act differently than what another one was. And so combining AI with a strong telemetry platform, I think is...

gives customers this magic combination for them to achieve their goals. 

Richie Cotton: Okay. I do like the idea of not having to wait for engineering teams to sort out the problems. So one of the big trends in data analysis has been the idea of self-service analytics, and I guess it's the same thing, so you can use self-service telemetry analytics as well.

So I guess related to this, so this is like a, a custom app platform. Now, at the start of the year, we had this whole idea of the SaaSpocalypse, where no one wanted standard software anymore. No more standard SaaS platforms like Salesforce or whatever, and everyone's gonna have personal software.

So how personalized do you think software can get? 

Ledion Bitincka: I think it it'll depend, right? I think it'll depend on on a lot of different use case. I think there's gonna be software that is going to be running specific workflows in specific organizations. I think that would have to be personalized.

I don't think it's gonna be personalized as within a team I have a version of Salesforce and Nikhil has his own version and Richie has his own version, 'cause then we would not be able to have the same conversation. But I think it is possible that within a particular within teams or even departments, you can have a platform above which you're, you are building the application to me- meet your needs.

Like internally, for example, our support organization is building on top of this, of the platform that they're able to get information from customers, we call them diags, and they're able to analyze those. And as the customer puts in the case, there is an agent that's running stuff on, in, in the background.

Now, that so far makes sense for our support organization No vendor out there would build that just specifically for ours. You can have, you could talk about Salesforce or Anthropic or OpenAI. They have different use cases. So the custom software market is huge, and a lot of people don't don't see it because this is software that is built by companies for themselves, so there's no vendor in the middle.

Sometimes we'll be having consultants, but it's in the trillion dollars size already, even though we don't see it. So I think it will be personalized and more personalized than what it is today. 

Richie Cotton: Okay. That's fascinating. I had no idea this was a trillion-dollar industry. That's a lot of money.

I guess when everyone's doing ca- their own custom things, maybe it's internal, you don't see it, but yeah, there, there's a lot of that going on. All right. To wrap up, I always want more people to learn from, so whose work are you most interested in at the moment? 

Nikhil Mungel: Yeah. I-- My one of my favorite authors is Jocko Willink, is a former Navy SEAL who writes a- writes about leadership and, how do you show up for your team and your teammates and your your organization.

And my... The, the one of the favorite books that that I've read by him, he's written a few books, is Leadership Strategy and Tactics: A Field Manual. And I feel like in today's age, when everything is agentic and, we all have several agents working for us, it's just so important to demonstrate what solid leadership looks like, both, for your team, for the people you work with, for the people you work for your customers.

And I highly recommend that book to 

Ledion Bitincka: everyone. 

Richie Cotton: Okay. That's fascinating. I hadn't really considered that there was a, an overlap between being a Navy SEAL and developing software, but l- leadership skills are strong in both cases. 

Nikhil Mungel: It's leadership, right? Ultimately, most of the leadership principles that we use in corporate America stem from, the armed forces, and it translates really well, actually.

That's absolutely fascinating. I'm now wondering if we can do special tactics and fun assignments and at the computer. 

Ledion Bitincka: I'll be following more d- a lot of developments in, 

Richie Cotton: especially in the in the hardware 

Ledion Bitincka: side of the house. From, AWS's, custom silicon to even NVIDIA's.

One of the areas that I'm very passionate about is I think The it, Cribl software runs all on or at least the service that we provide runs on Graviton, on Arm. As we look at, from a very high level, the, the cost of the energy cost that's needed to perform teraflops of operations, GPUs are an order of magnitude more efficient than CPUs.

However, there's problems with the, the memory bottlenecks on being able to get software to get data, at least in the space that we are, to get data l- large enough chunks of data into a GPU set. So I'm follow- following Nvidia and all the, the players in the space to see, when that, the threshold kind of crosses.

Because there is a lot of benefits for running telemetry systems on GPUs, and because of the cost, right? The, the cost per per compute unit is very economical. And, the, the growth of t- the growth of software is going up, growth of telemetry is go- is g- data is gonna be going up.

So I'm following these, these vendors pretty closely in the developments on memory cost, memory throughputs, and all that. Even though memory cost right now is going through the roof, it's doing the opposite of what you'd expect. But yeah. 

Richie Cotton: Yeah. It's fascinating. One of the benefits of the AI boom at the moment is like it's all layers of the stack.

It's like from software right down through yeah data centers and chips and power and things like that. So yeah, fun stuff at all levels. And yeah time series on GPUs, that's gotta be like a fun s- new sort of area for analytics. 

Ledion Bitincka: GPUs are great at crunching numbers and crunching, string data, which is, which what telemetry is all about.

Obviously you can also follow, data centers in space too. You could throw telemetry up in space, but that's a topic for another conversation. 

Richie Cotton: Yeah. We'll bring you back in a few years' time for the telemetry in space conversation. Nice. 

Nikhil Mungel: I'll make a, I'll make a quick plug here. I'll make a quick plug here based on what Layan just said.

Keep keep an eye out for, we have some cutting edge research that we are gonna publish soon on, processing telemetry using models that are running on GPUs and even on CPUs. So we should have that up on archive in the coming weeks. 

Richie Cotton: All right. Super.

Yeah, we'll get that added to the the show notes once it's live. Excellent. All right. Thank you so much, Ledion. Thank you so much, Nikhil. Great conversation. 

Ledion Bitincka: Awesome. Thanks, Richie. This was great.