Machine Learning for Ride Sharing at Lyft, with Rachita Naik, ML Engineer at Lyft

Adel Nehme: Rachita, it's great to have you on the show.

Rachita Naik: Yeah. Hi, thank you for having me. Yeah, it's, it's great to be on the show as well.

Adel Nehme: So you are a machine learning engineer at Lyft, and you've worked on many use cases in the ride sharing space. I'm excited to see how machine learning is impacting ride sharing, so maybe could you walk us through how machine learning is currently used in the ride sharing space?

Rachita Naik: I've been a machine learning engineer at Lyft for about two years now, and I can confidently tell you that it's widely leveraged for several use cases across ride sharing I think like Uber, Lyft, even DoorDash, they have several technical blogs about the different use cases that teams are working on.

Even at Lyft, We have teams working on some really cool stuff. So I guess the most important and major use case of machine learning has been the ETA prediction problem, which is the estimated time of arrival. That's basically the duration that riders see on the screen telling them when they can expect their driver to arrive.

But while riders just see that time estimate on the backend, there are like routing algorithms that need to. Crunch through like historical right patterns, even like real time traffic and weather conditions to be able to predict this critical estimate as accurately as possible. And as you can imagine, these algorithms need to adapt as dynamically as possible.

 Be it like to long term pandemic induced disruptions that we saw like a couple of years ago Or even like more short term, like on a daily basis, you have road closures or traffic accidents, et cetera. So I would say that's the most important use case, which we're still actively working on making more accurate.

But then there are also like teams working on dynamic pricing and demand prediction problems to be able to leverage Search pricing during peak hours, to balance the marketplace demand and supply conditions. We also have teams working on safety and security enhancements. on the ride sharing platform because with AI, are able to power easy verification for riders as well as drivers, or even like when riders are completing their ride, we can flag certain deviations from expected path so that we ensure that the riders.

Feel safe and comfortable as they are getting to their destination. Yeah, I think these are like some of the distributed use cases across the company and with the recent Surgeon Gen AI we also have a lot of NLP powered Use cases coming up especially with respect to like automating customer support. 

Adel Nehme: Okay, wonderful. And we're definitely going to unpack a lot of those use cases as well as how generative AI can be used at Lyft. Maybe let's deep dive into ETA prediction. I think it's kind of related as well to route optimization and how to think about that, can you take us maybe behind the scenes and explain to us what a typical pipeline looks like for an ETA prediction algorithm?

I'd love to see what type of data goes in. What are the nuances that we should think about? Because I think for a user, what they just see is, you know, five minutes. six minutes, right? Something along those lines. But yeah, walk me through kind of the complexity behind the scene that comes up with coming up with such a number.

Rachita Naik: I think there are different definitions of ETA when you consider the time series of a ride. So before the rider requests the ride, they also see an ETA when they like enter their destination, they see a bunch of options with the price and time estimates. So that ETA is actually an estimate on our end. 

When we don't even have a driver assigned to the right. So, in that phase of the ride, we are basically trying to estimate, okay, if you were to request a ride, what would be the expected time that you could expect the driver to arrive in. So at that point, since we don't have Any driver assigned we have, like, these dynamic marketplace conditions.

That prediction problem becomes a little more complicated since we have, all these variables, given that the rider has not even requested the ride. But once the rider requests the ride, once we assign a driver, that's the ETAs that, riders are, like, most concerned about, like, is my driver moving towards me, Can we expect them to arrive on time?

So we actually have a very team at Lyft that works on this whole stack, right from like routing to the EPA prediction problems. So if I were to like walk you end to end, then at the bottom layer, we also like have our own navigation system called LiftNav. So the mapping team basically curates all of this data in house using signals from driver GPS sources or even like open street maps, which is like this open source community that curates like these mapping databases.

So The mapping team basically is in charge of managing all this data, which contains, all the points of interest and road segments that Lyft needs to be able to build routes optimally. So this is the mapping layer. Then on the layer on top, we have, more real time signals associated with drivers, like their exact locations, real time speeds any traffic and road closures, like real time events, which help us analyze, like, The congestion on these roads.

And once we have like this data model, we're able to like convert it to a graph model to like run DTRA search to build routes optimally from say, point A to point B. So once we have like computed these routes, we have the ETA layer on top of that. So the mapping team has built really sophisticated ETA models to be able to predict this time accurately along the.

routes that we calculate given a certain driver. And as you can imagine, like we show the driver a route and we provide them the EPA, but the driver could end up taking another route, like they may take a turn. That's not, they're not expected to. And so like to be able to handle these.

real time updates we need to recompute the route and like recalculate the ETS on top of that. So we have like a final bias correction layer on top, which basically trains dynamically like every 15 minutes to be able to capture this error between our predicted estimates versus the factual.

So that helps us to incorporate like quick feedback loops into the system. Which will, like, account for any, like, sudden road closures or traffic accidents.

Adel Nehme: So there's a lot of, a lot of nuances here to the pipeline that go beyond just calculating the distance from A to B. And I think this segues into my next question. Set of questions, which I think pertain to the challenges really of building machine learning systems that are used by millions and millions of people.

you mentioned here the ETA prediction algorithm, but also dynamic pricing, demand prediction, thinking about supply and demand conditions and how to, allocate drivers to rider, et cetera. These algorithms are all operating within the Lyft application, but Lyft is being used by millions and millions of people.

So these algorithms are being called. In any given day, millions of time, so which must present a unique set of challenges, So maybe what are some of the biggest obstacles you faced in building these systems at scale? How do you approach solving these, challenges to ensure reliable performance and accurate performance of these algorithms?

Rachita Naik: you're right. Like, at the scale that we work at at Lyft where there are, so many services interfacing with each other to, like, be able to serve hundreds and thousands of requests per second. I think latency and scale are primary concerns. Like we want our system to be as reliable as possible.

So that's our utmost priority because it impacts user experience and loyalty. Like we want riders and drivers to return to our platform. we don't want any like single points of failure in the or at any point in the pipeline. So since we have a microservice architecture at Lyft, like we ensure that we have proper fallbacks at, both client and server side, across all services so that.

we can deal with transient errors or network blips that occur from time to time. Even like whenever we roll out any feature or a new product we ensure proper shadowing and testing. no deployment is directly rolled out at a hundred percent to the users because we don't want to hamper their experience in any manner.

 Talking about modeling specifically, like if I were to speak about building ML models, I think one of the common challenges that we've faced is like teams develop and prototype models offline, test out a new feature or an entirely new model for a certain use case. And they observe like really great performance on offline data.

But once you deploy the model online, there's some disparity in like these metrics and it does not translate to the same. Performance gains in the online world. So trying to debug and investigate that. That's one of the challenges. we also faced eventual model degradation over a period of time.

I think that's very common in the dynamic marketplace right sharing industry provides because you have these data drifts all the time. solving these problems, I would say is an iterative process because you encounter these problems and then brainstorm on how to come up with solutions.

But we also have solid infra and ML platform teams at Lyft that are able to like help out with a lot of these issues related to like deployments or ML infra are related to our services. And just talking to colleagues who have worked on similar problems in the past, that helps you gain perspective.

Adel Nehme: Wonderful. So let's maybe unpack those. One thing that you mentioned here is the disparity between online and offline data when training a model, I think this is something that's actually not talked about lot as a challenge in machine learning when you are, training your models offline, experimenting in a notebook on your local machine, and then you deploy it into the wild and you start seeing quite different performance in how the model is behaving.

So maybe first unpacking, why does that happen? And then two, how do you approach solving it?

Rachita Naik: I think debugging that can take a while because I think you need to take it step by step, like unpack several layers to identify where the root cause might be. But if I were to say like one of the common problems is that you basically compute your features offline for training your model.

Like you do a bunch of feature engineering steps use SQL to pull data from like different data sources and then feed that into your model. But you. Basically need to, recompute and, like, recalculate all of these features in the same way during online inference. So, you need to have that sort of parity between your feature sources, because if your feature source online has, a different data distribution as compared to, like, your offline analysis, then that I can, like, definitely tell you.

Will impact the model performance. At least when you're like deploying the model initially, you don't want any disparity in your feature computation steps. And I guess just having regular feature validation and data validation during online inference helps because sometimes a lot of these pipelines, which are computing features of online can like.

You know, break and like result in like null values or just all zeros, for instance. So you need to be able to intercept that in time during like serving these online requests. So the way Lyft tries to resolve a lot of these issues, we want to avoid this disparity between online inference and offline analysis.

To a great extent. And so the data platform team has actually built the centralized feature store at lift for serving ML features. So we have both streaming and batching pipelines to serve like real time and historical features. And then we also write these simultaneously to offline data sources so that even during offline analysis, you're basically using the same set of features computed using like a single source of truth.

Adel Nehme: Wonderful. And then you mentioned something as you deploy online and you realize that the results may not necessarily match with the offline training. How do you ensure as well that you're able to roll back any update that you've done in the machine learning system? So I'd love to see what are the safeguards, the guardrails set in place to be able, in case you deploy a machine learning model that behaves incorrectly, that you're able to roll it back and not harm the user experience.

Thank you.

Rachita Naik: I think rolling it out seamlessly is a priority for sure. So what we do is whenever we like offline prototyping, then we deployed the model to production, we don't enable it at 100 percent to all users. So we usually shadow the model. In the online world. So we have the production part and then we have a shadow execution part so that we are basically able to Analyze the performance of the model. before we deploy it at 100 and that helps us compare the parity between the Analysis that we had and like the online model in using production data and once we've confirmed The model performance by shadow analysis. We usually set up a B tests to have another layer of validation to test this new model and treatment versus an existing system and control.

And once we decide to ship. Based on like business metrics, if we end up improving them, then we don't deprecate like the existing stable version of the model immediately. Like we usually have it around for a month so that we are basically able to roll back to the previous stable version immediately if the new model creates any issues, but usually by then we're like pretty confident that this model will work in the online environment once it's rolled out to a hundred percent.

Adel Nehme: and when you mentioned that you've done A B tests, and I think that's wonderful. But when you look at A B tests here, what's the main objective? What's the main metric that you're measuring? Are you measuring some form of metric that relates to how users are, you know, depending on the use case, of course.

But is the metric usually an accuracy metric or, you know, a technical machine learning accuracy metric? Or is it sort of like a business metric, like, activations of users, ability, you know, customer satisfaction? Like, what's, what's the main metrics that you look at in terms of like the effectiveness of a machine learning model?

Rachita Naik: we ultimately want to see how it translates to business impact. So while like model accuracy is important and we usually like monitor that as we shadow during these experiments, we basically monitor business metrics and test the impact. suppose we have a model that improves this prediction problem, then we would want to monitor rider cancers or like.

The number of completed rides should be, statistically significant. So we have like a bunch of guardrail metrics, like, the top line metrics that we did not want to degrade at all. So like, depending on those guidelines, we sort of make the ship decision.

Adel Nehme: Okay, great. And let's maybe jump to another challenge that you mentioned here is data drift, which I think is really common in the online inference space. But also, I think every organization that had a machine learning model in when COVID happened encountered data drift, for example, because, A lot of norms changed and how we approach so many different things in the world, as a species after 2020 happened.

So maybe, how do you approach data drift both in detection, but also how do you approach detecting data drift within your model, but also how do you alleviate it and solve it?

Rachita Naik: I think data drift is a given since these commuting patterns and ride share, dynamically. And that's why, we want to be able to intercept any anomalies as quickly as possible. So we leverage a third party tool to set up these feature checks on the model inference site to ensure that we can manually intervene and investigate as needed.

So these checks could be like simple checks, like, you know, ensuring feature values always are between like a certain min and max threshold or within like three standard deviations from the mean. So simple checks like that, depending on the feature distribution can be leveraged. But since this is a given, like we usually set up periodic automated retraining with the most recent production data, because That's like the most recent relevant distribution that our model should be trained on.

So we set up like dynamic data collection of the production, like rides data essentially in offline tables so that we can then like run some scheduled cron job to retrain the model periodically to mitigate like any of these performance declines before they even happen. I guess, like, we also have model performance monitoring dashboards and alarms in place so that these alerts trigger whenever the performance drops below, like, a certain threshold over, like, say, a two day or a three day period.

Adel Nehme: I imagine another challenge here, which I think we didn't touch upon yet is, Especially when an application is used, by hundreds of thousands, millions of people at the same time is latency, the ability to provide almost real time inference at scale, It must be quite a big challenge because it can make or break the user experience.

If I need to wait three minutes to get the ETA of a prediction, I'm probably going to cancel the ride before I get the prediction. So what are the main challenges associated with latency? How do you make sure that you're able to deliver realtime inference for so many users?

Rachita Naik: so, latency is definitely like a primary concern at Lyft. And since, as you mentioned, a lot of these golden parts need sub second inference we need to be mindful of that, right from the prototyping phase. So you can imagine that when we're having these discussions around model architecture and what sort of model do we want to build in production we need to make that trade off because you could have like a complex deep learning architecture for your models that can result in really high accuracy.

But then if you want that sub second online influence loading these heavy models and memory per request that can get expensive. So. Then you have discussions around, like, can you cash some of these requests? Can you have, asynchronous model calls to be able to serve these requests in real time without, impacting any user experience?

So, depending on the latency budgets, we make our modeling decisions. And even, like, when it comes to the features that we use for the model, I think we often, like, Forget that, feature computation in the online inference request also takes some time. So if you have real time features that you want to use in your model, then these streaming pipelines that compute these features, every five minutes or every 10 minutes, that can get expensive, very quickly because of course, like these streaming pipelines need to run SQL queries on that volume of data every five minutes.

So, being like. mindful, like having that consideration in place when you're prototyping, I think that is also important. we have some strategies in place of course it depends on the use case as well.

Adel Nehme: Okay. And you mentioned here it really depends on the model architecture. I'm curious to see what are kind of, of course, the old adage remains the best model for the job is the model that performs the best. Right? It's not like there's a preference on the model, but what are the class of models that Lyft usually uses for these types of use cases such as ETA, prediction, et cetera.

Is it deep learning based architectures? Is it more, time series forecasting. I'd love to see the class of models that are used here.

Rachita Naik: Yeah, so, I think we use like a lot of tree based models for a lot of our use cases because we've often noticed that for these structured rightsharing, data sets like pre based classifiers are regressors. They are like super powerful. They are able to like capture complex, like nonlinear relationships and are at the same time, they have more lightweight and interpretable as compared to deep learning.

But then like the mapping team, I think uses a lot of deep learning models, like some computer vision even for ETA predictions, I think they're migrating to deep learning now. So we have a blend of both, but I think we also have like some reinforcement learning, in fact, at Lyft for like some of these pricing problems.

So I would say like, yeah, it depends on the use case at Lyft.

Adel Nehme: Yeah. I'm curious to see now, how's, reinforcement learning used in pricing? because you don't hear a lot of reinforcement learning use cases in production. So yeah, I'd love to see how you've approached it.

Rachita Naik: since we need to be able to optimize for multiple objectives in a dynamic environment, they basically employ reinforcement learning to be able to like learn through trial and error Through constant feedback. So, basically they try to maximize overall objectives like Profit customer satisfaction, even like driver availability.

So, the actions in the reinforcement learning system would basically involve selecting these pricing multipliers because you are trying to implement search pricing. So. do I double the price or do I half the price? Like what are those multipliers that I need to apply to be able to adjust the base fare in response to current conditions?

And then the environment would basically be factors like the demand and supply conditions the traffic conditions, time of day, et cetera. And basically the rewards. Which we talk about in the reinforcement learning system would basically be the effectiveness of these pricing decisions, depending on metrics like revenue or right completion and acceptance rates.

So, the reinforcement learning model is basically able to learn a pricing policy that like maximizes longtime rewards and is able to balance between like exploring new pricing strategies versus like exploiting existing successful ones and. Yeah, I think it's basically able to adapt and continually learn from new data to be able to define that to improve efficiency and responsiveness over time.

Adel Nehme: You know, one thing that comes out from a lot of what you're saying is, there's the modeling layer, there's the infrastructure layer, there's the embedding it in the user experience, the A B testing layer, right? And I'm sure there's also a design layer, right? Like, how is the, prediction served within the application in a way that the user, you know, immediately internalizes, and I think that requires a really strong collaboration between different teams from machine learning, engineering, product design.

So maybe could you walk us through what that collaboration looks like in practice? what does ideation to production look like across these different teams?

Rachita Naik: What I really like about Lyft is we have a bottom sub culture, so like a lot of ideas bubble up from individual contributors within each team and then move from ideation to productionization. So, like we usually have these quarterly or annual brainstorming. Sessions where like the teams get together to brainstorm a bunch of ideas that they envision themselves working on during the next quarter of the next year and depending on like some rough impact sizing we then like proceed to work with product managers to understand the requirements of the product so that we can like Come up with like model requirements and outline the development and execution plan.

And each project usually like requires cross-functional collaboration because you could have dependencies among multiple teams for a given product. So we also like work with PMs and engineers from other teams to build like a small. for that particular project.

And then in the prototyping phase, it's basically like more iterative fair, like, machine learning and data science, work together to sort of, offline prototype and work through like the model requirements, but also like talk to engineers because you need to understand like the feasibility of actually putting this model into production as well. So that's sort of an intuitive process. And then once we have a sort of developed system in place, then we talk to stakeholder teams understand dependencies like Train systems on the other end as well, because as you can imagine, like what happens a lot of times is we update our systems like launch some product within our service, but then if there are like multiple dependencies on downstreams and they don't update their models or their systems to like start consuming these new predictions, then like during a B tests that reflects because it does not impact like any business metrics or sometimes even impacted negatively.

So I think like proper cross functional collaboration like and communication that's also key so that you're able to basically shadow the pipeline and launch experiments with all parties on board. Yeah.

Adel Nehme: Yeah, that's wonderful. And you mentioned here one thing that you alluded to at the beginning, you know, working offline and then working online are quite different. I think one thing as well, given your background, working in academia versus working in real world machine learning is actually quite different as well.

you've made that transition from academia to real world machine learning. So I'd love to know what are the biggest breakthroughs. challenges within, machine learning biggest challenge you face during that transition, And what are common advices that you would give someone coming out of academia, going into a machine learning engineering job?

Rachita Naik: I think after I completed grad school and started at Lyft it was a little challenging since I started with COVID. So, I transitioned into the industry in remote work environment and I couldn't like meet with my mentor or manager to be able to like ask questions, collaborate with people.

So that was a little challenging. Having to rapidly adapt to a new technical stack learn the internals of like the company's infrastructure, even like ML pipelines. So there was a significant shift, I would say, like from my academic experience where I was like working with Clean curated data sets in control like Jupyter environments.

I was in that local Optima mindset focused on improving model accuracy, or like just tweaking model architectures. But in the industry, like you need to take a step back because the model is generally like a smaller component providing like value to a much larger system. So you need to be able to like balance model accuracy with real world constraints, like.

the scalability latency and like some of the challenges that we spoke about earlier.

Adel Nehme: Okay, wonderful. And you mentioned here the business skills and understanding, asking the right questions and being able to kind of navigate the environment. What are the soft skills and business skills you think you need to have to succeed as a machine learning engineer?

Rachita Naik: Yeah, I think just like not having any, inhibitions when you're starting off with a new job when it comes to like asking questions or like questioning like the why behind a lot of the design of systems. when you're starting out in a new company, it's very easy to get overwhelmed.

Even like if you join a team that owns multiple services, like you try to dive deep and then you have like 10 different architectures that you need to onboard onto. But I'd say like, your colleagues are usually more than happy to help and just being able to like talk to them and understand why certain systems have been built that way, like reading documentation, I think is also really helpful.

At least at Lyft, we have really rich culture of like maintaining documents for Whatever decisions that we make so reading up on like the history of why certain services were built or models or prototype. that's also super helpful. And I'd say like for machine learning specifically, it's.

Really important to like stay up to date with what's happening in the industry as well because like once you start working, you're sort of engulfed by your own project the tools that the company is using but as like the machine learning in the industry, like the deep learning world that's changing like rapidly every day, like every day.

There's some new model that's being released or that's surpassed like the previous state of the art. So just like being aware of the trends that are in the industry so that you don't find yourself stagnated or stuck after a period of time. I think that's, important as well.

Adel Nehme: And then, you know, you mentioned this at the beginning, like you're investigating generative AI use cases at Lyft as well. You know, as we look into the future, what are gen AI use cases that Lyft is working on at the moment? And how do you see this technology also shaping? The future of ride sharing.

Rachita Naik: I think there's a lot of buzz even within Lyft and like ride sharing broadly to identify ways that we could, like, leverage any AI at scale to, like, basically streamline the ride sharing app experience. So, I guess within Lyft there is a strong effort to, like, integrate these tools to improve internal productivity.

So the ML platform team has basically been trying to integrate chat GPT into tools like Slack and GitHub to improve like, employee productivity, because now we can ask questions against internal documentation or like data tables to get. Answers more quickly than like having to parse through a lot of threads as we used to earlier even like using it for summarization and sentiment analysis of rider driver feedback for gaining insight that has reduced manual analysis time to a large degree.

And as I mentioned, we've also been leveraging these advanced chatbots for like complex query resolution to be able to For common customer concerns without having to step in or like provide human support for each ticket that, customers raise. But I think going forward, there's still like a lot of scope.

 You can imagine like, hands free platform usage and it comes to lift where like you're using AI powered conversational booking systems to like, Request rides and manage your trips just by like speaking into the app or even like leveraging real time language translation when you're traveling globally and don't know the language of a certain country, but still want to be able to communicate seamlessly with your drivers.

Adel Nehme: Okay. That's wonderful. That's really exciting use cases. And beyond generative AI as well, what machine learning trends are you excited about the most and what are emerging technologies or methodologies in the machine learning space that you think will have a big impact on ride sharing as well?

Rachita Naik: I think, ride sharing could definitely benefit from, continual learning. I think that's something that's more isolated at Lyft today. Like, spoke about the ETA bias prediction layer where that model updates rapidly real time. But I think there are a lot of use cases across Lyft where the models could basically adapt to changes and improve Like more dynamically because these robust continual learning frameworks basically like will help improve the efficiency and responsiveness of our machine learning systems and basically more rapidly adapt to like traffic and weather changes which influence these rider driver matching decisions.

I'm also excited by reinforcement learning. I'd say like I've had the chance to like read about some of the algorithms that pricing uses, but I feel like even other use cases across lift like an agent learns to maximize multiple objectives learning from feedback that could benefit this dynamic ecosystem, because right now we're using a lot of supervised algorithms where you the model is trained on a fixed data set in time.

But since we have constantly changing variables, I think RL would be more suited for this type of environment. Like you could imagine more efficient rider driver matching for instance by considering factors demand and supply conditions and acceptance rates, et cetera, to be able to better match riders and drivers to reduce wait times and efficiency for instance.

So these are like some of the trends I'd say that I look forward to leveraging in the future.

Adel Nehme: That's exciting. And then maybe before we wrap up Rashida, do you have any final advice or notes you would share with listeners, whether it's about machine learning, your work at Lyft or anything that aspiring machine learning engineers. could benefit from.

Rachita Naik: I think I'd just like re emphasize on the fact that try to like engage in continuous learning. I think it's important to like build your skills by either like taking courses of these sites like Coursera or Udemy, for instance, or like reading weekly newsletters on the latest news.

Whichever industry like you're most excited about, it may not just be machine learning. But I guess at least in ML, I think experimentation is key. So like, outside of your work projects, also like trying to explore new ideas to build smaller prototypes that leverage these new technologies and trends.

Because I think that's how, you start thinking outside the box and try to apply those algorithms to your work as well, like that's how you sort of try to link use cases and like come up with new ideas that you can also maybe turn into a tangible product in production. it's also like important to like find what drives you at work because I think that's, important if you want to be able to work in a sustainable way over a long period of time, because for me, I think the most fulfilling aspect is, knowing the direct impact of my work on people's lives, like, every feature we roll out or, product we launch like, we can see the impact on a user, and Even like through these AB tests, for instance and like seeing those metrics move tells us, okay, the users are liking these features.

So that really motivates me and pushes me to sort of come up with new ideas to like resolve some of these pain points. So, yeah, I think hopefully our listeners had something to gain from these insights, but it was really great being here.

Adel Nehme: Yeah, they definitely did gain, I gained as well. Rashida and I, thank you so much for coming on DataFriend.

Rachita Naik: Thank you. Thank you so much, Adel.