Adel Nehme: Hello, and welcome to Data Framed! I'm Adel, Data Evangelist and Educator at DataCamp. And Data Framed is a weekly podcast in which we explore how individuals and organizations can succeed with data. Today we're speaking to Sam Stone, director of Product Management of Pricing and Data at Opendoor. For those who are not aware, Opendoor is an operator of an online real estate marketplace, used to streamline the sales process of home buying and selling, and they do that by using machine learning models to instantly price homes and provide recommendations.

Its millions of users. You know, oftentimes when we think about building great machine learning products like Opendoor, we get fixated on the nuts and bolts of machine learning, which algorithm to use, what are the features to be engineered and so on and so forth. But to build truly great machine learning products that reach millions of users, you need to marry great data science expertise with strong attention to user experience, design thinking, and a deep consideration for the impacts of predictions on stakeholders.

And building great machine learning products. The type that I'm speaking of is something Sam Stone has a deep understanding of. So I wanted to have a deeper conversation with Sam on what he thinks a great machine learning product looks like. Throughout the episode. We spoke about his principles for great ML product design.

How to think about data collection for these types of products. How to package outputs from a model within a slick user interface. What interpretability means in the eyes of customers, how to be proactive about monitoring failure points, and much more. If you enjoy this episode, make sure to subscribe to the show, and now on to today's episode. Sam Stone, it's great to have you on the show.

Sam Stone: Thanks. Excited to be here.

Adel Nehme: So you are the Director of Product Management Opendoor. You've overseen the development and maintenance of machine learning products that are extremely integral to the Opendoor value proposition. So maybe let's start at the very beginning. What do you think makes a great machine learning product?

Sam Stone: First before I even get into that, really excited to be here. Thanks for having me on the show and, and excited to talk about what we're doing with machine learning at At Opendoor. I think there are three things that makes for a, great enduring ML based product. Number one, building a product that stems from deep understanding of user behaviors, and that's true for every product.

Machine learning products are now in no way special in that regard. Number two, optimizing for the accuracy in your ML product in a way that's aligned with what users actually. And number three, building feedback loops into the product from the very beginning so that that product can be continuously improved.

Adel Nehme: That's very great and I'm excited to unpack all of that in our discussion today. But before, let's maybe focus on how machine learning drives value Opendoors. So maybe to start off, maybe what is Opendoor's mission? What is the business? Of Opendoor.

Sam Stone: Opendoor is an e-commerce platform for residential real estate. That allows you to buy, sell, and move with a tap of a button. So if you consider the traditional process of selling a home, at least in the US you hire a realtor, you have a bunch of open houses, you maybe get some offers, they fall through, you negotiate, you do repairs, you get more offers.

It's highly uncertain. There's a lot of friction, a lot of inconvenience, and the whole thing might take weeks or months or maybe even more. With Opendoor, you can go to our website, type in your address, answer a few questions, and very quickly get a real and competitive offer to, for Opendoor to buy your house.

And so that is much simpler, much more certain, much more convenient. Now, the way that we actually deliver on that, certainty, that speed, that convenience, is that we buy the home. We use our own balance sheet, our own. To purchase the home, which means that if we misprice that home, that mistake comes out of our own pocket.

And so that probably starts to give you some indication of why algorithmic accuracy in ML products are really important at Opendoor.

Adel Nehme: That's really great and the second half of that. Is, where does, machine learning come in? You mentioned the importance of algorithmic accuracy. Where does machine learning provide value for open?

Sam Stone: So I already mentioned pricing. And that goes all the way back to the creation of, the company. The very first lines of code that our technical founder wrote were a pricing model to value homes. And this is, before we were even operating. They were just like, Testing out or thinking about the business model.

One of the really interesting things about residential real estate in the US is that eventually all the transactions become public record. And I'll, probably talk more about this later. What that meant was that we could build a pricing model before we had bought any homes on that public data and see how accurate would we be, and then flow that through into a simulated p and l and say, do we think this business model will will?

And obviously we did and we launched the company and, it has worked. Pricing has continued to be a, a very large focus area for our ML products. And we've taken, the pricing problem, we've broken it up into, discrete parts. So one part is what is a home likely to solve for if it's sold on the open market today?

Another part is if we buy the home, we'll hold it for some period of months. What's likely to happen to the, the. Over that period. So that's very much a forecasting problem, whereas the first problem was a now casting problem. A third area of pricing would be how do we run an effective auction when we go to resell that home?

Beyond pricing, we've added ML models in a number of areas over the years. For example, we do recommendations and personalization. So for home buyers, based on their preferences, what homes are they most likely to be interested? We use it for trust and safety reasons, to detect signals that may indicate potential incidents in the homes that we own, where we need to go investigate and assure that, that home is clean, safe, functional, and safe.

We use ML for allocating our Marketing budget, and there's a whole bunch of other use cases as well.

Adel Nehme: That's a very holistic overview and I think it really begs to, unpack a lot of what you think are best practices in delivering great machine learning products. So maybe let's start off with the first Component that you mentioned, which is, developing products are really based on deep understanding of user behavior.

Of course it goes without saying data makes or breaks a machine learning product. But you mentioned how in order to create build machine learning products as well, it's really important to not just know the data, to really understand the user. Tell us maybe why that's so important in the context of a machine learning product.

Sam Stone: There's often something that is lost in between the real world and the data exhaust that comes out of the real world or the digitization of user behaviors. And I think it's, really important to start with the users and directly observing them and then using that to inform how you think about data.

So an example of this is that we have lots of data at homes. Oftentimes we have multiple different sources that are telling us about homes and those sources conflict. This is a problem for us. A, a very common pattern that we see is that we will have data on the square footage of a home from a listing on what's called the multiple Listing service or the mls.

This is essentially a database maintained by agents for listings with at least one or more MLS per market. And we'll have data on the square footage of home from county tax assessor or tax recorder offices. And those numbers will disagree. One will say, this home is 1000 square feet, and the other will say, this home is 2000 square feet.

Now if you look at that from a purely technical perspective, that's pretty perplexing. But if you think about what users are doing when they are creating this data, it actually makes a lot more sense. When you are selling your home, you have an incentive to position your home in the best light possible.

You're probably going to describe the square footage as the largest number that it would be reasonable to use. But when you're reporting data about your home to tax authorities, you want to minimize your tax belt, and you're probably gonna report the square footage that is small as could be reasonably const.

And so when you start to understand those behavioral elements, it gives you a lens in which to understand the data and to say, okay, I've got these two different sources. They disagree, but that actually makes sense to me. And I, I understand the bias and the data and how to start to reconcile that.

Adel Nehme: That's really great and I love that example that you mentioned because it really showcases as well the two-sided marketplace. The Opendoor works in, in a lot of ways from the perspective of the seller and for the perspective of the buyer. I wanna understand maybe in a bit more detail for you as a product leader and as someone who works with data teams as well, how do we reconcile these differences and how do you reconcile the user behavior with the data that's being fed into the machine learning model?

Sam Stone: There's a couple different strategies here. I mentioned understanding the, way that data is generated, that's strategy number one. The second strategy that I would think about here is understanding what users are really paying attention to that might not be easily available as data, and then going out and figuring out how to get that data or how to work with that data.

Sometimes the problem is getting the data. Sometimes the problem is you have the data, but you're not sure what to do. An example of this would be photos. So a lot of the time when people start to build models related to real estate pricing, they start with the data that's easy to work with. They start with structured fields like bedroom count, bathroom count, and square footage, right?

That's kinda like data science 1 0 1. I've got a couple of scaler values, and maybe I'll throw in some categor. But for anyone who has ever bought or sold or rented a home, you will know that no one is gonna move into a home without seeing it. Now, pre covid, that probably meant seeing it in person today.

Sometimes people only look at at photos or videos, but it's a hugely visual decision, right? It's people don't make that decision just. On the structured data, you, absolutely need to pay attention to the unstructured data. Now, that's a very hard modeling challenge and there's different ways to go about it, but any good solution to that problem, be it you're using computer vision or human in the loop systems, any good solution to that problem is going to stem from the, The principle that a model to be highly accurate needs to make use of that difficult, unstructured visual data that users pay so much attention to.

Adel Nehme: Have you felt that throughout, your time at Opendoor and collecting feedback from users did that user feedback alter and required updates to how you collect data and your data collection strategy, and how you structure data that goes into machine learning products at Opendoor.

Sam Stone: Absolutely. I think for any ML product, there's an evolution of You need user feedback loops. You need ways to show your, predictions or your recommendations or whatever you call the output of your ML product. To users. And at that point in the user journey, the user needs to be able to tell you something about what they think about your, prediction.

And I think part of that needs to be unstructured. Right? not a lot of users are gonna, do this, right? Unstructured feedback is, it's higher cognitive load but it will point you in directions that you never thought of and then, It's normally good to accompany that with some structured fields which are easier for users to fill out.

I'm selecting from a dropdown of, I think this is, off and here's the top three most common reasons. And that can be turned into analytics and that can be processed by ML development teams, based on aggregated insights. But I think you want both of those strategies for feedback loops that are embedded in the product at the point where ML outputs are exposed to users.

Adel Nehme: That's really great and we're gonna discuss feedback loose a bit more. You, as you mentioned, that as an important aspect of building machine learning products. But first off, let's talk about how you actually deliver. A machine learning product and within an application for a user, right? I think a lot of data teams tend to think about machine learning systems that go into production, as, just an algorithm, something that needs to be packaged into an api, but there's also a really important aspect of how the machine learning system is gonna be packaged in front of the user within an interface or a software.

 It's one thing to produce a prediction, but it's another thing to produce a prediction that is understood by the user in a way that creates user trust and where the prediction is presented as a cohesively, as part of a user journey. So maybe walk us through example where that's been important for machine learning products to Opendoor and how you've approached packaging machine learning systems as part of the Opendoor Applic.

Sam Stone: This is an area I'm, I'm really interested in. It's this kind of fast growing field of, model interpretability, and I think you hit the nail on the head when you talked about user trust and interpretability is key to user trust, and there's, there's a couple different benefits from that. First of all, users who trust ML outputs are more likely to.

Use the product or continue on in that journey. Second, if they understand something about the model out, but they're more likely to provide feedback that's useful to the team. And then third, I think interpretability is really important for development teams. As you think about the long-term arc of, iterating on an ML product.

Understanding why the model output is what it is at an individual inference level is really key to generating good research hypotheses that turn into model improvements. So at Opendoor we spent a lot of time thinking about model interpretability and how that goes hand in hand with long-term accuracy improvements.

When we show a, a price to a home seller, we don't just show that price to the seller. There's two broad parts of what we accompany that price with to make it interpretable. The first is actually relatively basic. It's what are the inputs that we used around that home to generate the price.

Now some of those input. The user, the home seller has told us. But there's a lot of home attributes and neighborhood attributes that go into our pricing model. And we don't wanna make the user answer a hundred questions if in a particular neighborhood we're using a hundred different inputs to value their home.

Especially because we have a lot of those inputs from public data. But it's important to show the user what we used so they can say, oh yeah, this looks wrong. Or maybe this was right up until a month ago, but this thing changed. And when they can provide us that kind of concrete feedback, we investigate it.

And a lot of the time they're right and we go and we raise their offer. And that's a very kind of deterministic I would say relatively straightforward path they can be implement. For pretty much any model, right? To show the inputs to the user that you are using. The second part of our interpretability strategy is showing how those inputs were transformed into the price.

Now this is where it gets much more domain specific and there's a lot of creativity and nuance that is, required. So I wanna step back and talk about how do actors in the residential real estate market, Produce prices. How do sellers come up with their willingness to sell? How do buyers come up with their willingness to pay?

Most people follow a three step process, and they might do this explicitly if they're an agent or they're an appraiser or they do it implicitly, just in their hat. The first step, they select comparable sales homes that have sold nearby that they think are. Second, they adjust the sale prices of those comparable sales.

So maybe the comparable sale is your neighbor and your neighbor's home is identical to yours, but you have a pool and your neighbor doesn't, and your neighbor's sold for $500,000. You're gonna say, okay, I think my house is probably worth $600,000 cuz I have a pool. And then the third step is they take a weighted average of a couple of adjusted comp prices.

So this looks very different, by the way, than just doing like a linear regression right. This is really a three step process, and maybe you would do a linear regression at each step, or maybe you use a neural network at each step. But the core part is that there are comps. You adjust them. And then you, take a weighted average and we expose information about each of those steps to our end user.

And that allows us, that allows them, again, to trust the process more and to provide us really useful targeted feedback where they can say things like, Hey, this comp that you used, It's actually not very similar to my home and here's why. Or you missed a comp. My neighbor literally just sold yesterday, 

Sam Stone: and maybe, it wasn't in your data set yet, but you should certainly consider this.

If we look beyond Opendoor, I think we're starting to see some really interesting applications of interpretability for especially complex model types, especially deep learning models. For example, in computer vision, we're seeing really interesting work where computer vision models can highlight what part of an image they're paying the most attention to.

And . In, in large language models increasingly the models are able to seamlessly explain their reasoning just by including the request to explain yourself in the prompt. Or you can say something even more specific. You can ask for citations and a language model can tell you specifically what sources it used.

Adel Nehme: that's awesome. And you mentioned at the beginning talking about interpretability and user trust. There's also an emotional aspect to this, right? People don't want to feel that their home is undervalued. It's a place that they've been in for decades maybe, and they've been, really working hard on making sure that their home has high value.

And I love how you leverage, model interpretability. Data science metrics in a lot of ways to package it as part of a product to create that trust. Are there any lessons that maybe you can share with other product leaders or data leaders around, incorporating, product management design thinking with data science metrics and data science interpretability as part of a mechanism to create more trust and to create, a stronger positive experience for users within a product.

Sam Stone: First and foremost, don't assume that what is interpretable to you or your data science team is going to be interpretable to end users. The people who are, who are developing the model are so close to it. They understand all of the nuances, all of the pros and cons, and. A lot of the time you need different interpretability strategies for end users versus internal users.

Adel Nehme: and what are ways you've been able to translate the findings of the data team? in terms of like model interpretability to make it and package it in a way that is much more empathetic towards the user needs and problems.

Sam Stone: One strategy that we, do a lot of is we evaluate. Individual ML inferences that we have some reason to believe were really, really off. Either a user provided direct feedback to us that they thought it was off. A member of our operations team flagged something. Or, and this is, this is a great aspect of the residential real estate market.

We can compare our price estimate versus. What the market actually transacted at. So we might have offered on a house that seller didn't take us up on our offer, they sold a month later, we can look at that price and say, how did it compare? And so we have a, a sorting process for all these different sources where we are identifying individual instances and then digging into those and.

How do we produce that prediction, which model inputs were leaned especially heavily on. Was there any that the model didn't weight heavily that we think maybe it should have? What was the sign? What was the magnitude? What were the interactions in the model? And all of that is being done at the different steps in the process.

Right? I mentioned one model that had a three step process. we're really going to that level of.

Adel Nehme: So let's jump to talk about metrics and evaluation. This segues really well here. As a product leader, I think it's very intuitive for you to think about North star metrics, And how there needs to be once. Single metric to optimize for, for the success of a product in general.

But I think a lot of data teams that may not be intuitive for them, because when you design a machine learning system, right, you have, dozens of metrics from like recall accuracy, precision, a u C, et cetera. So maybe to start off, how do you define a North Star metric for a machine learning product?

Sam Stone: I think it's good to be aware of and to track many metrics and at a product level. I think it's also very common that you have paired metrics one really common. Out of paired metrics would be something focused on, revenue or transactions or some notion of volume versus some notion of profitability.

Most businesses have that trade off and it gets reflected in most products. However, for most machine learning models in the actual training process, you have to pick some loss function that you are seeking to to minimize, and I think this is where. It's important for an M L P M to understand the details.

I think that in a well-functioning ML product team researchers or data scientists are gonna, the ones who are going to propose what that loss function will be. Maybe articulate a couple different options. Lay out the pros and cons and say this is the one that we would recommend.

But I think a PM should. Engaged enough with the details to understand those pros and cons, to ask probing questions and to really be a, a sanity check there. No loss metric is perfect. Anyone is gonna have, pros and cons. And some of those are Pure data science, pros and cons, and you need data scientists to weigh in on those.

But some of those are more kind of business related and that's where I think a product manager can be really, really helpful. An example of this at Opendoor was that when we first built our recommendation, Product for home shoppers, people considering different listings to, to go tour and ultimately to, to put in an offer on.

We tried to minimize loss in terms of listings that buyers clicked on. What this resulted in was a model that showed highly, visually unusual homes. A lot of them were mansions or they were homes that had something else in the photos that just really caught your. They were a lot of fun to look at, but they tended not to be homes that our users home buyers would actually go to or much less offer on.

And so we had to shift to a different loss function, a different kind of core metric that our model was, trained to to minimize well minimize the loss. But what it was, what it was doing was we actually trained it on offers that buyers submit. This is a big challenge for our data science team because there were a lot fewer offers in our historical data than there were clicks, so they had to deal with a bunch of data sparsity issues.

But at the end of the day, this resulted in a model that was much more aligned with actual user intent.

Adel Nehme: That's really great and maybe deep diving into that last example what are common ways to mitigate data, sparsity issues, especially when you're trying to optimize for outcomes that are not yet? Corded and you're trying to build out an initial product, right, as a product leader to be able to get that type of outcome that you want from particular users, provide, provide them with that value and you're still building it out early in the journey of the product life cycle.

Sam Stone: First, I think it starts from having a, a strong perspective on what matters to users. To translate that into data science terms. If you have a set of inputs, you should have a, probably have a, a strong. A priority perspective on the sign and magnitude of the relationship between each of those inputs and the output.

Maybe not for all of them, and there might be interaction effects, but you should know which factors you think are most important and how you expect those behaviors to occur. And if your data is really sparse, you might start with the set of heuristics and release a product that is purely heuristic based and use that to gather operating data or real world kind of user interaction.

I think the second level when you move on to an ML product is to use a relatively simple highly parametrized model. So take those, priors that you have about how the features should work and actually encode them to some extent in the model and maybe allow your model to, update as new data surfaces, but don't force the model to.

The entirety of your belief set just from from data. Because if you don't have much data or the data is inherently sparse that's just gonna lead to a weak model.

Adel Nehme: Okay. And related to North Star metrics, what do you think are common pitfalls, maybe when you're trying to define a North Star metric and as a product leader, what are systems that you can put in place to ensure that you're always optimizing for the correct outcomes?

Sam Stone: So I think there's, two parts to that question. In terms of common pitfalls on North Star metrics I, I think at the, the kind of more technical level Choosing metrics that are overly academic. Maybe it's, it's really cutting edge, but if it's difficult for your team to understand, or very few people deeply understand it, that probably outweighs the benefits of using, a loss function that might be slightly better than the more common thing.

You also asked about How do you actually monitor I think you said monitor inputs and outputs. Is that, is that right?

Adel Nehme: No, it's optimizing for the correct outcomes here. What are systems that you can put in place, ensure that you're always optimizing for, the correct outcomes when it comes to developing a model.

Sam Stone: I think they tend to be more. Process and, and people oriented than like technical because user behavior is changing the real world. And you want processes where you are soliciting internal and external feedback about particular predictions or inferences that look wrong. And you're doing individual investigations.

You're also looking at aggregate metrics and seeing how those are shifting over time, be it from back testing. Or aggregations of, user behavior.

Adel Nehme: So the third aspect here of building a great machine learning system is understanding, the product really well in creating feedback loop systems where you're able to get immediate feedback from users and to be able to improve the product, right? In a lot of ways, our conversation so far has touched on the nuts and bolts of building, an excellent machine learning product.

But let's maybe take a step back and think about how to continuously improve it. What does getting feedback look like in practice? What are the, what is the mechanism of getting user feedback look like? And walk us through some of the best practices that you've learned there.

Sam Stone: I think part of this comes from getting user feedback. Part of it also is about understanding. You're blind spots where you're not getting user feedback and reacting in those areas. On the the explicit user feedback side, I think it's important to embed opportunities for users to provide feedback at the same place in the user journey where they are, seeing or interacting with model outputs and.

I think you should provide users the opportunity to provide structured feedback, which is generally easier for them to provide. We tend to have higher completion rates around that if it's just say a dropdown menu and unstructured feedback so they can tell us about things that maybe we've never seen before or never considered.

Adel Nehme: In a lot of ways what you mentioned here requires a lot of overhead, right? Monitoring metrics, understanding what is going on in the wider world soliciting user feedback. Maybe walk me through what you've learned to make the machine of managing machine learning products more predict. Right. What can you share about becoming more efficient, about soliciting feedback about understanding the world around you when building and improving machine learning product?

Sam Stone: First, I think. Of good cross-functional partnerships, you need a, a data science and an engineering team that work very closely together. An anti-pattern here that I think is relatively common is to throw it over the wall approach where data science prototypes a model, sees that it works in back testing and then throws it over the wall to engineering, to productionize.

We try really hard to, to. Avoid that at Opendoor. And so in the very early kind of model prototyping discussions before any code has been written, we try to make sure engineering is involved. And conversely, we try to make sure data science stays involved as we get into Productionization and is there at every step in the way to help engineering think through all of the nuances and all of the challenges inevitably come from moving a model from prototype to product.

We also think a lot about the collaboration cadences and rituals that bring together our operations, our design team, our commercial teams, and get them to work with our data science and engineering teams around ML product ideation and, and creation and, and monitoring. And product managers play a big role in bringing all of these stakeholders together.

Ideally, it's not all going through product managers, that you have engineers who are directly talking with people from customer support or data scientists who are working directly with people from operations.

Adel Nehme: Yeah. That's really great and maybe expanding on that, what are great rituals that you find to be effective that improve collaboration between. To teams, engineering teams, design teams, and that has been able to create this cross-disciplinary function with an Opendoor that is responsible for machine learning products.

Sam Stone: We have a set of rituals around sharing work in progress and, and that can be uncomfortable because a lot of the time people like to share work, especially on the data science or research. Side that they feel is polished, they feel like it's ready for peer review, especially if you're coming from an academic background.

I think that's a very common inclination. And instead we push to say, no, we're gonna discuss your work before you have a strong opinion on what the right answer is, where you're truly receptive to, to feedback. And that's, that's very much like a kind of learned organizational skill because it runs counter to a lot of the things that.

We've been trained in from the corporate world or, or academia. I think another set of rituals that's really powerful is hearing directly from customers. So for us, that will mean getting out to our markets working with our operations team on the ground that is inspecting and renovating and maintaining homes directly, listening from customers.

Joining calls, watching videos. I mentioned before we're, we're profiling individual instances of where our ML output seems to have been off. So there's a whole set of rituals around, around that. And then there's planning and I think planning processes need to differ by, by the team and the organization.

But one thing they all. Share in common is that they provide opportunities for a wide variety of people to submit ideas. So an engineer should feel empowered to submit an idea that might not be pure engineering, but might be customer facing. And, our researchers should feel empowered to submit ideas that are not just research ideas, but might be about tech debt, pay down, or again, something that's more commercially orient.

Adel Nehme: Okay. And related to the example that you gave on regime change, right? And these types of, tectonic events that changed the performance of a model. An additional wrinkle that I found here is a lot of the learnings that improve machine learning model systems comes from a reactive mode, right?

There is a decrease in a North star metric, a failure in a metric and data leaders and product leaders have to react to that, and they. Tend to improve their product as a function of that. I'm definitely not saying that leaders or product leaders or data leaders are able to predict covid or are able to predict a regime change in how the Federal Reserve approaches interest rates.

But what are strategies that have worked with you to be able to go from to be able to be more proactive about improving and stress testing, machine learning systems and product?

Sam Stone: First, setting aside time in in sprints or planning processes to dig into the things that are unexpected. So it's, it's digging into bug is that one's relatively obvious. Most teams have an on-call. But maybe taking that concept that's common for engineering and applying it to data scientists as well.

It's really difficult if you, if you plan to spend your whole week as a data scientist working on, on model future X and you, you feel pressure to ship it at the end of the week. And then there's this really interesting, that bug that comes in, but your week is booked. That bug probably never gets investigated.

I would also say dig into bad customer experiences that may or may not be bugs. Sometimes there is a, a desire to triage such that only definite bugs get routed to technical team members because their time can be so scarce and valuable and there's certainly merit. But there's also merit to forwarding on some of the most uncertain pieces of customer feedback.

And I've seen really interesting hypotheses come out of digging into those areas. Finally, I would say in terms of becoming more proactive go on listening tours with your operators, with your end customers. And also pay a lot of attention to what other companies in your space are doing.

Both your, your direct competitors, but also companies that may be in a different industry, but maybe doing technically similar things. So at Opendoor, we're not doing anything related to trading equities or bonds, but there's a lot of kind of relevant technical work around those types of trading algorithms that we pay attention.

On the other hand, we're paying attention to a lot of innovation in the, the, the real estate startup space, even if technically it doesn't look as similar to, to our stack.

Adel Nehme: Okay. That's really awesome feedback. Sam, as we close out our episode reaching the 40 minute mark here I wanted to talk to you a bit about your career and how kind of advice that you can give for other people may be looking to be in a. Similar position to yours. In a lot of ways, your role sits at the intersection of product and data, right?

What do you think is needed to succeed in this type of role? Maybe walk us through some best practices that you can share with the audience here.

Sam Stone: I think the thing is that have allowed. Me to really enjoy being an ML product manager. And as I look at my team, the things that have enabled them to be successful are, one, they have a deep understanding and desire to understand users. Number two, they love data science. There's a, there's a kind of fundamental nerdiness and excitement about.

High quality data set or understanding the intricacies of a model. And number three, they have deep empathy for engineers. I think engineering is a, is a difficult role in general. And if you're an engineer working on ML products or the outputs, Probabilistic. It's especially difficult. And so as a product manager, you need to really understand and have the trust of your customers your data scientists, your engineers, and a whole bunch of other functions.

And I think those are three key components to, to having that and allowing you to be successful and ultimately enjoy the role.

Adel Nehme: That's great. And what are maybe. Of data science should you know, to be successful as a machine learning product.

Sam Stone: It will depend on the organization. I think you should have passion for it. And one test for passion is you've probably built some models yourself. So I think there is like a, a level of Personal ability to, to code and implement things and learn from that firsthand experience that is necessary.

Going a level deeper than just reading about it and knowing the concepts. And then the second litmus test here is you probably want to know more whatever your level of knowledge is. There's probably a handful of things that you would be really excited to go deeper on if you had.

Adel Nehme: Okay. That's really. Finally, Sam, as we close up our episode, do you have any final call to action to share with the audience?

Sam Stone: Yeah if you are buying or selling a home, or you happen to live in a home, so pretty much everybody out there, you should check out Opendoor. And even if you have no interest in real estate, you should also check out our blog, which is called Open House, where we have a ton of great posts on data science, engineering, and product management topics.

Adel Nehme: Okay. That's great. Sam. Thank you so much for coming on data.

Sam Stone: Thanks for having me.