Data Science at Stitch Fix

Hugo: Hi there, Eric. And welcome to DataFramed.

Eric: Hi Hugo, thanks for having me.

Hugo: It's a great pleasure to have you on the show and I'm really excited to be talking about data science at Stitch Fix, how it's woven into the very fabric of what Stitch Fix does, your role of Chief algorithms officer. But before that, I want to find out a bit about what you do. And I want to approach it in a slightly different perspective. I'd first like to find out what your colleagues, people you work with, think that you do.

Eric: Colleagues. So people at work, they would probably say we solve some problems and then I guess go beyond solving but implementing a solution as well.

Hugo: Absolutely. And they probably think you work with computers or something as well, technical stuff?

Eric: Oh, sure. They think that there's a lot of math and exotic code that we do. Sometimes their imagination is beyond where we actually are, thinking they're far more exotic than what we really do.

What do you actually do?

Hugo: So what do you actually do?

Eric: Well, I think the best characterization of it is applied micro economics. Well, it's my own bias because that's what I studied as an undergrad. But many of our problems are straight out of textbooks and even the data looks like it's straight out of textbooks, which is really fun to work with if you're a data scientist. Things that look Gaussian curves actually do behave like Gaussian curves. That's pretty rare, at least in my experience in the real world, at least for things that are not so bound by physics, like height and weight. More social properties tend to have different distributions, but by golly, we found some Gaussians at Stitch Fix.

Eric: So it is fun to work in that way where the stuff you learned right out of school is applicable. And it's like, wow, I thought that was just theory but here it is, I can actually implement this and make a difference. So when I say micro economics, I think you can think things like just making a business more efficient by changing the way we do various processes to leverage data.

Hugo: So that's the micro economics part. What about the applied part? What happens in your day to day or the people you work with that makes it so applied and how do you approach this?

Eric: Yeah, I think the big difference is Stitch Fix has sort of stumbled upon the data that's been long missing from retail. It's a nuance of our business model. I can tell you a bit more about how Stitch Fix works in a moment. But, the main thing to know is that our clients aren't picking out the clothes for themselves. We have a styling service that does that for them. And so, what happens is they provide a lot of valuable information to us, things like their age, and height and weight, and so forth. This data reveals all sorts of things about how to run your business better. This was always guesswork for other retailers, like how to do things like set prices, how to buy inventory, how to set the levels of supply.

Eric: All these things were just guesswork and now they're actually really possible in practice, and by golly, all that theory we learned in college actually works or at least most of it. You can actually frame problems as you would in theory and then you can use statistics and computer science to actually solve them and find relationships and then implement things to leverage that new knowledge that you just unearthed. And so this is what we call algorithms, automated decision making, things we put into production to run a certain business process.

How did you get into data science?

Hugo: And so, soon we'll be getting into exactly what type of business problems you solve and how you think about building types of algorithms you use. But before that, I just want to find a bit more out about your background. Maybe you could tell us a bit about how you got into data science originally?

Eric: Well, sure. Yeah, I think I was lucky enough to be interested in all the right things. My undergrad was economics, specifically micro economics. I was always intrigued by this, what I called a noble way to compete. If you can run a company more efficient, you make things better for both the company and the consumers. And so I always called it this noble way to compete. I naively thought that businesses leveraged this stuff, that they were optimized for these things. This was in 1995, it was perhaps the cusp of the information revolution, it was in full swing by that point, meaning data storage was becoming available and cheap and compute resources were becoming affordable, that companies can actually start doing some of these things, not just being theory.

Eric: And so, I graduated with an undergrad degree in economics and I somehow talked my way into getting a job as a statistician. We worked on various problems to make them more efficient leveraging data. But they're mostly in the vein of analyses, meaning I perform some analysis and that would kind of advise as what we should do, we should change these prices or move to those locations. So it wasn't anything automated about it. It was really just, output the results of an analyses.

Eric: What I learned was that you need to do more than that. I later got a job as a consultant doing this type of work, work for many of the large consumer packaged goods companies like Coke and Chlorox and Pepsi and so forth. And we would help them with certain things, again, performing analyses, perhaps how to do a pricing strategy across all your markets, what to price your products at and all the various sizes and so forth in order to optimize revenue or margin, whatever the objective function is. We would deliver those results in probably PowerPoint and they would say that's super helpful, this is great, this is going to help us really improve our numbers. As a consultant, you kind of move on to your next project that you worked very much project based.

Eric: Out of curiosity, I'd follow up with those clients like a month later and I'd call them up and say, hey, how'd you do with that pricing stuff we told you about? They would respond with, well, we never did it because we didn't know where in the system to make the changes. I'm like, oh my god, you got to be kidding me. And I happened to work for a consulting firm. We also had a software division that made the systems. And so, I knew where to, how to help them with this intellect. Let me come over and I'll show you where to change the prices and all that kind of stuff and how to actually implement algorithms to do this automatically.

Hugo: And I think that would implement there is key, right? That the challenge there was one of implementation. After a PowerPoint slide, how do you then build it back into your business?

Eric: Exactly. This is the part that was just missing from companies, like you could show them what to do but you couldn't make them do it. They didn't know what to do next. Back then, we didn't even call it data science, we called it data mining was the term. The output was just PowerPoint you shared with them. But that wasn't enough because people didn't know what to do with it. To make it really compelling, to actually make a difference because the analysis alone doesn't generate the value, you actually have to implement it to get the value. So to make it practical, you had to get it integrated with systems. That took more computer science skills than anything else. You need to know how to code and get in there.

Eric: But even that wasn't enough. You also had to work with their business teams to really embrace this and adopt it, meaning they had to have conversations about why the prices are being set the way they are and why they're going to be dynamic and not static. Even there needed to be narratives created so that they can work with the retailers they sold to because retailers would get very nervous about would you undercut my private label price, right? They didn't like that, so you have to teach them about how to do these things.

Eric: And so, I learned that it's far more than the statistics alone or even if you back up even further economic framing, how to frame this problem as a model in this case, we're probably if I recall, we're linear regression, and how to train those models took statistics. But it also required that all the tools of implementation, getting it done, integrating with systems and also giving the business context so you can get adoption both from the company and the company's partners.

Eric: That's where it really kind of led into what is now called data science, I believe. Again, that term didn't exist back then. And I recall, this was my foray into data sciences, just kind of this organic growth of wanting to make a difference. But I remember when I decided to go back to graduate school, I had to write a statement of purpose for Stanford University. I remember we didn't have the term data science back then. So I didn't know exactly how to tell them what I'm passionate about. But I wrote down something that went a little bit like this. Listen, I know how to write code but I'm not native to any one language. I speak statistics but I have an obvious accent. I know how to operate a business, but I certainly don't identify as a business person. So what am I? I left it dangling like that but I implied that I wanted to combine those things to create some new discipline.

Eric: And then years later, the word data science was coined kind of simultaneously by Jeff Hammerbacher, Hillary Mason and DJ Patil. And I go well, I think that's it, I think that's the word that I've been trying to come up with. So I globbed on to that several years ago back when I was with Netflix and I named my team that. It was called Data Science and Engineering was the name of the team.

Eric: But I was glad somebody came up with a word that captured that because statistics didn't do it and computer science didn't do it and business didn't do it, but it was some amalgam of the three.

Hugo: Exactly. And something else you spoke to in there was the fact that a lot of people when speaking about data science, they think of it as moving from data to insight, right? But I think a point that was a throughline in what you were just saying is that we also need the intersection of data science and decision science. It's not enough to go from data to insight necessarily, but you need to make informed business decisions, informed from the data science, right?

Eric: Yeah, that's absolutely correct. You'll find that a heavy weighting at Stitch Fix towards what we call algorithms. So my team consists of primarily three types of people. There's data scientists that are developing the algorithms. We have a data platform team that builds a lot of the infrastructure and tooling that can make the data scientists more efficient. Then we have the analytics engineering team. The analytics engineering team is both newer. They're only about five whereas the data science and data platform team combined are about 100. And that's a very different weighting than you'll find at most companies.

Eric: The reason for that is the insights are very important. We need to keep the company informed. But it's mostly to do just that, keep them informed. So our analytic systems, you know, the dashboards and reports that our business users use, they are important, but they shouldn't be made or shouldn't be used for a lot of decision making. Probably at the high level, it's okay, but for detailed strategic things, I really get nervous about it. There's a lot of confounding factors that we've revealed at Stitch Fix. These are the same confounding factors that have always plagued all companies. But I think the only difference here is we've actually found them here. We have the data to reveal them. And so now, we can no longer play by the rules of "ignorance is bliss". We actually know better and we have to be more responsible.

Eric: So we do have an analytics team but it's much smaller than the data science team. The reason for that is because there's usually a lot more quantifiable value of putting things into action as opposed to informing other humans. And then, there's a disconnect, you may inform a human but what do they do about it. The problem is twofold is one, you need to somehow, if you're informing a human, it's hard to get them back into the loop of a system to take action. But second, there's usually confounding factors that the human is not going to be able to diagnose and probably take into account.

Eric: And so sometimes, you're better off with a machine only route, meaning let's have the machines automate this, they can control for those things and then since they know that action to take, let's automate it. Other cases, you want that human decision making in the loop, right? There's something more that a human can bring to the table, whether it's values based decision making or ambient information or even the ability to improvise. Those are all qualities that are in the purview of humans, and sometimes we want to incorporate those things.

Eric: So we do have a myriad of solutions here. Some we call MTM or machine to machine, others we call MTH, machine to human. We leverage a mix of them including sometimes humans in the loop of the algorithms, meaning they're part of the algorithm as opposed to just digesting the output.

Hugo: Yeah, and I'm really excited to discuss the details of how you consider humans in the loop, particularly in the way you think about styling at Stitch Fix. One of the really interesting things about your work at Stitch Fix for me and for our listeners I think is that as we said earlier, that data science is inextricably woven into the fabric of Stitch Fix and that Stitch Fix is a company that leverages data science for part of its strategic differentiation. I was just wondering what this means to you and how does it differ from the use of data science in other companies.

Eric: When data science or anything for that matter is part of your differentiation, it really takes a more prominent role. So, the way to think about this is we are a retailer. Stitch Fix, if you were to define the business model, it'd be in the class of retail. That is because we buy merchandise at wholesale and we sell it at a retail price and the difference is called margin. So that's the exact same business model that has existed for hundreds of years. Of course, the way we do things is pretty different, right? The fact our clients are not picking out the clothing themselves. So they're signing up for this service to do that. That changes the game. I already mentioned earlier that means this delegation of responsibility and data from the consumer to us, they say, well, here, if you're going to do this for me, I'm going to have to tell you about myself. Height, weight, age and all that kind of stuff comes over to us and now we're responsible for picking up the clothes for them.

Eric: All right, so we have the transference of information that used to be once private to the consumer is now we hold in order to act on behalf of that consumer. So that's the big difference that shifts the data. But from a company standpoint, with respect to differentiation, it also changes the game. We set this company up from the very beginning, or, you know, the credit goes to our founder and CEO Katrina Lake, this was her brainchild. And she said from the very beginning, well, this is going to be a different type of service. We're going to have to differentiate very differently. Most retailers will pick some of the more common ways to differentiate such as price, having better pricing than your competitors will give you an advantage.

Eric: But we said we're not going to do that, we're going to be a full price model. We don't do any sales or promotions or markdowns, that type of thing. We don't mark the close up any higher than anybody would do the standard markups. But we won't discount them either, that we believe it sort of cheapens the brand, it's not what the experience is supposed to be. At Stich Fix, is supposed about getting relevant clothes, not the cheapest clothes. So, we're not going to differentiate on price.

Eric: Another great way to differentiate yourself as a retailer is brand. If you think of a company like Prada, it's amazing what they have achieved. They can charge 10 times more for their products than comparable products just because it says the word Prada on it. That is an amazing place to be in. But that's not what we're going to be setting out to do. Nobody's going to pay anything extra because the clothes are coming from Stitch Fix.

Eric: Other ways would be fast shipping. Amazon's wonderful at that. You want something fast, you should go to Amazon. If you need same day or even next day delivery, they're wonderful at this. That's not the value proposition of Stitch Fix. Those are very traditional means of differentiation, price, brand, fast shipping. And we said well, we're not going to have those. What are we going to have then? And we said, well, we got this rich data. Why don't we be the best in the world at data science meaning getting relevant clothes into the hands of our clients and running an efficient business. That's something we can differentiate on.

Eric: So, that's one of our advantages, we have others, but that is the one certainly relevant to me and why I ended up joining Stitch Fix over six years ago was wow, that puts me in a wonderful position to have every incentive to be the best in the world at what I do. And so I get to try these crazy ideas that just might work because if they do work they're going to have an outsized return on this company. And so, that's exciting place to be if you're a data scientist is anywhere where what you do really matters to the company.

How does the client experience work?

Hugo: Absolutely. And it sounds that data is so central to what you do with Stitch Fix. So, what I'd like to do now is kind of go through the Stitch Fix client experience and identify the roles data science plays in each step. But before we do that, perhaps you could just give me a quick overview of how the client experience works.

Eric: The model is this: you come to us when you need clothes. It's an e-commerce site but minus the shopping. There's no browse pages or product pages or search boxes on stitchfix.com or on the app. There's no shopping in the app or on stitchfix.com. Instead, you're going to go and put in your preferences. The things I mentioned before, age, height, weight and sizes, preferences for fit, preferences for style, etc. And then you're going to let the service pick out things on your behalf and send them to your doorstep before you even see them.

Eric: The experience is you get a curated set of merchandise to try on in the privacy of your own home, which is a much better experience than going to the mall. So you get to try them on the privacy on home with your own shoes and wardrobe. You get feedback from a loved one because it's pretty hard to drag somebody to the mall these days. You don't have to make a snap decision. You can take them off, try them on again tomorrow before you make a decision. You don't need to decide there in a dressing room.

Eric: So a much better experience, and the value proposition to the consumer is obviously convenience. You don't go to the mall, or even if you're shopping online these days, it's no picnic because you have to digest so much information about ratings and reviews and search and filters and all that, that it could take several hours to buy clothes online as well, especially if you're not clear on what it is you want or even should want.

Eric: So, that's the convenience property of Stitch Fix is it comes to your home and it's curated for you and you don't need to go anywhere. But the more compelling value proposition I think is these two other things. One is what we call discovery. Precisely because you're not picking up the clothes yourself, you're open to more or you're open to try new things. So for example, we hear this feedback a lot from our clients: well, I would have never picked this out for myself. I saw something like that in the store, I just didn't think it was for me. But you send it to me, you compelled me to try it on and turns out I loved it. I don't know what I was thinking. This is definitely for me, it's like my favorite thing ever, right?

Eric: So this is very consistent feedback we get that we do open our clients eyes to things that they wouldn't have found on their own. So that's the discovery aspect.

Eric: And finally is one of confidence. It's a little more subtle of a value proposition but it certainly resonates with clients who have tried the service and after they get going, on their third or fourth shipment, they just have to realize this value proposition of confidence. My narrative goes something like this. When we're picking out clothes for ourselves, we're never quite sure, is this thing currently in style? Am I wearing it correctly? Is it appropriate for my age or my body type? We harbor these subtle insecurities. But when something comes from Stitch Fix we know all that stuff has been taken into account and you can check those boxes off and you don't worry about it. And so it does instill this confidence in you that yes, this is the appropriate thing for you.

Eric: So, those three things combined are very different than what you're doing when you're shopping at other e-commerce sites or going to malls or even other shopping experiences where more commodity products, you know exactly what you want, you just go get it. This is different, where you want help with this, you want somebody to tell you what would be appropriate for you or to suggest things for you or what is currently in style. And so, it's a very different value proposition to the consumer.

Hugo: That's fantastic. I presume there's some sort of trade off between sending out to a particular customer 100% of things that work for them but also wanting to get feedback on stuff that doesn't work, right?

Eric: Oh, that's right. It's very important that we get both the good and the bad, right? In fact, you could argue you learn more by introducing more variation like that. If you're sending out only things that people love, you're not going to learn as much about them. I use a story from my Netflix days. In Netflix, when you hired a new hire that you introduce them to the company by sending out an email, one little quirky bit is you would put what their favorite movie is. So many people would say my favorite movie is Shawshank Redemption. And that doesn't tell much about the person because everybody's favorite movie is Shawshank Redemption. And so, that doesn't reveal much about your preferences or who you are.

Eric: But if you were to say, well, I really hate Quentin Tarantino. Okay, now we're talking, I just learned something about you that I can use, right? And so, sometimes what you don't like is as important as what you do like. It can reveal some of those preferences and that allows us to learn much better over time.

Hugo: I like that example. And of course, finding out that someone hated Shawshank Redemption would tell you even more about them.

Eric: That's true. That would tell they're an iconoclast. They don't Like what everybody else does.

Client Experience

Hugo: Exactly. So, I'd love to jump in and have you walk me through the client experience and just let us know about how data science influences each step.

Eric: As a client, it's not ostensible to how much you get to enjoy the benefits of algorithms because a lot of it, almost all of it is behind the scenes. But, if you go through the normal experience of filling out that style profile and getting your fix, that's what we call a shipment, it's called a fix, you're touched by several algorithms through the process.

Eric: So, first of all, we use a phrase at Stitch Fix: "there's an algorithm for that". Like I said, nearly every business processes can be augmented or enhanced or even operated by an algorithm, given the data sets we have here. And so, we've found places for algorithms in nearly every function of the company. So, one of the first things that's going to happen when you request a fix from Stitch Fix is, well, we got to match you to one of our many warehouses. We have a lot of warehouses throughout the country that we store our inventory at. And so there's an algorithm that does that. It matches you based on the inventory and its relevance to you. It also tries to minimize the shipping time to get it to you and that kind of thing. So there's an algorithm for that.

Eric: Then it's going to match you to a stylist, a human stylist. We'll get to this in a moment but we do combine machines and humans in our styling algorithm. But which of those humans we need to pick is important. For the machines, just pick any of them, they're homogenous. But humans are far more heterogeneous. It matters which one we pick for you. So, we have an algorithm that will pick the stylist with the highest chance of success for you.

Hugo: And is this some sort of recommender system of some sort. Is that how you tend to think about it?

Eric: Sure. The end result to the consumer will be a set of recommendations that they get in the flesh. They're shipped to their door and they get to choose from. Behind the scenes where using machine algorithms combined with human judgment to pick those that set out for the-

Hugo: I actually meant to pick the stylist out.

Eric: Oh, the stylist. It's a little stronger than a recommendation, it's going to happen. So, we do assign this stylist algorithmically to the client. Now there is some overriding that could happen if the stylist feels that there is a ... different styles that would be a better one. But those are rare. Generally they'll follow the algorithmic assignments, because the algorithms take into account so much. There's things like, well first of all, capacity constraints. Styles only have so much time they can work, we allow them to work very flexible hours. And so they tell us how many hours they're going to work this week. And so that's that's a capacity constraint, how many clients we can assign to that stylist.

Eric: But more importantly, relationships matter. This may be an existing client that has been working with a certain stylist for several years. If they've had a successful relationship, we'll try to keep them together. But there are also other things that you need to consider such as affinities between the stylist and the client. We have to learn these one by one such as what is the revealed style of the client as revealed through our algorithms and what type of clothes do our stylists like to send, right? So if we can match those up, we'll have a better chance of success.

Eric: And so, all these things are taken into account by our what we call the smart fix assignment algorithm or SFA as we call it internally. Some of the ways that they're matching, there's a whole suite of logic underneath it that matches clients to stylist. And then once they've been paired up, then we need to perform the styling process which is that's more where we're picking out the clothes for the person. That's the piece that is really a hybrid model between machines and humans. The two are collaborating. So machines take on all the things that have to do with rote calculations, which would include all the various algorithm techniques that you can think of. Things like collaborative filtering and mixed effect models and neural networks, etc. Machines will do all those things to recommend merchandise over to the stylist, and she's presented with all the algorithmic output rank ordered in terms of relevancy.

Eric: But we won't stop there. The stylist isn't going to just take the top five from the algorithm. She's going to layer in some judgment because humans have access to a little bit more information. They can leverage ambient things such as a magazine, a fashion magazine sitting on the coffee table, or observing who wore what to the Oscars. These are all things that are ambient and unavailable to the client. Other things that you need to leverage are more nuanced context. Imagine a client writing in a note, we allow them to request using text, add a little bit more information to what they're looking for. Imagine a client requesting something like well, I need some clothes to wear to my ex-boyfriend's wedding. All right, that changes the context. The meaning to that is going to be better digested by a human. A human can empathize with what that really means to this person, right? Even our best natural language processing algorithms are not going to get to the level of empathy that a fellow human can.

Eric: So we need to instill that into our recommendations for this client, right? And so, we leverage humans for that. That goes to our final thing, which is relationships. We want to have human to human contact so that we can foster a relationship. People treat humans a lot differently than they do machines. If you've ever tried chatting with a chat bot, you're typically pretty rude to it. You don't really answer all that synchronously. You give one word responses. You don't really check your spelling and so forth. But with a human on the other side, you're going to write a little bit more thoughtfully and you're going to engage more. So, we need that in all of our fixes. And so, that's how we're trying to leverage the machines for all the calculations, leveraging all the data and hidden relationships. But humans for curation, empathy, relationships, and even the ability to improvise occasionally.

Eric: Those things are in the purview of humans and probably always will be. And we need them both. So we've figured out a way to combine them in our processes.

Hugo: That's really, really good. I love the example you gave, because I was wondering, having humans in the loop for interpreting nuance language is incredibly important. But I'm sure you have a whole bunch of natural language processing stuff to deal with the less nuance typed of notes people can make. I suppose I was also thinking you probably have a whole bunch of image analysis or computer vision algorithms to deal with, photos of clothing and that type of stuff as well.

Eric: Oh yeah, we have an algorithm that we've experimented with that pulls down Pinterest pins. A lot of our clients make us Pinterest pin boards because it's much easier to express in an image of what kinds of things they like than rather than describe them with words. So they pin a bunch of blouses and dresses, jeans, that kind of thing to a Pinterest pin board and we're able to digest them with machines and train with computer vision algorithms to figure out how to strip out the backgrounds and that kind of stuff. And then find very similar items that we own in our inventory that might be good recommendations for these clients. And of course, our human stylists also look at those results and they're also varying their judgment on them. And so, it's a good collaboration between humans and machines.

Data Driven Fashion Design

Hugo: Fantastic. So Stitch Fix is also placed at the forefront of data driven fashion design. I'm wondering what data driven fashion design as a concept and practice means to you and Stitch Fix's approach to it in general?

Eric: Well, we do use algorithms for designing some of our clothes. It's the vast minority of the clothes, but we do have a program called Hybrid Designs that leverages algorithms for designing the next set of fashion styles. So this was the brainchild of one of our data scientists, a fellow named Daragh, who tinkered with this. He tinkered with a form of genetic algorithm where he's going to take all the clothes that we've ever sent in our history and he's going to look for gaps in the market: What clothing should exist but doesn't? To do this, he has to have an audience in mind. So say younger females that are looking for cardigans. And perhaps there isn't anything suitable for the younger females in the class of cardigans.

Eric: So what he'll do is say, well, we should really provide that and if there's no vendor that's going to make that, we'll design it ourselves. And so, the process for designing it is by looking in the past at what has been successful and we're talking millions and millions and millions of points of feedback. We're able to decompose the garments into elements, say, a collar type and a sleeve length and a color, a pattern etc. And so there's a couple dozen of these elements, each taking on between two and 30 values that can hold, like patterns could be plaid or they can be houndstooth, etc.

Eric: So taking all that data, quantifying each thing, separating and then recombining them. And again, this is where it's a bit generous to say they're a form of genetic algorithm. It's more or less the same process that Mother Nature uses for evolution by natural selection. That is really two properties, recombination and mutation. So, in nature, we recombine the genes of two parents to create a child. In clothing, what we're doing is combining elements from several parents to create something new. Say, we take a silhouette from one garment and the sleeves from another, maybe a collar from a third and a pattern from a fourth. And then we recombine them to create something that has never existed.

Eric: So we are recombining elements of clothing. So we may take a silhouette from one and a sleeve type from another and a collar from a third and recombine them to create something that has never existed before. And then we might also introduce random mutations, say a new pattern or a new color that might enhance this, the applicability of the style. And so, we have the algorithms doing the search cost. What they're going to do, I mentioned before, there's several elements, each with multiple values, and if you do the multiplication, that results in literally trillions of combinations that you could evaluate.

Eric: So, what we do is ask our machines to traverse that search space and come back to us with say the top dozen or so designs that would be most likely to succeed with our target audience. So they come back to us after they do their calculations and they present these designs and then we do let our human designers augment them. Let's say, well, they loved the pattern but maybe make it a little less symmetrical or change the color of red a little bit. Those types of things.

Eric: The result is a final design that we actually have manufactured and we send them out to our clients and they have been wonderfully successful. We started with just three blouses where we did small units, we're talking like 1000 each. They did so well that we created a dozen more and then we did another dozen more. We've done up to 100 design so far. I think it surprised us all with how well they've done. It was a bit of an experiment by this person Daragh a data scientists that, I don't know that any of us had all that much confidence in it, it just seemed like a neat thing to try so we tried it out and to our surprise, it really works well. So we'll keep expanding this program.

Hugo: That's really exciting. If I recall correctly, Daragh actually wrote a really fantastic post on it on your blog in I think probably summer 2016 or something like that.

Eric: That's right. There's a lot of great blog posts out there on Multithreaded, that's our tech site, multithreaded.stitchfix.com. And so, there is a good blog post on using algorithms to design clothes.

Hugo: Yep. And we'll definitely link to those in the show notes. So we spoke briefly about the role of humans in the loop with respect to data driven decision making at Stitch Fix. Some of the points you made which I find really attractive, you know, the ability to improvise, empathy and relationship building. That's definitely a huge part of what you're doing at Stitch Fix. I was wondering how you feel about the roles of humans in the loop in general in data science and as we move forward?

Eric: I think humans have a lot of value to add to processes, right? Again, there's a lot of things that, they have access to information that machines don't, such as values, what do we care about. Sometimes that doesn't manifest itself in the data. Or even smaller decisions, such as, we have an algorithm that actually helps to buy the clothes and they might be saying, we got to go buy 10,000 units from this vendor. A human might be able to look at that and say, well, that's true. That performs well for us, but that vendor is unethical. We don't like the way they do their business. We're not going to do it. And so, that's something that wouldn't be available to the data and the machines, yet the human will very much enhance that decision in this case not to buy it.

Eric: So humans do add a lot of value, but they also need to be mitigated. Us humans, we're not good with math. We're not good with statistics. We kind of lack this in our genetic code through the way we evolved, you know, back in our hunter gatherer days, we were almost rewarded for using heuristics. We don't have to take in so much data, we just make a knee jerk decision on things. This story I like to tell is one of them, imagine some ancient hunter gatherer that was very astute and liked to make measured decisions. If that person came across like a saber tooth lion and he said well, I wonder if that's the man eating type, is it a carnivore, is it angry right now. Yeah, that guy got killed, right? But the guy that made a knee jerk decision and saw Sabretooth, I'm out, he survived at a higher rate.

Eric: And so, the snap judgment decision making was rewarded and therefore manifest in our genes. The older part of our brain, the limbic system is filled with this type of knee jerk reactions and type of decision making where it's more quick, emotional response, almost involuntary versus the neocortex, the newer part of the brain. This is the part where you would use it to do math in your head. And as we all know, we're pretty bad at it. Like even ask me to hold 10 numbers in my head, I probably can't do it.

Hugo: And as you said, our statistical intuition is perhaps even worse also.

Eric: Oh, it's horrible. We just don't, we can't do the calculations in our head. We have recency bias and confirmation bias, every bias, all the cognitive biases from Kahneman and Tversky, we're guilty of them all. Again, there was a reward system for that and we survived at a higher rate for having those biases. But they no longer serve us well in the context of business when we have more data available to us to make better decisions. So we almost can't trust our guts sometimes. We like to make up narratives that are not true. We say, oh, this went up because I did that. But we don't have any evidence that supports that. And so often that we want something to be true that we fall victim to it, we conclude that it is true. It must be true because I did this thing and then it happened.

Eric: And if we take a more disciplined approach using data, it can often reveal how wrong we are. And so one of the things, of course, like any modern company, we do a lot of A/B testing at Stitch Fix. It's amazing how often our intuition fails us. We do things, like we collect what I call a priori judgments. Before we launch an A/B test, we collect from employees, hey, what do you think is going to happen with this test? And well, everybody will write them down, we record them in a Google form and then we run the experiment. And once we have results we compare to what we thought would happen and we're often wrong, just plain wrong, like not even close.

Eric: But this is new. This is a new practice, only in the last 10 or 15 years of doing these randomized control trials or A/B testing. That hadn't been done prior to 15 years ago, not in most domains anyways. To see our success rate or how successful our intuition is, which the answer is pretty bad, it's not great, it makes me think back to how many decisions were made for hundreds of years that were really horrible but people never knew. We had the benefit of "ignorance is bliss". You make a decision, you do it, you'll never know the outcome.

Eric: But today, with A/B testing, we can test all these things and find out, wow, we're pretty wrong about stuff. And so, it just emboldens us to leverage data science when you're making decisions to make sure that the outcome of our decisions is having an impact that we thought it would. Or even, we no longer, I no longer get too attached to the outcomes of making sure it's the way I wanted it to go, because even when you're wrong, you get to learn. An A/B test where you learn not to do something is as valuable as one where you should do something.

Hugo: Yeah, absolutely. I know, of course, the browser based interface is incredibly important. But also, is the experience of receiving clothes. I do think to improve the client experience, I'm sure you need to think about the future of fashion. And essentially, this is a prediction problem of thinking about how the future of fashion will evolve and your role in that and whether data science can be used for that or if that's more of a human task.

Eric: Yeah, that's a great question. So, forward looking on what's fashion going to be six months or nine months or 12 months out from now, it turns out that existing data isn't particularly valuable. It has some merits but it's not enough to really make that decision of what might be interesting to people six months or a year out. That tends to be more of a human task. So we have human buyers that are actually buying the new things, the things that we'll have in our inventory six months from now.

Eric: This has more to do with all those things that I said that humans are great at. Understanding social mores, leveraging ambient information like who wore what to the Oscars. These are things that humans are great at and we relegate that task to them because we don't have, it's not much that our machines can help with in those things. And so it's a human task and they have to take their leap of faith on buying things. And what we can do is once they've bought something and they get it into our inventory, the machines can take over from there and decide, well, should we rebuy, should we get more of this stuff?

Eric: And so, we've divided the work, it's another place we're combining machines and humans is in merchandise buying. The work is roughly divided between the algorithmic rebuying of things or once we know something works to go rebuy it, it's done algorithmically. But the exploring of new things is still a human endeavor. And so, the world is divided, or the world of buying at Stitch Fix is divided into explore, exploit. Humans for exploring, algorithms for exploiting. And so, it's a nice balance where, again, each resource humans and machines are doing what they're great at.

Hugo: That sounds like some sort of human in the loop reinforcement learning challenge as well, right, framed in that fashion.

Eric: That's right. The nice thing about humans and machines together in the same system is they get to learn from each other. We're not training our machines to act like humans, but we are training them on what works. So we do get some interaction and some feedback between humans and machines. In some of our internal interfaces, you'll have these little thumbs up and thumbs down things where our internal workers can click on and say this was a good recommendation, this thing wasn't. And so they do get some responses from them. And of course, the machines inform the humans the outcomes of their decision. through not just A/B testing, but all the outcomes of their decisions and in terms of metrics and performance of the things they bought in the past.

Hugo: Yeah. And an aspect of this practice and philosophy that I find really attractive also is what is essentially an axiom for you at Stitch Fix, to allow humans and machines to do what they do best because they're very different, they're orthogonal in so many ways.

Eric: Yeah, this is a very different perspective on AI. The narrative out there in the world is that AI is going to take over the world and leave us humans without jobs. It's quite the opposite at Stitch Fix. Where AI is intruding into jobs is actually making lives better for our humans, meaning they get to seed the rote tasks over to machines. These are all things you don't really like to do anyways. A lot of rote calculations and rote processes, let's give that to machines and let's leverage humans for doing the things that only they can do and the things that they do better such as improvising and relating to other humans and the ability to appreciate aesthetics. These are the things we like doing. And we get to do more of it, because we're giving away parts of the job to machines.

What does the future of data science look like?

Hugo: Absolutely. So I think that really leads well into my next question is, which is really another prediction problem that none of us have a concrete answer to yet. But I'm wondering your views on what the future of data science looks like.

Eric: Well, I think there's still so much to do at integrating it into processes. It's kind of we opened with it from that story I told from the consumer packaged goods company. This was 15, 20 years ago, where it was hard to even get them to change their prices. When you find an optimal price, how to get them to implement it. And that's still an issue in a lot of companies. Integration into what were formerly human processes, it can be a challenge. There's not a lot of robust APIs for things. We do have them for things like Google and Facebook, but there isn't an API for say buying clothes. What we need to do is we need to get those humans in the loop since they're the ones that are going to go reach out to vendors and purchase orders to buy their clothes. We need to get them on board. That means trust. This can't be black box algorithms. You're going to have to provide some transparency in how they work so that they can feel good about the decisions they're making.

Eric: It turns out that has a lot of implications for how you design systems and how you design the processes within an organization. So, you really have to take time to do this very diligently. There's a lot of interesting things that emerge out of introducing algorithms, such as you're going to force clarity on objective functions. For human processes, you can be kind of ambiguous on things. But machines need very clear objective functions. They need very clear constraints. No longer can you fudge things, you're going to have to very clearly provide instructions on what to do in every case. And sometimes it's going to be uncomfortable. You may be familiar with the trolley car problem where, I think the narrative goes something like if you're a trolley car operator and you're rolling down the tracks, and you have your choice, the brakes are out, right? So now you have your choice. I'm either going to kill that one person on that track or I can switch tracks and kill a whole group of people. Which one do you do because neither is a pleasant outcome? I've seen this come up in the context of autonomous vehicles. They have to be told something, what to do in this situation. And it's a very uncomfortable thing that as humans don't like to really have to talk about. We don't come up with those contingency plans. We just pray that we're never in that situation.

Eric: But if you're going to cede something to an algorithm, you're going to have to be clear with what you're going to do. Now, that's a very morbid example that we fortunately do not have to face at Stitch Fix. But we may have to come across a far less uncomfortable situation. Something like we have one more unit of this dress and we have a thousand clients that like it. Who should get it. And so, you do have to be explicit with those instructions, you have to have some way to break the tie. For humans, we can be ambiguous. We can do the first one, we can do the one we like best, whatever. But you're going to have to code something up if you want an algorithm to operate on that and be clear.

Eric: So that forces a lot of very interesting conversations that were kind of hand-waved away by either ambiguity or ignorance is bliss. It's actually interesting when you have to address these things to put in a system. You're going to have to get very explicit.

Hugo: I think something we do when we develop algorithms or APIs, we do encode certain biases and having humans to be able to help us figure out what those systematic biases are as well is incredibly important.

Eric: Oh, absolutely. Like figuring out what things we shouldn't be allowing these algorithms to do, that type of stuff?

Hugo: Exactly.

Eric: Yeah, I do get a lot of questions about what is known as the filter bubble out there. If your algorithms are getting really good at what works, won't they only send things that work? My response to that is that you can code them to do what you need to do. You can code them to take risks if you need to. There's lots of literature out there on exploration bandits and so forth or even just taking the upper bound of confidence interval to encourage risk taking on the part of machines. So you can train them to do whatever you need to do so that they don't narrow into niches or bubbles that you can bounce them out of them when you need to.

Eric: So I think most of that is addressable. And of course, there's the more nefarious bias in algorithms, which again, we're mostly removed from at Stitch Fix since we're not doing financial loans for people or anything. We're picking out clothes for folks.

Hugo: Yeah, and it sounds as though anecdotally at least you're doing a very good job from the stories of people responding and being like, hey, this is something I would have never thought of wearing. But you sent it to me and it actually works, so thank you.

Eric: Good. Yes, it's always good to hear those stories.

Favorite Data Science Technique

Hugo: So, we've dove into a number of different algorithms that you use it at Stich Fix. I'm wondering on a kind of a personal professional level, what one of your favorite data science-y techniques or methodologies is?

Eric: Great question. Personally, I'm less attracted to techniques. My favorite thing is framing. I love the art of framing a problem. When somebody comes to you and says god, here's the dilemma, we want to do this but we don't know about that. Usually these are couched as the need for some algorithm with uncertainty. So, there's lots of outcomes out there. Some don't have any uncertainty such as a sorting algorithm. That's usually not where we play. We love the algorithms with uncertainty. Those usually warrant a different framing. And so usually you get to do some real math, no numbers at all, just algebra.

Eric: It's an amazing feeling you get when you know you've nailed it, when it feels right. Wow, I think this captures all the phenomenon, all the relationships between things. Then you really get a nice little tickling feeling when you can use the math to do some intuition such as check boundary conditions, things like that. And it's just a wonderful feeling to know that I think I've captured this right. And then of course you have more work to do with empirically validating model. But that art of framing is really probably the thing that gets me going the most.

Eric: Now, if you ask me to pick a technique, I think one of my favorites is principal component analysis or PCA. The reason I love it is because, first of all, it's an unsupervised technique which means I can just throw the data in there and see what happens. But what I like about it is that again, maybe it's back to this paradigm of machines and humans working together, is that the machines take on all the rote calculations, finding the eigenvectors. That's a machine task that takes millions, if not, depending on your data set, billions of rote calculations to do.

Eric: But once it returns its results to me, it returns kind of the various components, it takes some human intuition, something perhaps I'm good at to name and label these components. Like, what is this direction it found it at and what is that thing mean? It's up to human interpretation to assign a label to that thing. I just love that interplay between machines and humans in that context.

Hugo: That's awesome. I love that one of the things you love about PCA is that a tells you immediately what explains the most variation, particularly in the framing of you're working with and at Stitch Fix where human bodies and measurements of bodies are so wild and various, and there are so many ways in which we can move, and thinking about it in those terms is really cool.

Eric: Absolutely. I love what it reveals and it's usually not intuitive. I mean, you got a lot of dimensions you can throw in there. Things like age and price and style and fit. It's almost always a surprise of what the direction explains the most variation is. It's usually not what you think.

Call to Action

Hugo: Yeah, absolutely. So Eric, as a closing question, I'm wondering if you have any final calls to action for our listeners out there.

Eric: Yeah, I'd like to offer up two. First of all, for consumers that don't like to shop, check out Stitch Fix. We serve men's, women's, and more recently, we've launched our kids line. So just in time for back to school. It's great way if you hate shopping to go find things that you probably wouldn't have found on your own and it will change the way you shop.

Eric: Now, also, I'll give a plug out there for data scientists. We have a nearly ideal environment for practicing data science and we're always looking for folks. We do what we call the full stack data scientist. They have to be pretty diverse in skill sets, we make you do all the ETL yourself but of course the model framing and the model selection and training and all that kind of stuff. You do though your own product management, project management, all that kind of stuff. So you have to be a full stack data scientist. You have to be curious and you have to be passionate. And if you are interested, go check out multithreaded.stitchfix.com and you can see some open roles we have there.

Hugo: Absolutely. And you actually have a really great blog post from a few years ago now called Advice for Data Scientists on where to work, which spells out a lot of different ways to think about where to work and actually says, when you feel the way you do and the way you've just explained why Stitch Fix would be such a great place.

Eric: Yep. Yes, it's a good read too because data scientists do have a lot of choices these days in where they want to work. So, make a good one, work somewhere where what you do is going to matter, you get to work with great data, and you get to have greenfield opportunities, things that nobody's done before. Those are the big three we like to talk about.

Hugo: Absolutely. Thank you, Eric, for coming on the show. It's been such a great chat.

Eric: My pleasure, Hugo.