Offline A/B Testing: Experimentation in Brick and Mortar with Phillip Paraguya, Chapter Lead Data Science at ALDI DX

And then I would say the, that a good AB test requires a good, good, proper design, so to say, just to make sure that we have enough statistical powered behind it to actually make sure that we can, well reliably recommend something there. It's nothing worse than having a badly designed test and don't have any reliable results at the end.

Adel Nehme: And maybe I wanna jump in as well, because, you know, you talk about reliability and the statistical power of a AB test. I assume this is quite challenging in a brick and mortar setting, So maybe walk us through those particular challenges of running AB tests in a brick and mortar setting for an organization like Aldi.

Philipp Paraguya: Sure, and maybe to, to give a short background for those who are not that familiar with ab B testing to have in mind where we AB test often our way B or B testing is coming from, and AB testing is most often used in. Like in marketing analytics settings, I would say like optimizing campaigns, assessing impact of website changes.

And these are of course very different from brick and mortar because you can very simply change a website. You can very simply change content and have it consumed by different types of people. In a brick and mortar retail chain like we are. This is difficult. Like, let's take a typical retail example of finding the right assortment.

So what's the correct assortment to fit the demands of what our customers want? Well. You could think, okay, let's AB test this. Let's try out different products and let's see which ones have the highest revenue impact, for example. However, the thing is, once you start thinking about how to set this up in an actual setting, you come to, well, okay, if I want to do different products, I need to buy different products from different vendors, so I need different contracts, and then I need different distributions from different warehouses because of course I have actual product.

And then I somehow need to have different assortment at different stores, which I wouldn't necessarily have otherwise. And maybe different promotions even. And so suddenly you're thinking about how this actually impacts your whole supply chain and thousands of people while in theory, you're just testing different products in different stores. And one thing as well, because in AB test, you need to compare the A and the b. Within the test. At the same time, like how do you do that in a store as well? Do you have separate spots in the store that are having different assortments or do you compare it between stores within the same area?

Adel Nehme: Like walk me through kind of as well, like how do you physically set up the test?

Philipp Paraguya: Yeah, exactly. It's exactly, that's the question, right? So usually we would say maybe to take a different groups of stores and, and test things there. However, you have to make sure that you can actually do this from a, for example, logistics point of view. And usually in an ideal lab B test setting, you would even switch them around again after a period of time to have a more stable answer.

But you can't really do this often in like a real store because you would switch logistics chains too often. So it's very much finding a good balance between effort that you're putting into the AB test and getting reliable results.

Adel Nehme: And then maybe outside of this particular example that you just mentioned of assortments, what are other types of problems in the brick and mortar space that lend themselves to an AB testing approach?

Philipp Paraguya: The classical problems from like a marketing analytics standpoint are things that are also interesting for us, and I'm not only talking about like a e-commerce or I. Online world, most supermarkets do have leaflets, for example, as marketing channels. And so promotion channels and seeing which promotion channels do actually work are things that we can a B test and that we can see if they have an impact.

Same with testing out promotions or testing out assortment changes. So everything where we want to basically. Test for a particular impact is something where we can, and we are doing AP tests,

Adel Nehme: So, yeah, if next time you, anyone who's listening, you go into, your supermarket and you see that there's a different leaflet from usual, there's probably a data scientist behind the scene deciding what's going on

Philipp Paraguya: might be the case. Yeah.

Adel Nehme: yeah. And then can you maybe share an example of like AB test that actually did maybe had counterintuitive results or like did have like a kind of semi significant change from what?

From resolve, from what you're expecting.

Philipp Paraguya: Sure. Although I, to be honest, I would have to go before I joined Al ldi, but I guess it doesn't really matter. So a couple of years back, I was the team lead for a marketing analytics firm. It was still a brick and mortar topic, but it was not Aldi stores. I. And back then we tried to optimize conversion rates for their e-comm area.

And we tried to change it based on customer target, group based website design. And so we, we did segmentations of customers and basically created different content for the website based on which group they were part of. And we did this, we did the analysis, and at some point we noticed. Oh, shoot.

We don't have enough customer data at that point in time of our customer journey because one thing we forgot to take into account is, well, at the point of people surfing our website, we don't have actual data of them yet because we, get data after they bought, we get data after they did something.

Right. So we had our. our, or we tried to do a segmentation based on data we didn't have yet. And that didn't really work. And however, we didn't notice this Right away. And this was something that just didn't work out at the end, I.

Adel Nehme: Maybe coming back to the brick and mortar example one thing I often think about is in a brick and mortar setting, in a physical setting, right, even just going beyond brick and mortar there's a lot more potential to have. Variables be corrupted by the outside world and the environment.

Like one thing that comes to mind, what if the weather in one area where you're measuring stores is much worse today than in another area, and that's affecting how many visits you get? how do you create protections against this type of effect and dynamic when doing AB tests in the physical world?

Philipp Paraguya: Yeah, yeah, absolutely. Right, because oftentimes in the physical world, we have lots of variables in there that we cannot. Really control, right? It's things like weather, as you said. It might also be just location, like in my data set, looking at two lines of store might be more or less the same and only a couple of attributes that are different.

But when you then go to the physical locations, it's very different because one might be at the mountains and one might be at in the sea or whatever, right? So it's very different and variables that we cannot really control. Foot traffic being another thing, and I think this is oftentimes a thing of being aware of this happening, being aware of these things happening, maybe getting data to try to have this in the data, like demographic data, data, geographic data, weather data being another thing, and to have them as an option in there, even though most of the times in my experience.

They don't really matter in most cases. There are some cases where it might be interesting, like when I look at, I don't know, then this is now a fictional example, but if I look at ice cream sales, weather for sure will have an impact, right? But when I look at sales across our whole assortment, probably not so much because people go through the store if it rains or if it's dry.

Anyway.

Adel Nehme: You mentioned kind the challenges in the brick and mortar setting. Related to you know, there's supply chain adjustments that need to be done. There's creating contracts with vendors. It's not just an ability to change the color of a button on a website. Right. As simple as that. So I feel like here you need to probably do 10 x the amount of effort to align stakeholders, align department leads. 

Walk me through the, challenges associated with alignment and creating kind of a common AB testing agenda and experimentation agenda. And how, how do you get people excited about doing experiments?

Philipp Paraguya: So I think, and this is a very, agile mindset. I think the Agile community came up with this sort of, and that is transparency is very. Vital, Keeping the stakeholders as close as I can from the beginning. have a feeling that especially in the last couple of years with data science becoming more and more used nearly a commodity with lots of the things we are doing, it's still an education job with people outside of the field.

It's still trying to make them understand how this works, how this maybe cannot work as well. Because I have the feeling. There is a spectrum from, don't trust this, this is the computer taking my job up to this is magic. This can do everything right? And, and there's like the spectrum and you can find both extremes, maybe even in the same project.

And so it's very much being transparent there, trying to educate and this is very much, which has become very important.

Adel Nehme: What's easier to deal with? Is it the folks that have really high hopes of out of data science and ai, or is it the folks that have really high reservations when it comes to data science and ai?

Philipp Paraguya: That's a very good question. this might be also personal preference. I feel like it's easier. The ones that have very high hopes, because at least they come with a very high motivation. I don't have to sell them on the idea. don't have to sell them on the idea. I just have to adjust what they're expecting while still keeping their motivation high.

To motivate someone and educate them on something is for me, at least, more difficult because you need to educate someone who doesn't really. Want to learn potentially as well.

Adel Nehme: And yeah, I, that resonates with me as well. Like if someone has bought in already on the importance of doing this, even if their expectations need to be adjusted, that's much easier of a battle maybe. what have you learned best practices when it comes to like shifting people's perspectives, especially the naysayers?

I'd love to kind of hear some best practice here because this is something um, almost every organization faces.

Philipp Paraguya: Yeah, true. And I, think that resistance to these topics often stem from either. lack of trust in a solution or maybe lack of understanding because they don't really know yet, how can this work? How does this work or should this work at all? And so I feel like working with examples simulations, maybe fictional examples.

Examples from other companies from the past might help there to gain trust in a solution. And also, this is why I, I want to have them close by so that they see the journey, that they see the steps and how we get, how we improve step by step, but also get a feeling of the effort that goes into this and the boundaries that we might have.

Adel Nehme: Yeah, and maybe zeroing in on AB testing itself, When it comes to this, have you ever had a situation where you would encounter resistance, even if. If the AB test had a definitive result that we need to change something, for example, like is that part of the process as well of the education process when it comes to changing hearts and minds within the organization?

Philipp Paraguya: Yeah, I think so because people often go into an AB test with a notion or with an assumption of how this is going to go right, because people already have an opinion. I. Of what's the ideal solution? And of course if the AB test then shows you that it does not actually agree with the, your hypothesis, you well at first run into the wall there, right?

Naturally. And so I think this is where having a good hold of or a good grasp of, quality measures and so on gets very important because once you ran against this wall. they will try to find the things that are not working.

Adel Nehme: holes in your IV

Philipp Paraguya: they, they, they, they're trying to see the hole and rip it open as far as they can because if there is one hole in it, why shouldn't there be another a hundred ones of them?

Right? And so this is why the analysis part, doing quality checks, doing quality control, good testing of my analysis is very important. and I mean, honestly, of course you will do. Mistakes here at the end, right? And this will happen. And of course this does not necessarily mean you're a bad data scientist.

Everyone does mistakes, and this is where then code reviews, having other people double check and those kind of things are getting important as well. I.

Adel Nehme: Okay. And then, you know, it's just like when the results of an AB tests are accepted, everyone's aligned. Especially in these contexts, right, where there's so many teams involved, there's so many business processes that need to change. How do you roll it out? What, what role does data science play in making sure that the outcome was dictated by the experiment actually is rolled out into the business.

I.

Philipp Paraguya: It depends, of course, a bit of what sort of thing we are talking about. Because we can ab test things that don't have a data science component. the testing itself is then the data science, but the, the thing we're rolling out might not be. Oftentimes we are a b testing, for example, software components that have data science parts in them, this is where, for example, then topics like machine learning, engineering and so on get very important.

I like to think of because people have different notions of what a data scientist is or what a machine learning engineer is. Right? And you ask three people and you get four different. Definitions, and I like to think of this a bit of like during the development of a model or something. My data scientist is similar to what maybe a chemist would be, a in a chemistry lab and like creating the new material.

And I'm in my lab and doing this, and now we are in a stage of we've tested this and now we want to roll this out. And then suddenly we have to become the chemical engineer. Like we have our example in our laboratory and now we have to scale it out to an industrial setting, to an industrial plant.

And this where oftentimes, for example, machine learning engineers come in.

Adel Nehme: I think this marks a great segue to discuss how data science is actually done in an organization because in a lot of ways when you look at, universities today, I think a lot of universities, teach data scientists to become chemists. Right. But not necessarily a lot of.

Universities are focusing on how to teach data science to become chemical engineers. Often that's learned on the job, or you become a software engineer who learns data science over time. Right. So maybe first focusing in like outside of AB testing, right? What kind of problems do data scientists Aldi or similar organizations simply focus on?

Because there's often this hypothesis that it's only gonna be chemistry, But maybe give me a bit of the chemical engineering side. What does that look like?

Philipp Paraguya: I think and looking at its core, we are all about creating a business impact, Because at the end of the day, we are data scientists working in a business, focusing on delivering some benefit to a customer while sustaining a healthy margin. For us as an enterprise, and this is maybe not the most magical answer to, but this is the reality of what we as hired data scientists are doing.

Adel Nehme: come from revenue.

Philipp Paraguya: Exactly. Salaries come from revenue. And if I want a data scientist salary, I have to create revenue somehow. And so this of course can be science, But it is important to keep in mind I'm doing this to create a business impact. And these benefits are, are what we have to keep in mind when creating solutions.

When I am in a university, this is not what I'm doing. This is not my mandate of what I'm doing there.

Adel Nehme: Yeah, that's, that's completely true. And then maybe if you expand on the actual problems that, like you may encounter that require a lot more engineering skills or machine learning engineering skills, as you mentioned that you may not encounter in university, for example, I.

Philipp Paraguya: In academia, my mandate is different. I want to generate knowledge, I want to answer questions and so on. So. Why bother with dirty data? Why bother with problems in my data set? Because these are harmful to what I wanna do, what I learn in reality, this is different industry is different.

Data comes from everywhere. Data might be the Excel file that someone in the business has created that has lots and lots of holes in them that I need to create some business of. And so data is messy. Data is incomplete, and we need to work on them. We need to engineer them to get to a point where it actually can build models where it actually can.

Well derived decisions from, and this is something that's and I work a lot with with students coming freshly from university, junior positions, entry level positions, and oftentimes they're very surprised in how much work we do in engineering, how much work we do in data cleansing and how little in comparison we actually do the scientific part.

Adel Nehme: Maybe focusing in on here, right? Like if, you were to give advice to someone, just like going into the field right now, what are the skills that they need to develop the most to be able to succeed in a data role? I.

Philipp Paraguya: So people that come from university are often very fascinated in the modeling parts, right? It's, and I agree, the, the having some ones and zeros predicting some complex context is very exciting and it feels a bit magical. I. But I mean, let's be honest, most problems that we have in the businesses are not using every state of the art method are very easy, are very simple.

A simple regression might not only be enough, but might be the most feasible option. So I would say as an advice to impact more on, on the one hand data driven skills, engineering skills, and so on. Skills that are more soft skills, storytelling problem framing and on those kind of things, understanding business problems.

Adel Nehme: Okay. And speaking of business problems and storytelling, I think communication. I often think about communication. Is probably one of the most important skills that data professional needs to have, especially if they want to grow in their career, cause you can be excellent technically, but if you don't have the communication skills that will put you as the most technical person on the team, and then you stay in that role, right?

Because you're the technical problem solver. And in your view, what are the core communication skills and soft skills like data, data professionals need to have to be able to succeed in a particular role? 

Philipp Paraguya: and I a hundred percent agree, I always feel like coming to a per to through certain seniority, the difference in the ones that go further are more the soft skills than they are the hard technical skills. I. And I feel like knowing your audience and knowing what do they need at the moment to understand what I want doing or to give me what I need.

Do they actually need to see my print of feature importances for type of loss functions and my model representation as a couple of trees? Or do they need to understand? What the business impact is of my analysis first, and do they need to understand what logical next steps come out of what I am actually proposing?

And I feel like understanding these things of what your audience at that moment needs and tailoring, then your message around these kind of things is a thing that can get you a step further.

Adel Nehme: Based on your experience and kind of in your own growth path as well, what are some best practices that you recommend for. Folks who become more better data storytellers, especially if they're presenting to non-technical audiences.

Philipp Paraguya: Over time, I feel like you, you built yourself a toolbox of different things that you pull out during these kind of things. And for me, I'm a fan of asking. So why am I presenting right now and what do I hope to get out of this? And what do my audience need or why are they listening to, what am I saying?

And that sort of focuses and frames me my presentation that helps me reflect on what do I now put into the presentation or what do I leave out of it? And, and this is I just said, feature importance is feature important importance. Interesting. Also, lots of the time for non-technical people, absolutely, but maybe not.

While I'm presenting results, trying to get approval for something this fits into another conversation.

Adel Nehme: Okay. That's more likely maybe something that you share with a senior data scientist or a senior data leader looking to. Verify your results, but not something that you would share with a business stakeholder. Maybe one thing that I see, especially with a lot of folks who are come from technical backgrounds, is they join data science because they love to code.

They're not necessarily people people, right? And then they often struggle. They hit a wall when it comes to improving their presentation skills. How can individuals grow their data storytelling skills? Like what are resources, tools, frameworks, that help you break out of that shell.

Philipp Paraguya: I feel like practice is one big thing here actually to do this regularly and get feedback on what you're doing. I like to, in terms of very actual frameworks and so on. I'm a fan of using the Minto pyramid, like doing a top down example rather than a typical storyline, which is very counterintuitive for someone coming from academia at first.

Then learning how to build very good one pagers, summarizing what I actually want to do and doing visual and visualizing data in a meaningful way. These are all things, tools that you can put into your toolbox and pull out whenever you need them, but I feel like practicing and getting feedback on what you actually presented from someone in the audience is one of the key things that can help you get better.

Adel Nehme: Do you wanna expand maybe a bit on the narrative storyline that you mentioned? Can you repeat what it was again and Yeah, expand on it because I think that's something a lot. Of folks struggle with.

Philipp Paraguya: So there is this concept which is pyramids presenting or Minto pyramid, which is you start with your key learnings, your key analysis, and you build, for example, a slide or whatever containing everything that you want to convey in one slide. And then the next ones will be going a bit more into detail, and then the next level would go more into detail.

Why do this? Well, you start with what you want to convey and you can make actually sure that you are conveying it. Because especially when you talk to upper management levels, it happens all the time that after, you have an hour of time and they say, I have to go in 10 minutes, what do I need to know?

And then you have a slide deck of 30 slides and what you want them to know is on slide 25. And so suddenly you have a problem. And so when you do this pyramid style, you don't have a problem. They don't get every detail, but they get what you want them to get. And this is counterintuitive because coming from academia, for example, we learn that when we write papers and when we present, we need to start with our motivation.

We need to start then go into definitions. We need to get into method, we need to get into how we did things. And then at some point we go into results and maybe risks.

Adel Nehme: hundred percent.

Philipp Paraguya: And this, is the narrative that all of us learn there in, in when we go through these university settings. And this is very, because it's a very intuitive storytelling, but in management presentations or in business presentations, this might actually be problematic because someone asks you a question along the way and suddenly everything derails and you've not even said yet what you wanna say, what you want to convey.

And this is where this pyramid. This pyramid presentation style comes in handy because you start with what you want to convey, and then from there you can go as much into detail as you need to.

Adel Nehme: Yeah. That's wonderful. I, I love that. And I couldn't agree more. And then one thing someone asked me recently, I was delivering a training on better data storytelling and how to, how to approach it is how do I know if my data storytelling is resonating with my audiences? Like how do you get feedback?

I. Whether your presentations work, like one thing that you can see is who's engaged or not, But like, is there kind of a more formal way as well to be able to understand whether your data story is sticking?

Philipp Paraguya: sure if you are in front of a smaller audience, you can see who's engaging and so on, right? And to be honest, since everyone is more or less, Not so much in front of people anymore, and oftentimes in front of the screen. This already gets harder. If people have the camera on, you might get a bit of an idea, but it's very different from standing in front of people.

And then there is a point of when too many people, you also cannot really see what's happening anymore. And so I feel like this is where then the feedback comes actually in. So what I like to do is when I do presentations, have someone also in the audience. sort of a plant that I plant into the audience more or less, right?

To either give me feedback after, how was it, can I improve? Did I convey what I wanted to convey? You could also do this, of course, beforehand, practice rounds and so on. Or, just ask after or ask also at the end, did you get what I wanted to do? It depends a bit of course, on what audience you have, but more often than not, you can ask for feedback.

People are most often keen on giving you feedback. 

Adel Nehme: So that, exactly resonates with, what I was thinking around that. And maybe as we look into the future a bit, we're talking about skills here. Let's maybe switch gears, talk about, you know, the big alpha in the room, which is generative ai. tools like change and copilot right, are definitely being integrated every day into coding workflows.

Walk me through your experience with them and then secondly, like, do you think coding will become less or more important? Before data scientist, as time goes on.

Philipp Paraguya: I feel like, you're absolutely right, in the last couple of months, years, however you wanna put it, these assistance in coding get better and better and become more and more available. And I've spent a good amount of time in the last couple of months being fascinated in how good coding support has actually become when I do my project.

But also understanding a bit where the boundaries, where we currently at, and that is often that while I do have to code less line by line by myself, I. Doing the review and knowing how things should work are still very relevant. It's still me having to look over it to find the hallucinations and to find the things that are not working out the way it's supposed to.

And I think this actually puts us in a very interesting spot because people always also ask, can we replace programmers at some point? And No, I don't think so. However, that being said, code that is generated is oftentimes, to be honest, as good as maybe entry level junior developers might produce in a faster time.

However, then the question arises when we think long term. If we need seniors to actually do the review and make sure everything works out, how do we get them if the getting experience part is not us, well crashing our systems every now and then to get to a point of understanding what's needed. Right.

And I feel this is a very interesting point, which I don't see spoken about very often. That, yes, code is getting good that we generate, but we need also the experience to have long-term pipeline of people that can actually still do the quality assurance and can ask the correct questions. I.

Adel Nehme: Why. What's the solution there? How do you imagine this? Like how do we, you know, if cause I imagine a lot of companies potentially in the future will opt for more coding support instead of hiring junior developers. How do you see the pipeline from junior to senior talent going, moving in in the future?

Philipp Paraguya: I have two perspectives here. So one thing is this might be the data scientist in me in like a reinforcement learning or in learning in general. There's this concept of exploitation versus exploration, right? Trying things out and seeing what works. So exploring versus taking the best and quickest route all the time.

And this dilemma you solve usually by. Exploring and exploiting at the same time. And when I transfer this, this could mean yes, you maybe have generative AI doing the coding part, but you also have juniors doing the coding part to get there. So this is then finding the correct balance. For your company of how much do I want to invest in my future seniors that do the quality insurance, and how much do I need the efficiency part, for example, this is one perspective and the other one is this is not very different from well, hiring consultants really, is it? Maybe the shift is more less consultants, more generative consultants or whatever, while having our own internal people that are also junior. This might also be a shift that can be happening there. And so I think these, are both potential solutions.

Adel Nehme: As time goes on, how do you see the kinda the future skillset data scientists that they need to develop? What, what makes, what will make a great data scientist?

Philipp Paraguya: So I. Understanding the code and understanding the foundations is, will still be very relevant. You will need this all the time. And I think what's getting more important is this notion of asking the right questions, which nowadays sometimes gets a bit well equalized with like prompt engineering, but I mean. Understanding what's my goal? Understanding how to get there, doing the task breakdown and understanding then the output. That may be my different generative functions. Give me, I.

Adel Nehme: As we close our chat, maybe I'll go back to the beginning of it. Do you see certain categories of problems being more automated or streamlined? You know, do you see generative AI systems being able to run AB tests? Right. Or experiments at scale? You know, I had a chat with Ben Stenzel once, right? And he mentioned to me that one thing he's excited about.

With generative ai, it's not being able to just do ab tests on your, with changing buttons or quantitative metrics, but being able also to run, for example, qualitative surveys at scale, right on your website. Right. And that kind of changes the nature of experimentation. So, yeah. Do you see generative AI changing experimentation as well?

Philipp Paraguya: I think so, and I agree with the point there. I. AI has always been automating things that we might find boring at some point or that that are very repetitive. And with AB testing, do I see Gen AI planning, how the best way for a brick and mortar store to run an AB test is in the foreseeable future. Not quickly because of the complexities that we already talked about, but do I see an AB test result being automatically analyzed and getting recommendations out of it?

Absolutely, and I don't think it will take too long because, but this is also the part which is very routine because I have the data, I run some tests on it, and then I get some results. The more interesting part is the other part, and so that's why I feel like, yeah, this is where automation can come in.

Adel Nehme: It's gonna be an interesting time. Maybe Philip, as we close out, do you have any call to action to the audience before we wrap up today's episode?

Philipp Paraguya: We talked a lot about about the IB testing and so on, and I think more of a mind experience, thought experience kind of thing. Maybe to the audience, think about where in your life could you maybe use an AB testing just in your private life to get more data-driven decisions. And you will find that there are a couple of things where you could AB test.

I don't know, I. Using one spice in your favorite dish versus another and see maybe data-driven, how the feedback is, and this will get you more into the mindset, okay, a B test is not that complicated, but rather something I can also integrate into my private life.

Adel Nehme: We could all use a few experiments in our lives. Cool, Philip, great to have you on the show. Really, really appreciate it. And yeah, see you on the next one.

Philipp Paraguya: Thank you very much.