Transcription
Dear friends, hello everyone. Uh, my name is Dima Kirillov. I will be the host of our, uh, AnalTical DS section today at DataFest. We will have speakers today, uh, from very different places, whom we wouldn't have been able to gather in one hall physically. No matter in which part of the globe he is located. I want to invite our first speaker, Dmitry Zabavin, to the stage. Yes, first of all, I wanted to thank you for the invitation to give a report. This is a great honor, sincerely for me. I really appreciate the Open Data Science community. For me, I will repeat again, it is a great honor to make some contribution to such a knowledge base. Uh, so, yes, the topic of my report is Uplift Rexis and a universal framework for developing recommendation systems aimed at increasing business profitability. Uh, let's probably move to the next slide. Here it will be a little clearer why, where this task generally arose from. And I'll tell you a little more about some details here. And, uh, Dima, thank you again very much for this introductory word. It became much clearer to me, uh, the context itself, and I fully share this vision. And moreover, it seems to me that when we talk about tasks related to increasing profitability, in particular, yes, which is what I most often, at least, work with, then here, of course, it is impossible to isolate these two areas from each other: analytics and directly Data Science, machine learning, any other mathematical methods. And today I hope that it will become clear to all of us why this is so. So, I am the Head of Data Science, and here I also have to say a few introductory words. The fact is that I participate in many projects, including international ones, and now the circumstances are such that it becomes very unsafe for some companies if I mention them, in one context or another. For this reason, I apologize, in principle, today I will not name any personal names, including, uh, I will only name my role in the team, Head of Data Science. And I will also say about myself that I have been engaged in custom development of Data Science solutions for 10 years, and this also imposes a certain touch on my experience and my approach to developing Data Science solutions as a whole. What is its difference? If an internal team can afford to experiment for a long time, uh, to develop, to test various hypotheses, and, in principle, even if none of these hypotheses are confirmed, well, fundamentally, nothing catastrophic will happen from this. In the case when we talk about custom development, if my solution does not bring effect, it means that I and my team do not receive adequate compensation. For this reason, the issue of ensuring profitability, proving through a pilot, through testing the effectiveness of the developed solution is an integral part. And what I will talk about today stems from, uh, first, from, uh, a rather harsh, I would say, real-life experience, real development of Data Science solutions, an attempt to comprehend it and develop, to build a guaranteed path to business effect. That's it. Uh-huh, I also have to say about myself, yes, that in my portfolio there are more than five hundred, well, actually close to seventy, I sometimes recalculate the current number of complex Data Science projects in industries such as product development, retail QSR - these are chains, fast-food restaurant chains, banks, loyalty programs, and even Game Dev and Travel. I was lucky enough to work with them too. I am also the lead methodologist of the A/B testing framework A/B testing, perhaps you have heard of it. And also the author and lead of Data Science courses in targeted marketing and immersion in A/B testing on its basis. Also, perhaps you have already managed to read the book cover. I am the author of a series of books on deductive theology. And the full edition was released in the twenty-second year. Here I am holding it in my hands in the photo. Every time I talk about this, with a certain degree of thought about whether it will be interesting, but every time it turns out that questions from this side also bother people, they also turn to me with them afterwards. That's it. Also, the QR code is a link to my personal website, where you can read more about my projects, publications, and conference presentations. Yes. Now, briefly about the content, what we will talk about today. First, we will start with a case that became the starting point for me on this path, which I have already mentioned. That's it. And it also seems to me that for you this case can become, well, for me personally, it truly divided my data science experience into before and after. Then we will talk about approaches to forming inherently profitable offers. We will talk about what concepts like the softness and hardness of offer conditions mean, how they affect profitability in general, in what context and how to manage the softness of offer conditions. We will talk about balancing strategies: increasing the probability of purchase and increasing the amount spent. Then we will look at the architecture of a universal recommendation system. And I will also outline some possible steps for further improvement, further development of this approach. In any case, this is what is personally in my backlog. So, the first thing, the very case that is designed to divide our lives into before and after. So, I was invited as a solutions architect to a project that had already started before me. And here is the situation I encountered at that moment. So, a model for predicting the probability of purchase was built. That is, the model is checked, it looks at the client's parameters, at the offer's parameters. This was a QSR company. That's it. So, the client is matched with various offers, and the probability of the client making a purchase is assessed. The goal of the system was to select an offer to which the client would respond with the highest probability. In principle, as it seems to me, this is a typical task setting, it most often sounds exactly like this. And my colleagues, they developed a model with quite high accuracy, that is, the G score, maybe even close to 0.8, that is, the model was indeed very accurate. A pilot was launched to assess the economic effect of applying this model, and the effect turned out to be negative. It was at that moment, when I first saw this case, that I really started to think about what the problem could be, why this is happening. That is, it seems we understand quite well what customers need, and customers really respond to the offers that we send them. Well, by offers, also, just in case, I will clarify that these were coupons for purchasing certain dishes. That's it. And we see that the response is indeed high. And yet, the overall economic effect, if it is, let's say, separated from other proxy metrics, then it is negative. And before we move on to considering the question of why this happens and how to prevent it, let's make an important remark right away. The first axiom in today's report is accuracy does not equal effectiveness. This is the first thing I learned for myself for the rest of my life. I want to share this insight with you. Accuracy does not equal effectiveness. This may seem like an obvious statement. In reality, it is not obvious at all, and its implications are much deeper than they might seem. Why? Because the classic representation of Data Science as a whole as a department often looks like this: there is business that generates some hypotheses, and Data Science builds some model for this hypothesis. And further, the use of this model is exclusively a business issue. It doesn't concern the data scientist themselves anymore, but it turns out that everything is completely different. And the accuracy of the model does not guarantee business effectiveness. And this means exactly that the data scientist is obliged to delve into business processes and understand how exactly this model will work and by what means it can bring an effect. I will also share some insight with you. Often, when I have to communicate with business representatives, they say: "Let's build this model and launch it." And the first question I usually ask is: "Can you please suggest a user story in which this model would really increase the effect?" And let's calculate what is the share of such user stories in the total pool? And I often hear the answer: "No, let's not do that, let's build models and see in an A/B test." This approach will definitely not lead to an effect. That is, the first thing to start with is to think about how the model can increase business effectiveness in a specific scenario. This is the first thing I wanted to share with you. So, where does the profit disappear? The thing is that we are making an offer here, a little bit from another project. I took the slide. I apologize in advance, because it's a collage in a sense, my presentation is from different projects. But nevertheless, the general idea here is quite clear. The thing is that when we offer various offers to clients, it is important to consider the dynamics the client is currently in. And depending on his current dynamics, the same offer can work both for the better and for the worse. So, a simple example. For example, there is a client Sergey, who made purchases daily. Now he hasn't been around for 20 days. Sergey needs retention, and we offer him a discount in the milk category, because he often makes purchases there, and for him this discount, of course, it reduces his natural spending, but it performs the function of retention. And for this reason, here we can talk about an increase in profitability. Now, the second example. Nikita makes one purchase a month, and he also hasn't been around for 20 days. Let's pay attention to the fact that both clients, the first and the second, have not been around for 20 days. But for him, this period without purchases is absolutely natural. This client does not need retention. And therefore, we give him a discount in the book category, where he has never made purchases, because here we are trying to ensure a development strategy. Because if in this situation the client is also given a discount in the category where he buys often, we will reduce his natural spending, and this will only be a loss. That's it. And here, why I also draw your attention to the fact that they have the same interval without purchases, 20 days, is because classic RFM approaches will also not work here, because they will consider these clients absolutely the same, their dynamics are the same. That is, we are already interested here, I am also gradually moving towards some statistical estimates. That is, here it is necessary, among other things, to apply additional models, additional statistical estimates that will answer the question not how many days the client has not made purchases, but how long for this client he has not made purchases. Yes. And the second axiom that follows from the above is that a profitable offer is any non-loss-making offer. It may sound naive somewhere, but in reality, there is a sufficiently deep thought here, that in order to form profitable offers, we first need to understand where the loss actually arises, and for what reason. The first step we will take here to answer this question will be the following. We will introduce the concept of soft and hard offers. I will say right away that here I am simplifying somewhat for the presentation. I divide it into soft and hard offers, but let's keep in mind that in reality it is a continuous scale. And fundamentally, any offer is somewhere on this scale, yes, it can be extremely hard, extremely soft, or somewhere in between. In principle, we can control this, say, switch, we can manage it, and we can also evaluate each offer for a client, how soft or how hard it is for him. Well, let's look at examples. A soft offer is a discount on a product that a client regularly buys. The offer format is a discount, yes, the mechanic, or a gift offer, that is, a gift product when purchasing for an amount below the client's usual spending. This is very important. Specifically below. That is, a soft offer is one that allows the client to reduce his natural spending. Now, a hard offer, for example, a discount on a product that the client has never bought before. Yes, he has never made such a purchase. And therefore, even if the client receives a discount in this category or for this product, it is a sufficiently hard offer. The same applies to the gift offer mechanic - it is a gift product when purchasing for an amount above the client's usual spending. And one more thought that I would like to fix here is that the same offer for different clients has different softness, different hardness. For different clients, the same offer, for example, a discount in the book category. One client has never bought goods in this category, for him it is a hard offer. Another client regularly makes purchases in this category, for him it is a soft offer. Let's look at the connection between client dynamics and offer profitability, as well as the softness and hardness of offers, what should be the connection between them. Well, here, I think this graph should be more or less understandable. How to read it? We have, uh, client profitability in general at different periods of his life cycle. Initially, it is increasing, the client gets acquainted with our products, gradually starts making purchases, profitability grows. Then at some point it may start to decrease. And in this interval, the client begins to need retention. If retention is not successful, then, accordingly, we enter a stage of zero dynamics, and further, reactivation is required for the client. Let's look at what kind of offers, in terms of their hardness and softness, we can offer the client at different stages. Well, a new client needs a softer offer, because we are trying to introduce him to our products more closely, increase the regularity, frequency of his regular purchases, and so on. Next, the growth stage. Here, at the growth stage, this is, let's say, the safest interval where the client does not need retention, and the business can, in general, offer a fairly wide range of mechanics, such as upsell, which is a hard offer. Because it implies that the client should spend more than usual. Cross-sell is also a hard offer, because we offer products that the client has not bought before. And the only exception is the additional purchase mechanic, when we can offer the client a soft offer, but with a short validity period. In this case, we will say, we hope that the client will make an additional purchase, and thereby increase profitability. Well, at the retention and reactivation stages, I think it is clear that here the need is primarily for soft offers. And the third axiom, which we arrive at, is that the profitability of an offer for a client is equal to a weighted combination of the strategy of increasing the probability of purchase and increasing the amount spent. That is, here we have already talked about the fact that at different stages of the life cycle we use offers of different softness to increase profitability. And here is a new perspective from which we look at this problem, and that is that different offers with different hardness, with different softness, provide two different strategies. The first of them is an increase in the probability of purchase. These are mainly soft offers, that is, we try to increase the probability of purchase. And there are mechanics aimed at increasing the amount spent. That is, these are the very, for example, upsell mechanics, cross-sell mechanics, when we try to expand the client's product basket. And others. That is, one way or another, all offers, besides the assessment of hardness and softness, can also be assessed in terms of which strategy they provide: an increase in the probability of purchase or an increase in the amount spent. And here I propose a metaphor, yes, that a tightrope walker is walking over an abyss. The tightrope is client dynamics, yes, we start from it, we balance on it. And our task to achieve profitability, on the horizon, yes, profit is visible, our task is to balance two strategies. This is an increase in the probability of purchase and an increase in the amount spent. So, the balance is important to us. Any violation of this balance, when, for example, we start giving slightly softer offers, as a result, we reduce the client's natural spending, although he did not need retention, yes, we start losing profitability. And vice versa, if we offer the client who needs retention, our offers start to lean towards increasing the amount spent, yes, that is, we start, let's say, sending more greedy offers to the client, that is, more directed towards the strategy of increasing the amount spent, then we do not ensure retention, we do not support the probability of purchase, we do not increase it, and as a result, the client with high probability will simply not respond to our offer. Here I have offered you this logic in another, more detailed form. That is, essentially, I have divided the dynamics into bins. There is, say, natural dynamics, and in it the probability of purchase is quite high. That is, when the client, well, in his natural dynamics, he does not slow down, does not speed up. This is his natural behavior. The probability of purchase is quite high. And it is clear that here we can shift the balance towards increasing the amount spent. Next, a significant increase in dynamics. This is most often observed in situations when the client was very recent, and we have an estimate of the interval without purchases. For him, well, the interval without purchases is very short. He was very recent, yes. And at the same time, yes, an increase in dynamics can indeed be observed, but in any case, the probability of purchase is quite high. Here too, we can shift towards increasing the amount spent. Not a critical decrease in dynamics. Here, let's say, the average probability of purchase still remains. That is, it is no longer high, but it is, in general, not far from zero, yes. And at this stage, it is important for us to maintain a balance between the two strategies: increasing purchase frequency, increasing purchase probability, and increasing the amount spent. So, in case of a critical decrease in dynamics, it is clear that the probability of purchase becomes low, and we need to shift more towards the strategy of increasing the probability of purchase. And let's have a few more insights here, let's get acquainted with them. They are listed below. Well, the first one, we have already discussed that the same offer for different clients has different softness and hardness, yes? Then, the same offer for different clients corresponds to a different strategy. That is, again, yes, the same offer, for some it will increase the probability of purchase, for some it will increase the amount spent. And here is another very, very important conclusion: there is no ideal or most profitable offer. Often, when we talk with business, and it comes to a pilot, we often hear such an argument: "How do we check that you are really launching a pilot according to your model? Maybe you are just giving everyone the most profitable offer." And it can be difficult to convey to the business that there is no most profitable offer, because, what does most profitable mean? The most greedy then those who needed retention will not respond to it. We will lose profitability. The most profitable is with the biggest discount. Yes, we will significantly increase the probability of purchase, but we will reduce the client's natural spending and also lose profitability. Therefore, there is no ideal most profitable offer. The very meaning of applying Data Science in this task is to find that ideal most profitable offer for each client. Now let's talk about the architecture of a universal recommendation system. So, it is generally divided into two large blocks. These are essentially two uplift models. One uplift model is responsible for the increase in the probability of purchase, the second is responsible for the increase in the amount spent. Each of these models is two-component. Why is this so? Because classic uplift assumes that we only consider the feature itself, whether communication was directed or not. But we are interested in the characteristics of the offers. For this reason, we need to have one model that will predict the probability of purchase given an offer with certain parameters. That is, at the training stage, it involves both client parameters and parameters of individual offers. Yes. And, moreover, these parameters, they can actually, there can be a lot of various characteristics listed in them, including the softness of the conditions of this offer for this client. The second model, in this architecture, is responsible for the probability of natural purchase, that is, without offers, yes? That's it. Because why is it needed? Because, if we talk about the first model, we cannot, well, if we transfer a case there, into this, try, yes, to use a case in this model at the training stage where the client made a purchase without communication, yes, without an additional offer, then it is unclear what characteristics to enter into the fields characterizing the offer itself. That's why we need a second model, precisely, which will be responsible for the probabilities of natural purchase. Well, or exactly symmetrically. One model is responsible for the amount spent given a purchase, the second is responsible for the amount spent, natural. That is, without an offer. And thus, by calculating the difference in probabilities, estimated by two models in one case, and the predicted amounts spent in the other case, we get two uplift values. That is, one uplift by amount spent, the second uplift by probability of purchase. And here is the most interesting part, how do we, how do we work with these two predicted uplift values? And here I propose such an approach that we can use the probability of natural purchase as a weighting factor. What does this mean? This means that the probability of natural purchase acts as a weighting factor, and it tells us how much we need to shift towards increasing the probability of purchase. That is, if the probability of natural purchase is low, we focus more on the uplift in probability of purchase. More, I emphasize, it works precisely as a weighting factor. Accordingly, it provides us with this smooth balancing of the two strategies. That's it. Accordingly, if the probability of purchase is sufficiently high, then the uplift in the amount spent has a greater weight, and we focus more on it. That's it. And how does scoring happen? We select a specific offer, score it with all models. Well, for example, in my example, here I took an offer with the ID number two. We received for it, um, uplift for one, for the probability of purchase, uplift for the amount spent. And, considering the probabilities of natural purchase, we drew a conclusion about, uh, we calculated the value of the objective function for this offer. That is, we looked, considering these, the balancing of these two strategies, how high a score this offer has overall. Then we score different offers and select the one for which the value of the objective function will be maximum. That is, the one that best ensures our strategy balance, corresponds to it most. As for the results of applying this framework, well, I applied it in a fast-food restaurant chain. And regarding the increase in RTO, the increase in revenue, we received an increase of 9% compared to the control group, and 3% compared to the current approach based on business rules. For those who know what retail is, who know what QSA is, I think they understand the significance of these values. These are large values. The increase in the number of orders, meanwhile, yes, compared to the control group, is 6%, compared to the current approach based on business rules, it is 2%. I emphasize that overall, I kept the focus on the monetary metric, because the goal was to increase profitability, but at the same time, for informational purposes, we also calculated the increase in the number of orders, and it also turned out to be impressive. Now, if we talk about the further development of this framework, here I will draw your attention to the fact that here, in fact, only one additional component has been added to the previous architecture. This is the balance of strategies. That is, in addition to the fact that the probability of natural purchase of the client affects the balancing, we also use for ourselves an additional, well, an additional shift of this balance. Based on what? Based on the fact that from the data we can pilot such an approach when, yes, we look at the probability of natural purchase, but we add some, some addition, yes, some coefficient that, randomly, we add an additional shift towards one or the other strategy. And we can collect such data at the pilot stage and train a model that, for a specific client, will tell us which shift is most effective for him personally. And thus, we introduce an additional adjustment or an additional balance coefficient into our model, into our objective function. And thus, we expect to be able to increase profitability even more. And also, one more additional improvement besides individual balance adjustment.
Strategies are additional factors of the client, offer, and context. Let's look at such a slide. I am also not providing an exhaustive list here, but it is simply what I would like to share with you. Well, client factors are the significance of the no-purchase interval. I understand that this sounds extremely abstract to you right now, but to make it clearer for you, I would suggest you scan this QR code and familiarize yourself with my report, which I gave in a community about the statistical profile of a client. It details the approach on how to evaluate the significance of the current no-purchase interval. That is, in addition to a quantitative assessment, to also get a qualitative assessment of how large or small the current no-purchase interval value is for a given client. And this same approach can be applied to any other metrics, that is, to assess how critical or natural the observed value of a particular metric is for the client. Offer factors. But here I am looking more towards retail potentially, yes, one can, in the direction of retail, transfer this framework, try to apply it in other environments, in other industries. Here, category brand dependence, the expensiveness of category goods relative to others, can also work. Client-offer connection factors are the forecast of changes in the basket structure. That is, here we can think about how likely it is that this product will end up in the client's basket. The rigidity of offer conditions for the client, we have already talked about this. Connection factors, client-category. Well, here too, one can consider the same significance of the no-purchase interval, but within a category. Client category purchase strategies are also quite interesting metrics, because they indicate that in some categories a client may, for example, adhere to a savings strategy, while in others, for example, they may adhere to a strategy of optimal price-quality combination, and in some, they may adhere to a strategy of choosing only premium goods. Discount sensitivity and so on, right? Connection factors, client-product, context factors. That is, context is, for example, the consequences of a purchase. For example, cannibalization of other purchases may occur. That is, we, for example, prompted the client to make a purchase in a certain category, but thereby displaced a purchase in another category. Here. And the delayed effect of a purchase is when, within a promotion, for example, a client stocks up on a particular product. And thus, we can encourage them to make repeat purchases in this category as much as we want, but they will not happen because the client has a stock. So, perhaps, this is probably the main thing I wanted to share with you. Once again, just in case, I have included this slide, which also has a QR code with a link to my personal website. Everything is there, in fact. Including, by the way, the opportunity to purchase merchandise with these logos. If anything, write, all contacts are also there on Telegram, or somewhere else. Yes, and I also wanted to emphasize one more important thought at the end, that this framework has already been implemented. And if any of you are interested in piloting it or conducting some joint additional research, writing a scientific article in a scientific journal, working on something else together in some context, I don't know, transferring this framework to some unexpected industry. Please, also, the contacts section is on my personal website. Contact me in any way that is convenient for you. I will be very happy to answer your questions and your suggestions. Thank you very much, Dima. Your applause. You see, here are the applause from our listeners. They very eloquently indicate that the report, in fact, I think the guys liked it. Well. Look, I have, as always for analysts, the first provocative question, right? Like, look, from your story, it follows that you have essentially three models. Well, one model, well, in the latest version of the framework, right, you have a model that is about increasing the probability of purchase, increasing the average check, and which balances, right? Uh-huh. Right. And then the question immediately from any data scientist is: "Why can't the issue be solved with one model with feature engineering?" That is, engineer features in such a way that, conditionally speaking, your offer takes into account this rigidity, softness as a model parameter. In a sense, yes. Here, this classic problem of uplift modeling. Let me restate it once again, just in case. So, the classic uplift modeling. We have client data, then there is a column, whether there was communication or not, and a target column, whether there was a purchase or not. This is the classic that is solved by a single model. Now, look, let's assume that we want to consider not only the fact of communication, but also the characteristics of this communication. For example, the discount amount and the category in which it was given. So, we already have two columns in the communication. And now the main problem is that if we fill in these columns without difficulty for clients to whom communications were sent, then for clients to whom communication was not sent, it is unclear what to enter there, because then you will perform imputation, and these columns will be filled with essentially random values. That is, it will turn out as if this client actually received some offer, even though he did not receive it. And here, this reason for this two-component nature is precisely that we need to separately predict the probability of purchase under the condition of an offer and in its absence. Now, the question, as I understand it, also concerns why we need two components with an increase in spending and an increase in probability. The whole point is that here we don't have a target, that is, we don't understand in which case spending increased and in which case it did not. Well, if we had a trained dataset where we knew that, say, this client, for example, well, again, there would still be a problem here, because the same problem with uplift. One can, in principle, yes, I agree. That is, one can, for example, get by then predicting only the amount of spending, without paying attention to the probability of purchase. One can do that, that can be done. I think one can get by in theory only with the right branch. In principle, in theory, it can be done. But look, here we will simply miss the important detail when the probability of purchase for the client is very low. And, you see, when we look at the predicted spending, at the predicted increase in spending, this model does not take into account, it will not take into account that, well, it may turn out that it will not take into account this problem of losing the client, that we are actually losing him, he needs retention, and, well, in general, I will answer briefly, one can try, I think, to leave only the right branch in this ensemble. Well. But still, I would recommend looking at both terms, that is, both the probability of purchase and its increase, and the increase in spending. Because by increasing the predicted spending, and here's the problem, by the way, I remembered, Dim, exactly, we tried, we tried this approach. And here's what happened, that when only the right branch works, there were cases where the spending forecast turned out to be very large. And then we started to check, how can that be? Well, that is, it looks at the probability, it's small. Yes, yes. That is, it significantly lowers the probability, but says, but if he buys, then he buys, right? And here, exactly, I remembered, yes, indeed, we initially built only the right branch, and this problem arose there, which is why we included the increase in purchase probability in the ensemble, so that we don't form these recommendations blindly, but take into account both terms. Understood. Understood. Thank you. Look, we still have a question from the chat from Alexey Natyokin. By the way, if I understood correctly, this module with uplift is trained by you separately on top of the companies' existing recommendations. And if it were trained jointly, well, the main thing is the offers, the candidate offers, right? How much would the effect change, according to your estimates? So, let me comment on this. Look, no, it was not trained on top of recommendations, it was trained from scratch. From scratch. That is, we initially formed a pool of offers, and conducted an exploratory pilot, and based on its results, we trained the models. But here, in fact, the topic is worthy of a separate report. This topic is about, let's say, strategic piloting. What is it about? The whole point is that we used such an approach. So, when you are still at the pilot stage, when you form a pool of analyzed offers, so that you don't take the entire available pool and try to do some kind of grid search, that is, how each offer will work in each segment. There is an opportunity, based on fixed strategies, to filter out demonstrably unprofitable offers. That is, essentially, it is an attempt to solve the same problem, but from the side of business rules. And it is clear that we set them a little less strictly, so as not to accidentally discard a good offer. That is, we, of course, incur some losses, but we still give away a significant portion of demonstrably unprofitable offers. This is, in essence, the classic problem of candidate generation in anything. That is, if you aim too much for accuracy, at the candidate generation stage, you will lose completeness, and so on. That is, it's the classic balance between completeness and accuracy. Understood, right? Okay. Well. And if you have questions, there will surely be some more questions for Dima, so you can always ask them in the room. And, Dima, please monitor the room chat. If you see questions for you about the report and not only about the report, then please answer them. Well. Let's thank Dima once again for an interesting start to this section. The physicality was felt in every slide. Thank you very much.