Transcription
We will now have a system design section, and something more high-level than what we've had so far. And, uh, we'll talk about technologies, what we had to work with a little bit about data. We'll finish with system design tasks. So, look, if you have a task like this, uh, well, let's say ClickHouse, a normal example. You have some kind of dashboarding, for example, for an end-user, who ultimately, well, launches queries, they go to ClickHouse. You need to periodically, for example, once an hour, ingest into ClickHouse. Well, let's assume some tens to hundreds of gigabytes of data of different types. That is, there could be some gzipped CSVs, there could be Parquet, there could be something else. So, how would you approach this task, what would you use for ClickHouse ingestion of precisely these large volumes at once? >> Okay. A file has arrived on S3. A handler for this event should be launched. This could be a Lambda or not a Lambda. So, I just gave an example. But the point is that we need to split the received file into chunks, in order to process these chunks in parallel. These files can, this file can be split into chunks as follows. We have, uh, let's say, some worker that monitors S3, and its task is to react to new files. A new file has arrived. Uh, well, yes, yes. Let's do it this way. Let's analyze separately how this worker will react to a new file arriving. Later, a new file arrives, and this worker starts. It begins to read the file from S3 via a stream, and its RAM size should be sufficient to, well, hold its code, right, in this memory, and to hold some buffer size in this memory. It starts reading via a stream and buffers. Well, it buffers, reading it. As soon as the buffer size fills up, reaches a certain size, it can, this means, essentially, put this data further. Further, for example, into Kafka. So, in this way, we will be able to split this file. That is, the worker reads via a stream, it does it quickly, asynchronously, without blocking, and fills Kafka, sending data to Kafka as a producer, so that, well, of a certain size. Thus, each message in Kafka will be of a certain size, and it can already be, that is, and it is already, well, split. A consumer group can be attached to this Kafka, whose task is to consume these messages, these events that the first worker, which is only responsible for splitting, has put there. As soon as, uh, there are some partitions in the topic, a consumer group is attached to this topic, each of the workers reads partitions, and processes its own, uh, its own, well, some piece. Uh, it might happen, and this is probably an important point, it might happen that this file contains duplicate lines, and then it probably doesn't make sense to do anything with them. Uh, and because of processing each line on every little thing. What if it's a duplicate? Here you can do it so that this worker, which reads these lines, also reads from Kafka in batches. Look, for example, at some ID and by ID, take the latest value with the overall value with the last timestamp. Well, for example, like this. So, let's imagine that such a task, well, such a case can happen. >> Exactly. Then it also fills its buffers, in the buffer, well, processes it somehow, fills the buffer with this data. And as soon as the buffer in the consumer reaches a certain level, it inserts all of it into ClickHouse in a batch and empties the buffer. >> How? Well, that is, through what will it ingest, for example, the ingestion process itself into ClickHouse. Is it some Python code or what? Here we have Kafka, right? Which wasn't there before, right? >> And, well, I understood, so this is for some kind of batching, right? Uh, then collection. And at the end, what will we use for insertion? >> Some library, uh, preferably asynchronous. In the project I mentioned earlier, I do this directly, like, with a raw query through the connector. But now there are a huge number of Python libraries. And there, I think one of them is called, uh, either Python ClickHouse, or something like that. Well, in that project, I don't use these libraries. There, as I said, I use raw queries through the connector. But in general, if we do everything asynchronously, we can choose an existing library and send batches from Python. Well, I just don't use these libraries now, so I don't remember the names by heart. >> Okay. Is there a cheaper way? So, here we need Kafka, here we need to do some kind of buffering of consumption, and also put it into Kafka with a limit of 1 MB or >> 8 GB, depending on the Kafka engine. We need to, well, somehow split this data, right? Well, if we have, well, in one file or in several files, this data on S3 that we can ingest, is there a cheaper way to ingest into ClickHouse? >> Well, look, uh, at the very beginning, I actually didn't clarify the requirements. So, if this task were, let's say, formal, right, for system design, it would be good to clarify the requirements, right, functional and non-functional. You are now asking: "Can it be cheaper?" Well, I will ask you the next question in response: what do you want to sacrifice? In the sense that the method I proposed, as it seems to me, is fault-tolerant, fast, reliable, lossless, and so on. You now want, uh, how, well, you want it cheaper, well, then what will we sacrifice? That is, what in this method doesn't suit you? And what would you yourself want to save on? Let's start with that. Money doesn't suit me in this method. Money doesn't suit me. >> So, what then is sacrificed? What >> sacrificed? You tell me what is sacrificed. So, I'm a business person, right? So, I'm a business person, and you tell me, to make it cheaper, we have to sacrifice this, right, like orchestration. >> Okay, here we can sacrifice. Here we can sacrifice, for example, the bus. That is, we can remove Kafka from here and simply connect S3 to the worker. And the worker itself, within itself, essentially, within its memory, has queues, has buffers, and will itself dump data into ClickHouse. But I didn't propose this method. Well, keeping in mind that, most likely, you are interested in something like what will be bad in my pipeline compared to yours. And if we remove, as you say, >> uh, so my pipeline where Kafka is, your pipeline where Kafka is not. >> No, not mine, the cheaper pipeline, right? So, I don't like the overheads. And what will I sacrifice specifically, non-functionally or functionally? >> Uh-huh. Functionally, you most likely won't sacrifice anything. That is, it's like, well, or rather, if you have a high load, then functionally it might break, right, in general, right, billions of something messages or billions of lines in this file, right, functionally, nothing might work for you at all, but if we put that aside and take non-functionally, then probably so. As soon as we remove this bus between, uh, essentially, the target database and the initial, well, how to say, initial location, right, where the file is stored, we immediately strongly tie the target database and deprive the initial file of the ability to replay. In case of, well, something, right, we found, we found a bug in our code. Kafka allows us to replay. Then we remove this. there is this file. So, we can also, >> yes, but that again requires reading it, spending time on it. Uh, well, yes, I agree, but I would say that it again requires reading it, spending time on it. And one more thing came to mind, that this bus, right, between the target ClickHouse and the worker is essentially also a buffer, which allows us to add these very workers, which can parallelize the task. Because in the case when we have, and we can scale them infinitely, let's say, and in the case when we don't have this bus, as in your pipeline, for example, our scaling mode is limited only by vertical scaling, probably, of this server where this worker is located, which streams, reads from the stream and writes to ClickHouse. At its level, we can only scale it, let's say, by processes and so on. Well, or within processes, well, or threads, well, depending on how we do it, right, if we do it through asynchronous programming, right, we can scale it with threads, each thread has its own event loop, but there are also limitations on one server. So, I would say so. That is, as soon as we, that is, my understanding is that if reliability, throughput, and, I don't know, uh, well, failures too, right, then Kafka gives us, if it's a Kafka cluster, we are protected from failures and so on. So, if all this is not there, then it turns out that if something fails somewhere, we can lose data, we can lose messages, and so on. Uh, well, I would answer like this. So, I understand that, let's say, if we are building something from scratch, we have little data, we might not need Kafka. But if you are asking in the context of a data-intensive application, and this should all be in production, then, well, Kafka is probably needed here. Okay, let me ask you a question directly. Let's say I'm a business owner, right, and I have an architect friend, and he asks: "Why don't you want to insert directly into ClickHouse from S3? That is, you can just dig into S3 files in different formats and so on. Why don't you do that? What's stopping you?" >> That's a really good question. I'll answer this right now. I have no idea why I don't do that, because personally, I've never used a service that can natively access S3 and natively write to ClickHouse. But if I received this information, that there is such a service in Amazon, maybe it's Amazon Athena, maybe it can do it, but I'm not sure, to be honest. I used it a long time ago to, roughly speaking, run SQL queries on S3. But whether it can connect to ClickHouse and so on, I don't know. Therefore, at the moment, I have no idea. But if I'm given this information, I'll probably go and see what can be done. >> No, no, I'm not talking about a service here, I'm just saying that ClickHouse itself has had this feature for quite a long time. >> Through this. Honestly, I didn't know about that at all then. Okay. >> All okay. Okay, I see. >> But, wait a minute, please, but then I have a counter-question. How much will it be, well, as I said, Kafka gives us fault tolerance, separation, buffering, and so on. How beneficial will it be for us in terms of cost, and well, no, it's a bad question, how convenient will it be for us to do it, that is, will ClickHouse cope with such queries, and won't it choke, well, that is, can it batch? >> My question was actually about this, right? That is, a familiar architect said that you can do a select >> star from S3 parenthesis, right? And how, but what impact, that is, you are protecting, oh, he doesn't understand anything, right? This will break this, this, this, right. >> I understand. Well, yes, it seems that ClickHouse might choke on large data if done this way. >> Uh-huh. Can you isolate workloads there, right? So, you have, for example, analytics, right? So, directly, well, customer-facing, analytics on ClickHouse is often done, like, customers do it. >> Uh-huh. >> And your users use the dashboard and want second latency in general, right, and you have something like this on the side, which is directly through S3. So, can you isolate and like >> workload. Look, how it's implemented at the ClickHouse engine level, I don't know. I, well, haven't dug into it, haven't looked. But I assume it's possible, uh, because it would be strange if it wasn't possible, because it's obvious that ClickHouse, when it's in working condition, involves a huge number of read operations from clients, right, well, various dashboards, right, as well as a huge number of write operations. It would be strange if the ClickHouse developers at the engine level made it so that a read operation blocks a write operation. On the contrary, a write operation blocks a read. Therefore, I assume it's possible, but I don't know how. From the perspective of resource consumption. Here I'm not asking about direct blocking in the database, but about resource consumption. >> Yes, yes, of course. Any query, well, reading from ClickHouse implies accessing the disk one way or another. And the disk immediately means I/O, disk operations, depending on whether they are sequential or random, well, also have their impact. That is, if the data is scattered, then, I think ClickHouse can fragment it somehow. But, for example, if the data is fragmented, then random reading also has an impact. Interaction with the device impacts the fact that we are reading, so writes will slow down at the low level, at the low level. So, well, yes, in this regard, of course. >> Uh-huh. >> And plus CPU and memory too, all these expenses and so on. >> Uh-huh. Let's probably move on to this system design task. Let's try web scraping, right? So, look, you have some kind of SaaS product that does competitive analytics for e-commerce. What is competitive analytics? For example, it compares prices on different marketplaces, on different platforms, sites, etc., online stores. And, uh, well, it sells it, for example, right? And the key thing on which all this is based is web scraping, right? So, it has, for example, >> 50+ e-commerce platforms that it regularly scrapes. It collects product descriptions, some assets related to the product, multimedia in terms of images, maybe video someday, some PDF files, suddenly. Well, and metadata of the product itself, some SKU, model, name, description, and price, of course, right? And it scrapes, let's say, with this whole system, that it needs to scrape and track changes no older than 24 hours. That is, within 24 hours, we must have fresh data for all products we have. Let's assume. We have tens of millions of products in total, right, and they can be on different platforms, of course. And we need to scrape them regularly. And, well, a typical scraping volume, let's say, a million products a day. And what else? So, here, it's important for us to design precisely this kind of data architecture. Perhaps, if there's time left, you'll also consider how we'll do the actual scraping, if you have any thoughts, like how to bypass IP blocks, >> Uh-huh. >> bot prevention, Cloudflare, and all this stuff that blocks such scraping, and all these smart algorithms. But in general, the most important thing is the basic scraping architecture, data architecture in terms of what the data will look like, how it will be structured, and how we will perform these checks, that within 24 hours we need to check if anything has changed. If it has changed, then we parse all of it. So, what will be the final consumption? Another important point, right? So, this is, well, you need to understand how it will be used. It will be used by some, uh, BI tools, analytical queries, right, for example, some ML that will build a time series of how the price has changed over a year, for example, retention, we can also do a year. >> Right. But we need to see the historicity of the data, right? So, it's important that it's not just the last change, right? But we need to see this timeline. So, if it's, I don't know, like Wayback Machine, compare, right, the state of the site at certain moments in time, when it changed. And we need to, accordingly, with the price, with all other attributes, also have this timeline of what changed. First of all, the price, and secondly, other attributes too. So, that's the task. What questions do you have? Uh, ask. >> The first question is organizational. Are you interested in, well, in observing all this formality of a system design interview, where I go through this plan, collect requirements, draw APIs, draw get and post requests, what goes where, and so on? Or is it interesting to you, should we spend time on this, or not? Tell me. >> No, my questions are more important. So, as you find it convenient, I'll tell you. >> Well, look, it's convenient for me not to observe all this formality, because there's no point, well, I know and understand that it exists, but honestly, I don't see the point in spending time on it. >> Okay. Okay. So, what's important to me, probably, is your, uh, drawing, uh, I'll put it this way, right? So, how you understand, uh, these requirements, how we will, what pitfalls there might be. Well, things like that. And I'll also ask some questions as we go. >> Yes. Yes. Okay. Okay. Then let me go over it again. So, well, honestly, regarding functional requirements, everything seems clear. So, we need to design this competitive analytics that connects to a huge number of platforms, scrapes data, collects this data. Well, and the SLA should be that the data is no older than 24 hours, and the number of products is tens of millions. Well, honestly, there are no questions about the functionality, because it's all described here. Because, what does this competitive analytics do? It scrapes, it saves somewhere in some storage, which will then be used by some model. So, if we are talking about this specific narrow product, then, well, there are no questions. If you've hidden some Easter egg here in terms of functionality, which I don't realize because I haven't worked in this area, then please, tell me yourself. >> Uh, no, probably nothing like that. Well, I don't know what an Easter egg is called, right? Well, I don't think there's anything super tricky, right? So, regularly scrape data from external sources. >> Uh-huh. >> Okay, then let's think about non-functional requirements. So, uh, so, what we have, uh, regarding non-functional requirements, uh, so, well, as we said, that changes should, well, that is, our window, the update window, probably, is 24 hours. So, we need to see this, uh, then the number of products, tens of millions. This means that within these 24 hours, we need to, well, somehow scrape, like, tens of millions, >> right? So, uh, how to say, calculate the rate, well, from practice, that 90-95% and above doesn't change, right? So, maybe a banner, some advertisement, or they change the layout, but the valuable information in 95% of cases doesn't change. And we need to account for this too. >> Okay, so, 95% of products don't change within these 24 hours, roughly speaking. Okay. Uh, so, so, since we are scraping, right, and not receiving requests. So, when we receive requests, we calculate RPS, right? So, when we receive RPS, right, here it will be, essentially, RPS from us, right, or the number of requests. How many will there be? So, tens. I don't know, let's take, just to be sure. Let's take 100 million, just to be on the safe side. Let's take 100 million. Can I take such a number to just fix it? >> Uh, so 100 million in 24 hours. So, well, let's just estimate. 100 million is, uh, so, 24. 24 is approximately 2 x 10. That's how many per hour, right? So, remove ten, remove a zero, remove two, leave five. So, we get 5 million per hour, per hour, uh, products. 5 million products per hour need to be scraped. Uh, and if we convert it to seconds, I don't know how much that will be, 5 million divided by 3,600. Uh, well, what do we get? So, remove zeros. Remove two zeros here. 36 is 10 times, well, roughly four. Remove another zero. Leave four here. Uh, 5,000 / 4. Well, that's 2,500,000. Well, it's 1,250 requests per second, right? Uh, well, >> approximately, >> yes. How much is that? Uh, well, that's a good question. For example, if 2,000 RPS were hitting our system, then one Nginx would easily handle it. Uh, well, of course, with proper configuration. But in general, in general, this is not even 10K, right? So, this is a very small load. So, that's why. Uh, okay, fine, I understood that. Next, are you interested in these calculations about storage sizes that we need, or not? Or is it less? >> Well, I don't know, if you want to use some specific storage, then maybe. If it's S3, then no. >> Uh, well, okay, let's do that later, right? So, later, uh, later storage size. >> I'll tell you a little nuance about RPS, right? It's probably a bit >> tricky, not tricky, right? So, you might be blocked very often, and you'll get it on the third try. >> Okay. >> No, well, wait, if, well, okay, right? So, let's multiply by some coefficient, but it's clear that it will probably be less than 10,000. >> A coefficient? Uh, well, okay, fine, that's clear. Let's calculate the storage size a bit later, when I choose it. So, well, in general, in general, what about reliability? Reliability, so, as you said, we need to be able to store data for the entire history. So, well, we reliably store data. But in terms of reliability, but in terms of losses over these 24 hours. Hmm, so, in terms of losses over these 24 hours. Well, let's assume that we would like to lose nothing. Lose nothing during scraping. So, if there's an update, it's important to us. And what else? Fault tolerance. Tolerance. Uh, well, we'd like it so that if we deploy all this somewhere in the clouds or not in the clouds, that if, say, a data center goes down, then our scraping continues to work, and the data is not deleted. So, well, I would say here that we need some kind of, uh, what is it called? Multi-AZ deployment. So, that's what we need, so that if something goes down somewhere,
We, we did not lose. And so, reliability, to fail. Well, and the price, let's also consider the cost. And the cost, that is, we would like, the less, the better. Uh, the better, that is, if we can save on something, then we save. For example, I immediately foresee here that, most likely, for scraping tasks, what are called spot instances will suit us. And if we take some cloud, then spot instances can suit. Here. Further. How not to get banned? Yes. Let's do it this way. Working with bans. Working with bans. So, first. Well, first of all, a rate limiter from our side. What? Well, like, from our side, so as not to, well, as it were, not to hit limits. If we can not hit limits, let's not hit limits. Secondly, even if the rate limiter does not save and we get banned, that is, if we are not banned by IP address, our rate limiter saves us. If we are banned by IP address, then we need to somehow switch IP addresses. Well, I don't know, what is it called, dynamic dynamic proxies, which allow, uh, that is, we can buy from such providers, a pool of IP addresses and just reuse them periodically. Here are ipools. Here. And you also asked how to bypass all sorts of cloudflares. To be honest, I have no idea how to bypass Cloudflare, because everything is complicated by ML there, and Cloudflare tracks behavior, how to say it, behavior. That is, it immediately understands whether it's a bot or a person by behavior patterns. Therefore, how to bypass Cloudflare, I'll say right away, I have no idea. So I'll skip this. >> Ah, well, I don't know how to deceive them, yes, like that. Uh, what else? Working with bans. Uh, so, 24-hour metadata. Well, I think that's all. From these requirements, probably all, so as not to waste time. It seems like if something comes up later, well, yes, yes, we'll clarify. Ah, okay. So, so, so, so, well, this requirement is certainly interesting. 95% of products do not change in 24 hours, but we don't know which ones. That is, well, we can hardly guarantee that a product will always be constant, right? Well, no, of course, they can, well, as it were, change randomly, but in general, the percentage. This means that we most likely, that is, it won't work to cache some products and not scrape them, because there is no guarantee that this product has not been updated remotely. Therefore, here, honestly, off the top of my head, well, off the top of my head, the assumption is that, despite this condition, we still need to scrape these, well, scrape everything, because there is no guarantee that this particular product will be constant. Here. Well, that's how I understand it. If there are any ideas, if I'm missing something, please guide me. >> Well, look, I'm more interested from an engineering perspective, right, that is, that we somehow need to quickly check the content of the key properties for us. That is, not the entire HTML, the entire page, including all its nesting, but specifically, say, 100 properties that we need, right? And from a data perspective, we want to collect it ourselves every time, right? Or is there a way to quickly check if something has changed or not? That's the question. Uh, well, only this, well, how to do this reconciliation, right, as it's called, that is, we have, well, as it's done in Kubernetes, for example. We then, when we deploy, we have a desired state and an actual state. And then we have, say, 10 pods, right, or we specify 20 as desired. And then in Kubernetes, a worker is triggered by an event, right, some worker that deploys additional pods so that there are 20. So, here, essentially, this looks like a similar task. That we have, uh, that is, our, how to say it, desired state is what we want to be current, and the actual state is what we have now, which may not be current relative to the remote. Therefore, here, probably, well, some kind of worker is needed. If it's just a part for us, some kind of worker is needed, which, one way or another, requests data from the remote and looks, well, and looks, right, between what, well, between what is in the remote, what it answered us, and what we have. And if it differs, then we update ours. Well, I don't know, it seems like this is a very similar task to this. >> Uh-huh. >> Here. Well, okay, let's probably finish with this. And since there are no requests coming to us here, I won't say anything about the API. Probably, because, well, we are requesting instead. And it looks like you've already drawn it here, thank you. Yes, that is, we obviously, yes, should have some, uh, so we should have some workers. So, I'll probably make a copy here. I'll make a copy. Uh, so, okay. So, first, we should have, uh, let me move this so it's more convenient for me to draw. Uh, we should have, so, first, these proxies, right? That is, we clearly have some of these proxy IP pools. And this proxy, it should be available somewhere. Next, we have some rate limiter. A limiter from our side, which reacts, well, that is, a key that, which limits our traffic to the remote by some key, which can be, I don't know, what is the key? Well, the website URL, probably, for now. Let's say the site, right? Or better, let's do this. site page, right, so it limits all this by this key. A rate limiter, for example, on a fast Redis, where the state is stored. And this rate limiter is actually, well, okay, let's get to that. So, next, we should have some workers, workers, which, in fact, do all the work. SCRAPING, well, scraping workers, which do all the work. As I said, they should, for us to have fault tolerance, these workers should be deployed in different availability zones, in different data centers. And so that, uh, in case, uh, in case when we, uh, so that they auto-scale when our load increases. What, how to understand how our load increases? This is, for example, probably, uh, probably, probably, to understand if the load is increasing or not, we can by how often our data is updated. That is, if, roughly speaking, you say that in 24 hours we should, figuratively speaking, scrape 100 million data, 100 million, well, meaning, scrape data for 100 million products, then this can be a metric by which we should scale workers. That is, if this metric falls below, say, 100 million in 24 hours, we, well, or again, we can divide, right, get this RPS, roughly speaking, right, that is, if this metric falls below a certain level, we add a worker to scale it. Here. So, let me write here that scraper worker with auto scaling, auto scaling by, uh, well, RPS matrix, that is, but RPS means that, let's do this. Desired desired RPS. Here, that is, we want to maintain some RPS to ensure this 100 million in 24 hours. So, let's choose this as a metric and scale these workers by this metric. Well, if we have Amazon, we can create an auto-scaling group here. If it's Kubernetes, then do auto-scaling in Kubernetes. Here. Uh, so, the limiter in this case can be middleware for these services. Just this middleware, well, stores its data in Redis. For example, Redis is fast, well, I choose Redis because it's fast, it has atomic interaction, and so on. Here. Plus interaction with this proxy IP pool. Good. Uh, so, so, we parse with. Well, so, about the data model, let's talk a little later, let's talk architecturally for now. Uh, so, here, so, workers parse. How do they parse? So, they, obviously, should be asynchronous. I'll write it like this, that each of the workers is, let's say. If a worker is a virtual machine, then on the virtual machine, a process is running, say, a thread pool, and in each thread pool, there is asynchronous interaction, that is, an event loop, an event loop. >> Uh-huh. >> Thus, we parse asynchronously, and since we have an I/O-bound task. Here. >> So, listen, let's go up one level. I'm interested in the orchestration and scheduling of all this. That is, look, platforms, that is, how it will be divided by platforms, right, considering that platforms can be different. We can parse Amazon, and next to it we can parse, I don't know, some relatively small Dada, understood. Okay, now >> e-commerce, right. And, uh, how will this load be distributed throughout the day and how will we guarantee it, not guarantee it. >> Uh-huh. >> That is, will we just parse it with the capacity we have, or how will this flow be? That is, I understood the rate limiter, but tell me more. >> Yes, okay, let's do this. If this is, well, the task, how this, well, rather, what you just asked for is probably a classic task of either a priority queue, right, or a two, well, not two, but several queues. That is, as soon as we have something in the system that is somewhat similar to each other, but different in terms of, say, performance or something else, then, well, two options. Either a priority queue, and we, figuratively speaking, assign priorities to tasks and distribute them according to priority, or different queues. Therefore, I will probably choose the option here that if we have different platforms, say Amazon, I don't know, something else, then for each platform I will have my own set of these scraping workers, each of which, well, how it's deployed, scheduled independently, uh, and has its own metrics, uh, metrics that allow using this metric as an auto-scaling metric. Therefore, well, so, let's, let's, let's do this, right, so, okay, so this proxy remains, and for each, uh, for each such, how to say, data flow, right, I will have a separate set of rate limiters with their own state and, собственно, their own metrics with their own scaling key. What's the advantage? So as not to mix everything up, because different platforms imply different performance. That is, figuratively speaking, to parse a simple website, we need few resources, to parse a huge Amazon, we need a lot of resources. Therefore, it makes sense to separate all this by tracks. That is, it's a track per platform. I would say so. Uh, well, answering this question. Mm, okay. So, then, then let's do this, let's move on, so as not to touch. Now let's. That is, okay, workers somehow scrape. Let's see what we can have here about the data model. How can this data model look? Let's think. Uh, well, I, let's say, for my convenience, I'll write data model here. That is, well, what is it, probably, it's a URL, URL of what? Well, probably, of a product. Product URL, probably, it's needed. Next, the product name. It's its SKU or some identifier. Here, next, the date and time, when we, well, did the scraping, uh, then, for example, images, uh, well, some pictures and, for example, also, as you said, we need some description, right, so description. Here, description. So, what is description? It's, for example, some text. And it can be of any size. Images are some set of, uh, binary objects. Date time, well, it's some, well, meaning, it's no, let me delete this here. That is, it's text, it's a list of binary objects. Date time. Well, it's some date object. SKU is a string. Uh, well, so SKUs are usually strings. Well, of fixed size, not text, right? So, well, let it be for simplicity, UID 4, right, which has a size, I think, 30 characters or 32 characters. Here. Well, let it be for simplicity. Name? Well, I don't know. Well, well, an infinite name is unlikely to exist. So, well, also a string of some size. Well, and URL, right, also a string. Here, so, we would like not to make the URL text, because all these get parameters, right, that marketing campaigns, all this, can be removed, right, leaving only the raw URL to the product, so it can also be somehow limited. Uh, well, here, well, what else can be here? >> Well, that's enough, I think. >> Yes. Here's how to store it? So, look, as far as I understand the specifics of the task, uh, this product with its fields is like a document. So, for this specific task, I don't see the point in using a relational database. Uh, that is, we don't have an N number of entities with relationships, with, well, with relations, right, and so on. Uh, there are products, there are fields, and it seems like each product is a document. Therefore, probably, it makes sense here, specifically for this, for the case of storing products, to use a document-oriented database. A document-oriented database. And now, now let's do this. We just found out that the RPS from our side, right, and even at peaks less than 10k, and we want to save money. And if we know for sure that, say, in the next year, it won't exceed a hundred thousand, right, or a million per second, well, per second, right, then, perhaps, there's no point in offering engineering solutions like we discussed at the beginning, when I suggested Kafka, you said, I want to save money. Here, I think a similar principle can be applied, that there are quite a few tasks, right, and we want to, so to speak, uh, well, save money. Well, let's not add Kafka here then. Let's, probably, say, our workers, they scrape, they get data. And here we have the final, well, I don't know, let's call it an analytical database. It's somewhere at the end. Well, I'll choose either ClickHouse here, or StarRocks. Here. And we need to put the data here. Well, as we discussed at the beginning, it obviously makes sense to do it in batches. Uh, now look, it turns out that, uh, let's discuss this. 95% of products don't change. This means that if, well, what I said, this reconciliation problem, right? That is, we have some data locally and some data remotely. And how to understand, well, that is, how to achieve alignment, right, between Desired and Actual, right, that is, we need to compare. >> Mhm. But but how to save on requests? How to save on requests? Pam-pam-pam. Okay, I'll think about it a little later. Uh, so, okay. Let's do a simple solution. So, each of the workers scrapes, it does a batch insert into, well, I'll move this here, but it's not a queue, but just instead of an arrow, batch insert to ClickHouse. Well, I'll choose ClickHouse, probably, because, well, I've worked with it, I'm familiar with it, and so on. But yes. And it turns out that each of these does an insert to ClickHouse. And then, since we have different tracks and want fault tolerance, then it probably makes sense to do all, well, our ClickHouse cluster, also with multi-deployment, uh, and for each track it's separate, so that in case of failure of one track, other tracks, well, as it were, do not fail. Therefore, I will probably propose here to just copy, oh, just copy these tracks. And the difference between them will only be in what power, right, that is, what performance the instances will have here, how they will scale, by what factor, uh, and what performance the cluster will have, right? That is, for a small online store, everything will be minimal here, for a large Amazon, it will be maximal. Here. Uh, so >> so, listen, I'll interrupt you a bit, because our time is running out a bit. Let's ask a couple of questions. So, it's good in general, right? Look, a couple of things. Uh, here, uh, the first thing, right. Should we save raw data? If so, where? >> Here. Uh-huh. Yes, yes, yes, good question. For raw data, of course, this is a classic pattern. For raw data, we always do S3. Uh, and so, scraping can take raw data, enrich it, parse it, enrich it, process it somehow, and so on, and put it into ClickHouse. Raw data, yes, we should always send to S3, for example, uh, because it will probably always be convenient for us to return to it if necessary. If we don't do Kafka here, then if we did Kafka here, we might not put it in S3, because everything would be saved in Kafka, for example. Here. But if we don't do Kafka, then yes, I'll say, yes, raw data should be put in S3, so that >> In what format? In what format? >> Uh, in the simplest way. That is, figuratively speaking, what the worker, when the worker makes, well, a GET request, right, to the remote server, uh, it receives a binary representation from the server, well, that is, the HTTP response is essentially a set of bytes. Uh, and this set of bytes can be put into S3 without any additional, probably, uh, processing, or anything else, right, so as not to waste resources. Or, or the second option is, since the worker parses it somehow, right, then some, well, say, JSON or XML. Here, here's that option. Here, probably. >> Uh-huh. Okay. Look, and the second question. Uh, if sites return 429, how will you implement the retry mechanism? >> Yes, here a retry is definitely needed. 429 is when rate limit exceeded. It usually returns information in the headers when to retry. Here. And in each worker, specifically, we should have some decorator, right, that wraps the GET request. Uh, and the decorator should have an exponential backoff with jitter, preferably. Uh, and if we have, for example, a 429, it is accompanied by an exception, the exception is caught, this decorator understands that it needs to resend, but resend after some time, asynchronously sleeping. And this asynchronous sleep time we take as the difference between the retry attempt and the current moment. >> Uh-huh. Uh-huh. Okay. Good. So, well, good, thank you. Uh, because the interview time is already over. Do you have any questions? I have at least 5 more minutes to answer something? >> Yes, yes, yes, thank you. Yes, it was interesting to talk. Yes, I've even written down a couple of things for myself. I'm very interested, literally 10 seconds. >> Ah, so, where is it here? You use ML very heavily in your work, since you are an architect. Tell me, please, how specifically is it done at your place, how do you deal with it, yes, I assume you save money, possibly you use spot instances. Here. And if it's all like that, then how do you deal with the problem that Amazon takes away Spot Instances, and the ML model is in the middle of its work? Here. That is, how do you handle it? >> Well, actually, there's no drama. Models are actually the most resistant thing to spots. That is, for you to understand, probably 80-90% of the workload is on spots. That is, well, it's managed, there are its own algorithms, its own tools, how to choose instance types and in those places, in those AZs that are less prone to interruptions. Well, figuratively, for ML, we have full code there, and figuratively, a terabyte and a half or now two are requested per instance. I'm talking about memory. So, the same can probably be done on 256 maximum, I think. Well, uh, our ML engineers write the code themselves, they don't have time to optimize, so they take it very often, right, so it runs for about an hour. The probability that it will be taken away is about 50-60%. Well, you configure it, you make the run idempotent, so that if it runs 1, 2, 10 times, it doesn't matter. Well, and you configure it, and everything is orchestrated through Airflow, it's configured, I don't know, three to five, however many retries. Well, that's it. And it seems to be more or less, there were complaints at some point, but rather these complaints are actually their own fault, how the code is written, right? That is, most of the problems with the infrastructure are solved quite easily, right? It's difficult to find GPUs now, or probably impossible in SPE. But as for CPU workload, it's actually not a problem.