Transcription
Many, when studying programming, overlook, perhaps, the most important thing – its purpose. After all, code doesn't end with pet projects and job searches. Therefore, today, using a real example, we will look at how programming can simplify your personal life and automate certain tasks. Our backstory is as follows, based on real events. By the way, I was looking for a fifteenth Pro Max as a gift. They are no longer produced new, so there is only one option left: to look for used ones in some listings. And when I was sitting like this, scrolling through my phone, constantly checking if anything new and good had appeared, I felt like a turkey. After all, I perfectly understood that this could be automated in literally 30 minutes. Therefore, today we will do just that. First, let's look at the platform I personally used. It's called ofram. What is its essence? I'm sure it's some kind of type. They just quickly coded this flea market on the fly using JS code. All it does is crawl through TG chats and collect thematic listings that fit a specific category. For example, iPhones themselves. And for us, this is a huge benefit, because from three different flea markets, for example, it sits, parses, and posts them here in the form of listings, and we will then parse these listings for ourselves and look for profitable deals. So, let's look here. Select a city. Let's take Bilis. We are interested in smartphones. We select iPhone and see all the listings. Immediately, we are interested in the structure. As developers, we look at the structure. We have a catalog, cards here, and pagination at the bottom. So, how pagination works, I'll show you right away. By default, we see the iPhone category, but when we click on the second page, page 2 immediately appears. This is important. It's important to specify this. Again, we will write, of course, through a neural network. Page 1 is the first page we saw. Page 5 is the fifth page. In general, there is nothing supernatural here. Let's go back to the first page and open the first listing we find, a fourteenth Pro 128. This is what we will parse. Next, we take the photo, description, and price. Plus, there is a button to contact the seller. It leads directly to the TG account of the person selling. There are many listings here, and this is, of course, great, but if we enable sending these listings to TG every time something new appears, then we will essentially get spam on our phone that we have loaded ourselves. The logic will be as follows. We take the last 20 pages and calculate the average price for a specific model. If a new listing has a price for a specific model that is more than 10% lower than the average price among all listings, then we send this listing to ourselves in TG. Thus, we get listings only with good prices, personally for us, which may potentially interest us in purchasing. And we extract, our parser will extract the price, the main information. We already talked about this, description, photo. send it to us in TG via TG bot and immediately send a link to the seller, because, you see, this button, I hovered over it, and at the bottom of the screen I have it, but the video immediately goes to the person's account so that we can, even without wanting to, just click a button in Telegram from the TG bot and write to the seller, "Hello, is it for sale or not?" We have conducted a brief overview, in general, we have targeted the structure, links, and so on. Now we need to create a technical specification for the project. If anything, I have already opened the folder where I will be working. This is phone monitoring. We need to create a technical specification. I write to him, there is such a site, they sell used iPhones. I send the link directly. This is the main page, I'll sign it. With pagination, the page has a tag 2345, which we have already seen. I send the link. The detailed product page looks like this. Let's go to the same iPhone, just copy this link, paste it here. And now we need to describe what? Describe the stack, how we want this to work at all. Describe where we will deploy this. Deployment, by the way, would be on a VPS, so that our script constantly runs and sends us listings. This is a mandatory point. You can't just run it on your computer. That's not ideal. I'll make a rough sketch. Look. I'll send it for generation so as not to waste time. Oh, I need to do something. One second. And, damn, you can't edit here in antigravity, by the way. I'm using antigravity. Here I chose the Opus model because I have an AI Pro subscription. And AI Pro, I found out about this very recently, it gives access to Opus. And Opus has more limits, of course, than the regular code for $20 directly from Anthropic. But honestly, this Opus is a bit dumber than native Claude code. But in general, for such a simple project, it will do. Therefore, here I duplicate. So, gather all my requirements, all my requirements into a technical specification for development in one .mfile. MD. I forgot to add that. And I will also add that for the correct compilation of the technical specification, you can make requests to the site we are parsing. I send the prompt. Let's go over it again. There is a site for used iPhones. I sent the link. How pagination works, I also sent the link. The detailed product page looks like this. I sent the link. I want to write a script in Python. Python is the best option for such projects. Maximally simple, fast, convenient. Neural networks write wonderfully in it. that will parse this site for iPhone sale listings. My script should parse the last 20 pages once a day. That is, we take the last 20 pages, collect the listings into one array, after which we calculate the average price for each iPhone model. We will start from the eleventh model, but what's below, like the tenth, eighth, why do we need it? Also, all Pro and Pro Max versions must be considered. Once a day, a large parsing of up to 20 pages of pagination is carried out to calculate these average prices. That is, once a day we do a large parsing of 20 pages, calculate the average prices among all these listings for a specific model, and then every 20 minutes we do monitoring. If during such monitoring on the first page, a listing appears that has not been added to the database before, meaning we haven't parsed it before, and this listing has a price lower than the average price for this model. That is, it's lower than the average price minus 10%. Then this listing is parsed, its detailed page, after which it is sent to me in private messages via a Telegram bot. That is, the Telegram bot sends such a notification: here is a good listing, the price is 20% below market, for example, and the listing should contain all the main information that is indicated on the detailed page of the listing. Compile all my requirements into a detailed technical specification for development. One tech file in MD format. Why is it the most convenient? Regular markdown files. I love them. And the code should be secure. Corresponds to OP DRI for correct compilation of the technical specification. You can make requests to the site we are parsing to improve the structure, so that the neural network sends requests directly from our computer and sees what it returns. So, the composition of the detailed technical document. There are several open questions. Should memory size be considered when calculating the average price? No. Second. Should battery health be considered? Also no. Should we create or use an existing one? Of course, create. Create. Is a notification history needed? No, it's not needed. Oh, wait. Answer the questions. I clicked the wrong button. Upgrade. Ah, in general, they gave me i pro credits in antigravity. And in this example, you can also see that the same Opus, but in antigravity, it's really dumber than from the codes themselves, because we go here, I sent him the same message, copied, sent it. And what did he do? If in antigravity he just made a plan and asked questions, but at the same time made a plan. He didn't make any requests to the site, because if he had, we would have seen it here during his work. But the code is just a web version, I sent him the same thing. What did he do first? He went and got the structure, and he did it himself. Usually, the web version of the code is now writing us a technical specification. That's why I think Claude code or just code directly from Anthropic is much smarter than intermediaries like antigravity. We see that the code has written us a technical specification. Let's open it and see. This is an MD file where everything about our project is described. Let's go from top to bottom. First, the goal is stated: development of an automated monitoring system. That's good. Main functions: parsing, monitoring, sending notifications, data storage, storing data in an SQLite database. That's fine. URL scheme, data structure, structure for the detailed page, supported models. So, everything is written here. Architecture, applications, he also wrote. And you see, this architecture corresponds to OOP. Why? I, by the way, in the prompt where I wrote this, I'll show you now. OOP. Why did I ask it to correspond to OOP? The thing is, very often neural networks like to write some scripts, simple bots, right in one file. And then you end up with a huge file of 800 lines, which is just, well, not very pleasant to look at, wouldn't you agree? Therefore, I always advise breaking down projects like scripts, bots, parsers, and so on using the OOP method, so that we have a simple and understandable structure. Next, we have a diagram. Of course, it's not mandatory, but okay, let it be. A database schema to create the database. The client itself, which will interact with Telegram, from where we will parse everything. And what else? Message format. The message format suits us: profitable offer, profit, average price, model, description. Seller. Immediately a link to the seller on Telegram, so that you can go directly from the bot and a link to the listing itself. Task scheduling, configuration. In general, we have received a competent technical specification that we can feed to the neural network, and it will write. Well, a wonderful result, because everything is described here. And when we have a competent technical specification, the neural network writes much better code because we have described everything to it, given it a ready-made instruction, and now we need a folder where we will store the project. I just opened a folder called Public Projects. It doesn't matter, name it whatever you want. You can open your own folder. And we need a folder where our project will be located. Let's call it MicDer folder TG. Well, in honor of the TG bot, I'm sitting and writing CD TG. I find myself in this directory. Here it is shown on the left. And first, I want to create a Python 3 virtual environment, since I have MacOS, or on Linux also Python 3, on Windows just Python. Venf VF. And now we activate the virtual environment. Source Venf bin activate. Okay, we are in our project. Virtual environment is just for our convenience. And now we have copied the technical specification. Let's paste it where? Here in the files, right in the TG folder, we will create a new file, let's call it tech.md, i.e., technical specification. Paste our technical specification here. Let's remove the further development option just in case. Well, not just in case, but let's save some of our context, because everything you feed into a neural network, it consumes context. We have received our technical specification. It is in the tech.md file. And for convenience, we are now in the publics folder, I opened the TG folder. Everything, here is the directory of our project. Here again is the technical specification. And we will use Cloudcode as the neural network. Why Cloudcode? Opus 4.6 has already been released. In general, Cloud CD has objectively been the leader in code writing neural networks for the last year. And of course, you have antigravity with GM 3 Pro. If you want to use it, use it. After all, code requires a subscription to use it normally. And I want to warn you right away, Opus in antigravity is dumber than Opus from Cloud Code. Yes, I may have said this already, but I have to repeat it. I'll open the terminal, I type CD public project, I get into our project directory. And here let's have such a simple prompt: read the project's technical specification, after which fully implement it. Yes, the technical specification has been fed. The task is there. It's writing, writing wonderfully. When installing libraries, use VENF so that it doesn't download libraries directly into my computer's global memory. Use VENF. And definitely enable plan mode, because this way the neural network analyzes everything first and creates a plan, and then executes this plan, and it writes code better this way than, well, just in normal mode. Therefore, I recommend using plan mode in this case, at least. And before that, let's initialize our repository. We will need Git. I type the trivial Git init. And what do we need? A .gitignore file. gitgnor. Here we will put our VF folder. It must be ignored 100%. And now git add dot. Add all files. Git commit -m init. This is our repository initialization. And let's not link it to GitHub repository for now. It still needs to be created. I'll just launch the prompt, let it do its thing. In any case, eventually, everything will have to be sent to the repository. Why? Because we will host the bot on a VPS. And to send it to a VPS, you simply need a Git GitHub repository. The most convenient option. Here I click Accept. Let it do it, execute its plan. We just wait for the result. After that, we test this result. You can ask the neural network to write tests, run them, and we will see the finished result. And after that, when we have the tests ready, when we see that the project is ready, we take a VPS, deploy it there, and get a ready-made bot that works around the clock on our server and provides us with information. The neural network tells us that it has coped with the result, what it has done. All imports work. The model detector works in five test cases. The database is initialized and creates tables successfully. And all dependencies are installed in our virtual environment. And now, to run and test, well, locally, we need to copy V Example to UNV, fill in the Telegram bot token so that the bot sends us this information specifically, and run Python Mind Pie. Okay, no problem. Let's do this: create a .ENV file, as it tells us. .ENV. I type ENV exampмle copy, paste it right here. And now we need to get the Telegram Bot Token. This is as simple as possible. Go to TG. Of course, you will need BotFather for this. Create a new bot and come up with some username for it. Create bot. After that, you get the token. I think you will figure this out perfectly. Now we need the user ID, our chat ID. Telegram chat ID, you can also get it somehow, ask the neural network how to get it if you don't know. Now I will just paste my user ID and continue. I have inserted our ID, by the way, don't pay attention. I just don't want to show my ID, so type python main p. Wait, before that, we need to activate our venf. Source venf bein activate. Activate and python. Okay, let's run it. Let's see request fils certificate. Yes, this is a default error on MacOS, so I copy it, paste it here. And here, let's turn off our process for now. Such, I type such an error. Error when trying to deploy. Oh, when trying to run locally, okay, it will fix it quickly, because I constantly encounter this problem with Macos, yes, and it will work. Okay, it confirmed that there was a problem with MacOS. Add CertiFй. We run MINP and see what's happening. Parsing the first twenty pages. And as you can see, it's happening successfully. It's not blocking us, because the site, well, let's say, doesn't have the best protection against bots. Now we will look at our result. It should finish 20 pages, calculate average prices for everything. You see, it saved Overage Prices for 24 models. In total, we parsed 2,480 listings. And now, in theory, if something new comes out, it will send it to us in TG. I don't see any errors, so instead of sitting and waiting for it to send something, or writing custom code for it to send something, what will we do? Deploy it to a VPS. And, well, I'm just sure it will work, as it's a maximally simple script, so there's no point in even checking it much. Make me a guide in a .md file on how to deploy this project to a VPS so that the bot works constantly. Send it, it will create an MD file. We will just copy-paste commands to our VPS. And yes, as you can see, now with such powerful neural networks, it's not just rote memorization of code that is valued, but rather analytical thinking to correctly build the workflow. Because a neural network itself, of course, does nothing. It needs to be directed correctly and to extract a good result from it. And for that, you need to think a little. Depource done, let's quickly go through it. Server preparation is good. Installing Python, creating a user, uploading the project. Then we have environment setup, launch check, system disсвиice. And you see, this is something, well, long, right, let's agree, sitting and setting this up manually is not very desirable. And let's ask such a simple question. What would be easier: if you write me a script to automate deployment, or if you configure a docker container for this? That is, again, neural networks are a huge knowledge base, and you periodically, when you don't know how to implement something, can ask it a question, and it will offer you a couple of options. You can write it like this. It offered a couple of options, but don't write the code yet. But don't write the code yet, I want to choose. Well, in our case, it decided, ah, to write docker, because docker compr, because Docker has always been a convenient thing. Three files have been created: Dockerfile, Docker Compose, and Deploy. Dockerfile. Here's Python, it installs, ah, configures our libraries and launches the project. Docker compose simply opens our service. The service that handles monitoring. Here it is. And it updated us. Now we need a VPS where docker is installed, and configure docker comprd build. That is, we have much fewer commands than before, as you can see. And now we need a VPS, of course, where we will host all this. Here I can recommend Cloudcell. This is a marketplace of servers where offers from various providers are presented, collected in one place. We will need a European server 100%. Then from one processor core and from one RAM. Then we will definitely have enough for everything. We see offers again from different providers on one page. We can compare them and then purchase the necessary VPS. After purchase, there will be a dashboard here. I was logged out of my account. I don't want to log in again. You copy the server IP address, login, well, by default everywhere it's root and password. After that, we log into our server. To log in, we have two ways. The first way is to use some SSH client. I usually use Termius. I like it the most. You can use any client, it doesn't matter, or connect directly from your computer's terminal. You can ask the neural network how to do this. And the second option is to use an IDE. I've been using Z for the last six months instead of VS Code. I've completely forgotten about VS Code. And Z has built-in SSH connection to a remote server. We open this thing, Open Remote Folder. And here you configure your SSH server. Only I'll tell you right away, if you bought a server with 1 RAM, 1 CPU, it's better not to use this. Why? Because it's a full-fledged code editor. And you, roughly speaking, on the VPS, will open the same IDE Z. You will open a code editor on your VPS, and this will, of course, load it. It will consume RAM, a little CPU. If you have a server that you bought just barely enough, i.e., one RAM, one CPU, then your server simply won't handle the load that the project gives. Well, the docker container itself in our case, and the load from our IDE, it won't handle. Therefore, if you have a weak server, please connect via SSH Termius, again, or through the regular terminal. Also, so as not to forget, before deploying, we will, of course, need to create a repository where we will host all this. And let's call it TG parser. I'll write TG parser public and create repository. I click, I don't need anything else. Now we need to link our repository. It needs to be linked. And look, we have a .cod folder. A folder has appeared in the project. It's not in Gitignore, so we need to add .cod here so that this folder from the neural network doesn't end up in our repository. Yes, it's not needed there, wouldn't you agree? Nothing else unnecessary gets into our repository, so I type git dot again, git commit -m and I'll write deploy. I'll just write deploy. Okay. After that, we return here, copy these two commands, and send all our changes directly to the repository. Reloading the page, we see our project here. Here it is. And now this project needs to be sent to the VPS. It's public, so you can download my project directly. Public repository. No need to configure SSH keys. We just need to open our VPS. Here, by the way, I am on the VPS. The IP address is written here. The IP address is written here. And all commands, all files, all changes that I create or edit now in this IDE window, they happen directly on the VPS. That's why I like using Z for connecting to a VPS, because it's maximally convenient. You open the IDE, roughly speaking, on a remote server, and all changes you make here happen on the server. Therefore, let me create a folder MDIR Parser. I'll click CD Parcer, I'll click, go to CD parcer. And here we can clone our repository. Git clone. I paste the link to the repository here. And we go into TG-parser. TG parser. You must have docker installed on your VPS. Look how to install docker docker complз through the neural network. There are literally three or four commands to copy, and everything will work. And according to our MD guide, we remember what we need to do. First, .ENV, right? We type Touch Touch.v. Here, let's type vim.v. And remember, I configured the bot locally in our local project directory. Let me just copy that ENV and paste it here. Here it is, my local V. I select it completely, copy it, return to the tab where I am on the VPS. I paste it directly here using Vim and save all our changes. After that, docker compose up compose compose up - - build - -. Okay, we build the container, and launch it immediately. And after that, our bot should be working. We see that everything has launched successfully. Just in case, you can type docker compl and see if there were any errors. No, there were no errors. And we see that it's already scanning the seventh page, it has scanned it for us, so after that it will send us notifications about new listings whose average price is below market. Okay, we have completed the task, got a normal configuration, a normal project. And what I wanted to show with this video? The project itself is maximally simple, agree, especially when you do it through a neural network. What I wanted to show is that programming can be studied not only for pet projects or endless job searches. You can simply make your life easier with such simple projects. That is, you have some need, boom, you write it through a neural network, automate something, and it becomes easier for you. Because I often notice that people who don't have much experience yet, they write and write and write. But how to use it in real life? Well, they don't really understand. Therefore, please think of ways to use your skills for personal purposes. This will help you a lot. And with the help of a neural network, you can write anything if you find the right approach to it. By the way, after a couple of days of testing, here is the result. It really sends phones, and the average price is below market. This is true.