Transcription
Greetings everyone. Today we have, well, a very intriguing topic for discussion. We are diving into a world where code is written not only by programmers but also by machines, artificial intelligence. This is called coding. Yes. Agentive coding. It sounds futuristic and, I must say, the potential there is enormous. Exactly. But as always with new technologies, there are nuances. Pitfalls. And one of the most noticeable is how AI agents handle, well, the appearance of applications, the user interface. Yes, precisely. They can write logic brilliantly. Complex algorithms, please. But when it comes to how it all looks on the screen, that's where difficulties often begin, shall we say. Exactly. It's as if they are coding blind, writing code that works functionally, but they have no idea how it will render, how the end user will see it. Yes. There's no visual control that a human developer has. And today, we have, so to speak, material for analysis in our hands: a video from a developer. His name is Leon Van Zyl. He encountered a very typical, as it turns out, problem. An agent made him an application, and there were, um, visual glitches, seemingly minor things, you know, but the kind that spoil the whole impression. Imagine, a beautiful interface, dark theme, all stylish, and the letter in the logo is cut off. How can that be? Indeed, it immediately catches the eye and reduces, perhaps, trust in the quality of the product. So, our task today is to delve deeper into why this happens. Why are AI agents so blind to visuals? And most importantly, can they be taught to see, to notice and correct their own interface errors? An interesting question. That is, can they be given not only the ability to write code but also, um, a semblance of sight? Exactly. And we will examine one specific tool that Leon Van Zyl proposes in his video as a possible solution. It sounds promising. So, let's get started and dig deeper. Let's start with the example he shows, so it's clear what we're talking about. Yes, excellent idea. So, imagine an agent, well, let's say an advanced model like GPT-4 or Claude, receives a task to create a web application. What kind of application? What should it do? The concept is quite simple but useful. An application where you can upload some unstructured data. For example, just the text of an article or a product description, or even perhaps an image with text. Yes. Something that is difficult for a machine to understand immediately. Exactly. And the application should process this data and output structured information, that is, break everything down, for example, in JSON format. Understood. JSON is a standard format where data is organized as "key-value" pairs. Very convenient for further processing by other programs, for databases, for APIs. Exactly. Suppose you upload a product description and get JSON with fields for name, price, characteristics. You upload an article and get a title, author, key takeaways. Quite a useful thing. And did the agent manage to create such an application? On the functional part, yes, it wrote the code, thought through the data processing logic, and even made the interface. Moreover, as Leon shows, the interface looks quite good at first glance. Modern, yes, quite. It even has a dark theme, which is fashionable now and pleasant for the eyes. All controls are in place, a download button, a field for displaying the result. Everything seems fine. And you see, the logo in the site header. Beautiful, stylized. And one of the letters, the letter G, its lower tail is simply cut off, as if it didn't fit. Hmm, strange. So, not the whole letter is visible, right? The lower part just disappears. And it's not just in the logo. Leon shows that the exact same problem repeats elsewhere in the interface where the same font or style is used. Another cut-off G. So, this is already looking less like a random rendering error and more like some systemic problem in CSS, most likely in the styles responsible for text display. Exactly, some setting, maybe line height, or margins, or the overflow property, which determines what to do with content that doesn't fit into a container. Something from this is set incorrectly. And this is a typical example of what AI might not notice. For it, G is just a symbol in a line of code. It doesn't see that this symbol is displayed incorrectly on the screen. Yes, and it seems like a trifle. Well, what's a letter's tail? But you'll agree, it immediately reduces the overall impression of the product. It looks unprofessional, sloppy. Absolutely, it's like a stain on a perfectly white shirt. Functionality isn't affected, but aesthetics are compromised. And the user immediately wonders: if they missed such trifles here, then what about more serious things under the hood? Exactly. And the video author, Leon, emphasizes that this is not some unique problem of his specific agent or his specific application. He says he has encountered similar visual glitches, well, defects, when working with other platforms. Low-code, no-code approaches, and says he has seen similar artifacts even in examples from other developers who are also experimenting with AI code generation. So, the problem is not in the specific implementation, but in the approach itself, in how these AI agents currently work. It turns out so, it's some kind of systemic blindness. And here we come to the most interesting question. Why? Why are they so great at logic and so, um, inattentive to visuals? And here, I think, lies the very essence. Modern large language models, on which these AI agents are based, they were trained on vast amounts of text information, including millions of lines of code from GitHub, Stack Overflow, etc. Yes, they absorbed all these patterns, structures, syntax of different programming languages. They perfectly understand program logic, how functions, classes, modules are interconnected. They can generate code that compiles, that performs a given task. But they lack what we humans call visual experience or visual perception. That is, they were not trained on interface images. Not entirely. Some multimodal models can process images, but it's one thing to recognize objects in a picture or even describe a screenshot in words. And it's quite another to understand how the code they wrote will turn into pixels on the user's screen. To understand concepts like harmony, balance, layout neatness, text readability, convenient element placement. Yes. So, they lack this intuitive aesthetic judgment. Exactly. They don't see the result of their work the way we do. They can follow instructions. Make the button blue, center the text. But they cannot assess how well this blue matches other colors, or if the font is too small for this text, or if this text overlaps with the neighboring element at a certain screen resolution. It turns out they operate with abstractions, code, styles, but without understanding their specific visual embodiment. Absolutely. They code blind. They follow rules and patterns learned from training data, but without that final visual check and correction that a human developer or designer always performs. They look at the result, think, "Hmm, something's not right here." They open developer tools in the browser, adjust styles, move elements, ah, and they can't do that, at least not on their own. Understood. But, of course, there's a workaround. As Leon himself says, you can take a screenshot of this problematic letter G. Yes. Send it to the same AI agent and say, "Look, there's a glitch here. The bottom part of the letter is cut off. Please fix the CSS so it's normal." Yes, such an approach is possible, especially with multimodal models that can look at an image and understand what's wrong. But that's, well, that's manual labor. Take a screenshot, write a prompt, send it, wait for a response, check. And if there are not one, but 10 or 20 such bugs, it's incredibly tedious and eats up all the advantages of this fast code generation. Agreed. It turns AI from an automated assistant into a tool that still requires constant manual supervision and correction specifically at the visual level. Not very efficient. Exactly, and the logical question arises: can't this process be automated somehow? Can't the agent be given the ability to look at the page it generated itself? To find this visual bug itself, to give it eyes. Exactly. And here Leon Van Zyl presents a solution he found and tested. It's a tool called Playwright MCP. Playwright MCP. Sounds interesting. Playwright is a well-known framework for browser automation from Microsoft. It is. It's very actively used by QA engineers for writing automated tests for web applications. It allows programmatic control of a browser, opening pages, clicking buttons, entering text, taking screenshots, checking element content. In short, simulating user actions. Yes, a familiar thing. And what is MCP? MCP stands for Multi-Capability Provider. Leon describes it as a free, open-source server that he himself, it seems, developed or improved. And this server acts as a bridge between an AI agent, for example, Claude or GPT, and the Playwright library. That is, the AI agent can send commands to Playwright through this MCP server to control the browser. Exactly. Essentially, this gives the AI agent direct access to a real browser. It can say, "MCP, open this page." MCP, through Playwright, opens the page in the background. Then the agent can say, "Take a screenshot." MCP takes a screenshot and gives it to the agent. The agent can look at the screenshot, if it's multimodal, or analyze the page's DOM structure, which MCP can also provide. So, uh, this is very interesting. So, it's not just browser control, it's getting feedback on how the page actually looks and is structured. Yes. And here's the key point you touched on earlier – how to give the agent eyes and hands. Tell us more about this analogy. Eyes, apparently, is the ability to get a screenshot or the page structure. Exactly, the agent can see the current state of the page. Not just the code it wrote, but how that code rendered in a real browser. It can analyze this visual or structure. And hands? And hands are the ability to interact with this page through Playwright, click a button, enter text, scroll. And most importantly, hands are also the ability to make changes to the code. How? Directly from the browser. Well, not exactly from the browser. The agent, after analyzing the problem, for example, through a screenshot or DOM, can decide what needs to be changed in the CSS or HTML code. It generates this corrected code. And then, the details can vary, but it's possible that through the same MCP or another integration, it can update the project files and reload the page in the browser to see the result of the changes. Valuable. The cycle looks something like this. One. The agent receives a task, for example, fix bug X. Exactly, and that's valuable. The cycle looks something like this. Two. Using MCP and Playwright, it opens the page and looks at it, gets a screenshot, DOM. Exactly, and that's valuable. The cycle looks something like this. Three. It analyzes the problem, determines what needs to be fixed in the code. Exactly, and that's valuable. The cycle looks something like this. Four. It generates a fix, for example, new CSS. Exactly, and that's valuable. The cycle looks something like this. Five. It applies this fix, updates the code. Exactly, and that's valuable. The cycle looks something like this. Six. It opens the page again through MCP Playwright and looks at the result. Exactly, and that's valuable. The cycle looks something like this. Seven. If the problem is solved, great. If not, it returns to step three, tries again. And all this is potentially without human involvement or with minimal involvement at the start. Ideally, yes. A person sets the task at a high level, here's a bug. Or test this scenario and fix bugs if you find any. And then the agent itself runs this cycle. Analysis, correction, verification. It sounds really powerful. This is no longer just code generation based on a text description. This is much smarter. Exactly. This is a demonstration of the very transition we talked about, from working with abstract code to working with a concrete visual result. The agent doesn't just guess what might have gone wrong in its code; it reacts to an actual visual defect that it saw itself with the help of MCP. Yes, it's like a programmer sitting next to a designer or tester who points to the screen. Fix this here. Only here, the agent is both the programmer and the tester itself. And this opens up huge prospects. Imagine how much time developers spend on this minor UI tweaking, on fixing such visual bugs. If AI can take on at least part of this work, it will free up time for more complex conceptual tasks. But the question arises: okay, it handled the simple letter G. It's a relatively isolated, understandable bug. But can an agent with such eyes and hands handle something more complex? For example, testing entire user scenarios. Not just looking at one page, but going through steps like a real user would, and catching and fixing visual problems along the way. Ha, that's much harder. Here, it's not only necessary to see the static result but also to interact with the page, click, enter data, navigate between sections, and at the same time evaluate the visuals at each step. Exactly. And Leon Van Zyl decided to test this. And he gave his Claude with connected MCP a new, more complex task. What was the task? Something like this: "Okay, Claude, now let's test the main user path. Use Playwright MCP. Start from the main page. From there, go to the dashboard. In the dashboard, find the API keys management section. API keys are special strings needed to access the application's programming interfaces so that other programs can communicate with it." Yes, absolutely. Often, web applications have a section where a user can generate a key for themselves, copy it, and then, if it's no longer needed, delete it. Standard functionality for many services. Understood. So, go to API keys management. What next? There, Leon tells the agent: "Create a new API key. Name it 'testing'." After you create it, immediately delete this 'testing' key and then report to me that everything went successfully, the entire scenario is completed. That is, a full cycle, navigation, creation, deletion, verification of the main key management function. Yes, but that's not all. Leon added important conditions. "And please, if during the execution of this scenario you notice any visual bugs or UI problems, fix them, and then try to run the scenario again to make sure everything works and looks good." Wow, that's a serious claim. That is, not just to pass the test, but also to act as a QA engineer and UI developer along the way. Exactly. To find it yourself, to fix it yourself, to re-verify it yourself. So, how did the agent cope? And this is where the most interesting part begins. The agent accepted the task and started acting. Again, through MCP and Playwright, it began to simulate user actions. Clicked on the dashboard link or button on the main page, navigated. Yes. On the dashboard, found the API Keys link or something similar. Clicked, navigated to the key management section. So far, so good. There, it found the "Create New API Key" button. Pressed it, a modal window appeared with a field to enter the key name. The agent entered the text "testing" there, pressed the Create or Generate button. So, the key was created. It should have appeared somewhere, right? And it appeared. According to the description, it was displayed in a pop-up modal window. And here's where a new UI problem arose. The very one it was supposed to notice and fix. Exactly in this pop-up window, where the newly created key was shown, and keys are often long strings of characters, the text of this key and possibly some accompanying information, they didn't fit within the window dimensions. That is, the text was extending beyond the container boundaries or overlapping with the OK or close buttons. Yes, something like that. Leon describes it as "content was overlapping the container." A classic layout problem where they didn't account for the possible length of the content. It looked, naturally, sloppy and unprofessional. And what about the agent, did it notice this glitch and fix it, as instructed? Well, no. And this is a very telling moment. The agent, it seems, was so focused on completing the main scenario – create the key, then delete it – that it either didn't notice or ignored this visual defect. Despite the direct instruction, fix bugs along the way. It seems so. It saw that the key was created. Found the delete button next to this key or in the same window, pressed it, the key was deleted. The scenario, from a functional point of view, was completed successfully. And it reported back to Leon. Yes, it wrote something like: "Scenario completed successfully. I navigated to the dashboard, to key management, created the 'testing' key, and then deleted it. All steps are completed." And not a word about the visual bug in the pop-up window. Not a word. It simply omitted it. Hmm, this is a very important nuance. It turns out that even with eyes in the form of Playwright, MCP, and the ability to see the page rendering, the agent can be imperfect in detecting all visual problems without explicit specific instructions. Exactly. Perhaps it considered this bug not critical for completing the main task of testing key creation and deletion. Functionally, nothing broke, the key was created, deleted. Yes, it's similar to how human testers sometimes work, especially with automation. They write a script that checks functionality. The button was clicked, the element appeared, the text matches expectations. But the fact that the button is slightly crooked or the text overlaps the border, an automated test might not notice this if it wasn't specifically targeted for it. Right. So, the agent coped well with the main task: go through the scenario steps, click around, enter data. Playwright and automation have been doing this for a long time. But with the task of also being a sharp QA, noticing all visual defects along the way, it faltered here. And this shows that full automation in such complex tasks, combining functional and visual testing with correction, is apparently not yet achieved or requires very precise tuning and very clear instructions for AI. It even emphasizes that human-AI interaction is still critical for fine-tuning, for catching such non-obvious or non-critical problems from the AI's perspective. A person would immediately notice this text overlap and say, "Okay, stop, this doesn't look good, it needs fixing." But an AI, focused on the task, might miss it. It turns out that Playwright MCP is a powerful tool, but not a magic wand that finds and fixes everything perfectly on the first try. Exactly, that's what Leon shows further. He saw that the agent ignored the bug and decided to give it another chance, but with a more specific hint. That is, he prompted it where to look. Yes, he wrote something like this to the agent: "Thank you for completing the scenario. However, after creating the API key, there was a small UI problem in the pop-up window. The content was slightly overlapping the container boundaries. Could you please use the MCP server again, go through the steps up to the point of key creation, and this time fix this specific UI problem in the pop-up window?" Yes. Now the task became much more specific. Not just fix bugs along the way, but here, in this place, there's a specific bug. Fix it. Exactly. And here the agent showed its best side. It took on the task again, launched Playwright again. Through MCP, it went through the steps up to the point of key creation. The key was created, the problematic pop-up window appeared. Yes, now it knew what to look for. It analyzed the situation, the content didn't fit, and started looking for a solution. Moreover, as the video author describes, the solution didn't come instantly. That is, it tried several options. It seems so. Leon says the agent approached the task iteratively several times. This means it likely tried different CSS changes. Maybe it tried to increase the width or height of the window, maybe reduce the font, maybe add line breaks. Such a trial-and-error method, but based on visual feedback. It changed the CSS, looked through MCP how it turned out, if it didn't help, it tried again. That's exactly how it looks. Such an automated iterative UI debugging process. It tries different approaches until it finds one that solves the content overlap problem. And what did it come up with in the end? Ultimately, the agent found a solution it considered optimal. It decided not to change the window size or fonts, but to add a vertical scrollbar to the pop-up window. A scrollbar. Ah, well, that's a classic solution for cases where the content might be more than what fits in a fixed container. Exactly, it added the CSS property `overflow-y` to the pop-up window container. And now, if the content didn't fit vertically, a scrollbar appeared, and the user could scroll to see the entire key. And the overlap problem was solved. Yes, the content no longer extended beyond the boundaries. The agent checked the result again through MCP, made sure everything looked neat now, and reported back to Leon about the successful bug fix. This is very impressive, the ability not only to find a bug with a hint but also to iteratively find a working solution for a layout problem. You'll agree, this is a step forward compared to the first example with the letter G. That was a simpler fix, and here it's choosing a solution for dynamic content. Yes, and if we connect this to the overall picture, we see how tools like Playwright MCP truly expand the horizons of AI agent capabilities in development. This is no longer just code generation based on a text prompt; it's much more. It's automated end-to-end user scenario testing. Dynamic error detection, including visual ones, and even dynamic correction. Even if not always on the first try and sometimes requiring human input for non-obvious bugs, yes, but imagine the potential workflow. A developer describes a feature, and the agent writes the code, creates a basic interface. Then another AI agent or the same one, using MCPR, runs a set of tests, goes through all the main scenarios, finds functional and visual bugs, fixes them itself, re-verifies, and the developer receives not just working code, but code that has also passed basic visual testing and self-correction. Exactly, this is a step towards creating more reliable, self-correcting applications using AI. This can potentially significantly reduce the burden of manual QA, especially routine interface testing on different devices and resolutions. Yes, this testing on different devices is also a painful issue. With Playwright, you can emulate different screen sizes, different browsers. And the AI agent could run tests and fix layouts for all of them. Theoretically, yes. If given appropriate instructions: "Test this scenario on desktop, on tablet, and on mobile. If the layout breaks anywhere, fix it." This opens up possibilities for automated responsive design and testing. It sounds almost like science fiction, but Leon's example shows that basic steps in this direction are already being taken, tools are emerging. And this is very important because without such tools, without eyes and hands for AI, its application in UI development would remain limited to generating initial code, which would then still need to be painstakingly refined and corrected by humans. It turns out that Playwright MCP or a similar solution is the missing piece of the puzzle that allows AI to become a more complete participant in the UI development process. Quite possibly. At least, it's a very interesting and promising approach. Okay, let's try to summarize this discussion. What is the main conclusion we can draw from Leon Van Zyl's example and his use of Playwright MCP? Well, I think the main conclusion is that this problem of AI agents being blind when coding interfaces is real, it's systemic, but it's not insurmountable. That is, there is a solution, or at least one is emerging, right? Tools like Playwright MCP can truly become a kind of vision prosthesis for AI. They give it the ability to receive feedback on the visual result of its work. And what does this allow AI to do? It allows it to move from simple code generation to meaningful debugging at the visual level. It can find and fix specific layout defects like that cut-off letter G. And not just static bugs, but also problems that arise dynamically. Exactly, as we saw in the second example, such a seeing agent can perform automated testing of entire user scenarios. And most interestingly, it can simultaneously fix found UI bugs, especially if given a sufficiently clear hint, as in the example with API keys and the pop-up window where it added a scrollbar. Yes, so this is no longer just a generator, but to some extent a debugger, and even a bit of a QA engineer. Okay, this is what we see now with this tool, but it inevitably leads us to questions about the future: where is all this heading? And here the most interesting field for reflection begins. If AI agents truly learn not only to code but also to independently and effectively see, test, and fix user interfaces, how will this change the entire software development landscape? The first thing that comes to mind is the role of humans. What will happen to developers, UI/UX designers, QA engineers? Exactly, if AI can polish interfaces to perfection itself, find and fix not only obvious bugs but possibly even some aesthetic flaws or usability problems, if it's taught to do so. What will be the role of a UI/UX designer then? Perhaps the designer will not draw layouts pixel by pixel, but rather set general principles for the AI: a design system, rules of visual harmony, and the AI will follow them and generate specific screens. Quite possibly, the role will shift from direct creation to defining rules, guidelines, and validating the AI's work. The designer will become, so to speak, an art director for the AI executor. And QA engineers, if AI tests and fixes UI itself, their role may also transform. Perhaps the focus will shift from manual searching for visual bugs, which is very labor-intensive, to more complex things. Testing non-standard scenarios. Human perspective testing, performance and security testing, as well as developing and maintaining these AI testing agents themselves. They will need to be trained, configured, checked to ensure they test correctly and effectively. So, the work won't disappear, but it will change significantly. New skills will be required. Most likely, yes. Skills in working with AI, prompt engineering, understanding how these agents think, how to best guide and control them. And the developers themselves, those who write code, for them this could mean, on the one hand, freedom from routine layout and fixing minor UI glitches. They can focus more on architecture, on complex business logic. Sounds good. But on the other hand, it may be necessary to understand more deeply how AI generates code and interfaces to interact with it effectively, to customize results, to fix errors that AI itself couldn't fix. The developer might become more of a system operator for development. And one more big question. How close are we to this future, to fully autonomous cycles of application creation from idea to visually flawless, tested product created by AI? I think fully autonomous cycles are still far off, especially for complex innovative applications. The example with the missed bug in the API key scenario shows that AI still lacks human intuition, attention to detail, understanding of context, and the importance of aesthetics without explicit instruction. So, problems remain, of course, understanding truly complex or poorly formulated requirements, creativity in design, creating something truly new, not just compiling existing patterns, handling very rare or unexpected use cases, ensuring compliance with specific brand requirements or accessibility. All of this still remains difficult for AI, and therefore, in the near future, we are talking more about a hybrid human-AI approach. I think so. AI takes on more routine tasks: template-based code generation, basic layout, automated testing and fixing of simple bugs, while humans set tasks, control the process, solve complex problems, contribute creativity, and ensure high-level quality. And tools like Playwright MJRCP are precisely what make such collaboration more effective by giving AI the ability to better understand the visual side of development. Exactly, they help overcome this barrier of blind coding. And what other challenges or opportunities does this open up, besides changing roles? For example, the issue of development speed. If AI can generate and debug interfaces faster, perhaps we will see accelerated release cycles for new products and features. That's a plus, but the downsides? Perhaps the risk of excessive design unification. If everyone uses the same AI tools, trained on the same data, won't all applications start to look alike? Hmm, good question. Loss of uniqueness, right? Or the black box problem. If AI wrote the code itself, fixed it itself, and then something went wrong, how easy will it be for a human to figure out what happened inside? Debugging AI-generated code can be a non-trivial task. Plus, trust issues. How much can we trust AI to have not just fixed the bug, but done so in an optimal and safe way? Exactly, new approaches to validating and verifying AI agent work will be needed. In general, the picture is complex, but very dynamic. We are on the verge of significant changes in how software is created, especially its visible part, the interface. Yes, and tools that give AI vision play a key role in this process. They open the door to closer integration of AI into the full UI development cycle. Well, I hope our discussion today has helped listeners better understand both the problem of blind coding and the interesting solutions like Playwright MCP that are starting to emerge. And, of course, it has provided food for thought about the future of development. Yes, the topic really makes you think about many things. How we will work tomorrow, what tools to master, how the very essence of professions related to creating digital products will change. There are still more questions than answers. And that's normal for such a rapidly developing field. We will follow the news, new tools, and approaches. Thank you for the interesting conversation. Thank you. Our deep dive today comes to an end. Until our next immersion into the world of technology. M.