Transcription
It’s getting quieter, the reverse of the song, yes, the stands are getting quieter. And here I am.
Let’s all settle down. I will officially start now. While I’m getting ready, my wife says, “Use it after you take a photo.”
Last year’s photo, by the way. It’s funny, people stood up and left.
Alright, hello everyone! My name is Artem. I am a coach, a pig coach, and I’ve been involved in transformation since 2017 across various businesses and companies. We use different methods. Back then, it was all about Jile, which was trending. Now, thanks to one of the companies I worked for, I’ve actively immersed myself in data. Currently, the company Solar is leading in production transformation, and I have a hobby that I will talk about today more than anything else: artificial management.
I call it management through artificial intelligence. How do we use data? How do we use large language models?
Let’s take a look at the audience. Who here has ever written a prompt to generate images using large models? It’s amazing, right? And who understands how it works?
We will discuss this because we will touch on it a lot. It seems to us that large language models are some kind of magic. We put something in, and something comes out. Believe me, the authors think so too.
In fact, a large language model takes something as input, processes it, and then produces something that no one can predict. By the way, even the largest language model. We will talk a bit about GPT agents and how we at Solar use these agents to transform production.
I already asked how they work. Now, the version at the top is mine, and the version at the bottom is the language model itself. I communicated here with ChatGPT and then created a presentation. You can read a lot of text there; the essence is that a large language model is essentially a predictive machine.
What does it do? We are talking about text generation specifically. It has been trained on about 3 billion parameters. Imagine a huge linear function with 3 billion parameters, and it uses these parameters on trained data to predict the next symbol in English encoding.
In our Russian language, it’s a bit more complex because the language is encoded differently. But the task is essentially to predict the next token. Then the wonders begin because initially, this was designed to work with text, to generate text and search for information.
Unexpectedly, for example, translations were not initially included in large language models. This was not their main task; it became a consequence because the data clusters loaded into the first models were in different languages, and the model learned to understand and predict the symbol you input.
As for programming, we actively discussed this during the break with Alexey. He mentioned that there are networks that can program. This was also not the main task; it happened as a side effect.
When it first happened, no one understood why it worked. Then they started to figure out what was inside this black box. What interests us is data analysis and the tasks of managers.
What tasks can managers solve here? Project management, decision-making, improving customer experience, strategy, and managing the backlog, especially if you are involved in products.
But remember, it’s very difficult to raise a high-quality language model on some system unit or laptop. You need a computing cluster. When you use computing clusters from well-known vendors, if you don’t have a closed loop in your company, please do not upload personal data or data with commercial secrets.
Why? Because we don’t know all the data paths. Have you heard about the intelligence announcement in June of this year? Elon Musk said that iPhones will no longer be at Tesla factories because it serves as a layer between Tesla, Apple, and OpenAI, which provides this software product.
I started the morning by asking, “What are large language models now? Are they products or just a piece of products we are already familiar with?”
Let’s try to answer these questions. How do we start working with a large language model?
First, you need to determine the tasks you want to solve. Do you want to generate text, create some instructions? Then, determine how much resources you are willing to spend—time for training, money for interacting with the models. Most good-level models are paid.
Every token, every generation costs money. You need to learn how to compose prompts. By the way, in 2020, when GPT-3 appeared, everyone rushed to generate all sorts of things, and everyone said, “This is nonsense; these neural networks are not very smart.”
But the problem was in the prompt. You need to learn to communicate with the network in a language it understands. This is a new skill for managers: setting tasks for your digital agent, your digital employee. You must provide context.
I will give examples. You must tell what you want to get as an output. If you don’t set these boundaries, you will get complete nonsense, believe me.
As soon as you prepare and learn something, there are open and paid courses on prompt engineering. It all depends on your level of imagination.
First, we look at what model it is. Is it open? Does it require access? If you choose ChatGPT, you need a VPN now. They are playing a bit of politics. For some tasks, I will talk about them today, a subscription is needed, which will be more expensive.
Prompting and results are worth it, especially in the early stages.
Let’s talk specifically. Now, ideally, you should see it on the well-known tablet. This will be my request. I’ve displayed my prompt in large so we can see how it works.
The model’s response is displayed first, just by copying it. Why does it write to us? Yes, you can insert it into your documents. But then there will just be screenshots.
So, the task is: we want to work with the log. We need to formulate this as a prompt. We give context: “Imagine, dear model, that you are the product owner. The product is called digital reporting.”
This is a real case, how we and the guys here created the first dashboards. The product is an interactive dashboard. I set the context that must be taken into account. I want to know the throughput, and I only want methods from here. Please don’t give me any other sources.
And then I ask who will use it. Look at the model’s response. First, it gives some small MVP dashboard steps. What needs to be done, what should result.
Then the prompt sounds like this: “Let’s present this in the format of an epic and user story.” I want a user story, please. It gives us an epic and the sequence of stories that should be implemented, along with filters, etc.
But the most interesting part for us is whether it is feasible or not. Then we take our product team, start communicating with them, and begin to change things.
But the model initially sets priorities. Look, high priority—these are Epics 1, 2, 3. Only Epic 1 fits in the screenshot, and so on. The development team can gradually implement the necessary product features.
What did we get?
Now, about GPT agents. It’s important because what was on the previous slide is working with a regular language model. There are no specific skills required here, except that ChatGPT works more or less the same way.
Now you will understand why. ChatGPT in examples has no special structure.
Next, we will talk about a special structure called GPT agents. The topic of GPT agents appeared about a year ago and is actively developing now in leading Russian and foreign companies.
What’s the essence? You now train the model not on random data, but on specific cases. For example, image generation only for images, or data analysis only for data analysis.
You input Python code, data processing algorithms. Each of us can register on ChatGPT and set up our agent. We did this in our personal account. You can customize GPT agents, but OpenAI has already rolled out about 20 agents that are paid.
Once again, an agent is a specially trained neural algorithm that responds to your specific requests.
What requests can there be? About setup, training, and interestingly, there is decision-making analysis.
What requests can there be? The agent analyzes information from the given context.
Next, what requests can there be? Important context: Power BI. Who makes dashboards on some system? Who uses Power BI?
There are guys, you know, right? Microsoft created their own language, then integrated it into Excel.
I also have an agent. When I ask it, “I need to build a diagram, give me an algorithm, please,” it says, “Look, you need to collect data first.”
Then it says, “I don’t know how to do that. Collect the data for me.”
Next, we create a calendar table. The biggest plus is that I didn’t have anyone in the company who worked well with Power BI. I could use these functions myself.
At some level, I know the basics. But here it tells us how to do it.
What’s interesting about this agent? In principle, we can get results like the first screenshot. That screenshot with beautiful graphical bars was my first result after about 4 hours of communication with the model.
Remember, I practically didn’t know how to work with Power BI. I knew some basics. But here, I collected data in Excel.
Thank you! I managed to export it. Then we already had an analyst, and we handed him the template. Just yesterday, I had a meeting with him, and in the evening, Danya brought a ready dashboard that can be used.
Look, those who have worked with it understand how much time it used to take to build a working dashboard, roll it out on the server, and experiment.
Let’s say, in two weeks, we got a working dashboard that we will present on Monday.
Case number two: how to work with data. Here, there are nuances. You need a GPT+ subscription because the data analyst is a paid agent.
The total cost of this pleasure is about 4 rubles a month.
The interesting thing is that the agent takes on the task of working with files. You upload Excel or some other tabular format. I use Excel; you can also use PDF, but it works worse with PDF.
You describe to it what the data is. Notice that I didn’t describe it in detail; it was very brief.
Then I say, “Look, can you please describe everything you see?”
It starts analyzing the release values, and it says, “Look, the release accuracy is this, and there are negative percentages.”
Then there’s a back-and-forth conversation about what it meant. It says, “This means you performed better than planned,” and so on.
The data analyst can build graphs right here. You can copy them.
What’s interesting is that it can create presentations in PowerPoint. You can say, “Dear GPT, export to PowerPoint,” and it will give you a presentation that you can work with.
For those who have data analysts who will use this, it can export Jupyter Notebook files for summarizing releases.
The data we collected manually didn’t differ much from what we gathered with the agent. But when we collected it manually, it took us a day and a half to two days. Here, it took just an hour of communication with the agent.
Now, how does the team look? This is a humorous slide, but in reality, all the guys have the opportunity to work with agents.
We have implemented a gateway in the company where there is a closed loop, and you can safely upload commercial data. It definitely won’t leave the company.
The team structure: where there’s a person, that’s a direct employee; where there’s a robot, that’s our agent.
For example, we found agents for Agile. What do we do? If there are any questions, we can send them to the team.
If we conduct training, we quickly gather information for that training. We don’t write slides ourselves; we upload the main material and questions to the agent, copy and edit what we need for our context, and quickly produce it.
So, preparation for training has been reduced by almost four times.
There’s also a consultant agent. What we struggle with is that we don’t have enough administrators, and sometimes generating complex GraphQL queries can be challenging.
We found an agent for consulting as well.
The results we achieved with the agents are significant. By the end of the year, we will have a backlog of product features, user stories, and knowledge bases.
I’ve already told you about this. It’s a closed opportunity to communicate with the network.
What are the prospects for all this? Not all companies want to implement it; not all companies are excited about it.
But we understand that now there are various software products. There’s one that is not very respected in Russia, but unfortunately, we cannot ignore it as a breakthrough.
What the company did by uploading its beta version of three digital agents—a marketing specialist, a product owner—is very cool.
You gathered some stickers with the team, did a retro, highlighted what you think, and how the team performed.
We conducted experiments, created an Impact Map, and asked the agent, the product owner, to highlight everything and create user stories.
It turned out to be roughly what we did manually, plus or minus what exists.
There’s a great moment: open codes in networks that you can copy give you the opportunity to customize your agent.
We uploaded the knowledge base of SAFe. We are now actively implementing SAFe as one of the frameworks for scaling.
The external training team left us a database, and we uploaded it to the agent. Now the SAFe agent is based on the training materials they provided.
The next point is code refactoring. For example, Sberbank is currently experimenting with giving some pieces of legacy code to their agents, specifically targeting that programming language, and they rewrite pieces of code.
This is still experimental, but I’ve seen a team of four agents write a game from scratch in three hours.
The product owner sets the task, and the person working with the agent says, “Hello, you are the product owner of this game. It should be like this, with these screenshots.”
Okay, production happens, and in three hours, a playable game is ready. You just export it, install it, and you can play.
There are also full-fledged digital employees. There are experiments where you can connect digital employees to your conference. They start onboarding new employees.
You can constantly reach out to the agent 24/7. It answers questions about the company, call center sales, even assists managers.
Top managers take such agents to conduct surveys, respond to emails, and make calls during conferences.
Agents are probably the most developing technology in AI right now.
Let’s dedicate five minutes to answer your questions. If anyone is interested in everything I’ve talked about, please subscribe to my Telegram channel. I often publish interesting things there.
These are my contacts. I hope you can see them. While you take photos, I will answer online because I’m not just a speaker today; I’m also reading and leading this track.
So, online silence. Let’s check.
Yes, I said, “Hello.” Thank you very much for the report; it’s very interesting.
Tell me, what is the biggest problem right now? If there’s a lack of knowledge, the neural network starts hallucinating. Why does AI management stall?
If we look at the idea, it’s not new; it dates back to 1984 with artificial intelligence. It needs to be tied to management, but who knew there would be hallucinations?
If someone doesn’t know what hallucinations are, a neural network has the property of filling in context for you. For example, if you go to ChatGPT or another bot and say, “Please write me a recipe for pig wings,” there will be no doubts.
It will say something like, “There are no wings on a pig.” It doesn’t care.
So, this is where we place a human over the agent to check for adequacy; otherwise, it doesn’t work.
Maybe there’s another way, but we approach it this way.
I see a question. Thank you!
It’s a great board with many ideas on where to apply it. But one thing really bothers me: when I say, “Please make me this,” the product will always be cool, and a lot of work can be avoided.
But how do we get a product that probably already exists and is clear to whom it’s needed at the moment?
First, can we get good, beautiful requirements that no one needs?
Let’s start from the end because this problem concerns all models, not just ChatGPT.
We often see stories where people think, “The neural network will replace me.” No, it won’t replace you.
For a neural network to replace you, it needs to learn to create new content, not just extract new content from training.
Now, the problem with neural networks is that they need a base for training.
The most important thing is that people should start using it as a tool, not as a replacement for themselves.
Cool artists who know how to use neural drawing use it as an additional tool. The same should be done by product owners.
We get a standard product. We get a standard ChatGPT, but nothing prevents the product owner from taking some basic things that are needed and adapting them to their context.
That’s the most important thing I want to say.
And by no means should you perceive this as a reason not to hire people.
We had a situation where we simply didn’t have money, and there was little faith in the company that this would take off. We did what we could.
We hired people for some initial basic steps to create an MVP. We created something like this, but now, of course, to develop it further, we need people or more advanced models.
Last question. Thank you!
Yes, thank you!
Artem, tell me, what was the biggest difficulty you faced when training the agent?
There were many difficulties, actually.
Let’s say the biggest challenge in training is that the company training the agent and the company using it have different contexts.
That’s why what I’m saying is that what you generate with the model must go through people so they can adapt the generation to your content, primarily to your context, so it doesn’t diverge from the logic you’re telling.
Thank you, friends!
To the last two questioners, I would like to give these pop sockets. Please come to me.
We have a technical break now, and we will start the last report in this track in just two minutes.