Transcription
Hello. In this video, a brief summary of a Stanford study where they analyzed the work of over 100,000 developers. We raise the main question: does artificial intelligence really increase productivity? The results are surprising. >> In January of this year, Mark Zuckerberg stated that by the end of the year, all mid-level engineers in Meta would be replaced by artificial intelligence. I think Mark was a bit optimistic. And probably, he acted like a good CEO to inspire employees and maintain Facebook's stock price. But Mark also created many problems for CTOs around the world. Why? Because after Mark said that, almost all CEOs in the world turned to their CTOs and said, "Marcus says he's going to replace all his developers with artificial intelligence, where are we now?" And the answer was probably, honestly, not very far. And we are not sure we can do that. So I personally hope that this will not change. But I don't think artificial intelligence will completely replace developers. At least not this year, let alone beyond that. Right. And I think artificial intelligence increases developer productivity, but there are cases where it decreases it. So artificial intelligence, or using artificial intelligence for coding, is not a one-size-fits-all solution. And there are cases where it should not be used. So for the last 3 years, we have been conducting one of the largest studies on software development productivity at Stanford. And we are doing it in terms of time leads and cross-sectional data. This means that even if a participant joins us in 25 years, we will have access to their git history, which means we can see data trends over time. We can see COVID, artificial intelligence, all these trends and events, and also in cross-section, because we have over 600 companies, participants large, medium, and startups. This means that now in our dataset, we have over 100 software developers, tens of millions of commits, and billions of lines of code. And most importantly, a large portion of this data is private repositories. This is important because if you use a public repository to measure someone's productivity, that public repository is not self-contained. Someone might be working on that repository on weekends or from time to time, right? Whereas if you have a private repository, it is much more self-contained and it is much easier to measure the productivity of a team, a company, or an organization. So at the end of last year, there was a big debate around ghost engineers. This was from the same research group, our research group. And Elon Musk was kind enough to retweet us. But we found that about 10% of software developers in our dataset, about 50 at the time, were so-called ghost engineers. These people get paid but do virtually nothing. For some, this was very unexpected, for others, it was not. For example, in our research group, there is Simon from industry. He was a CTO at a Unicorn company, which he left. He had a team of about 700 developers. And as a CTO, he was always the last to know about problems in his developer team. Right? And he thought, how can I change this? I have been working at Stanford since 2022, and I am doing what I call data-driven decision-making in software development. And in my previous life, I was doing digital transformation in a large company with thousands of engineers. In our team, we also have Professor Kachinsky from Stanford. His research is on human behavior in digital environments. And if you remember, he was the whistleblower for Cambridge Analytica. Today, we will talk about three things. We will start with the limitations of existing research aimed at quantifying the impact of artificial intelligence on developer productivity. We will show our methodology, and finally, we will dedicate most of the time to the results. How artificial intelligence affects developer productivity and how we can interpret these results to make them more meaningful. There is a lot of research on this topic, but many of them are conducted by vendors who are themselves trying to sell you their AI coding tools. Right? There is a conflict of interest sometimes. I see three main limitations. Many studies focus on commits, pull requests, and tasks. We have done more commits, more pull requests, the time between commits has decreased. The problem is that the size of the tasks is different, so doing more commits does not necessarily mean increased productivity. In fact, we have often found that with artificial intelligence, you are adding new tasks, which are bug fixes for what artificial intelligence has just coded. In this case, you are just treading water, right? It's quite funny. Second, there are many studies that say, "We took a group of developers, split them into two groups, and gave one of them AI, and the other not." Usually, these are tasks where they are asked to create something from scratch without context. And of course, artificial intelligence beats people without it, but that's because artificial intelligence is really good at generating boilerplate code. Right? But in reality, most software development is not boilerplate and does not always start from scratch. Right? Usually, there is an existing codebase. Usually, there are dependencies. So these studies may also be applicable to such situations. And then we have surveys, which we found to be ineffective for predicting productivity. We did a small experiment with 40 developers, where we asked each developer to rate themselves relative to the average or median in five percentiles from zero to 100. Then we compared it to their measured productivity. We will explain what that means later, but we found that asking someone how productive they are is almost as effective as flipping a coin. The correlation is very weak, right? We found that people misjudge their productivity by about 30%. Only one in three people rate their productivity within their quartile. I think surveys are great. They are useful for identifying morale and other issues that cannot be identified by metrics. But surveys should not be used to measure developer productivity, let alone to measure the impact of artificial intelligence on developer productivity. You can measure it to see how happy they are with using artificial intelligence or anything else, I suppose. Great. Now let's move on to our methodology. In an ideal world, you would have an engineer who writes code. And that code is evaluated by a panel of 10 or so experts who, separately, without knowing who is evaluating what, rate that code on quality, maintainability, output, how long it would take, how good it is. It's sort of a set of questions. Then you sum up the results, and we found two things. First, experts really agree with each other. It turns out that one expert engineer agrees with another expert engineer when they are discussing objective code in front of them. And second, and this is perhaps the most important, you can use this to predict reality. Reality is predicted by this group of experts. The problem is that it is very slow, it is not scalable, and it is expensive. So we created a model that essentially automates this and correlates well. It is fast, scalable, and affordable. It connects to git, and then the model analyzes the source code changes of each commit and rates them on several parameters. Since each commit has a unique author, a unique hash, and a unique timestamp, you can understand that team productivity is essentially the functionality of the code that they have created over a certain period of time. not the number of lines of code, not the number of commits, but what that code does. Right? Then you can display this on a dashboard and get something like this. Great. Now let's look at some of our results. In September, this company implemented artificial intelligence. This is a team of about 120 developers. And they were testing whether they wanted to use artificial intelligence in their daily work. Here are the bars, each representing the sum of the results obtained per month using our methodology, not the number of lines of code. We see that green represents added functionality, gray removed, blue refactoring, orange rework. Rework and refactoring change existing code, but rework changes more recent code, which means it is inefficient. Refactoring may or may not be efficient. From the beginning, you see that the implementation of artificial intelligence leads to an increase in rework. You feel like you are creating more code because you are writing more code, doing more commits, more changes, but not all of it is actually useful. To be clear, I think that, judging by this chart, overall productivity has increased by about 15-20%. But most of the increase is essentially rework, which is a bit misleading. If I could distill this into one chart with many divergences, it would look something like this. With artificial intelligence, you increase your productivity by about 30-40%. You create more code, however, you have to go back and fix some of the bugs that the code introduces, and deal with the mess that artificial intelligence creates, which in turn gives you an average productivity increase across industries, across sectors, of about 15-20%. There is a lot of new data here that we will see in a second. Here we have two charts showing the distribution of productivity gains when using artificial intelligence. On the Y-axis is the gain, it starts from -20%. Note that. Then it increases. Here are four datasets. Blue represents low complexity tasks, red high complexity tasks. and on the left. The chart on the left is a task from scratch. The chart on the right is a task with existing code. So, from the outset, we conclude that artificial intelligence performs better on simple tasks. This is good. This is proven by data. This is great. The second thing we see is that for low complexity tasks from scratch, the distribution is much more stretched out and on average much higher. Keep in mind that this is for corporate settings. This does not apply to personal projects or creating something from scratch for yourself. There, the improvements would be much greater. This is for real work conditions in a company. And the third thing we see is that if you look at high complexity tasks, they are on average lower than low complexity tasks in terms of distribution. But in some cases, they are more likely to decrease engineer productivity. This could be due to many reasons, but this is what we see in the data. Right? The main reasons are not entirely clear to us yet. If we translate this into a more understandable chart, then the bars and columns will be the average or median gain, and the line represents the interquartile range. The bottom of the line is the 25th percentile, and the top is approximately the 75th percentile. Here it is very clearly seen that we have more gains from low complexity tasks, less gains from high complexity tasks, and in tasks with existing code. It is more difficult to use artificial intelligence to increase productivity compared to tasks from scratch. So, if you want to show this to your executive team, this might be the chart, or here we have a matrix, very simplified. In reality, it's a bit more complex, but here on one axis is task complexity, low and high. On the other is project maturity, from scratch or with existing code. We see that for low complexity tasks from scratch, the gain from artificial intelligence is 30-40%. For high complexity tasks from scratch, the gain is a more modest 10-15%. For tasks with existing code and low complexity, 15-20%. And most importantly, for high complexity tasks with existing code, 0-10%. These are approximate recommendations based on data. And I forgot to mention that this chart represents data from 136 teams from 27 companies. It's quite representative, and it will continue. Or is this chart based on this data? Here we have a similar matrix, only at the bottom is the popularity of the language. For example, COBOL, Elixir, quite obscure languages, and at the top Python, Java, JavaScript, TypeScript. And we see that artificial intelligence does not help much even with low complexity tasks for less popular languages. It can help a little, but it's not very useful. And it turns out that people just don't use it, because if it only helps two out of five times, then you won't use it often. What's funny or interesting is that for less popular languages and complex tasks, artificial intelligence can even decrease productivity, because it writes code so poorly in COBOL, Haskell, or Elixir, that you just work slower. Right? Of course, this is not very common, but it can account for 5-10% of all development work. Right? Most development work probably falls into the highly popular part of the chart. And here the gain is 20% for low complexity tasks and 10-15% for high complexity tasks. Now let's move on to something more theoretical, less empirically proven, but more in line with what we see in the data. Right? This is an illustrative chart where the Y-axis is the productivity gain from artificial intelligence, and the X-axis is a logarithmic scale of code size from 1,000 lines to 10 million. And we see that as the code size increases, the gain from artificial intelligence sharply decreases. Right? And I think most codebases are now somewhere in this range, depending on your use case. Right? But they are more than a thousand lines of code. Unless you are a startup or something that emerged a couple of months ago. Right? And this is because there are three reasons for this. Context window limitation. In a second, we will see how productivity decreases even with an increase in the context window. Signal-to-noise ratio. This confuses the model, if you will. And of course, larger codebases have more dependencies and specific logic. And here, borrowing work from a paper called Nolima, which shows how coding tasks are handled on a scale of 0 to 100. You see that as the context length increases from one to 32k tokens, productivity decreases. And we see all these models here. For example, Gemini 1.5 Pro has a context window of 2 million tokens. And you might think, "Wow, I can just load my entire codebase into it, and it will do everything perfectly, right?" But we see that even at 32k tokens, productivity decreases from 90% to about 50%. Right? So what will happen when you go from 32 to 64 or 128 tokens? You will see very low productivity. In short, artificial intelligence does increase developer productivity. You should use it in most cases. But it does not always and not equally increase developer productivity. It depends on factors such as task complexity, code maturity, language popularity, code size, and context length. Thank you very much for your attention. If you want to learn more about our research, you can visit our research portal Software Engineering Activity at Stanford. You can also contact me by email or LinkedIn. I would be happy to discuss this topic at any time. Thank you very much.