Transcription
[Music]
in January of this year Mark Zuckerberg
said that he was going to replace all of
the mid-level engineers at Meta with AI
by the end of the year
i think Mark was a bit optimistic and he
was probably acting like a good CEO
would to inspire a vision and also
probably to keep the Facebook stock
price up but what Mark also did was
create a lot of trouble for CTOs
worldwide
why
because after Mark said that every
single CEO in the world almost turned to
their CTO and said "Hey Marcus says he's
going to replace all of his developers
with AI where are we in that journey?"
And the answer probably was honestly not
very far and we're not sure we're going
to do that and so I personally think
hopefully this is not um you know going
to going to change but I don't think AI
is going to replace developers entirely
at least at least this year let alone at
at at Meta right but um and I do think
that AI increases developer productivity
but there's also cases in which it
decreases developer productivity so AI
or using AI for coding is not a
one-sizefits-all solution and there are
cases in which it shouldn't be used
and so for the past three years we've uh
been running one of the larger uh
studies on software engineering
productivity at Stanford and we've done
this in a time series and
cross-sectional way so time series
meaning that even if a participant joins
in 2025 we get access to their git
history meaning we can see trends of
data across time we can see covid we can
see AI we can see all of these trends
and and things that happened and then
also cross-sectional because we have
more than 600 companies participating
enterprise midsized and also startups
and so this means that we have more than
100,000 software engineers in our data
set right now dozens of millions of
commits and billions of lines of code
and most importantly most of this data
is private repositories
this is important because if you use a
public repo to measure someone's
productivity that public repo is not
self-contained someone could be working
on that repo on the weekend or once in a
while right whereas if you have a
private repo it's much more
self-contained and much easier to
measure the productivity of a team of a
company of an organization
so late last year there was a huge um
controversial
uh thing around ghost engineers so this
came from kind of the s the same
research group our research group and
here uh Elon Musk was kind enough to
retweet us but what we found is that
roughly 10% of software engineers in our
data set at the time about 50,000 were
what we called ghost engineers
these people collect a paycheck but
basically do no work
so that was very surprising for some
people very unsurprising for others and
so some of the people in this research
team are for example Simon from industry
uh so he was CTO at a unicorn which he
exited and he had a team of about 700
developers and as CTO he was always the
last person to know when something was
up with his engineering team right and
so he thought okay how can I change this
myself I've been at Stanford since 22
and I focus on what I call datadriven
decision-making in software engineering
and in a past life I was looking after
digital transformation for a large
company with thousands of engineers
part of the team is also professor
Kasinski who was at Stanford and his
research focuses on human behavior in a
digital environment and basically he was
the Cambridge Analytica whistleblower
back in the day if you recall that
so today we're going to be talking about
three things
we're going to start off with the
limitations of existing studies that
seek to quantify the impact of AI on
developer productivity
we're going to showcase our methodology
and lastly we're going to spend most of
the time looking at some of the results
what is the impact on AI on deaf
productivity and how are ways we can
slice and dice these results to make
them more meaningful
and so there's lots of research being
done on this topic but a lot of it is
led by vendors who themselves are trying
to sell you their own AI coding tools
right and so there's a bit of a conflict
of interest there sometimes and the
biggest three limitations that I see is
that a lot of these studies revolve
around commits and PRs and tasks hey we
completed more commits more PRs the time
between commits decreased
the problem here is that task size
varies right and so delivering more
commits does not necessarily mean more
productivity
and in fact what we found very often is
that by using AI you're introducing new
tasks that are bug fixes to the stuff
that the AI just coded before so by that
case like you're kind of spinning your
wheels in place right so that's kind of
funny
secondly there's a bunch of studies who
say well we grabbed a bunch of
developers we split them into two groups
and we kind of gave one AI and one of
them we didn't and what usually happens
there is that these are kind of green
field tasks where they're asked to build
something with kind of zero context from
scratch and there of course AI decimates
uh the non-AI people but that's because
AI is just really good at green field
kind of boilerplate code right but
actually most of software engineering
isn't green field and isn't always
boiler boilerplate right and so there's
usually an existing codebase there's
usually dependencies so these studies
can't be like applied too too well to
these situations either
and then we also have surveys which we
found to be an ineffective predictor of
productivity by doing this small
experiment with 43 developers whereby we
ask every developer to evaluate
themselves relative to the global mean
or median in five percentile uh buckets
from 0 to 100 and then we compared that
to their measured productivity we'll get
into what that means later but what we
found is that asking someone how
productive they think they are is almost
as good as flipping a coin there's very
little correlation right and so we found
that people misjudged their productivity
by about 30 percentile points
only one in three people actually
estimated their productivity within
their quartile one quartile and I think
surveys are great they're valuable for
surfacing you know morale and other
issues that cannot be derived from
metrics but service shouldn't be used to
measure developer productivity much less
the impact of AI on developers
for productivity cases you can measure
it to kind of see how happy they are
using AI or whatever I suppose um
great so now let's dive into our
methodology so in an ideal world you
would have an engineer who writes code
and this code is evaluated by a panel of
10 or 15 experts who separately without
knowing what every person is uh
answering evaluates that code based on
quality maintainability output how long
would this take me how good is it right
kind of like a bucket of questions and
then um what happens is that you
aggregate those results and we found two
things the first one is that this panel
actually agrees with one another so it
turns out that one engineering expert
agrees with the other engineering expert
when they're talking about an objective
code in front of them and secondly and
probably most importantly is that you
can use this to predict reality and
reality is predicted by a panel like
this the problem then is that this is
very slow it's not scalable it's
expensive um and so what we did is we
built a model that essentially automates
this correlates pretty well it's fast
it's scalable and it's affordable
the way it works is it plugs into git
and then the model analyzes the source
code changes of every commit and
quantifies them based on a bunch of
these dimensions
and then since every commit has a unique
author a unique sha a unique time stamp
then you can kind of understand okay the
productivity of a team is basically the
functionality of the code they delivered
across time not the lines of code not
the whatever commits but the fun like
what that code is doing right and so
then you can kind of put this in a
dashboard and uh overlay it across time
and get something similar to this
great so now let's dive into some of our
results
so here in September is when this
company implemented AI this is a team of
about 120 developers and they were
piloting whether they wanted to use uh
you know AI in their kind of regular
workflow and we have here um these bars
and every bar is the sum total of the
output done in that month using our
methodology not lines of code and we can
see that in green it's added
functionality in gray it's removed
in blue is refactoring and in orange is
reworked and so rework versus
refactoring they both alter existing
code but rework alters code that's much
more recent meaning it's wasteful
refactoring could be wasteful could be
not wasteful
and so from the get-go you see that by
implementing AI you get a bunch more of
rework
what happens is that you feel like
you're delivering more code because
there's just like more volume of code
being written more commits more stuff
being pushed but not all of that is
actually useful to be clear I think
there I mean based on this chart and
overall there is a productivity boost of
about 15 to 20% but then a lot of the
gains you're seeing are uh basically
this kind of rework which is a bit you
know misleading
so if I could summarize it into one
chart with many discrepancies it would
be something like this so with AI coding
you generate or you increase your
productivity by roughly 30 40% right
you're delivering more code however you
got to go back and kind of fix some of
the bugs that code introduced and kind
of you know fix the the the mess that
the AI made which in turn gives you an
average productivity gain across all
industries all sectors everything of
roughly about 15 to 20%
there's a lot of new ones here which
we're going to see in just a second
so here we have two violin charts and
they plot the distributions of the gains
in productivity from using AI and so
kind of like the y-axis is the gains it
starts from minus 20% take note and then
it goes up and here we have kind of four
pieces of data being shown in blue is
low complexity tasks
and in red is high complexity tasks
and kind of like your left uh the chart
to the left is green field tasks the
chart to the right is brown field tasks
so right from the get-go the first
conclusion we have is that sure it seems
like AI performs better in coding with
simpler tasks that's good it's proven by
data that's awesome the second thing we
see is that hey it sounds like for low
complexity green field tasks there is a
much more elongated distribution and a
much higher distribution on average keep
in mind that this is for enterprise
settings this doesn't apply for kind of
like personal projects or vibe coding
something for yourself from scratch the
improvements there would be much bigger
this is kind of for like real world
working company settings
and the third thing we see is that if
you look at the high complexity tasks I
mean they're lower than the low
complexity ones in average in terms of
the distribution but also in some cases
they are more likely to decrease an
engineer's productivity
now this decrease could be for many
things many reasons but that's kind of
what we see in the data right the
underlying causes are still not super
clear to us
if we translate this to a chart like
this which is a bit more digestible you
have uh in in the bars and the columns
kind of like the average or the median
gain and then the line represents the
interquartile range so the bottom of the
line is the 25th percentile and the top
of the line is roughly 75th percentile
and so here it's very clear to see how
we have you know more gains from low
complexity tasks less gains from uh high
complexity tasks and then brownfield
it's harder to leverage AI um to make
increases in productivity there compared
to green field
so if there is maybe a slide that you
could show to your leadership team it
could be this one or it could also be
this one so here we have a matrix really
simplifying things you know reality is a
bit more difficult than this but here we
have kind of on one axis task complexity
low and high on the other one project
maturity green field versus brownfield
kind of if we see that hey low
complexity green field 30 to 40% gains
right from AI
high complexity but green field more
modest gains 10 to 15
brown field and low complexity pretty
good 15 to 20% and most importantly high
complexity brownfield tasks 0 to 10%
these are orientative guidelines based
on what you see what we see in the data
and I forgot to mention uh this slide
has a sample size of 136 teams across 27
companies so pretty representative
um and then that's going to derive or
this chart is derived from that data
then here we have a similar matrix
except at the bottom we have language
popularity
so in low we have examples such as Coobo
Haskell Elixir really kind of obscure
obscure stuff and high is things like
Python Java you know JavaScript
TypeScript
and what we see is that AI doesn't
really help even with low complexity uh
tasks for low popularity languages it
can help a bit but it's not terribly
useful and what ends up happening is
that people just don't use it because if
it's only helpful two times out of five
you're just not going to use it very
often
what's funny or interesting is that for
low language popularity and complex
tasks AI can actually decrease
productivity because it's so bad at
coding in Cobalt or Haskell or Elixir
that it just makes you slower right
granted this isn't very like this
happens but it may be five or 10% of the
kind of global development work if that
right
most of the development work is probably
somewhere in the langu in the high
language popularity kind of part of the
chart and here you have gains between
20% for the low complexity and 10 to 15%
for the high complexity
so now moving into something a bit more
theoretical less empirically proven but
more so kind of like what we're seeing
in in in the data right this is like an
illustrative chart which has kind of
productivity gain from AI on the y-axis
and a logarithmic scale of the codebase
size right from 10,000 lines of code to
10 million on the x-axis and we see that
as the codebase size increases the gains
you get from AI decrease sharply right
and I think most code bases nowadays are
kind of somewhere in the depending on on
your use case right but they're bigger
than a thousand lines of code unless you
are a YC startup or something that's
like kind of spun out a couple months
ago right and that's because you know
there's three three reasons for this
really context window limitations we're
going to see in a second how performance
decreases even with larger context
windows the signal to noise ratio is
kind of confuses the the model if if you
will and then of course larger code
bases have more dependencies and more
domain specific logic present
and so then borrowing work from this
paper called no lima which shows you on
a scale of 0 to 100 how LLMs perform on
coding tasks you see that as context
length increases from 1,00 to 32,000
tokens
performance decreases and so we see all
these models here for example Gemini 1.5
Pro has a context window of 2 million
tokens and you might think whoa I can
just throw my entire codebase into it
and it's going to retrieve and and code
perfectly right and what we see is that
even at 32,000 tokens it's already
showing a decrease in performance from
90% to about 50% right so what's going
to happen when you move from 32 to 64 or
128 right you're going to see really
really poor performance here
and so in short AI does increase
developer productivity you should use AI
for most cases but it doesn't increase
the productivity of developers all the
time and equally it depends on things
like task complexity codebase maturity
language popularity codebase size and
also context length
thank you so much for listening if you'd
like to learn more about our research
you can access our research portal which
is software
engineeringproductivity.stanford.edu
you can also reach me uh by email or
LinkedIn super happy to talk about this
topic at any time thank you so much
[Music]