Transcription
You are many things, and I can't explain it properly, so you should introduce yourself. But yeah, your microphone's right here. Y press hard for the next slide, backwards, and a pointer if you need it. Great. Thanks. Thank you so much.
Hi everyone. Um, I'm Peter. I'm going to kind of take things in a bit more of a philosophical kind of direction. Um, you know, I work for a research lab these days, so that's kind of my uh tendency. Um, the lab is called Inc and Switch. Uh, we are interested in carrying on kind of the work of like Engelbart and Licklider and Alan Kay and these people. We're interested in how computers can augment human intelligence, and uh, you know, that's very high-futin; it's a lot of fun. We get to dig into all kinds of weird corners of software and experimental interfaces and so on.
But before that, I have written a lot of production software, so I'm not just speaking to you from kind of an ivory tower perspective. I was one of the early employees at a platform as a service company uh called Heroku. I've run a lot of Postrest databases. I've worked in game development as well. I've built titles for DS and GBA. I worked on this one Teenage Zombies Invasion of the alien brain thingies for a Victoria BC game company, and I've worked on desktop software like uh Songbird, which was a cross-platform media player that lost fairly decisively. Um, and besides that, I've also uh worked in uh research, um, spending some time uh on, for example, this ship here, this for Wilford Laurier. It's a Canadian Arctic uh research icebreaker, so I spent some time in the Arctic, and that actually helps inform my perspective on things too, but we'll come back to that.
So I want to start by kind of saying, hold on a sec, before we get into kind of talking about why we can't make simple software, we should probably define our terms a little bit and talk about what complexity actually is. So the first thing I want to clarify is that complexity is not difficulty. Right? Just because something is hard doesn't mean it's complicated. Like the ideas of calculus might be very difficult to internalize, but they're actually very simple and elegant. And I also want to draw a distinction between something being complex or just being kind of big. A box of Lego is not necessarily a complicated thing, though you can make a lot of things with it. Complexity occurs when your systems interact with each other, okay, and when it takes a lot to uh get things done because of that. And more specifically, I think the problem with complexity is that your systems can become unreasonable. And when I say unreasonable, I don't mean in kind of like a sort of colloquial sense; what I mean is that you literally cannot reason about them; you don't have the ability to predict what's going to happen. And a lot of the time that's a problem because you got work to do, and the thing is crashing, and you don't know why. But a complex system can also be generative and surprising if you harness that complexity in positive ways.
Last, before we get into it, I want to make just a very small uh detail point for the uh more pedantic in the audience. Some people draw a uh very specific difference between uh complex systems from sort of chaos math and like emerging complexity. We'll talk a little bit about that, as distinguished from complicated systems where you have a lot of just kind of mess. I'm kind of going to float back and forth across both of these without a lot of distinction because I think in this context either one can cause similar effects. Okay.
So we're going to begin with a non-comprehensive look at complexity, and more specifically, we're going to kind of like pick some, cherry-pick some examples sort of from fake industrial sort of scenarios where you can begin with something simple and end up with something complicated. So I'm going to start with like the most basic of all reasons why your software gets complicated, and that's, you know, the first version of your code. You just write the happy path; everything's glorious, and and software can be really simple when it lives in an idealized world, and everything is fine. And so you know, here's like a really simple imaginary, you know, pseudo-code web thing. Great. We got a GET request to our application; you know, we're going to go fetch something from the database and say hello there. This is a Hello World app of web apps, server apps. Great. It works fine. You know, if user 12 is me and user 10 is my friend KO, and user fu is… wait, what? Ah, right, we got to validate the input. Okay, so we'll validate the request now. Okay, so we're validating the request, and and that's good. And oh geez, I'm not really handling it if we don't validate the request. And wait, do I have to do this on every request? I guess I have to do this on every request. And am I doing it the same way on every request? In fact, this exact pattern of validating user input and checking your arguments can lead to something that uh Bratus and Patterson call shotgun parsing. And shotgun parsing is kind of the idea that, you know, you're parsing your input, but it's like someone's taken a shotgun to your code base because it's just blown through the whole code base, and it can lead to very serious security vulnerabilities because you might carefully check an input in one place but not in another. A great talk from BRWN 2012 about this.
So of course, you know, now we're going to be a little more careful, so we're going to also, you know, if the database is down, we don't want to send a 500; we're going to fail elegantly. And as you go, the more of these cases that you have to handle and the better the quality of your code is, right, the more conservative, defensive, thorough that you are, like at this point now like the the actual logic of this method is beginning to disappear into all the edge case handling. Right? We're we're well off the happy path now. And so you know, you can clean this up a little bit. So for imagine here in this case that, you know, we have a slightly uh better set of libraries, more thoroughly written set of libraries. We've moved the complexity down out of this method, and we're sort of saying like, well, the database won't throw an exception, right, because exceptions are ugly and kind of subtle to handle. Well, it'll just it'll return null if it can't do the job for whatever reason. And we're going to say, well, we'll, you know, there's like a good type system that's handling parsing the input args before they get to us, so you know, we can we don't have to worry about that anymore. And the value of well-written uh libraries and languages, type systems and so on is that they can really simplify your code, make it less complex by preventing you from having to carry around all of this mental state everywhere that you go. And you know, kind of my observation of this world is that like vigilance is not a strategy. Like we talk earlier about how, you know, if a system fails, it's not oh, the blameless postmortem; it's not that the the developer failed; it's that the system failed. You know, if you have security vulnerabilities in your code because you didn't handle input, the answer isn't to berate yourself for not handling the input; you know, it's to step back, to level up and say like, well, hold on, why do I have to do this manually every time? So so to strategically, you want to minimize the scope of failures and and kind of design them out of your system. And one way to do that is have a type system so you don't, you know, forget to check your nulls. There are other approaches; I don't think this is universally true, but it's one way.
Okay, so let's talk about scale. You know, we all know as an array gets bigger, you can have an O(n log n) kind of sort algorithm. Great. Okay. Well, let's, you know, scale's easy, right? So let's just imagine an admin panel for a web application. You list your users; looks kind of like this: go to your database, select all the users, and then in your front end just show them all. Perfect. Easy. It's fine. You've only got 10 users; who cares? Okay. Well, now we're at 10 to the four. Okay, you've got a thousand users or so. Okay. Well, okay, we can't just render all that on one screen. Okay. Well, uh, we're going to use one of those offset SQL things, right? We know about this, and then we're going to need pagination, I guess. And you got to click through. Okay, yeah, sure, we can still do this; this is fine. Yeah, you don't want to request all that; it's pain to scroll through. Okay. Well, now we're 10 to the six users; we got a million. Actually, that offset thing, that's a problem if you have a lot of users because in order to generate, you know, page 8850, it actually selects all of those rows and scrolls through them and just takes all of those results and just chucks them on the floor until it gets the ones that you asked about. And actually now, even though nothing has changed about our problem statement, like not only are the performance characteristics of the system starting to change and the user interface is starting to change, but the the way that we interact with the problem is changing too, because if you've got a million users, the way to find a user is not to just like keep clicking next until you find them. One page after oh, page 350. Okay, I'm on the J's. Okay, click forward to change the query, AR to force you… We've all done these kinds of things when the tools didn't keep up with our admin systems. But like the point I'm trying to make here is that, you know, once you start to reach like a 100 million users, you know, it's not even the same kind of problem anymore, right? The complexity that snuck in here is now that we have different responsibilities; there are ethical, there are legal, there are policy responsibilities. You probably have a team of people at your organization who are responsible for dealing with abuse; some of these people are terrible, and you want them off your platform, and you can't find them, right? Like it's a different thing, and it's complicated because the environment has changed as much as anything else. So everything changes when your scale changes; it's not just like an algorithmic thing; it's also the kinds of things that you're doing change. And like in general, my advice to someone who's built a lot of high-scale production systems through many orders of magnitude is that it's as harmful to build for the system of the future as it is to build an insufficient system for the present. You know, if you build the tools for 100 million users when you have a hundred users, they're going to be completely unusable too, and you're going to waste so much time trying to solve problems you don't have yet that you're going to miss out on all the benefits of being able to just look at all your users and see who they are and talk to them, right? So you need to be scale-appropriate and kind of be looking ahead.
Another way software can get complicated is with leaky abstractions. And so here we have a platform that's a little bit skewed, um, and so complexity can kind of bubble up from under the water line through these imperfect abstractions. This is how you copy a string; it's a nice handmade, low-level… This is an excerpt from Kernighan and Ritchie 1988, second edition, The Programming Language. And this book was such a revelation for me, like how it showed me the way kernel uh functions were really implemented… No, I'm just kidding; they're not really implemented like this. And they say here that this is how, you know, a C programmer would prefer to write string copy. The C programmer in question has never forgotten to null-terminate a string, I see. Um, but this is how string copy is actually implemented as of 2022 for Alpha in uh in Linux. And actually, this isn't how string copy is implemented; this is like one page of assembler; there's like 300 lines of assembly. And really importantly and and intriguingly, although the interface remains the same, modern CPU architectures have so many more constraints around memory alignment and everything else that although the interface is preserved, you can pass any pointers in and get values back, the kind of performance characteristics of like doing an unaligned versus an aligned string copy will be dramatically different. And so complexity here, right, like I think we can kind of call this a win, right? Like we've got this superpowered computer under the hood now, but it still kind of feels like that old galap they were running in 1988, you know. And like that's cool that that it works, but in a sense, this API that's so simple is actually undermining the value of the system, and there's a lot of magic happening to kind of hide this. And so I feel very mixed about this because on the one hand, you know, it's sort of like SQL databases where like you can add an index and change nothing else and things get faster, but there's like a really important subtle like lie that's being told here. But maybe this is good, and maybe it's bad; this is like a value judgment kind of question. And whether you understand this is happening could be completely irrelevant, or it could be the difference between the success and failure of a project.
Okay, so these are kind of like technical complexities that come in. I want to talk about other sources of complexity, different kinds of complexity. And one really big source of complexity in my projects—maybe you're smarter than me, and that's cool—uh, but it's when you have a gap between the problem you think you have or the problem you used to have and the problem you have right now. And again, we're going to use a toy example to illustrate this. So you've got a users table, just same app as before, and you know your users; there's a bunch of fields; you got a first name and a last name. And how do you get the first name and the last name? It's easy, guys; you just split on a space. As a van Hardenberg… You can see how this might get you in trouble. And in fact, on my California driver's license, it said that my middle name was van for many years. And the problem here is that the model that you have for the problem doesn't actually map the problem domain successfully. So what do you what do you do, right? And there are lots of things you can do; you can rewrite from scratch; you can patch around it. And actually, just as an aside for everybody in the room, the W3C has a wonderful um essay about personal names around the world, and just, you know, if you take one like little thing from this talk, um, don't use first name and last name; don't use family name. People's family names aren't accurate; some people have them and others don't. Uh, what you do is you have one field that is, you know, what is the full name, and then the other one is, if you need it, what should we call you? Um, it's like a much better model. Um, also Unicode, I know that's its own set of complexity, but uh… And so you know, there's only so many things you can do when you have this problem, right? And like I had a really specific concrete example of this in a distributed system recently where like I had a mental model of how the system behaved, and my tests demonstrated that it behaved that way, and then in my production environment it behaved completely differently because of behaviors that were not modeled by my understanding of the problem. And this is part of why distributed systems are so prone to this, is because there's so many new free variables and and like articulation points where things can be fast or slow or where they can fail or, you know, arrive out of order, and all these kinds of things. And so when you have these kinds of model-reality gaps, you have to bridge them somehow. And so what can you do? Well, you can fix the problem; you can improve your understanding and rewrite everything. And like when you can do that, it's probably best; you can't really always do that, though. And more to the point, you may still not actually—you could see the problem, but you may still not actually understand the solution, right? And so what can you do? Well, you can hack around it, right? Like I've put van Hardenberg into a lot of text boxes without a space in it, and you know, where the first letter is capitalized and the H is lowercase, and that's not how I spell my name, but it is how a lot of systems spell my name. You know, or you can ignore the problem, right? Like maybe it's fine. And like genuinely, it could be fine for your use case; you're just like, well, I guess, you know, that Peter guy can suck it; he'll deal with having his name spelled wrong. And I do. You know, it's these are options that you have. And there's like a great, you know, like meme series out there on the internet: Lies Programmers Believe About Pretty Much Anything… like all time zones are 1 hour apart, uh, half an hour later in New Zealand. And so I think this is a this is a very fundamental source of complexity, but where things get really bad is when your problems start to multiply off each other, and this of course is, you know, where you have kind of like compound interest on complexity. So what happens if your problems um dimensionalize against each other? So again, imagine this web application; you've got a bunch of different browsers to support; you've got a bunch of different runtime environments to support; different screen sizes, network speeds, different OS or browser versions. And so if you want to actually understand what's happening, you know, all of these things multiply against each other, right? And so you don't just have one runtime environment; you might have one codebase, but you have lots of different contexts. So how can you know that you actually have correctly functioning code? Because you know the old joke about Docker: like, well, it worked on my computer… Well, I guess we'll ship your computer… Well, even that doesn't really work because once you get out into the real world, there's different memory, you know, contexts or you're up against different loads or whatever else, right? You don't actually control the whole environment. And so this is what really starts to kill you with complexity, right? And it's where all of those smaller inconsistencies play off each other to create an unknowable. Like you don't use the native APIs because God help you if you're trying to like figure out how to map all of these totally different environments down to one consistent thing. And like one solution here is just to only have one thing as best you can and to minimize the difference between those environments. And that's why you see sort of like ostensibly lazy Electron apps from big companies because even worse than having to support all these different things is having to coordinate all the different people and teams to try and build features, right? And we've all, many of us have been there, right? Like if you have an iOS and an Android team, like how do you get them to ship the same feature in the same quarter? Right? It's tough; it's tough.
Okay, so I hope I've convinced you now um that complexity is like a a complex problem itself and that it manifests in a lot of different ways. So we're going to change gears a little bit and talk about seat belts, and more specifically, why seat belts don't save lives. And I haven't asked risk here because this is a little bit of a controversial um research topic, um, but the sort of upshot of this is that in at least some studies they found that after seat belts became mandatory in cars, uh people still died on the roads at similar rates. And that was really surprising because like, you know, seat belts are are a good idea; like they keep you safer in the car. And so somebody proposed this idea uh called risk homeostasis. The idea behind risk homeostasis is that actually we die at a rate on the road related to how much risk we're willing to take. And so if you now have a seat belt and you're driving a car that's safer, you're going to go a little faster, and you're going to take the corners a little tighter because you feel better, right? So the the issue with making people safer was that they then took on more risky behavior, conserving their total risk tolerance. And so I think we see a similar thing in software, right? We—I call this complexity homeostasis, which is to say that if you have kind of a system that's evolving over time, you know, everything's fine; we're going along; life is good. And then we add some more things, and you know…
We're still happy, and then oh, oh yeah. Now it's, I don't like it anymore. It's not good; it feels bad. Now it's time for that rewrite that we were hearing about, right? It's time to bring things back. Ah, okay. Now we can go back to making things complicated again.
And so this is kind of like a set point. So homeostasis is the process where, or it's any process where you sort of maintain a set point, and it's commonly used to talk also about how our bodies regulate temperature. But I think our organizations regulate complexity, right? And so all of us have different intuitions, and when people talk about wanting things to be simple, there's like an aesthetic preference here, and different people perceive that and pursue that differently. And like Deen's, you know, quest for just the right number of op codes that we heard about yesterday is a great example where, like in a certain sense, you know, the correct number of op codes is one, and you could do that if you really wanted to, but then you've got kind of like the CIS, you know, model where actually the correct number is hundreds because you're maintaining backwards compatibility, and anything that can squeeze you, you know, either better numbers out of a benchmark or more CPU sales is acceptable.
And so, you know, how you perceive what constitutes complexity or how much complexity you're willing to tolerate is an individual or an organizational decision. And you know, some things can actually move that up. Like I have abandoned projects because they got complicated and annoying to work on, but like some people might just tolerate that and not perceive it. I've worked with some brilliant people who write the most complicated, insane, convoluted code from my perspective, but when I sort of go to them, I'm like, "This is so complicated and weird, why did you do it this way?" They don't see it because they're just much smarter than me; they're able to hold that complexity in their head comfortably, and so for them it doesn't feel that way. Another way a system can become more complicated is if it's worth a lot of money, right? Because like you can just hire another poor schmo to sling Java into the code base. I mean, has anybody here worked on an EA Sports title? Yeah. I'm so sorry. I have heard some things, man. And it ships every year, on deadline, without fail, right? Like that's not an environment that leads to like healthy refactoring or reduction of complexity, but you know what? It makes a boatload of money, and so they'll hire people to work on it because they can afford to, and you'll work on it because they'll pay you enough, because they know, they know, right?
And so part of this as well is that also if you just have more people working on things, you know, you can tolerate more complexity. I can hold part of it in my head, and you can hold part of it in your head. And I kind of want to distinguish: like there's sort of the breadth of complexity, and if you have a well-factored system, you can decompose the complexity either into layers or modules, right? Like each individual Lego brick is simple, but you can build very complicated things from it, right? And so that's sort of the system complexity versus the component complexity. CSS is incredibly complicated, and by that what I mean is it's complex; you can't tell from looking at one rule what it will manifest as in a final document. So okay, well, we can just solve this problem with better tools, right? Well, no, because as we've talked about, we, you know, we have this homeostasis point that we fall towards over time, and we can choose where to set it, but we do move towards it. And you know, the research on this actually goes all the way back to like the 1860s. This is uh, Jeff's Paradox, which was like, "Why did coal consumption not decrease when engines became more efficient?" The answer is people did more work, right? And so I feel very much that this is sort of the inevitable conclusion we have to draw from looking at the problem, which is that the degree of complexity of a system is tied to who we are and what we're doing over time, right? And so when we buy back some complexity by using better tools or by picking a simpler environment, we're going to spend that out again eventually.
Okay, so let's talk a little bit about some theories of complexity, because, you know, this, I'm from a research lab, and I like to, like, read papers, so this is not a new, this is not a new problem. Have, have people heard of [Laughter] [Applause]? Cost and be problematic is worth thinking about. In fact, people have been looking at this and thinking about it since the 1980s. This is uh, Mayor-Liman in the 1980s, uh, IE Volume 68. So like, this is not a new domain. And so even back then—this edition was from 1980; the first paper published was in 1974—on these laws of software engineering, we're not going to read these like super close; it's more that I want you to realize that like this kind of recognition that systems grow to meet growing needs if they're successful is not a new idea. It's a common problem, and whether you're working, you know, in building tools or languages, you know, you might start with an elegant and simple thing, but as a system has more demands on it, it responds by adding, right? That's, that's a very common, and it's a reasonable consequence, and if you don't do those things, the thing will often sort of starve and die. Um, one great paper, if you haven't read it yet, on this sort of topic is "Out of the Tar Pit" by Mosley. This is from '06, and uh, this paper differentiates between what is like accidentally complex versus essentially complex, right? So like essential complexity is like the irreducible, like non-eliminable part of your system. You know, if you are trying to model certain physical processes, like, you know, smoke or fire, like the physics part where you're actually doing the work, you can't get rid of that; you don't want to get rid of that; that's what you're here for, right? But the accidental complexity is all that other stuff, like, "Oh God, we got to compile this so it works on like Windows 11 has changed the ABI for this DL," we, like, all that stuff where you, you know, we heard about Deen hoisting the um, uh, phone up the uh, the mast to download 11 gigs of something, right? That's mostly accidental complexity for our purposes. And so it helps as you're working and thinking about complexity to think about where you're spending that budget and whether it's, whether you're being deliberate in terms of how you're adopting complexity.
The other thing I want to uh, refer to is this great um, this great essay from the Berkeley DB team in "The Architecture of Open Source," and honestly, you, I highly recommend reading this if you just like software engineering. "The Architecture of Open Source Software" is a, a cool series, and basically what they do is interview open-source communities about their software and have them write essays about what they've done, and then they publish and share those. But I think this Berkeley DB chapter in particular is exceptional, and I, I think about this all the time, which is that software architecture degrades with changes made to software, right? So you might have the most elegant, brilliant, carefully planned system in the world, but it does not exist at a single point in time in a vacuum, and as new demands come along, this architecture will decay and so requires a constant shoring up. And when you have big interfaces that you've invested heavily in and they're straining, it can be extremely expensive to change them. I also just love this—this is a little more handmade-specific kind of vibe—the Excel team motto, at least according to Joel Spolsky, is "Find your dependencies and eliminate them." Uh, I have been told on reasonably good authority that Excel uh, actually is built with its own C compiler that they, they didn't even want to rely on other teams within Microsoft, and you know, to the degree where they've built their own compilers, I'm not saying you necessarily should do that; I'm not saying you necessarily shouldn't. You got to think about how you're spending your budget.
And the last kind of point I want to make sort of on this theme is again that complexity isn't necessarily bad, right? Like complexity can lead to all kinds of like wonderful emergent properties, right? You, the Legend of Zelda, um, you know, Breath of the Wild, it has had so, such a wonderful community grow up around it precisely because they managed to tame complexity with their chemistry engine. So there's a lot of emergent gameplay properties and experiences that come out of interactions, complex interactions between systems, but the systems are factored in a way that enables and empowers this. Um, and of course, if you're a roguelike fan, you know, people have been doing this forever. Okay, so how well, now what, you know, we, we have now looked at complexity. I've told you you can't get rid of it, so we're just going to have to live with it, and you know, better tools won't save us. So how are we going to spend this, this complexity budget that we have? I mean, you know, one approach is you just put your head down and you work away and pretend it's not a problem, and you know, that, that's a pretty common solution to all of our problems in life, so we could do that, but I'm going to maybe try and propose a few ways we can cut the Gordian knot here. And so um, you know, the, the story of the Gordian knot was that there was this sort of um, this ox cart with a really complicated knot on the handle, and people were like, "Oh, whoever, whoever unties this knot will rule all of Asia," and many people had come, and then Alexander the Great came along, and depending on the version of the story you hear, he just like, he like, okay, cool, and he cut it with a knife, right? And so instead of solving the problem by like trying to be really smart or work really hard, like, can we just cheat and change the rules? Like I think that's a better approach.
So one approach is to do what folks here like to do is just start over. You know, you can't actually get rid of this problem, but you can reset the clock on it, right? Just build a new one; start from the beginning. Everything's easy in the beginning; it's only when you have users and features that you have complexity. So like, you know, make a new programming language, start a new VM. It's good. You can't really change this like long-run pressure, but like genuinely you can reset the clock. And I think that's part of why things like Excel are popular and successful is they don't have package managers, right? Like every time you make a new Excel document, it's a brand new universe, you know, with none, none of the misery or mistakes that you made before. It's like the forgiveness of the blank page, you know, that's, and there's something to that. Um, I went to a talk by John Romero uh, at Strange Loop not that long ago, and he talked about how id Software had been making these like shovelware games for as contractors, or the id Software team, him and him and uh, Romero and Carmack had been making these sort of shovelware games, and their attitude was like, "Well, you should just always start from scratch. You get really good at it; you can be really fast, and each time you do a little bit better than the one before," and we do, I think, do better. I like, I'm not, I, I love learning new languages and building new ecosystems, and I think a big part of the reason why they're successful is because when you have a new language, you do have the opportunity to like clear away um, like the standard library and also just the ecosystem of ideas and people and sort of start from a fresh starting point. Now this won't cure complexity in the long run, but you know, it can get better, right? At least in the midterm, and we can get further. So I think, you know, that's pretty cool, and it's also pretty fun, so like I'm, I'm all for that. Please, please do more of that.
And to some extent it's sort of like, you know, it feels like we live in these uh, unbreakable regimes where like Unity, you know, rules, or where like the big companies, Google and Microsoft and everybody rules, but that's the kind of thing that's true until it isn't. And there's this lovely quote from Ursula K. Le Guin about capitalism, which is sort of, you know, the generalization of these problems. You know, "Capitalism was itself an invention, and um, you know, I'll just read it: 'We live in capitalism; its power seems inescapable. So did the divine right of kings. And any human power can be resisted and changed by human beings.' Right? So we, we should not doubt that the environment can be changed. The environment was created; it will be changed. We don't know how or when, but it is inevitable." Okay, so you can start over, revolution, burn it down, or, you know, just do less with less. Uh, do I have my Playdate? Does, do people have these? Has, anybody got a Playdate? Yeah. This is great. It's black and white; it, it's, there's only one platform; it's small; it comes in one color; it only has one color, black or white, depending on how you think about it; doesn't have a lot of buttons; doesn't have much RAM; doesn't really have any—there's a little bit of networking—and there's only one hardware platform, so you don't have to worry about compatibility problems. Is there, there is the emulator, which behaves differently, and due to like sourcing problems, they had to get a different microcontroller, but it transpiles natively. They're doing a really good job, anyway. It's a heck of a lot simpler than building for PS5, you know, Xbox, PC, Switch, 3DS, all on one code base. Like it is genuinely awesome, and because it's so small, it's like, you know, kind of appealing, and when you have less scope, you can choose to spend that energy, that complexity budget, you can put it into polish, right? And like on a small platform, a small idea can shine.
So okay, well, we can't solve complexity, but we can make things worse. This is actually from "Augmenting Human Intellect" uh, by Engelbart in 1962, and it's demonstrating um, that you know, you can, in fact, change people's experience um, with uh, design interventions. I said change, not improve. So you, you can, we, we've made things pretty bad in terms of software development. This is the uh, my favorite slide always to show; this is the uh, Cloud Native Cloud, Cloud Foundation, Cloud Native landscape. I always get this wrong. Um, this is ostensibly, I think, 20 trillion worth of companies, anyway. Please memorize all of this; it'll be a quiz at the end. If you want to make a web application in 2022, I had to zoom out the browser just to fit it all on screen for fun. So I'm going to talk a little bit now about our research, which kind of ties back into this because I think it might be interesting. Um, our approach is less the kind of like reboot uh, uh, model and more of the maybe do a little less with less model. And one of our sort of research interests kind of builds on these sort of tools, tools for thought. A big part of um, what we're interested in as a research group is not simplicity versus complexity or decentralization or anything like that, but we do find that we have to work in that space because what we want are tools that we have agency over, that we have ownership of, that are ours and can't be taken. And so we call our research there uh, local-first software, and the idea is basically that instead of having to go learn that whole chart of technology that you know, you can't actually do any of that stuff if you just run it on your computer, 'cause that stuff's all in the cloud. So if you want to build software that works on your computer, not only do you not get to use all that stuff, you don't have to use all that stuff. And so this, you know, it's sort of like simplification like via amputation, so we just cut off most of the cloud, and then we build things locally. And so, you know, we've dabbled in a bunch of different platforms over the years. Um, right now we're building things in the browser but storing everything in like IndexedDB, but one of our kind of core beliefs is that um, you know, collaboration is such an important part of um, thinking and making and doing that it should be like a really fundamental part of our platforms. And so we've been exploring technology that allows you to build software that runs on your computer but then collaborates with other people, you know, even allowing online-offline kind of cuts. And um, the model underpinning that is a data structure called Automerge. It's sort of like a portable, versioned JSON-like data structure. You could think of it as like Git for your data. Um, I, I don't, I'm happy to talk at length about this stuff, uh, but I don't really want to harangue you too much on, you know, sort of our particular kind of like development interests, but uh, it's really incredible just like how it feels to build software that's fast and simple and runs on your computer. If you, like me, have spent the last, you know, chunk of your career building cloud services and and you know, running things in the sky, I had this like really memorable outage where Amazon turned off us-east-1 dirty because the generators didn't come on in a hurricane, and you know, we were sort of like 3 days into some God-forsaken like system rebuild from backups of everybody's databases, and my coworker turns to me and he goes, "I'm fixing computers that don't exist in a data center I've never been to for people I've never met," and he just had the like thousand-yard stare in his eyes, and I was like, "Yeah, do you need some more coffee?" Yeah, but. And so, you know, like in a sense, working on this stuff is almost like penance for me because I created all these single points of failure.
Anyway, so let's recap a little bit of all of this: How do we live our lives in this complicated world, right? So complexity occurs when our systems have internal interactions, right? Complexity doesn't mean, "Oh, there's a lot of stuff." It's when all that stuff starts to bump against each other and cause unpredictable outcomes. And complexity is also a natural consequence of system incentives. If you have a lot of people using a thing and you're listening to what they need and you're, you know, like evolving as you learn more, you're going to end up with something complex. And better tools won't change this, right? The complexity is a consequence of who we are and the choices we make. Sure, you can, you can burn through your budget faster, but like ultimately where your project ends up on the complexity scale is more about how much time and how many ideas are invested into it than anything else. So we can't beat complexity, but we can get beaten by it, right? And so, you know, what are our coping strategies? Well, we can start over; we can eradicate dependencies; we can cut scope, do less, right? We can simplify our architecture. And a really big one is being conscious of and learning to kind of identify when you're getting into these like multiplicator environments, right? Like it, once you start porting things to multiple platforms and having to build those like per-platform abstractions, how do you manage that complexity back down? How do you isolate complexity? And a big part of success is isolating complexity. But I also want to, you know, give a shout-out to gazing into the abyss, and you know, you can, you can uh, go for it, right? Like embrace complexity, harness it. Yeah, it's a deal with an elder god, and you may accomplish great and terrible things, but at a great and terrible price. You know, that's fine, but you, when you do this, you got to be real careful, right? And and you want to be really deliberate, but deliberate about how and when you uh, take on that complexity. So I guess in, in closing, we can't solve complexity, but we can build better software, uh, or to put it in the words of cyclist Greg LeMond, "It never gets easier; you just go faster." Uh, there, and that's all.