Transcription
I think you can't ignore the fact that the sums of money going into this industry are truly gargantuan.
One gigawatt of a data center for AI basically costs $50 billion in capex. On an annual running basis, it costs between like another 8 to 9 maybe even 11 billion to run. Companies are trying to do deals with anybody who has any capacity. In the short term, what many GPU data centers are getting powered on is just uh gas turbines.
It's wild that we've come to the point where we just want like an AI that works on our computer, but like to get that, you need to have so many more powerful systems collaborate with you. That was the first time you had a system that could show its reasoning. Since then to now, the progress is pretty astounding.
Hi, I'm Matt Turk from First Mark. Welcome to the Matt podcast. Today, I'm excited to welcome back Nathan Bernes, founder of Air Street Capital, to discuss the 2025 edition of his state of AI report, a must readad on where the field really is. We cover a lot including why power is a new bottleneck, reasoning and chain of action robotics and the business reality revenue margins and what it means for builders and investors. Please enjoy this great conversation with Nathan.
>> Nathan, great to have you back.
>> Thanks for having me.
>> The state of AI 2025 is out and um as always, it's essential reading for anyone who's serious about understanding AI. This year it's 312 slice of goodness. A bit of a big year in AI.
>> Every year I try to cut it down a little bit. Um but this year I just felt like we were sharing it with various sub communities of the AI of the AI community. Uh and each time we did that the robotics folks would be like hey it's a little bit light on robotics. Can you add some more? And then we sent it to the bio folks and like why don't you cite this paper or that paper? And hence the inflation.
>> Amazing. All right. So we're certainly not going to cover everything in this conversation. Obviously, as always, the report is available in its entirety for free at stateof.ai. So, um we're going to riff on some of the most important topics and ideas in the report, but obviously people can go and check out the report directly for more.
>> All right. So, uh starting from the top in the world of research, you mentioned that 2025 was a year reasoning got real. Uh so how far have we come in the last 12 months?
>> I'd say pretty far. Um about 12 months ago or so we had I think the very early inklings of it with 01 preview uh potentially around like this time last year. And uh that was the first time you had a system that could kind of show its reasoning, show its step-wise process to get to a more complicated answer. And this has generally been the dream in AI for a long time. And uh and since then to now I would say like the progress is pretty astounding. One of the areas that that progress has kind of unveiled itself is in mathematics and other verifiable domains where you can like explicitly say yes the system works or doesn't work and you know we saw gold medals on the international math olympiad by a couple labs including openai and deep mind that area probably with uh if you'd asked experts again how long it would have taken would probably been a decade then areas a bit closer to my heart in biology and science we've seen reasoning models uh kind of be used as a as an AI co-scientist so just as a human would reading lots of papers, planning experiments, running the experiments and then doing data analysis and then reformulating their hypothesis as a result. There's examples of uh models doing that in lie of human which is exciting because there's way way too many papers uh to read you know AI people kind of complain that it's like 50,000 papers a year and say in biology and chemistry and physics is probably an order of magnitude more than that. Um and so uh deep mind has shown that you can integrate this kind of reasoning model to sort of decipher new targets for disease new mechanisms that were actually also proven in a wet lab scenario um postfacto. We've gone from systems that were kind of dumb stoastic paris to now they can solve pretty meaningful challenges that I'd say like even a smart human couldn't.
>> And still in research you talk a little bit in the report or a lot in the report about robotics and this evolution towards uh a system of action or chain of action going from chain of thought to ch of action. What's happening there?
>> Yeah, I mean the gist is probably two years ago robotics was kind of a dead end. Um openi had disbandled its uh its robot team that was famous for solving the Rubik's cube using uh locomotion with the hand. And so now robotics is probably you know going through a cambrian explosion. There's so much excitement uh and just as how language models informed biology, now language models are also informing robotics. So what you're referring to here is a sort of reasoning uh process for robots where a system is no longer just perceiving the environment and deciding what to act and sort of acting, but it's uh we've separated those steps. So now you have a reasoning model that looks at a task and tries to plan steps that a robot would need to do to execute that task and then passes that plan over to uh an actuator which goes and actually implements the plan and that's what's called chain of action. Uh and here the Allen Institute was one of the first to really push this and very swiftly thereafter uh Gemini also followed and and uh we have some companies including Serak that are implying this uh into the real world. So it does genuinely work. It's not just like a research thing. So we think the big moment for robotics is uh upon us because we all collectively have been uh talking about this for a very long time.
>> Yeah. Yeah. Well, I'd say it really is upon us in the industrial sector um in logistics and warehousing uh kind of more constrained environments or very repetitive tasks. There is the sort of more holy grail of this kind of embodied um uh humanlike form factor and putting a model on that might even be the same model that's been used in uh in warehousing. Uh a lot of money is going into that. But my personal bet is I is I think it's going to be the humanoid space is going to look much more like self-driving where we have some very good isolated demos, but the longtail will kill you. Uh
>> hopefully [laughter] not uh literally.
>> Yeah. Um, and uh and so we we're going to go through many false starts. I think this is just the start.
>> Okay, great. So a big year in uh robotics and reasoning. Um for uh people listening to this, if you're interested in deep dives into reasoning and RL and the evolution of AI systems, uh we've done a bunch of great episodes recently with Shto from Enthropic, Jerry from OpenAI, and then Julian from Enthropic. Uh if you're curious to learn more, let let's move on to the business uh of AI. Uh you mentioned in the report that the business of AI finally caught up with the hype. Uh what caught your attention in terms of fact stats uh in the last 12 months?
>> Yeah, a couple of them. Uh again like where we came from one or two years ago was just tons of money going into this segment building models lot of usage but not clear where the revenue would come from. I think it was maybe open was making $50 million or something two years ago. it was very unclear how they would ever hit like billions of revenue. Um, and nowadays I think if you sum sort of the top 20 or so uh major AI companies from the labs to the most popular kind of vertical applications, you know, across them they're making tens of billions of dollars of revenue. Um, you can look at the smaller scale companies which you know are growing from zero to 20 million or 20 million plus. Uh as a group they generally grow about 60% faster on a quarterly basis than nonAI companies. Um, we've all seen like the famous charts about ARR or non ARR it's unclear. But uh, you know, very steep curves for various coding companies. Um, and perhaps most interestingly across a segment of 43,000 or so uh US customers we work with ramp to show that retention of uh subscriptions on AI products across this customer set has really improved marketkedly since 2022. Around 2022 was around the 50% after 12 months. Uh and now in 25 it's hitting around 80%. Um, and the second stat in that analysis that was interesting was the total spend on AI products uh per customer kind of went up from $35,000 or so uh maybe two years ago. Now it's around half a million and it's predicted to hit a million dollars next year.
>> And you mentioned in your ramp stats 44% of US businesses now pay for AI tools. So they pay more but there's there's there's a ton of businesses using it.
>> Yeah. Exactly. And and there might be some sampling bias slightly to you know what kind of companies uh use ramp in the first place.
>> Yeah. So slightly more modern you know tech forward companies but a leading indicator I think of where things could go.
>> And then you had your own survey right of 1200 AI practitioners.
>> Yeah. Yeah.
>> And what did that say?
>> Yeah that was I I was surprised. Uh obviously bias is more towards uh you know pretty well educated US European professionals. Uh a lot of people in there have at least undergrad, master's degrees, maybe even more. But it's like 95% of people use AI in their personal life and in their professional life. U, about 76% of people pay out of their own pocket for it. It's like 10% of people pay more than 200 bucks a month for it. Uh and then looking at the organizations that they work at, it's like 70% of those organizations are spending a ton more or more than they did in the past on AI. the reasons that they gave for what why they might not be spending more or what problems they have. It's like all the classic like new technology stuff like it's a bit hard to configure. It's uh you know I haven't really figured out the ROI yet because I need to do more customization. There's like some data privacy issues that I have and I think all these things are kind of solvable like it's not rocket science how to solve these things.
>> Yeah, it feels like we very much live like this. You're in the world of shadow AI uh in companies where I mean to reconcile it's imperfect but reconcile your two stats. 44% of businesses uh use AI yet 95% of of people individually use AI. So there's a bunch of people as as you as you're alluding to that use AI at work without being officially authorized.
>> Yeah. And I think there's still a big like education gap. I mean there was a a study banded around a couple of weeks ago where you know it said 95% of businesses like get no value from AI.
>> Very very controversial survey.
>> Yeah. Yeah. But I think
>> 95% is the number right. Like everything is 95%. [laughter]
>> But I think that there it turned out it was like not the models that were bad. It's like the implementations of them were not great. Uh so I think there's just a big education gap for how how you should like you know update your view of your own uh day-to-day tasks and and apply what capabilities you know models have and then uh and think about like hey should I be doing this task myself or can I farm it out to a model and and there's definitely a delta of companies that really get this done well and others that are like basically clueless.
>> What do you make of the margin debate uh as an investor and intory analyst? Maybe maybe uh recap uh what what that debate is and then what do you think about it at a high level?
>> Basically the the margin problem is um for many many customers of uh large model companies their margins are basically dictated by how much the model vendor charges them for. Uh now here there's some issues because right now model vendors are charging the same amount per token. So if you're a hedge fund analyst and I'm a student you know your use case is clearly more financially valuable than mine but we pay the same amount for the token assuming we use the same model. There are some use cases that are more reasoning heavy towards what we discussed before and they consume a ton of tokens and the pricing that a customer uh pays for that product might not be fit for the amount of work the AI system is doing. Um and so there are cases where um these kind of vertical products are making margin gross margins of like 30%. And sometimes they get worse with scale uh because you do have some edge users that like really pump the system and you can't like price discriminate or they haven't managed to. And then you have uh some segment of of um of model users that don't that have a both a paid plan and a free plan. And it's not clear whether they include uh the costs of running the free plan in their gross margin.
>> So they sort of just look at their paid customers. Uh there's you know some creative accounting standards going on there. Uh and then you have the model vendors themselves and what is their margin. Uh and I think what's interesting in the last year is you've seen um CEOs of these model companies say hey if we if we basically look at um sort of in in financial analysis terms like a layer cake of like what revenue is generated by each vintage of model over time um it looks like prior models are uh profitable. So the amount of money we've spent to build them is less than the amount of money that we've generated with them over time assuming a certain margin of inference cost. And so so really these these labs are like not not profitable because vastly more resources going into developing next generation systems than than the prior ones. But as you and I both know like there are companies here that are making very very good margins on uh serving uh their AI systems like 70 80 sometimes 90% depending on the modality. Um and so like with everything the average number sucks. Um, but like when you look at the best companies, it's really good.
>> And just to drive it home, the the the the companies using those models, we're talking about the, you know, in part the the all the what used to be known as thin wrapper. So the vendors that happened to be powered by those models. So uh the the cursors, the wind surfs yeah Reddit like all and and all the whatever legal financial AI um startups >> um as a as as examples. Uh the other big debate uh in the business of AI of course uh is the bubble uh question. What's your >> what's your take? Are we in an AI bubble? Are we not in an AI bubble?
>> Yeah. I [clears throat] think like with most things in in markets, there are probably localized bubbles all over the place. And I think at a at a high level, what's interesting in terms of vibes and who's calling bubbles and who's not like the finance crowd in New York is definitely talking about bubbles a lot more than what we're talking about in San Francisco where their their view is like this is the golden era of AI and a lot of things are working. We have so much more to to do. You know, compute buildouts are enabling us to experiment a lot faster. Uh you know this huge flood of like talent that's built the consumer internet and cloud computing is moving into AI and with that is bringing a lot of optimization techniques and knowledge that AI researchers didn't have when they built the first generations of Chad GBPT etc. But I think you you can't ignore the fact that the the sums of money going into this industry are truly gargantuan. Um, you know, like 500 billion to build uh Stargate and then uh you know couple hundred billion here, a couple hundred billion there. Like pretty soon it's real money. And then the um and then like the circularity of these deals is like is interesting. Uh of course Nvidia is at the center of this and it has incentives to use its its balance sheet to sort of spin the wheel faster. Uh and then perhaps more concerningly, you have this sort of offloading of of debt um from big companies. For example, Meta that raises tens of billions of dollars to fuel it data center ambitions, but that doesn't sit on Meta's balance sheet. Some of this is like catnip to financial engineers. But but yeah, it rests on certain assumptions that everything is going to keep going up and to the right and that rates don't materially change. And just given how like uh I suppose precarious various aspects of the economy are and how like sensitive geopolitics are, things can flip like quite quickly. But I think that's like the major risk. The the risk I'm less worried about is the stuff doesn't work >> because I think it does work.
>> So it's a more question of uh timing to play it back that the supply uh phase of the market is met by an equally strong or hopefully stronger demand side.
>> Yeah, there's that and then just the just the the nuances of like the terms on the debt and what trigger events are whether rates get repriced and then you know like investors behave very differently once rates change and and um and flows of money can be quite like violent.
>> It's interesting what you're saying about the, you know, the dichotomy between um Wall Street and the the West Coast. Um also because uh when you think about it, there's actually not that many fuel play AI companies in public markets, right? A lot of the action is happening in >> private markets. So effectively, if you're a Wall Street/hedge fund investor, you you invest in Nvidia, you invest in the Mag 7. That's pretty much it, right? Palanteer C3 AI >> maybe you buy soft bank for its position open AI >> yeah pretty much like a lot of it is indirect or you invest in power and energy or like related players coreweave I guess but it's it's very it's very small so it feels like there that tension as well
>> yeah but I think it's also the crowd that you hang out with
>> um I mean and I
>> do you live in a house in San Francisco with correct two other or three other AI geniuses
>> >> correct correct correct or do you just consume the outputs uh of that of those kinds of conversations on Twitter and then try to like
>> >> yes
>> >> piece together your own world view and and I think the other part of this is like I I don't think some of those individuals are really shilling that much anymore I think they do genuinely believe what they say and they are at the core face of the advancements of these technologies and so if you know they've been saying for the last 50 times like hey this stuff is working there's lots of implementations we can improve or like things we can tweak or new experiments that'll yield better capabilities and that has happened. At some point you got to be like maybe they're right.
>> Another aspect of this that's fascinating to me is the again like the the the sort of dichotomy between some of the I would call them the old guard and the newer younger kind of folks. So, you know, from Rich Sutton to Yan Lan to um, you know, obviously Jeffrey Hinton, a lot of those guys who are absolutely the godfathers of the space and built this entire thing and are still extremely active uh, today on top of everything say that um, LLMs are just not going to get us there >> uh, or that we should uh, just do everything with RL. And then you know meanwhile the the younger guys and they tend to be at places like anthropic and open air so maybe they do have an agenda but they're all saying well we're just scratching the surface of what we can do with those modern systems.
>> Yeah. Yeah.
>> I think do both. Um but but yeah I think for me it's it's mostly um what are kinds of new problems that you can that you can work on and solve with this technology. And I think it's becoming more popular to believe like the overhang of of like problems we can solve in enterprise for consumers and science with the tools we have today is huge. Um and so even if a lot of this compute buildout doesn't go towards like dreaming up the next uh transformer architecture but goes into uh improving the unit economics of serving AI systems for everybody and makes it easier so you don't have to be like some prompt master uh to elicit a behavior you want for your task. I think that's not good.
>> All right, let's uh switch to the physical reality that this whole uh stack sits on. So infrastructure, data centers, uh energy. You you mentioned in the deck that uh power uh has become the new bottleneck. What is your sense of the state of play in the energy procurement game?
>> The the biggest stat for me is one gigawatt of a data center for AI basically costs $50 billion in capex. Um and uh on an annual running basis it costs between like another 8 to nine 8 to9 to maybe even 11 billion to run. And so uh when you have just you know casually a 10 gawatt data center uh that's like a lot of money. Um and um and so one of the problems is like where does this energy come from? You know traditionally it would be from I don't know coal uh or natural gas um potentially solar or ideally at some point in the future nuclear and what we're seeing is it right now companies are trying to do deals with anybody who has any capacity. So we chronicle some deals with uh future uh nuclear uh you know reactor companies then that would take maybe a decade or two decades to deliver. You know famously
>> yeah that's Google inking a PPA deal.
>> Yeah.
>> Uh with CFS to buy 200 megawatt of electricity from a planned
>> fusion plants. So the the the plan does not exist.
>> It does not exist. Yeah. And then last year we documented the sort of uh restarting of uh 3M island. Um
>> yes
>> the nuclear facility which was controversial in the past. Um, and um, and then uh in the short term what many uh GPU uh data centers are getting powered on is is just uh gas turbines and because these can get set up a lot faster but that has other issues like they're super loud um and there's demand outside of the US for these things and so now basically US tech companies are paying more to repatriate like uh some of the supply that should have been shipped abroad. the the other issue is the the grid and like to what degree the the grid can even tolerate data centers getting plugged into it. Now obviously like these turbines are off-grid. Uh so it has some advantages but uh in China for example we do some analysis between like eur between uh the US and China with regards to energy and China has a lot more like slack in its system to plug in um for any unpredicted uh demands in energy. Uh the UK famously cannot really tolerate more data centers on its grid. wrapping all this together is driving um some of the like offshoring of data centers towards uh energy-rich countries whether that's the UAE uh or even uh Norway and uh and then with that comes a lot of like geo geopolitics of uh are these nations your friend or or potentially not and how do you ensure uh access to this regardless of your administration change and other things. So it it yeah it's it's wild that we've come to the point where you know we just want like an AI that works on our computer but like to get that you need to have so many more powerful systems uh collaborate with you.
>> Yeah and I was just looking for the slide as as as you spoke uh especially for United States versus China we're talking about the dramatic difference where uh the capacity added in 2024 for the US if I read this correctly was 48.6 6 GW whereas China was 429 gaw
>> the other thing that's interesting is um the at least the states in the US uh or actually also internationally that um that are good for hosting data centers because there's energy typically are extremely dry and uh and we also chronicle the water usage that's needed for cooling of these data centers and so if your state is super dry where do you get the water from uh is that actually going to detract away from human populations that need the water? Then you have this whole like recycling of water which could potentially like yield just like bad quality water getting circulated into the water system.
>> So the sustainability aspect to all of this seems uh extraordinarily important >> yet uh under discussed at least that's the my my perspective. Is is that is that correct? Do do people actually care and do something about the sustainability aspect of this?
>> Well, a year or two ago, big companies did make commitments to be green as of you know, 2030. And then as soon as they started inking deals with uh you know, nuclear companies and uh and and various energy providers for data centers, all those commitments basically got like washed away. So it seems like maybe they care, but the corporate priorities of [snorts] making AI work have way outweighed the environmental constraints. That's what's happened. But I think again going back to like the politics side of things, I don't think everybody's very happy about this. Particularly there's this like growth of nimiism, this like not in my backyard. Uh and uh and and I do think that uh people generally don't want to have a data center in their backyard. Uh and I think that's going to drive some of the political agendas like going forward whether it's in the US or or other countries. So yes, people do care about environmentalism. companies have sort of washed that away but it's going to I think it's going to come back.
>> If we talk about infrastructure uh obviously we have to talk about Nvidia feels like it's been uh another extraordinary >> uh last 12 months for uh Nvidia.
>> Y
>> do you uh see Nvidia continue to break away as like the undisputed number one in the market or or do you think that sooner or later we're going to end up with a multi-ilicon kind of world?
>> I think it's gonna be 955 to 95%.
>> Um,
>> revisited.
>> Yeah, exactly. Yeah. Um, yeah. So, you know, for context, when we did the executive summary last year, we put Nvidia, you know, hit 1 trillion for the first time and now we had to change that to 4 trillion. We look at uh all the open source AI research papers every year, which is about 49,000 or so. And then, uh, programmatically determine which chipsets are used in those papers. So we know like hey an AI researcher is doing a study on uh I don't know some new model and in their in their experimental setup they say you know we train the model for x number of GPU hours on uh whatever chip and uh if you do that analysis you basically find that 90% of uh all papers make use of a Nvidia chip out of that same analysis we did find that AMD is sort of popping up a very little bit um Apple silicon is as well I think it's just because the computer the MacBook is getting so good that people are doing local training uh and experiments on their computer
>> and Broadcom is experiencing a a resurrection of some sort as well, right?
>> Yeah. Yeah. Exactly. Yeah. I think it's maybe a decade ago they bought a company that now is kind of the internal team doing this custom AS6 for uh Google's TPU and uh you know more recently they announced a deal with OpenAI also to do uh a custom chip. And a high level what's interesting with the rise of Broadcom is basically GPUs have been the the dominant chipset for a long time as the uh kind of nature of the neural network or other kind of AI system that you're running on the hardware was still changing very rapidly but as soon as you get to a point where there's some convergence on an architecture that's looks like it's stable and is revenue generating and developers are coming to uh sort of work on it and confirm that it is like the then you can flip towards doing a custom chip that's built to extract the most value out of that architecture. And so the rise of Broadcom basically tells you like there's strong forces that are saying like the transformer is the thing. But at the end of the day like we also look at how would your dollar be best used as an investor if you wanted to bet on chip companies. And uh and in the graph in in the report, we look at sort of six of the major contenders uh to Nvidia and basically said, you know, if you bought Nvidia stock on the day of uh the announcement of all the like private uh rounds in these companies, what would the value of your stock be in Nvidia versus these companies? And uh if I recall correctly, it's basically 12x in Nvidia versus 2x in um in these competitors. And the trend was roughly the same last year. Uh so I think I think it's a little bit of a diff difficult beast to to bet against.
>> Yes, I was uh I was looking for that uh slide as you were uh speaking. It's for anybody that looks at the report that slide 166 that says um what would have happened if investors had just bought the equivalent amount of Nvidia stock at that day's price. The 7.5 billion would be worth 85 billion in Nvidia stock today. 12x >> uh versus 14 billion2x for its contenders and the contenders being Grobra, Samanova, Celestial, Graphcore and in China, Cambercon has experienced you know a big run. Um this is you know a private company that then went public on on Chinese stock exchange to build custom AS6 for for AI and and that was driven mostly by uh the geopolitical sort of zigzagging on policy with regards to exporting custom um Nvidia chips to China the H20 which at some point was deemed to be okay by the government and then deemed to be not okay but then okay if uh 15 to 20% of the revenue was passed back to to the US government and then and then um someone in the someone high up in the US administration said you know our goal is basically to ship the like crappy stuff to China and at that point the Chinese said like no thank you and >> and effectively said no one can buy Nvidia chips and then camera stock rips >> and that's Huawei as well right that's the emergence of a separate Chinese uh full stack from the the models which we'll probably talk about that at some point in this conversation of open source but very much at the player. That's what you mentioned and then Huawei whatever the model is becoming the sort of default chip for the Chinese stack.
>> Yeah. Yeah. Yeah. And there's some interplay between the government trying to get Deep Seek and other labs to to run their models on uh on Chinese chips. And there's been rumors that this is why a lot of the new generations of Chinese models uh have slowed down. Particularly Deep Seek like there people are waiting for for the next uh next R1. So like R2 and uh and allegedly it's because it's just hard to run it on Huawei.
>> To to double click on on on uh something that you mentioned a few minutes ago. Um
>> talk about uh sovereign AI uh and uh what you've seen >> people do. It seems to have been a big theme of the year. You mentioned open AI in uh Norway, India and and UAE. What's happening in that world?
>> Yeah.
>> That part of the world.
>> Yeah. Yeah. So the idea with sovereign AI is that nation states want to be uh at like able to control basically their fate with regards to AI. So that's uh running models as training models um having chips uh and um this is basically because you know nation states want to have control over their energy control over their currency, control over their infrastructure and AI is deemed to be uh kind of equivalent to those categories. Um and so ever since the White House announcement of 500 billion in January uh various nation states have followed suit saying you know we have our own initiative and it's to the tune of billions of dollars etc around the world and Nvidia has even started marketing this as like a like a a new kind of product line basically for its business that currently generates I think around $20 billion worth. So it's it's real money. Um and so they're forming partnerships with various nation states uh to provide data centers there that are run locally. Um and uh and in theory that should give like countries comfort that uh their access to AI can't be turned off. That's the idea. I personally think it's a bit more of a of an alignment between political agendas where particularly in the US it's really about re-industrialization like onoring of key industries and building you know manufacturing and things like that which is I think one of the reasons why these AI data centers are getting rebranded as AI factories and so that's the the political part um and that's getting aligned with um uh just the need of countries to get access to this technology so I think it's more marketing than it is like a real policy because at the end of the day if you buy your stack from uh from from the US and you're not an ally of the US at some point then they'll just switch it off. And so part of this is like sovereignty washing I think and it also like oversimplifies the very interconnected nature and ecosystem aspect of of AI where not just about the chip it's about uh the developer ecosystem um how you actually run it where your training data comes from um and uh and all the like infrastructure like data tools and and whatnot that that sit around this
>> although uh that's where open source plays an important role right if get your AI from OpenAI and indeed uh you are a USLI but you no longer are USLI for whatever reason there's a risk that you could be turned off but if you have sovereign data center and with a bunch of like chips running and then you run open source on top of it like presumably you are safe
>> which is then uh interesting because where is the most popular open source coming from now?
>> Yes, China. China. [snorts]
>> Um
>> although interestingly I think since you uh published the report there's been the announcement of a very large investment in reflection AI which is >> a New York and San Francisco based uh company that just raised 2 billion >> uh to build uh the US equivalent of the Chinese models in in a world where Llama >> and Meta have sort of um gone in a different direction.
>> Yep. Yep. I think this is this fascinating um because part of the AI action plan uh that was published by the US government a couple of months ago now um you know articulated the need for having this American AI stack. So they're moving away from like diffusion controls and more towards just buy our stuff. And then one of the other aspects of that action plan was around open source and like and and sort of leading in that direction. And of course as you said like meta stepped back and into the fold came Quen. Um I think 50% of all model derivatives um being downloaded from HuggingFace or Coinbase now um hundreds of millions of downloads partially because they're very they come in very accessible shapes and flavors. So as a result of that we sort of predicted in the report that uh uh that a major you know AI lab would lean back into open source to win um basically brownie points with the government and then the next day this financing happened.
>> Oh amazing great uh great timing. Yeah. And I think you you said in the report as well that um your sense was that OpenAI was sort of um forced for lack of a better term uh into releasing an open source model to be on the on the right side of history.
>> Yeah, I I think that's one of them and then the second one probably dovetales with their announcement with AMD and I say that because uh you know quite recently semi analysis uh kind of published this benchmarking data set where they run models on various clouds to sort of benchmark them. Um, and actually GPT OSS like looks pretty good on AMD. Um, and so one could imagine that um, like there were some uh, optimizations and there actually were optimizations to to GPTO OSS. So it runs nicely on AMD. It has support from their framework from day one. The uh, the parameterization of the model is is uh, to the point where you can run it on a single AMD chip. And there's some other nuances to their attention mechanisms that they customize to make it work really good on AMD. And to the point around like the circular economy stuff that we discussed a little while ago, um, there's uh, like another financial sweetener in the deal where OpenAI has warrants in AMD if the stock price hits 600. Um, and so you can see how how there's a lot of like incentives to this game of both like aligning with US government, helping developers, which is a good thing, but also like helping one of your vendors uh improve, which frankly it does need help and it and it should improve, but also getting some financial sweetener as a result of that, which could help you kind of make the flywheel spin faster.
>> And uh since we're talking about circularity, uh talk about concentration as well. So maybe as an echo to the conversation about the bubble a few minutes ago, it does feel like this um AI economy has a lot of um depending on how you look at it from funky to scary things.
>> Yeah. Yeah. Well, a lot of Nvidia's revenue comes from uh the major uh you know hyperscalers or or um or neoclouds. So you know it's like Meta like XAI, Google, Amazon um then Cororee and then a lot of Cororee's revenue also comes from Microsoft on the way back. I think it's just this challenge with with AI progress that we've uh you know very meaningfully shift from shifted from I think the GPT3 era to now of basically scale like rate limits your progress and it's no longer like a couple of people in a dorm room that can really build something transformational if they want to advance like AI capabilities. It's really um it's really big boy land now. Um and so w with that comes just different dynamics like you have to be good at capital raising. You have to align yourself with uh with nation states. You have to align yourself with Wall Street. These are all I think contributing to the big vibe shifts that you've seen in in the culture of AI labs.
>> What what do you mean by that? What
>> well you know for example there there were some labs like Anthropic that were built you know to to really push the safety agenda. Because you know if we didn't do that the rational went that you know we could lead to the extermination of humanity right and I think quite recently like Dario was interviewed by uh Mark Ben off just this past week and asked about like some of these data center buildouts and and you know he said something along the lines of yeah there's a lot of money going into this a lot of cost but at the end of the day the only thing that matters is revenue like I don't think he would have said that you know on the founding day of anthropic and you know it's it's just the reality that that the table stakes in this game have changed and with that you know entrepreneurs have to update their priors and and you know change their strategy a little bit. And so we document some of this in like sort of the blooper section of the report uh which is uh which is just like how how much of a sort of pendulum um swinging we've we've noticed in um in corporate priorities at AI labs um as a result of the extreme financialization of the sector.
>> are you uh encouraged or discouraged by some of the stuff that's happening at the app layer in in in in particular uh you know whether that's uh AI slop uh or uh yeah focus on on on revenue and um you know versus the ideal like do do you think that's um inevitable but good or what do you make of it?
>> I think I think we're just at such an early era to like see how you can maximally extract value and create interesting experiences for people with this AI uh technology that you know we have to try a lot of different things. At the end of the day, you know, if if you're a lab that expends tens of billions of dollars on uh on R&D, you do have to have a way to generate money to to fund that. I think that's just reality. And I think like this the slop thing, I mean, if it's bad, people won't look at it. And uh if it if people look at it and they enjoy it, then you know, good good for them. Like I don't necessarily have like a huge problem with that. As long as uh as long as like where I'm spending my time uh, I find is uh is useful. And so that's why I end up spending a lot of my time on like enterprise software automation, biology, like doing new discoveries and drug discovery, like defense technology and autonomy, u robotics. I think these are all like very important macro drivers of of the economy. And as we move into an era where like intelligence is, uh, you know, increasingly cheap and accessible, uh, there's just so many different like instantiations of products that we need to build that are really meaningful. And you know, if a byproduct is that is you have a social media app with like AI videos, like that's fine, too. You know, we all have to like unwind, [laughter] right?
>> You mentioned safety a minute ago. I'd love to, uh, riff on that theme a little bit. Uh, IP rights, safety, regulatory, uh, a little bit like the sustainability thing that we were discussing earlier. It sort of feels like that that whole world uh has um sort of slowed down in terms of like progress maybe starting with regulatory. Do do you think that regulatory is anywhere near catching up or providing an adequate response to what's going on?
>> Yeah, I'd say like a big 180 on that one. I mean clearly the Trump administration unwound a lot of the Biden era policies uh whether that was on uh diffusion you know trying to push a lot of state level legislation against AI the uh over in Europe like the EU AI act has had um delays in implementation there's only three member states that have actually implemented it and now we're finally seeing how even its authors are saying maybe we went too far particularly as we look at progress uh the speed of progress in the US in China compared to Europe. You know, famously this bill in California, um, you know, rate limiting AI progress was really watered down into what eventually became SB53. Um, there were, you know, many, many proposed bills. I think over a thousand, 10% of them actually made their way into laws. So it's still kind of patchworky, but like at a meta level looks like we traded regulation for just going faster. It was perhaps like best encompassed by uh by the shift between the AI safety summit in the UK which was at Bletchley which basically pledged like a whole network of uh AI safety institutes and conferences that would happen over the the coming years. to then the subsequent event in Paris which was called the AI action summit completely different than AI safety summit >> and JD Vance saying something along the lines of basically like AI progress is
Not going to happen if we keep hanging over AI safety. The US basically didn't show up to a few of the subsequent conferences, and we have this in the like, safety RIP section of like, very few people seem to care about it anymore.
And to the, to the vibe shift, like, even the more, uh, doomerist parts of the ecosystem have, uh, kind of quieted it down, right? It feels like the debate has gone from "kill all of us" to more like, "well, is RLHF the better way to get to AGI?" kind of. So the, the naysayers have like shifted their, their kind of like approach.
Yeah. Yeah. And I think it's become, yeah, less about this existential crisis and more about which capabilities look concerning in models. And, you know, there's been some kind of interesting, uh, data points that we chronicle in the report. Like, for example, uh, models can increasingly know that they're in a simulation or, or know that they're in an evaluation and then change their behavior as a result of that. There's examples of, uh, models trying to like exfiltrate their own weights.
There's another, uh, piece of work that we show which is around the cybersecurity capabilities of models, which is basically measuring how long does a human take to solve various categories of cyber tasks and then putting models at the against the same tasks and saying, you know, how long would it take for them to solve it at a 50% pass rate. And there, it looks like again, the capabilities on cyber tasks of models are doubling every six months. And then so this is cast against the fact that independent safety organizations, there's maybe like five or six. These are usually nonprofits, uh, that are still nonprofits, uh, or private companies. They spend on average $134 million a year in total.
Total budget across all of them.
Yeah. Across all of them. Exactly. And that's cast against roughly like $92 billion, uh, across all AI work for the major labs. So basically, like the same amount of money, uh, that a big lab would spend in one day is spent in an entire year across these safety orgs.
$130 million, aka a seed round in a week-old AI startup.
Correct.
Correct.
What about, uh, data rights? That was another part of that just general kind of like policy universe that was very, uh, sensitive and controversial. There's been some, some evolution in the last year, right?
Yeah. Major changes. I think it looks a little bit like the sort of on-demand commerce war of, you know, the Uber style of do something that's a bit dodgy for a long time, get to scale, and then once you're at scale, you're kind of too big to kill. And so, and so similarly in AI, like a lot of companies took slightly dodgy practices, uh, to acquire training data and then got to scale. And they were subject to many lawsuits in the last year or two, uh, particularly in in the media sector, whether that's, um, you know, music or video, uh, and books.
And then there's the biggest, uh, settlement that happened in the last few months with Anthropic that agreed to pay out $1.5 billion. Uh, and this was settled out of court. So it can't be used as precedent. But, but generally shows the, the rough price, uh, tag that's affiliated that's associated with, uh, with human works, uh, in the context of AI training. And then separately, there's been, you know, dozens, if not hundreds, of organizations that have agreed content licensing deals with various model companies. As I think the power shift has has really happened.
$1.5 billion still being a drop in the bucket for a company like Anthropic. Interestingly, does that create a moat over time? Meaning that you have to be large enough to be able to afford that kind of money, uh, that you're going to pay to data rights if you want to do pre-training? And does it make it harder to start a company that needs to do pre-training from scratch?
Uh, in one sense, yes. In another sense, uh, if you can exploit the knowledge of these frontier models, particularly from open source, and then generate synthetic data, could be a way to get to capable models faster. And also, I think, I mean, you'll have many guests that go deep on this, but, um, but even the nature of pre-training and what, uh, information is included in the corpus and at what point it's kind of like data mixtures, as people call it, has been evolving over time. So I think we're just getting smarter about how to do pre-training rather than just shoving everything we have into a bucket and like seeing what happens. And so as a result of that, you might not necessarily have to spend the exact same amount of money to get a capable system. And, you know, some of this kind of came out from the DeepSeek paper.
You mentioned cyber. Let's riff on this a little bit. Obviously, AI creates new attack vectors. What should people know?
I mean, as of a couple years ago, people were obsessed with deepfakes or like these videos of people saying things that they didn't actually say, and they were still kind of grainy and not awesome. Uh, clearly, those deepfakes are getting a lot better. Um, although quite positively, it looks like we're actually quite good at detecting them and and realizing, oh, that's like.
Um, but there's more advanced, uh, approaches now where, you know, models can be capable of coercion, um, particularly for some individuals who are sensitive to this kind of, uh, risk. There's been examples of, for example, North Korean state actors, uh, trying to infiltrate other states using AI systems. Uh, you could potentially even package a language model in malware and then have it installed in a computer, and then it kind of wakes up. And because it's not dumb, it's a language model, it can do things on computers, and that's kind of scary.
The rise of MCP, I think this Model Context Protocol, which is kind of like a USB stick for, uh, for all sorts of, of, uh, data connectors, is cool because now models can be smart. They can integrate all your stuff, uh, across your digital life. But do you necessarily trust the creator of that MCP server? Like, where is that data getting sent? There's tens of thousands of these things now, and cybersecurity risks that, uh, that, um, result, um, because of this. And also some changes towards APIs of, you know, of model APIs that sort of trade off whether the user or the model vendor manages state. And depending on that, that's another like risk that you have to think about. And so I think at a, at a high level, like there are lots of security issues that are that are coming to the fore here. But it's, it's sort of still unclear whether there's a good business to be built in cyber, um, for AI because it's still so early. Like we haven't, um, necessarily like felt the pain of all these things yet reputationally and financially. And a bit like insurance, until you have actually felt the pain, you know, you sort of like prefer to divert your money towards just like improving and making more money than protecting your downside.
Yeah. Interesting.
And it's another area where the, uh, incumbents are not asleep at the wheel.
Yeah. And all the big labs. Yeah. Exactly. So, if you're really good at, at, at security, do you want to, it's a bit like AI safety. If you're really good at these things, do you want to be in the belly of the beast and be able to like see how the sausage is made and like influence it, um, because of the proximity, uh, or do you want to be on the other side, like receiving the artifacts and maybe at best doing collaborations with labs on pre-launch safety testing, like, uh, they do in the UK with, in the US, or at worst, just like literally trying to sell as a cybersecurity SaaS to people who are consuming these models? So I can understand like why that imbalance occurs.
And to your point about it being hard to sell before the pain is, uh, felt, feels like there's a whole generation of young startups that have got acquired pretty quickly by the,
Palo Alto Networks and Checkpoints of the of the world.
Yes.
Before they got a chance to get to scale. I mean, you know, something that feels like it probably turned out to be great for the for the founders, but in terms of building large, self-standing, sustainable companies, not not so much.
Agents. It cannot be a 2025 conversation on AI without talking about agents. What is your sense of the reality and the state of play?
There's some vertical products that are really good. Clearly, search, um, is actually pretty good. You know, replacing consulting, uh, replacing market research, or augmenting all these, uh, these areas that were previously, you know, very heavy human, uh, you know, knowledge working tasks is getting extremely good. I think, uh, coding agents clearly are are getting really good. There's other metrics around like how long they can work autonomously. I think with the new Haiku release, it's 30 hours or something, and it can make a pretty decent version of Slack.
Yes. Although a controversial, uh, number, but, uh, yes, up to 30 in lab testing. Okay.
Uh, exactly. What, what does even hours mean in an agent of like a computer running it? Like, is that equivalent?
Yeah.
And then, uh, and then some of the scientific reasoning we talked about, uh, is agent-based. I think that's also quite neat. I think the, the biggest problems just become like this kind of compounding error of, you know, an agent is like 95% like good, and then 95% times 95% times, etc., etc., sort of decays quality over a long period of time. And then there's some contention now about like, do you build these harnesses, i.e., like nerd speak for sticky tape, um, between [laughter] um, between like models to like make it work in enterprise, or do you just wait until the next model generation hopefully becomes better out of the box?
I think a ton of excitement and at some point, basically, just as desktop software became SaaS, at some point, SaaS will just become an agent because it's no longer really like a human that's that's actually doing everything in the software product, but a software that's running the software product itself.
Which I think is cool implications for like, uh, search and and, uh, and product discovery and and this whole, like, uh, ecosystem of like online content. Like, is it humans that are reading it anymore, or is it agents that are chewing it and then serving it to their human, uh, for that, like the whole, um, evolution away from, you know, we go on a website to buy a product versus, uh, you know, answer engine/search engine that's largely OpenAI that enables us to buy natively.
I'm not so enthusiastic about the, oh, we're going to have agents that will book flights for us and travel. I feel like that's just like a niche problem that, uh, sort of like the, the sad canonical use case in San Francisco. But, uh, but I think this, what's what's telling so far is that traffic that's generated through conversations in AI search onto a commerce platform converts at a higher level than direct traffic. So the intent is really high because there's already been like background research that's been undertaken in chat. I think that's really powerful and you can't ignore.
And then the, the next question on that is, okay, so what content is the model actually, uh, consuming to serve recommendations or information to its user? You know, people say, "Oh, Google Search is dead." I think that's like probably completely wrong because ChatGPT references Google a ton as it shifted off of Bing. Um, and so maybe it's not like the front page of Google that's being consumed by a human, but by sort of an agent that represents the the user.
If you're a company that that has a new product and you want it to be recommended, then there is like this flywheel that, uh, you should probably get on as soon as possible because the more you make your content and your website and your product accessible to agents that can try it, you know, even like a demo environment for an agent to go test your new SaaS product, the more it will be able to learn about your product and provide recommendations to relevant, uh, prompts from human users. And then if you kind of go the next step, which is, uh, all this like reinforcement learning and environments and preference learning and things like that, then that flywheel like accelerates even faster. So I feel like it's kind of inevitable. It does kind of open up this, uh, agent experience rather than just pure user experience sort of craft within within software companies that is yet another like piece of alpha that, uh, one should jump on sooner rather than later.
Where does it all leave you as a, as a VC? We have been talking about the State of AI report, which is, uh, your annual labor of love and content and, you know, which I think everybody in the industry very much appreciates because there's so much going on. So tying everything together in one document is, uh, incredibly helpful. But you're first and foremost a VC wearing an Air Street t-shirt, as people can see if they're watching the video. But otherwise, trust me, if you're listening to this on Spotify. Very, very nice logo. Kind of retro, a little bit. Kind of, kind of.
Yeah, it's inspired from like old US Air Force.
Very nice. So what, what are you, um, excited about? So you mentioned like a bunch of like deep tech robotics. Is that, is that what you invest in? Uh, where do you think value can be built for founders and and the VCs who love them, going forward?
Yeah. Yeah. The meta thing I care about is, uh, is how do you build and make use of AI to create like new kinds of product experiences, new kinds of companies. And for me, that's like best expressed by companies that are AI-first. So that's like both in terms of the product that they build. If you rip out the AI, the thing doesn't work. But also in like how they approach their like company philosophy, the types of people they hire, where they allocate resources.
And then I've generally just tried to follow areas of industry that are, uh, increasingly ripe for getting value out of AI. So traditionally that would be, you know, lots of data for a task that they care about, not enough people to do that task, but where there's a clear ROI, uh, if that task gets automated or increasingly automated. Uh, and so that led me, you know, 10 years ago or so to first do like fintech-style investments. And then after that, you know, biology really came online, uh, into this new wave called tech bio. So I made some investments there, like Balance Discovery, that we sold to Recursion, and also we sold to Xentia. Uh, and then more recently, Profluent, which is, uh, kind of leading the charge for these language models in protein design, developing the first like, uh, CRISPR genome editor that an AI has created.
Then like another segment, uh, that really came online in the US was in defense, uh, and more recently in Europe, uh, after the Munich Security Conference in February, kind of unwound a lot of assurances that European states had for US security guarantees. And that like led to a big influx of, "Holy, we need to defend ourselves because no one's coming to save us." Uh, and so I have some investments there, like Deli and Alliance Industries in the UK and Greece. And then in robotics, as we discussed, so a team in, uh, Stuttgart called Seract, which is developing kind of these general-purpose, uh, AI models for, uh, robotic manipulation and increasingly going to other form factors.
And then I've been obsessed with voice. I think we talked actually about voice the last time I, I was here, and I'm still just like amazed at how.
The magic demo, I think you were saying, like, if you want to impress your, your smart but non-AI-literate executive friend, you show them voice.
Yeah. Yeah. Exactly. So I've definitely used our company, uh, 11 Labs, to like, uh, create, uh, audio of me speaking Korean. Like, I've A/B tested this, and apparently it sounds pretty good. But I have this, like, you know, newer company called Delpha, which is building tools for clinical trials, like starting with actually just calling back patients who want to be part of your trial and need to be consented. Uh, and these are conversations in lots of different languages. A lot of like, kind of esoteric medical terminology. You know, patients forget what drugs they were on, so they have to call you back. And this is like super laborious human work that agents like 11 Labs and others in audio, like solve really well. So I'm excited to see where this goes at the limit.
And then perhaps like the more sciency stuff, like these generative world models, I think are pretty amazing. Um, whether it's Google's, you know, Genie or VO, or Odyssey's system, or you sort of like imagining this world and then you can take actions in it, and the actions are physically plausible because the system was trained with video plus actions. Um, and then maybe taking that even into scientific discovery, um, um, for just trying to explore like the frontier and being a bit smarter with, uh, with what experiments we run, um, because now foundation models are not dumb.
Okay, fantastic. All right. So to close the conversation, um, of course, we have to go into your predictions. So, uh, each time, uh, you do the State of AI report, you boldly, uh, come up with a prediction for the next 12 months. So without going, uh, into all 10, and people can check them out mostly on slide 304, pick like, you know, maybe three that you're passionate about.
Yeah. Well, I think, um, one is just how politically charged a lot of the kind of AI compute data center buildout actually becomes because of energy, because of water, because of money, because of geopolitics. And I think that that's becoming too large of an issue for voters to ignore. Um, and so we predict that this kind of NIMBYism, not in your backyard, will kind of take precedence in, uh, in major political campaigns in 2026.
I mean, the, [clears throat] the other one that I think is, uh, is interesting is like a fully end-to-end, uh, designed or developed, uh, scientific discovery. I would honestly predict Nobel Prize, but the, the 12-month window is a little bit too short. I think the, the AlphaFold Nobel Prize is probably the fastest in history.
A Nobel Prize won by an AI.
Yeah.
Versus the recent Nobel Prizes were for like AI researchers using AI to, uh, come up with better with with breakthroughs. But that was a human powered by AI. Here, what you're talking about is an AI actually winning.
Yeah. Yeah. Uh, last year, I mean, we predicted maybe like a step towards this, which was a fully AI-written research paper would be accepted at a major conference or workshop, and that actually happened with this paper, AI Scientist V2, I think. So I think we're, we're getting there because this is what the nerds are really wanting to work on. Like, as, as a meta point, you know, I think there's all these like software industries where, uh, you know, analysts think like, oh my god, it's, it's going to be dead because of AI. But I think part of the reality is, um, what's not going to be dead is the problems that like these AI people don't want to work on because it's so boring to build that software.
[laughter]
That's such a fantastic heuristic.
Yeah. Workday is safe. Um, he was funny, like actually that CEO, because I think he said recently in response to, is OpenAI, Anthropic, or etc., etc., a threat to your business? And he just replied, "They're all my customers." [laughter]
Yeah.
All right. That's true. Uh, pick another one. I mean, it's kind of cheating, but the open source one, I think happened. You know, whether this particular company is a is a leading lab or not, um, is beside the point that, uh, basically like aligning yourself with political agendas is the way to go. And, and I think you could maybe take this even further and say, like, similar to how Nvidia has been monetizing sovereign AI, a way for nation states to kind of guarantee access to AI services is for them as nations to invest in one of these labs. There's obviously still a risk that due to export controls, the US can just like tell OpenAI to switch it off. But I think it's interesting that, for example, the Albanian government invested in Thinking Machines. Uh, obviously the CEO comes from there. And so we, I wrapped this kind of prediction or this this topic in a prediction that said, you know, some countries will basically abandon their, uh, their efforts to achieve AI sovereignty and declare AI neutrality.
It's a bit similar to like the, um, the defense posture where some nation states are just too small or don't have enough people or don't have the money, etc., with the capabilities to develop weapon systems to defend themselves. And so they have a strategic security guarantee that they get from a larger neighboring nation. I think it doesn't seem that inconceivable to me that, um, you know, various countries would say, "I can't build this stuff. I need to have a formal alliance with another country that is sovereign."
Well, Nathan, it's been wonderful. Thank you so much. The State of AI 2025, again, is available at stateof.ai. It's, uh, remarkably comprehensive and detailed, uh, yet approachable. So thank you for doing this. Thank you for coming on today, sharing predictions. Hopefully, I get to embarrass you at least a little bit, uh, for the next one when some of those predictions, uh, turn out to not have panned out. But this was wonderful. Thank you very much for the opportunity.
Thanks for running it back. Appreciate it.
Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.