Transcription
If you think cloud code has gotten dumber over the last few weeks, you were not hallucinating. In fact, a week ago, uh, Enthropic issued the statement identifying two issues that are causing quality degradation in claudris. But now they have released a detailed postmodern of the three recent issues, which I'll highly recommend to read. It has a lot more details than what you would expect from a company like Enthropic. They talk about the timeline of what those issues are when they happened. What was the severity and corresponding effect and how you should even think about your own evolves if you're building with large language models. But the bottom line is that inference at this scale is extremely hard. Tropics is not the only company OpenAI had similar issues with hosting their models. But I think with the recent news of performance degradation of cloud code, OpenAI has really capitalized on building and improving codecs and I have seen a lot of people either switching from cloud code to codeex or talking about it a lot more. Now in this video we're going to see what those issues were and what we can learn from this.
Okay, so here's kind of the rough timeline. According to Enthropic, the first major issue occurred on August 5th. The second occurred on August 25th and the third one August 26th. These seems to be independent of each other and they were identified at different stages. The last one was fixed on September 12, which is almost 5 days ago. Now before looking at those issues, here's a tweet which directly addressed the concern. They said that we would like to address a concern we have heard in the community. We never intentionally degrade model quality as a result of demand or other factors. So this is a statement directly from clone. So before looking at the issues something very interesting, Enthropic is internally using a combination of AWS Nvidia GPUs and Google TPUs. I think AWS and Google are shareholders in entropic but just thinking about the complexity of serving the same model on three different platforms is kind of nightmarish.
Okay, so let's quickly go through the issues and what we can learn from these. The first one is what they're calling context window routing error. So it seems like they were configuring sonnet 4 with 1 million context window and misouted some of the queries to this new server. This started happening in August 5th. Now we're not sure whether those were short queries or longer than 200,000 tokens. Initially it affected less than 1% but impacting almost 16% of sonnet 4 requests which is a really substantial amount. One thing you are going to learn throughout this blog post is that infra and inference is extremely hard. So if you're a back-end engineer and you have misconfigured things, don't feel too bad. Now, interestingly enough, there has been a lot of reports regarding the performance degation specifically for cloud code. Now they say that approximately 30% of quad code users who made requests during this period had at least one message routed to wrong server type resulting in degraded responses. Interestingly enough it's happening only on the entropic servers. So Mrock and Google Vortex AI customers were not really affected by this. Now the resolution that they have done is that they say that we fixed the routing logic to ensure short and long context requests were directed to correct server pools. So I would suspect that in this case probably even shorter queries were being routed here and that resulted in issues.
Now the second one is output corruption. So this is more related to how you configure your sampling parameters. So on August 25th they misconfigured cloud API TPU servers that caused an error during token generation. Now the issue was they were I think trying to optimize performance but this resulted in occasionally assigning a high probability to a token that should rarely be produced given the context. So for example, some people who were sending English prompts probably have seen Thai and Chinese characters in their outputs. This mainly affected Opus 4.1 and Opus 4 on August 25th and 28th and seems like it also had impact on Sonnet 4 as well. Again, third party platforms were not affected. Now this is supposed to be fixed by September 2nd.
Okay. The third one is what they're calling approximate top K XLA TPU miso compilation. So on August 25th they deployed a code to improve how cloud selects tokens during text generation. This change triggered a latent bug in Excel TPU compiler which has been confirmed to affect request of haiku 3.5. Now this is specifically related to TPUs and they go into a lot more details on what exactly happened here. So when an LLM generates text it calculates probability for each possible next word then randomly choose a sample from this probability distribution. So if you have uh set top P what it does is that it only considered words whose cumulative probability reaches a threshold. So let's say if you set it to.9, it will only consider tokens where commutative probabilities add up to 0.9 and then it randomly samples from those tokens. So it seems like it kind of misfired when the temperature was set to zero. It would occasionally drop the most probable token because of this compilation error in the TPU XLA. But the cause is very interesting. So their model compute next token prediction probabilities in 16 bits floating point. However, TPU process everything in natively in 20 32 bits. Right? So this has caused some mismatch. In fact, if you look at the defeating non-determinism in LLM inference article from thinking machine, they talk about this specific issue of mismatch between inference and training quantization level and that could be a contributor to nondeterministic outputs in LLMs. Although in that article they report that nondeterministic outputs are mainly caused by how we batch our samples. It's a very interesting read which I highly recommend if you are interested in understanding why these LLMs are nondeterministic or more probabilistic in nature when it's generating tokens. So here they have some examples of how they were trying to fix the bug. I am going to put a link to this if you are interested in reading more.
Now something very interesting which they highlight in their blog post is that why they were not able to capture these issues even though there were some reports in the community. So they say that our validation process ordinarily relies on benchmarks alongside safety evaluation and performance metrics. Engineering teams perform spot checks and deploy to small canary groups first. Right. So it seems like the benchmarks that they have do not actually capture issues like these. Now this is very important especially if you're building anything with LLMs or machine learning systems in general. You don't want to have static benchmarks. Your benchmarks needs to evolve based on the issues bugs that you see in production. It complicates things for them because of their internal privacy and security limits. They cannot just look at interactions that are happening with cloud. They say that we relied too heavily on noisy evaluation. Although we were aware of an increase in reports online, we lacked a clear way to connect these to each of our recent changes. So as a result of this, we're going to be producing more sensitive evolves to help discover the root causes of any given issue. We have deployed evaluation that can more reliably differentiate between working and broken implementations than quality evaluation in more places. They plan to run them continuously on true production systems to catch issues such as context window load balancing and then faster debugging tooling. So very interesting things from the entropic team.
So a few takeaways first is inference is extremely hard especially at this scale. Now if you are putting anything in production apart from your evolves that tests your system at deployment time you want to have continuous evolves and you need to update those based on the feedback that you're gathering from your users or the type of issues that you encounter in production. Now at the end I'm actually really happy that Enthropic shared this detailed timeline along with the issues that were potentially causing the degradation. Now, are these three the only reason or there is something else? We don't know. So, do let me know your thoughts. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.