Transcription
Well, I'm not sure who saw this coming, but Quen 3 is here. I think most of us were expecting DeepSeek to drop something new, maybe their new reasoning model. But plot twist: it's not DeepSeek. It's another Chinese open-source AI model that is surprisingly good. Let's take a look.
So, first and foremost, the naming conventions for AI models are just absolutely insane. You know this; I know this. Quen 3 takes it to a whole different level. So, the flagship model—this is what we're going to be talking about a lot today—that's the Quen 3 235B A22B. So that might be super confusing. Really fast, Quen 3 is the family of models; it includes the big flagship models as well as some other faster or smaller models. The 235B refers to the number of parameters in the model; this is kind of what we think of as the size of the model. However, this model is a mixture of experts, meaning that depending on what you're asking it to do, different sorts of parts of the models can be called up to answer that question—sort of a different expert. This means that instead of engaging the entirety of the model for any given query in a given prompt, you're only engaging a part of it. So that's referred to as the activated parameters. And so the A22B—that's the activated number of parameters, 22 billion.
But let's get to the big headline here. The point is, this is competitive to the other top-tier models, such as DeepSeek R1, the 01, the 03, Mini Gro 3, and Gemini 2.5 Pro. So, with this model, you can have it do extended thinking or not. So, meaning you can think of it as a reasoning model or just a regular model; you're able to switch that on and off.
And so here's the flagship Quen 3 model; it's here on the left, and on the right side we have Gemini 2.5 Pro and OpenAI's 03 Mini. So, the Gemini 2.5 Pro is probably the one to beat; this is probably one of the best competitors across the board, so kind of a good sort of apples-to-apples comparison. The 03 Mini is another very strong one, although we now have slightly better models: we have the full 03, the 04 Mini. But notice Quen 3 beats the 03 Mini and is very close to 2.5 Pro on the Arena Hard, on the Aimeme 24, which is a high-end sort of mathematical competition for 2024. Again, it places between Gemini 2.5 Pro and 03 Mini for the Aimeme 25; same thing, it's it's it's up there, very, very close to the top. Live Code Bench: it actually beats out Gemini 2.5 Pro. Code Forces: again, beats out both Gemini 2.5 Pro and the 03 Mini. And it's very competitive, or even beating Gemini 2.5 Pro and the 03 Mini on the other tests that they've chosen to include here. And again, benchmarks are just one piece of the puzzle; you can't just look at it. Sometimes developers choose to sort of like to overtrain on it and focus a little bit more on, you know, sort of gaming these benchmarks, but it doesn't really contribute to sort of real-life use cases.
Now, additionally, there are six dense models that are going to be open-weighted. So, dense models are the opposite of mixture of experts. So, mixture of experts is sort of different parts combining to one model; dense is just one big, chunky model. And here are those models, ranging from 32 billion parameters all the way down to 6 billion. Looks like the post-trained models and the pre-trained counterparts are available on HuggingFace, Model Scope, and Kaggle. And they're hoping that release and open-sourcing of these models will significantly advance the research and deployment of large foundation models. Their goal is to empower researchers, developers, and organizations around the world. Interestingly, one of the developers on the team is saying that Quen 3 has some really intriguing features that are not written in model cards. I believe it'll open new room for both research and product. So we'll see kind of what he's referring to, and once people get their hands on it, I'm sure we're going to see what this thing is good for, what it's going to be useful for. But obviously, dropping all of the different versions of the models, the base, the post-train models, as well as—you'll notice they're going to be talking about how they have achieved some of these results—this is, of course, incredible for research and development, for advancing AI globally. Open-source AI really allows everybody to sort of move forward.
Now, some of the key features: Number one, is it supports both the thinking and non-thinking modes. So you can think of this as a thinking mode being it's sort of a reasoning model, right, so thinking through everything before putting out the final answer; and the non-thinking mode where the model just provides quick, near-instant responses. One thing that they notice is that integrating these two models greatly enhances the model's ability to implement stable and efficient thinking budget control. So the model is better able to expend more tokens thinking about a problem where it's required, like for example on a difficult problem, or it can scale back to sort of answer quickly without thinking. So here are the performances on Aimeme 24 and 25, and this below that's sort of the thinking budget in thousands of tokens, right, so 1,000 tokens to 32,000 tokens. So as you can see here, the red line is the non-thinking mode; it has only sort of one result across the board. However, with the thinking mode, as we allow it to think deeper and deeper about the problem, as you can see here, I mean all the way up to 16,000 tokens, we're seeing a pretty sharp improvement. So the more it thinks about solving the problems, the better it gets. At 16,000 tokens, it's maybe 85% or thereabouts, and even slightly more at 32,000. Same thing kind of on the AIM 25, Live Code Bench; also we're seeing pretty smooth improvement upwards, and the GPQA diamond. So after 2,000 and more tokens it sort of like really improves. It has 119 languages and dialects and improved agentic capabilities. At the end, I'll play a quick—I think it's like a one-minute video or under one minute—that shows some of these things from the Quen 3 team. But the point is they've improved its abilities for coding, for agent capabilities, and also strengthened the support of MCP—so that was kind of Anthropic's um, model context protocol for interacting with various software tools.
The data set for Quen 3 has been significantly expanded compared to the Quen 2.5. Quen 2.5 was pre-trained on 18 trillion tokens, and this model, Quen 3, uses nearly twice that amount. This comes from the web, but also PDF-like documents. Interestingly, they use Quen 2.5VL to extract the text and then Quen 2.5 to improve the quality of the extracted content. So they're using the previous models to kind of gather the data, to kind of filter through it, to improve the quality of it, and to increase the amount of math and code data. They've used Quen 2.5 Math and Quen 2.5 Coder to generate synthetic data; this includes textbooks, question-answer pairs, and code snippets. So this follows with that idea that we've been talking about is that each sort of generation of models is then used to help build the next generation of AI models, making them better with each iteration.
And there are three stages to the pre-training: S1, the first stage, it was pre-trained on over 30 trillion tokens with a context length of 4,000 tokens. This gives it basic language skills and general knowledge. In the second stage, we improved it by increasing the proportion of knowledge-intensive data: STEM, right, math, engineering stuff like that, coding, reasoning tasks, etc. The model was then pre-trained on an additional 5 trillion tokens. In the final stage, we used high-quality long context data to extend the context length to 32 trillion tokens. All right, here's the post-training. Right, so you got the base models that we have. Stage one is long chain of thought cold start. So chain of thought is its ability to think through the question before answering. So if I understand correctly, the long chain of thought cold start is basically where they give it a small amount of examples of how to reason through stuff that kind of sort of kickstarts it, like cold start its ability to do reasoning. Stage two is reading reasoning reinforcement learning, right, and so when it's getting the correct answers, right, we're sort of positively reinforcing it, etc. Stage three is the thinking mode fusion. So again, since it has two modes, the thinking and the non-thinking, stage three is kind of the fusion of the two. And stage four is general reinforcement learning. And of course, what's produced is the big model, the 235B A22B. I, that's a crazy model name, but okay. And the Quen 3 32 billion parameter—so this is the dense model—and this is the model mixture of experts. And then for the lightweight models, so they're using strong-to-weak distillation. So basically they're taking the outputs of these models and they're using that as synthetic data to train these smaller, lightweight models. And what we've seen from past research is that produces, you know, very small, inexpensive, fast models that are can be very, very good in their own sort of area of expertise. So the bigger models are sort of the teacher models; these are the student models. And so you get these which are significantly smaller, right, much faster, but probably retain a lot of the abilities, right. So we lose a little bit of the ability, but it's still it's still good, but with the benefit of likely being able to be run much faster. Some of the smaller ones can be used on edge devices, on on phones, etc.
So here they briefly kind of talk about the first stage: they fine-tune the models using diverse long chain of thought data. The process aimed to equip the model with fundamental reasoning abilities. The second stage focused on scaling up computational resources for reinforcement learning, utilizing rule-based rewards to enhance the model's exploration and exploitation capabilities. So we're trying to get it to develop its own problem-solving skills, and the goal is for it to sort of try a bunch of different stuff out, figure out what works, what doesn't; also like when it's time to get the right answer, you know, figure out what strategies get the right answer. So if you've seen my videos where we train that little snake to play, you know, the snake game, a simple game of snake, you know, while we're training it, we're like just just go for it, do whatever you want, just try stuff out, see what works, right, explore. And then when it's finalized with training, we're like, okay, now it's game time, now don't mess around, just do the stuff that works, go. So interestingly, DeepSeek kind of came up with this idea; they called it the GRPO, the group relative policy optimization, which allowed for much more efficient reinforcement learning training by kind of skipping the critic model that critiques the outputs and then does kind of reinforcement learning, kind of positive or negative reinforcement. Instead, it kind of estimates the baseline from group scores, reducing the computational costs. Thanks to Hassan789 on Reddit for this write-up. We'll go deeper into what the GRPO is, probably when DeepSeek drops their model. I don't know, tomorrow, I guess. I mean, it's probably going to be all this week, which is kind of crazy to think about. But if I'm reading this correctly, they did not go with the same sort of approach that DeepSeek, the company, went with. And then in the third stage, they integrated the non-thinking capabilities into the thinking model by fine-tuning it on a combination of long chain of thought data and commonly used instruction tuning data. This data generated by the enhanced thinking model from the second stage, ensuring a seamless blend of reasoning and quick response capabilities. So it almost seems like they're building parts of this model in like a parallel and then like fusing them together into one. And then what's I think is more important is they're laying it all out here in these blog posts, and also they're going to have a paper about this; they're going to put out, put out. I haven't seen it yet, so I'm not sure if it's out currently. But everything that they've developed, any sort of a new trick or strategy that they've discovered, they're going to be sharing it with the world, probably most of it, if not all of it, allowing other people to replicate the results to continue this sort of cycle of innovation, which is phenomenal. And then finally, we have sort of the general reinforcement learning in order for it to be get better at instruction following, format following, agent capabilities, etc. And interestingly, at the very end of the blog post, they're saying, "We believe we are transitioning from an era focused on training models to one centered on training agents. Our next iteration promises to bring meaningful advancements to everyone's work and life."
So really fast, we're going to take a look at this video that they provided that is showing the improved agentic capabilities. But it seems like they're really doubling down on the whole open-source ecosystem, on providing various open-source resources, not just for people to use, but also for researchers, for people building products with it. This whole thing is running on the Apache 2.0 license; you're able to use it for commercial purposes. So if you're building something on top of this, like a business, you are more than welcome to do it. I quickly did a Google search just to make sure I get everything, so it's permissive commercial use; you can modify and distribute sort of the outputs of it; you can you can create derivative works, so you can build something on top of it, sell that as your own; you have to have the proper attribution, but you don't have to have your derivative works to be licensed under the same terms. So let me know what you think about this. Do you think DeepS, now that it's releasing, is it going to be even better, is going to be even one step above this? But whatever the case is, let me know what you think about this whole thing, and let's take a look at this agentic capabilities video. Hey, hey, hey. I feel...