Transcription
So, Anthropic released their latest model Fable 5 yesterday and I had some time to play around with it by adding in some features to an existing project just to see how well it does and also what is the credit usage that it ended up using on my project. Now, I'm going to skim through this blog post.
The first thing we'll talk about is the benchmarks, which at this point I don't know if I even trust. It seems like these models are getting tunnel vision on the software engineering bench pro that they're being trained against. And although they will perform well in these benchmarks, when you use them in your day-to-day, they seem to skip stuff. They seem to still generate code that's suboptimal, but overall it's saying that it's an 80% on the software engineering pro benchmark, which compared to GPT 5.5, which is 58.6, it's much much higher.
Now, I will say that even though these benchmarks say, for example, Opus 4.8 was 69.2%. I still have to fall back and use GPT 5.5 for a lot of stuff because for some reason even though it's scoring better, it cannot figure out these complex bugs and reason about uh issues in my codebase compared to how GP 5.5 does it. So, so you guys can trust or not trust these benchmarks based on your own opinions, but it does do better across the board on everything.
So, it says now the second chart that I think is more interesting to talk about is the accuracy versus cost. So they took the hardest 50 tasks of their 150 task benchmark and they plotted out the cost and the accuracy. Now it turns out Fable 5, which is this orange line here, is overall with the medium effort level, is still more accurate. It gets a higher score than the extra high-end max with Opus 4.8, which is pretty good. I mean, you can actually use Fable 5 on medium settings and still use less tokens and get higher accuracy than the high in the max effort levels of Opus 4.8.
Now, I will say that you don't get the main benefit until you go up to extra high or max. But if you look at the cost of max, it's almost double almost $20 compared to Opus 4.8's $10 max. Okay, so you are paying a premium almost twice as much if you were to stick to the max plans for getting 20% higher score on whatever these subset problems are, which probably don't even map to your own codebase very well. But overall, that's a chart and they're basically trying to highlight that GPT 5.5 sucks on everything. Although when I use it in practice, GPT 5.5 is actually really good for a lot of the stuff I ask it to do. So again, I don't know how accurate these charts are. I don't know if I trust these charts. Um, but that's my overall experience using these models.
Now, the very last thing I want to talk about is the credit usage. So, down here they say from today through June 22nd, Fable 5 is included on Pro Max Team and Seatbased Enterprise plans at no extra cost. So, if you want to test it and try it, you should probably do it now because that's going to expire in about 2 weeks. Right on June 23rd, we will remove Fable 5 from those plans. Using it after that will require usage credit. So, you do have to pay for the API usage, which honestly is kind of expensive for one prompt to try to add in a new feature to mission control. I think it was uh, you know, it was about $68, right? So, it's pretty expensive for one single feature and we will talk about that now.
So, that's all we're going to talk about for the blog post. Let's dive into the experiment that I ran. So, one of the things I added into Mission Control a while back was the ability to basically switch work trees and do work inside those isolated work trees. So, as I'm switching around, you'll see that every work tree has a different view of the sessions and the sessions will run inside the work tree location, which you can see here, work trees g. If I switch over to a different work tree, you'll see that it switches to work tree yolo.
But there are a lot of edge cases with this feature. For example, we need to go and be able to delete a work tree. When you delete a work tree, it needs to delete it from disk. When you run a task, it should show a toast, which you can then click to get navigated back to the work tree. So for example, if I kicked off a task on main, it should show a toast which I can click to navigate back to that work tree that finished the the task that I gave it, right? I think I prompted it hi, it's done. I click open and that doesn't work, right? Also, you can create terminals which are scoped to that work tree. So if you see over here, that is actually printing out the work tree gg location. But if I switch back to main and I try to click new terminal, it just doesn't even work. So that's also broken.
We also have the ability to run the project inside the work tree. So if I press this plus button, it's actually going to run a setup launch command that I have. So it's just going to run npm rundev. You can configure this to be whatever you want. But in terms of the work trees, when you spin up a new work tree, typically you want to initialize something. So if I go over here to configure edit project, you'll see that there's a work tree setup command here which you can specify so that when I create new work trees, it will actually run mpm install to spin up that work tree and start in installing the dependencies. So I think those are all the edge cases for how work trees kind of work.
And I will say that when I added this with Opus 4.8 and GPT 5.5, it took me around I think 10 or 20 prompts. Like it just kept on missing things over and over again. Like the inability to create new terminals, the scoping of sessions. I found that like I would go and I'd prompt and I do a long plan. I go through and let it run. I come back and stuff was just broken or not working. But with Fable, I would say that it did a pretty decent job. I'm going to read through this prompt real quick. I need to add in work tree support. There should be a way for users to create a new work tree off a main. Users need a way to delete the work trees. Users need a way to switch between work trees. All sessions should be scoped to the work tree so that if a user switches work trees to see a new set of sessions. Terminal should also be scoped to work. So here's the important one that's not working for some reason. Running the project will run the from the workree directory. So it would help if I could prompt better. All work trees must be stored in trees directory. There should be a way to configure a work tree net script so that I can tell it to run mpmi or whatever on new work trees. Only allow running one work tree at a time. Determine the best way in user experience for where we should add this ability. Do not look outside the directory or get logs for info on how to implement this.
So the last sentence I added in, I ended up doing a shallow clone of a g a commit shaw of an old version of my mission control application so that it didn't have git logs to look through because I did already implement work trees and I wanted to try to do a retroactive experiment to see how well it added it in. Okay, so get shallow clone. I told it do not look outside the directory because I didn't want to go and find my actual mission control project and find the implementation that I already have. So after running this plan, it did ask me some questions and it took a very long time to do this plan and then also I told it to implement the plan which overall I think it took around like 40ome minutes uh of API wall time to actually plan it out and implement it. So if I scroll down to the bottom we should see that it turned for some time.
Okay, so crunch for 38 minutes and 27 seconds. But actually at the bottom if you go to usage we can see how much did this thing cost and also how long did it take in total. So I'll just keep scrolling down to the usage that I ran. You can see that it took about $46 to do a single request a plan and an implementation for work trees. And then overall the API time was 46 minutes. So you can be the judge of how valuable this feature is. Was this feature worth $46? Depending on where you live and what the salary is of a software engineer where you live, you may say yes, absolutely because this would have taken me a lot longer than 40 uh how long was this? 46 minutes, right? It ends up being like a dollar per minute these things are running, which can add up because I still have to prompt this a couple more times to fix those other bugs that I talked about. And I'm sure there's other edge cases that I'll run into at some point that, you know, your initial verification, you just don't catch it.
Now, outside of cost, this thing took about half of my 5 hour session. So 51% of my five hour session. I am on the $100 a month plan. So you know I don't have as many tokens as like the $200 a month plan. But overall one prompt took 46 minutes and I'm almost completely locked out of my session. So overall you'll get about two plans/implementations for a feature that's kind of as complex as the workree stuff that I added in. And just to verify, you can see that it used basically all of Fable 5 to do this. It used a little bit of Haiku for some reason. I'm not really sure why it used Haiku. Um, but just to kind of verify that it did use the model that we had set, it used $45 of Fable 5.
All right, so that's all I want to kind of talk about. Hope you guys enjoyed this overview. Uh, let me know if there's anything else you guys want me to experiment with. I'm going to try to experiment with maybe different effort levels next to see how they perform, but you all can let me know what you guys want to see in the future. Have a good day. Happy coding.