Every single model I've tested that is supposed to be anywhere near Opus, wasn't there. There is some magic sauce in Anthropic models that these benchmarks aren't demonstrating.
A lot of these models are likely benchmaxxing the existing benchmarks that are out there. Some perhaps even by accident.
I think looking at some more niche benchmarks helps, I created one and it does show Kimi K3.1 Pro as the most intelligent of all open-weight models, though it is still significantly below Opus 5/5.5[1]
It's been suggested elsewhere that the AA-Omniscience benchmark [1] is more representative than the rest. It puts K3.1 about 6 months behind the frontier and the new version of DS 4.1-flash about 12 months. That roughly lines up with my gut feeling as well, at least when it comes to the trickier problems. Still rather impressive, but not quite frontier.
I think the secret sauce is about communication. It understands you and you understand it. Just a better coworker to work with. Benchmarks don’t measure that.
Opus 5.5 is flat out amazing, particularly how cheap it is in practice (I'm on a subscription and use Claude Code and I almost never even approach the cap).
I would love a benchmark suite that better reflects real-world coding performance, as having Muse Wonder in the top three makes no sense to me.
I spent the last 4 months on a mix of chatgpt and glm building some prototypes of product ideas I've had for years because claude was in the phase of obnoxious conversation and verbosity.
I took the output and original specs and handed them to Claude today and the level of insight and concise detail in it's criticism and problem solving was fantastic. Dropped the others immediately
Same. Every time I prompt some other new/best/benchmaxxed model, I think to myself: wow, finally, this is amazing. And then I give the same thing to Opus and the bastard is always better. It's annoying to a point.
It's not just 5.5, every iteration of the frontier moving - Opus has had some edge. I'm sure it's not 100% in every scenario, I love interacting with Codex models more, but the quality of the output, in most scenarios that I've been exposed to, has been the best.
I know the log scale of that chart is valuable when surveying the full breadth of possibilities, but I would encourage people to make decisions between nearby models based on linear scaling. Linear cost is often more useful for making the actual decision.
Would you please elaborate on "really decent"? Really decent for what use exactly? I want to love it (and use it as a cheaper research model in my harness), but I really cannot get any real development done with it? It stumbles and fumbles and over-complicates, where astra just plows ahead (often in less overall time) with excellent solutions.
I really want to love this model, particularly since it's good to have an open weight model on my team so to speak, but I think I might be holding it wrong.
I can have 2-3 sessions running at the same time for pretty much the entire working day with DSF4.1 where the responses are fast, cost is pretty negligible, and the results are good enough 95% of the time.
On an overall speed/cost/quality comparison, it seems unmatched in my experience these days.
If you have well-defined, long running tasks that are relatively disposable but requires constant ML monitoring and overwatch e.g. web crawlers, monitoring prices, logs etc. DS4 Flash is perfect for those.
For tasks like crawling and real time log analysis etc., the costs really add up at scale.
I think looking at some more niche benchmarks helps, I created one and it does show Kimi K3.1 Pro as the most intelligent of all open-weight models, though it is still significantly below Opus 5/5.5[1]
1 - https://bench.killswitch-lang.org/
1. https://artificialanalysis.ai/evaluations/omniscience
I would love a benchmark suite that better reflects real-world coding performance, as having Muse Wonder in the top three makes no sense to me.
I took the output and original specs and handed them to Claude today and the level of insight and concise detail in it's criticism and problem solving was fantastic. Dropped the others immediately
I really want to love this model, particularly since it's good to have an open weight model on my team so to speak, but I think I might be holding it wrong.
What are you using it for?
On an overall speed/cost/quality comparison, it seems unmatched in my experience these days.
For tasks like crawling and real time log analysis etc., the costs really add up at scale.