I noticed Fable decline in quality a couple weeks ago. And I've noticed Opus drop over the last week, even after bumping to a default of Extra High reasoning for almost everything.
Tonight has been maddening with the kinds of absurd mistakes it's repeatedly making. It's so frustrating.
I'm the kind of person that wants something to be consistent and reliable. And Anthropic has been great, except for in this way. Which is enough of a recurring frustration (about every couple months this year) that I genuinely can't wait for other options to catch up. Then I'll be leaving as soon as I can.
I've been using Fable for a couple months as a relatively independent, long-horizon task executor and it worked well. About two weeks ago it completely stopped working. Like ~85% success rate down to ~10% in my workflows.
Totally possible that I'm just hallucinating/projecting/etc or my workflows somehow changed in terms of difficulty, but it's been kind of shocking.
I have the same exact experience. Working on one project, brainstorming. It was brilliant in July and August, like a seriously smarter, more experienced version of myself, reading in my mind, one step ahead of me, when pointed out a mistake it would remedy on the spot with the exact brilliant solution.
Now it's like talking to my hungover self, after 3 days bender, who just wanted to be left alone.
Same for me, especially the last couple of days. It was really frustrating and it felt like Claude was sick of working with me and just didn’t give a f*ck anymore
I read somewhere that the test might switch models mid conversation which stops the other from reading the previous model's reasoning. I don't know if that's true, but it sure feels like that quite often in the last days.
It blows my mind how people still build their workflows around depending on services like these, with how easy it is to just self host and forget about all of this. You can buy a GPU that can run Qwen 3.8 flash next for about 9 months worth of the highest tier Claude Code subscription. You'll never have to worry about token limits, the model changing out from under you, getting banned, etc.
Superior? Sure. But do people need the capabilities beyond what can be reasonably achieved with local models? I think the people that actually sit in that gap are vanishingly small, and if your title contains "Engineer" or "Developer" or something like that, you are not one of those people (at least for doing your job)
As for faster, maybe? It depends on the metrics and what you're doing with it. If you mean TPS, no, I get much higher TPS and even faster TTFT on my local setup than frontier models. If you mean time to complete tasks... maybe. If you're throwing whole projects at it to vibecode, yeah, a frontier model will dump slop so fast it'll make your head spin. If you're doing actual methodical development, like doing spec-driven development and actually reading the specs before spinning off agents to implement them, rather than just having another agent review it and yolo, you'll find that frontier models don't really push the needle that much, and you'll be making high-quality projects.
i dont understand how people keep comparing local models with frontier models, they dont belong in the same discussion especially for coding usecases. they solve different problems under different constraints.
Compared to Opus local models Opus 5.5 is far superior, also for something like a dgx spark at 5k that would be 25 months ignoring power costs if you chose the $200 plan.
And that’s looking at a large model, to host the equivalent sized model as Opus 5.5 you’d need more than a single dgx spark. A fairer comparison would be to Open models hosted remotely or comparable weak models like Luna which you would get away with a $100 plan or less. That also doesn’t account for token prices continuing to go down for the same level of intelligence.
I can buy a GPU that can run Qwen 3.8 Flash Next for 9 months worth of the highest tier Claude Code subscription, but then I'm running Qwen 3.8 Flash Next when I could have had nine months of unlimited Opus 5.5.
> then I'm running Qwen 3.8 Flash Next when I could have had nine months of unlimited Opus 5.5.
By choice, maybe. New models release faster than every nine months. And, evidently, you won't be running nine months of some fixed meaning of "Opus 5.5". What that means changes over time, out of your control, and without ever even being officially told.
If Anthropic indeed nerfed Opus 5.5 on November 3, it could be because they wanted to be prepared for inference cost going up due to the data center moratoriums and the election results. I really hope this is false, but I don't see how inference cost can go down in the near future.
I noticed the same, and posted about it three days ago.
Opus is suddenly fumbling basic tool calls and failing to finish tasks even at high, where I had been permanently using it on low effort with success since it came out. It is very frustrating.
My guess is they quantized it and/or reallocated DC capacity to train a new model, this has happened before.
Not just frustrating, it can also be dangerous. What if you trust Claude with certain security-sensitive things (because experience), and suddenly you cannot trust it anymore like that?
The reverse happen with Deepseek. Getting smarter and faster especially if you dont mind the China API. Gemini Argon getting dumber compare to older 3.8 and significantly slower too.
Some teams I know started moving all linting and validation into LLMs and markdown files, the reasoning was that they asked agent to review changes that were later found to be buggy and agents flagged these bugs. And that’s great but my immediate worry was that: it finds it now, but will you check every new model for all these bugs? And what if you need to switch providers? Or your provider silently tweaks the model to improve their margins? The answer was “well also build deterministic linters for all these problems”, but you know that’s just wishful thinking - if you can get 99.9% done with just a little bit of fuzzy text and the next 0.1% is a lot of work it won’t happen. So you have model behaviour lock-in that will bite you suddenly when you least expect it.
Or maybe LLMs will just keep getting better and cheaper? It’s all a casino right now.
We need a consistent way to tell level of service for LLM services. Without they can play with us and steal money by providing different/quantized models than we paid for. I believe this may be a good next AI company if they will be reliable.
I moved from Claude to Codex and am happier. The prose is less grating and results comparable. As much as I don't like Altman I am tired of Anthropic's antics.
I tried this and while the prose is indeed snappier I find the code quality of Codex truly horrific. It's dense and uninterpretable. I almost wonder if they have made it purposefully opaque so that it's harder to resume work without Codex once you're using it agentically. Claude is still much better at writing quality code that is fit for human consumption.
I don't think Astra's code is quite as bad as you say, but yeah while it's much better at non-annoying prose, the code definitely isn't quite as good as Opus 5.5.
Nope, the issue was not the 5 h limit, but that the 5 hour limit was gone in some minutes and that this burned trough 50% of my weeks fable limit. Before I could do roughly one full 5h session (fable till max) per day.
They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.
Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.
I'm not sure how you're arriving at "if you use the same seed with the same version of the software on the same hardware with the same drivers you get the same result is an exaggeration", it's wild that you're a teacher and you seemingly do not know this.
Yes. That was the weekend my token usage blew up and used up my weekly allowance in a matter of hours. It was so stark.
After I figured out, like this chap on X, that it wasn't ME, I diversified my model usage and now a lot more work has shifted to Astra/Sol and I'm learning to make my workflows more resistant to the whims of each model.
Definitely a significant change. I noticed it in just daily work. Quality of work down, mistakes up, repetitive errors in scripts - rare earlier, now a few every session.
> I noticed this too as did many others. Had to bump my default reasoning but it’s not the same. Seems to often happen before new model drops
I heard that Qwen4 was dropping some time soon, I wonder if Anthropic is trying to announce a new model immediately after Qwen4 - both for their IPO, and to kill the competition.
This has happened to consistently across OpenAI and Anthropic models that I can't believe people are still denying it. Obviously they're silently quantising their models or making other efficiency changes. If they can save 40% RAM and only hurt intelligence by a few percent in their benchmarks, they're going to do that. Unfortunately their benchmarks are obviously not representative of the real world, and these downgrades are very noticeable for people who rely on these models for work.
Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.
Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.
I guess the question really is: is this malice (trying to trick people into thinking new model is better by nerfing old one), or is this capacity constraints, where they obviously need to make room in their infra for inference of the new model by shoving the other models into a smaller box?
People are denying it because it's simply not true and for sure has not happened consistently.
In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.
Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.
When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.
I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.
I guess it's just capacity problems. They probably have a fixed amount of compute and they need to make space for new models that will be released. So they quantize existing models e.g. in half so that can use 50% of current capacity for new models. They have such massive scale that you just cannot dynamically get that amount of new compute online so they need to work with what they have. No provider has thousands of GPUs idle.
You can still control it if you can trivially spin up a node with eight B200s and 2TB of memory to run your Deepseek v4 instance. Nobody on a subsidized plans is doing that. Local models that fit 128GB memory don't come close even to a "nerfed" frontier model.
Actually Qwen3.8-flash-next comes very close to a "nerfed frontier model" the way I host it locally at 4.75bpw with int8 kv. It is pretty much equivalent to opus 4.6 up to 4.8 on coding. What it lacks is some factual knowledge - this is to be expected, it's a far smaller model. But if you run the kind of coding tasks where you have to spec them in high detail anyway (large very custom environment) this model is very usable.
I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.
I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.
Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.
The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.
I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.
But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.
I've been very impressed by it and recommended it to everyone. 32gb ram, 12gb 3080ti, and I run 3.8 flash next coder (half the experts ripped out, leaving in coding) int8 kv 128k context at 35-55tk/s and I agree with the opus comparison; I personally say 4.7 is what it seems like the most.
I've seen big drops in Opus 5.5 (on high/x-high) & Fable. Both in terms of following instructions / workflows execution and overall code quality. Also the token burn-rate increased a lot, I've reached limits of 3 Ultra subscriptions during last week & had to spend 1.5k on API usage during the weekend just to be frustrated with the stupidity of the output.
I am using Opus 5.5 high almost exclusively. I definitely felt this and looked for nerfing watch sites. It felt like downgrading to the old version of deepseek. Things have improved a bit now.
Hm, I’ve seen a degradation of 5.5 yesterday. I’ve had a session running for around 50 days and it just started to generate garbage and ignored all rules. A fresh one feels the same as always
this is the reason i cancelled claude after 4.6, non-determinism aside, anthropic just doesn't seem care about maintaining sane baseline for their products
Anecdotal, but I tend to get by with medium thinking on Opus and I noticed that as of sometime last week it really did feel like it had become noticeably slower to think about things.
I wouldn't say I've especially noticed its effectiveness having been reduced, it's just slower.
LLMs are such a great business, they are non-deterministic, hence you can do whatever you want, because customers can't do anything about it:
* nerf them based on supply and demand, when you have a high demand, nerf it, but keep charging same price.
* do you want to show revenue growth to investors? increase thinking tokens, customers will keep paying, because again, this is non-deterministic system, sometimes it thinks more
* do you want to increase profits? start serving 4-bit, 5-bit quantizations and keep charging same price
* want even more? publish a paper "Advanced Auto Super Smart Prompt Router - AASSPiRation", and start routing requests to Qwen3.8-27B or GemmaE2B from time to time on the edge
* Not enough? introduce "Complex Chained Workflows - CCW" in your coding harness, which spawns multiple deterministic workflows (which you can't stop or control) and they will use Haiku to write the code, then use Opus 5.5 high to verify the code (which is obviously wrong) and Opus 5.5 produces recommendations on how to fix the code, delegates to Haiku, which again fails writing proper code, and keep doing it 5-6 times, you made lots and lots of money (don't forget, all should run in parallel, to pick one best result among 10 outputs).
Eagerly awaiting Artificial Analysis to redo their Anthropic benchmarks and kick it down 15-20 index points, in the interests of accuracy. Those guys love accuracy. They moved Heaven and Earth to kick Kimi off the #2 spot, out of their deep, abiding passion for having the most accurate benchmarks in the industry.
I have no doubt they will ensure their users get an accurate picture of Opus 5.5 within hours.
IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.
The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's language from the EU "dual quality" food scandal, which saw food brands (Coca-Cola was one) adulterate their recipes to use cheaper ingredients in select markets—though the external packaging was kept the same.
If I were a cynic I’d also say that it wouldn’t be particularly difficult for them to detect when a benchmark/eval context is active and to route those requests to larger models or higher quants.
Each new model version hits your dopamine receptors very hard when it first comes out, but in the nature of things it stops hitting after a while and you need novel models to fire them off again. That's why people think the models are getting nerfed after a couple of weeks; it's actually their endocrine system getting nerfed.
This'll likely be the reality moving forward - they allocate more compute at launch to get good test scores, reviews, and news coverge, they get people excited and used to it, then they cut costs.
I'm reminded of the video game industry where they would launch a game without any in-game purchases at launch, get the reviews to make note of it and give it a higher rating because of it... then add in-game purchases anyway. I think that stopped again though.
My paranoia tells me they A/B test users which means consistent benchmarks go through the high performance pipeline while regular requests fall back to the quantized / experimental / low cost version. I can see the main reason being the oldest one in the book, which is making more money.
It could also be a regional effect where systems are under higher load and hence dynamically routing to / provisioning lower precision instances; or some other factor where one is affected whereas someone else isn't.
Is there a practical reason this link uses twitter.com rather than x.com, or is it a nod to the old name? If it’s the latter, I don’t really see the point.
Tonight has been maddening with the kinds of absurd mistakes it's repeatedly making. It's so frustrating.
I'm the kind of person that wants something to be consistent and reliable. And Anthropic has been great, except for in this way. Which is enough of a recurring frustration (about every couple months this year) that I genuinely can't wait for other options to catch up. Then I'll be leaving as soon as I can.
I just don't know how long that will be.
Totally possible that I'm just hallucinating/projecting/etc or my workflows somehow changed in terms of difficulty, but it's been kind of shocking.
Now it's like talking to my hungover self, after 3 days bender, who just wanted to be left alone.
https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
As for faster, maybe? It depends on the metrics and what you're doing with it. If you mean TPS, no, I get much higher TPS and even faster TTFT on my local setup than frontier models. If you mean time to complete tasks... maybe. If you're throwing whole projects at it to vibecode, yeah, a frontier model will dump slop so fast it'll make your head spin. If you're doing actual methodical development, like doing spec-driven development and actually reading the specs before spinning off agents to implement them, rather than just having another agent review it and yolo, you'll find that frontier models don't really push the needle that much, and you'll be making high-quality projects.
And that’s looking at a large model, to host the equivalent sized model as Opus 5.5 you’d need more than a single dgx spark. A fairer comparison would be to Open models hosted remotely or comparable weak models like Luna which you would get away with a $100 plan or less. That also doesn’t account for token prices continuing to go down for the same level of intelligence.
Personally I was forced to use only GPT-5.6 Sol at work instead of Opus 5.5, and it was so much worse.
Benchmarks that control for benchmaxxing indicate all open-weight models behind 5.6 Sol and roughly at the level of GPT-5.5.
And that's for the latest large ones, much less local Qwen 3.8 Flash Next.
I would regret such a downgrade.
By choice, maybe. New models release faster than every nine months. And, evidently, you won't be running nine months of some fixed meaning of "Opus 5.5". What that means changes over time, out of your control, and without ever even being officially told.
Now, if GPUs didn't cost an arm and a leg, the economics would have been different.
Opus is suddenly fumbling basic tool calls and failing to finish tasks even at high, where I had been permanently using it on low effort with success since it came out. It is very frustrating.
My guess is they quantized it and/or reallocated DC capacity to train a new model, this has happened before.
Not just frustrating, it can also be dangerous. What if you trust Claude with certain security-sensitive things (because experience), and suddenly you cannot trust it anymore like that?
then $corp should remove your access to these security-sensitive things ;)
I'm so happy to not look at bogus jargon that Claude made up.
This is the #1 thing i HATE with LLMs. They are not predictable like a compiler is.
6 lines per model? Can't you put that 6 lines in a function/macro/decorator/library so you only have to call it?
After I figured out, like this chap on X, that it wasn't ME, I diversified my model usage and now a lot more work has shifted to Astra/Sol and I'm learning to make my workflows more resistant to the whims of each model.
> I noticed this too as did many others. Had to bump my default reasoning but it’s not the same. Seems to often happen before new model drops
I heard that Qwen4 was dropping some time soon, I wonder if Anthropic is trying to announce a new model immediately after Qwen4 - both for their IPO, and to kill the competition.
Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.
Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.
In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.
Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.
When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.
I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.
Specifically
> YOU get to decide when the model changes
I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.
I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.
Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.
The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.
I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.
But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.
Also, how much power does that consume at full tilt ?
Truly insane times
That's around 1% of the price of 8x B200. At current prices, it's as low as $6200. At last year's prices that's $3700 or so.
Weirdly enough, Sonnet quality seems unchanged.
You rewarded Anthropic's failure with more dollar spend. I get why you did it but...
I wouldn't say I've especially noticed its effectiveness having been reduced, it's just slower.
If they did, there's absolutely no way that remains a secret.
If they didn't, Anthropic would be making their own service noticeably shitter than AWS, which I can't believe.
* nerf them based on supply and demand, when you have a high demand, nerf it, but keep charging same price.
* do you want to show revenue growth to investors? increase thinking tokens, customers will keep paying, because again, this is non-deterministic system, sometimes it thinks more
* do you want to increase profits? start serving 4-bit, 5-bit quantizations and keep charging same price
* want even more? publish a paper "Advanced Auto Super Smart Prompt Router - AASSPiRation", and start routing requests to Qwen3.8-27B or GemmaE2B from time to time on the edge
* Not enough? introduce "Complex Chained Workflows - CCW" in your coding harness, which spawns multiple deterministic workflows (which you can't stop or control) and they will use Haiku to write the code, then use Opus 5.5 high to verify the code (which is obviously wrong) and Opus 5.5 produces recommendations on how to fix the code, delegates to Haiku, which again fails writing proper code, and keep doing it 5-6 times, you made lots and lots of money (don't forget, all should run in parallel, to pick one best result among 10 outputs).
I have no doubt they will ensure their users get an accurate picture of Opus 5.5 within hours.
The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's language from the EU "dual quality" food scandal, which saw food brands (Coca-Cola was one) adulterate their recipes to use cheaper ingredients in select markets—though the external packaging was kept the same.
https://ec.europa.eu/commission/presscorner/detail/en/qanda_...
(I suppose Coca-Cola's CEO would prefer we talk about what they did as "A/B testing").
I'm reminded of the video game industry where they would launch a game without any in-game purchases at launch, get the reviews to make note of it and give it a higher rating because of it... then add in-game purchases anyway. I think that stopped again though.
That would be the less bad reason.
https://github.com/ninjahawk/livenerf
I wonder as well whether would this pop up on nerfbench
This company does not deserve to be successful and has no moral high-ground.