The cost per task chart is telling me that I should _never_ use Sonnet 5 above medium effort level - Opus always performs better for a given cost. So I guess the takeaway is that if Sonnet 5 medium isn't good enough for you, switch models, not effort levels.
They're actively trying to use lobbying power to make open weight models illegal. So I'm just not going to use their services at all anymore. I don't think they're a net gain if you're a skilled senior, and the hidden cost in terms of technical debt and skill atrophy is just being swept under the rug. I'll be okay without their bullshit generator.
> I don't think they're a net gain if you're a skilled senior
I'm a skilled senior (I'm 54 and been coding since I was about 8; I've been 100% AI-generated code for at least 6 months now and have produced a combination of speed and quality that has astonished me; my velocity is apparent at https://github.com/pmarreck/) and this has been a massive net gain, so your claim is now officially in sheer defiance of reality.
In a skilled senior's hands, this is like an expert power tool. In the hands of someone less-skilled, it is likely also... less-skilled. It's a magnifier.
> and the hidden cost in terms of technical debt and skill atrophy is just being swept under the rug.
Nope, no it's not. It's being reviewed, measured, and controlled against. Because... you WILL need more controls to take full advantage. Look, I even invented a whole new control methodology around it called MFIC: https://gist.github.com/pmarreck/b30aa3ca69cb70a5526f8a63ab8...
I think I might have written a comment similar to yours maybe 6 months or a year ago. I'm not quite sure to respond to these sorts of replies. I have used LLMs/Claude Code quite extensively professionally and was a very early adopter, have built tooling around LLM/agentic development, and genuinely embraced it. They aren't useless, but the short term gains you think you're getting come at a very steep price that you may not actually account for consciously for quite some time, if ever.
I think the uncomfortable debate is not about skill atrophy as a general phenomenon (it’s happening anyway, doesn’t matter how much we debate it) but rather, _which_ skills are atrophying and if these skills are now superfluous/worthless or not.
If you don’t use a skill, it’s like a gene a species doesn’t need anymore, it will atrophy.
Is that bad and if yes, why? Skill atrophy is not intrinsically bad. I don’t know how to make tinted glas for church windows and I will never learn it because there are machines doing it now.
But I would for example think that critical thinking would be a catastrophic skill atrophy. As far as I know, there is no proven link though (and one would have to define what is “critical thinking” in the first place). Writing assembler without any autocomplete, I’m not so sure it’s such a problematic skill atrophy.
One could argue that the cumulative atrophy of skill around writing CPU assembly has been problematic in some respects, but it’s also completely unreasonable to lament what we’ve gained in return.
As far as I’m concerned, so long as we can be happy with AI we can run locally, AI is no different to the rise of scripting languages or the pocket calculator. It’s only problematic if the calculator is rented to you as a service.
Hence only let your skill atrophy to the extent where if all you had were your local laptop you can still be competent. Relying on paid subscription services for your skill is a fool's errand.
It’s not one single skill being lost, it’s about many and how they interact.
I just did a big refactor with opus, it went ok, some bugs. The normal stuff. One of the bugs was in a part of the code no longer needed, which Opus had just filled with comments more or less.
Asking it fix the bug worked, but then I really looked at the code and realized just that, this is pointless now.
I’ve only been coding for 20+ years so I might be more susceptible than the author, but I’m quite terrified about losing skills in writing code, but also designing good structure, coherency and system overview. These are the things people claim you need more of with LLMs, but is what you outsource the most, even if you think you are describing it in detail.
We are all collectively growing the skill of complacency and laziness though, and those are not great ”skills” to have. And I’m just as guilty as anyone.
Since compilers became a thing Assembly language knowledge atrophied[1] across the workforce.
Since automatic memory management became a thing memory management and pointers knowledge atrophied[1] across the workforce (although not nearly to the same degree).
I think the pattern here is that compilers almost always output better machine code than humans, automatic memory management doesn't output better machine code than skilled humans can very (especially with modern languages that give you a lot build-time safety checks).
And even then, there is still demand for assembly knowledge in the workforce, it is just very niche.
I don't think LLMs will ever be good enough to "almost always" output better code than humans. But, like automatic memory management, it will likely make some types of programming more niche.
The key thing here is that compilers are deterministic, deterministic tools have way less variance in output quality. Automatic memory management is not as deterministic as a compiler because it happens at runtime. LLMs output build-time code, but the can be drastically different if I sneeze too hard.
[1]: as in % of the workforce, not absolute numbers. Hard to get exact figures on this, but I think we have more experienced people actively using Assembly today than we had before compilers became the default (late 80s). We probably have more active C/C++ programmers today than before Java became popular (early 2000s).
I might have written a comment similar to yours maybe a year ago.
Yes, some skills will atrophy, but the learning curve for LLMs is also steep and you will acquire new skills that will pay off the costs many times over.
We see this in discussions like these where you have people running the gamut from using them as glorified auto-complete or babysitting them (usually a net loss in speed, though it'll feel less draining) via people running multiple agents in several different tabs (a gain) to people prompting for harnesses rather than tasks, and putting the agent in the resulting harness (where the multipliers come in) and even people at the peak of experience with them today are only scratching the surface.
I'm very aware that just as my assembler skills are not what they once were, my skills in the languages I'm now writing less will not be what they once were a few years down this line.
But I produced far more before I started using LLMs through the force multiplier of modern languages and frameworks than I did in assembler in the 80's and 90's, and I produce far more now with LLMs, and I will produce even more in the future by learning how to take advantage of new capabilities.
I have Claude refining a system that wasn't tractable a few years ago in another terminal as I'm writing this. I don't care if it would take me a bit longer to get back up to speed on a C codebase again if I was stripped of all access to LLMs any more than I care if it'd take me a bit longer to get up to speed on programming assembler on a Commodore 64.
I get what you are saying, but how can we be talking about skill atrophy when our main skill is changing from being able to produce code ourselves to being able to leverage LLMs to write that code.
At the end of the day there are goals achieved with coding. Coding is a tool to reach either your business needs or some personal aspiration.
When it comes to businesses I don't think a business cares if you used the best stack possible, or you've written it in assembly, as long as it works.
Judging from the biggest coding drivers out there, most of the code produced globally and the biggest apps out there have had skilled engineers writing code but its not always perfect. As long as it works. Lets not forget that the web is build in php and js.
So again my argument is that, are you atrophying a skill that is going to exist in the next 1 to 2 years, or is everything going to shift towards LLM code writting.
Personally I think that LLM code writing is the winner, whether we like it or not, it accelerates business objectives, which at the end of the day its what is the deciding factor.
And yes I do miss the days I was writing code and I was solving complex problems myself.
> At the end of the day there are goals achieved with coding. Coding is a tool to reach either your business needs or some personal aspiration.
This is your opinion and I even share it, but there are many people here for whom writing the code was/is the whole deal. You would not have languages and heck - even editors! - holy wars otherwise.
Yes for sure there are people that writing code is the whole deal.
How many of them are driving goals and businesses though?
Like if we take for example an ex coder like Gabe, someone from his team comes up and says to him "we can launch this game in 1 year by using LLMs, codebase will be meh/okish" or "we can launch this game in 4 years we'll hire the top engineers and write the best software piece ever, a technical novelty".
I already know his answer... even if his answer was the 4 years which won't be, his board would disagree.
So yes there are people that love and enjoy writing software, but the truth is that business is leading software, not the other way around.
For me personally nowadays I don't need languages, I am so deep into coding using LLM's, not vibing, that I don't really mind on what is written.
Also we are just at the start of this thing.
I am consistently underwhelmed by the output, I can't really explain it besides LLMs have no taste about what is easier to read for humans.
I usually start a task with an LLM and then do small refactors using the LLM and then do some manual refactors before I am done. But often for more complex tasks the manual refactors are quite large.
Maybe it is because they can read walls of text so easily, so they output walls of text that are hard to read for humans.
I feel quite sad because a lot of my fellow colleagues are not putting this extra effort in to make things easier to understand by humans. PR review is basically me just doing this extra effort for them and their LLM implementing my comments.
And that is when I can even pinpoint the bad taste in the code structure, sometimes it is not something you can easily describe in a PR comment besides "no human would structure the code like this".
It’s hard to know if your experience is relevant from 6 months to year ago. The models are getting better every couple of months. My current experience is mostly like the other senior above. For me the last 4 months I’ve gone from mostly writing code by hand to writing almost no code by hand. I guide the LLM and it’s a force multiplier. I review its code and discuss with it how to test and what needs changing and it does it. I point out things it didn’t handle and it handles them.
Uh, I didn't say I stopped using LLM's 6mos to a year ago. I have to for my job, it was just that long ago that I began to understand that they aren't what they seem and definitely aren't a "force multiplier," more like a "debt generator."
Technical debt due to accumulated excessively verbose, badly architected, often redundant, feature-bloated code which always looks good, even upon earnest review, but actually sucks and becomes extremely difficult to maintain in ways which are not obvious in code review. The issue is this: your tooling can help, and can make you feel better, and you might think you wrote all the prompts and made all the tools to mitigate these issues, but you haven't. If you're not consistently seeing it generate code that is very very close to the way a skilled senior dev such as yourself would have done it (with similar line count, etc), that is a red flag even if the code looks great and works.
> ...badly architected, often redundant, feature-bloated code which always looks good, even upon earnest review, but actually sucks and becomes extremely difficult to maintain in ways which are not obvious in code review.
I can only judge from my own experience but with or without LLMs, these are the codebases that I have worked with during most of my career. To me, much of the question is whether LLMs produce worse code than the me and my colleagues have done in the past and I don't think that's the case. It is however very common that people hold LLMs to a higher standard than human colleagues and then it's not a useful comparison.
LLMs can produce bad code in ways no human ever would. I remember working on headscratcher problem of walking a tree while gradually building up state, and since LLMs are supposed to be good at algorithms, I asked Claude.
It came up with a correct LC-hard tier solution that involved dynamic programming, and was essentially an unreadable dense mess that was impossible to reason through as a human.
It worked, but it was so bad, that I sat down and realized after a bit that with maintaining a small cache, and being very particular about how the nodes are traversed, I reduced the solution to like a 10 line modified DFS, that I could understand.
I’ e written bad code and I’ve reviewed bad code by other humans. In both of those cases the code gets re-written. In the first the reviewer tells me the code is hard to understand, can it please refactor. In the 2nd I tell the person I’m reviewing the same thing, maybe with some suggestions.
I do the same with the LLM. I tell it that solution is convoluted and hard to understand, if I have a concrete suggestion I suggest one, otherwise I ask it for ideas. We get there just like I do with humans
In my experience, even if the code is not worse, there is so much more of it that people quickly stop understanding what is going on. Now you could argue „the LLM knows“, but that comes at an insane token cost, even if it works well.
This is in the interest of big AI companies: if they quasi-monopolize the skills entire sectors of the economy need in order to function, that will be great (for them).
Everyone keeps comparing this to compilers, but I don’t need a multiple-hundred dollar subscription to use LLVM. And people didn’t stop understanding how computers work either, just because they used C. And yeah, maybe local LLMs will become the norm, and I hope so. But market forces (hardware prices) certainly are working against that right now.
Yeah but I think going back to hand writing bespoke code is not coming back, the genie is out of the bottle.
But we could build much better tooling around keeping the agents honest. The problems you are describing are absolutely real and I see them every they.
One friend of mine had almost a mental breakdown when he just went ahead and drilled a bug producing Claude to the point that it itself admitted it was “a piece of shit”. He knew that arguing with an LLM agent is more than useless, but it was cathartic for sure.
When I encounter a situation like this I always go down to - have I done everything I could to catch these errors in my automated validation, and update it as needed.
Agents are also more than happy to spend tokens refactoring, once you have such a test harness be good enough, producing successively better and more general abstractions is quite easy.
The old rule of thumb of “make it work, make it fast, make it pretty” still applies , just with much much faster iteration speed.
It seems with agents people have forgotten the last 2 steps since they produce a _working_ solution, and it might be hard to justify spending time “cleaning it up”, but this still remains essential.
The cost is skill atrophy. When was the last time you wrote something entirely from scratch by hand without AI assistance? It’s a skill entirely separate from prompting and reviewing. And it atrophies when you stop using it.
I hear what you're saying but I'm not sure I buy it in the context of this thread (a response to someone who is 54 and has been coding since they were eight).
I am in a similar boat, having been coding full-time for fourty years. The way I use the current tools is that I own all architectural and design decisions but let Claude Code fill in the blanks. I reckon the quality of the output is about 90% of what it would have been had I done everything myself, but I get a lot more done (easily 3-5X).
Will I forget how to write a "for" loop just because I haven't been writing many of them by hand lately? Those skills are so deeply ingrained that I seriously doubt it. I can ride a bike after a multi-year break, or converse in a language I haven't regularly spoken for several decades. Or write using pen & paper even though I hardly ever do it. I don't see why coding would be any different.
I have a greater concern about societal skill atrophy.
I also am not about to forget how to for(;;), that said, as a result of some years invested in aligning old pre WGS84 mapping with modern GPS and improving digital mapping, there are fewer people per capita with the skills to navigate via paper maps in the absence of GPS.
Old farts coding since age 8 (in which I include myself with a decade+ over a sprightly young 54) will retain coding skills for as long as they apply them - the fear is that fewer and fewer others will develop and exercise such skills due to AI.
It remains to be seen if that's a bad thing long term.
I am not worried about the loss of skills per se. Over the centuries the average person has become less skilled at, for example, butchering animals. Is that bad for society? I don't know.
What I am worried about is us becoming dependent on tools that we as individuals neither own nor fully control, and gradually losing our ability to function without those tools. This, I think, is a huge societal risk.
The skilled seniors better stop downplaying what actually led them to be skilled in the first place, and realize that the conditions to develop that skill has been gone and almost deemed unproductive in today's workplace.
Not disagreeing that LLM's are a force multiplier, but I highly doubt whatever value will end up finding multiplying in the next generation of seniors, at this rate. It's surreal to me that I have to point out that recognition AND recalling are both necessary components of skill acquisition, because humans largely knew this since the dawn of education.
I have been thinking about it - paid apprenticeship is the answer - just as it is in other professions like medicine.
Seniors should be paid to actively introduce juniors to the trade over couple of years. No more bootcamp entry.
And it would be significant $ for senior to agree to expend his time and energy on software engineering apprentices. There would be also very limited number of places with good seniors. Exactly like medicine for a long time now.
In fact it is already happening in some companies I know about - seniors geting their bonuses tied to juniors being under their wings.
As an educator (UK) it's very hard to find people with experience to join us. The pay is SO crap and the workload is huge. Trying to teach complex, large systems is impossible, because we don't have them. We can't throw 1,000 files and 20,000 lines of code at a student and teach them how the system was engineered. We can show them toy systems.... which is fine if there is a work route that takes them and nutures them for 5 or so years. But without that pipeline, shrug not sure what we do about it
I believe you are miscalculating the effect of skill atrophy, there is benefit and actual experience gained by doing the work yourself. You are an experienced dev and already have a lot of tools and knowledge under your belt so at the moment it is hard to see the actual issue, as this is just a productivity multiplierfor you. But give it a couple of years working under these conditions, your tech savvy nature will be severely diminished.
Indeed. I think it's a much bigger issue for juniors, who haven't yet had a chance to build that systems design muscle.
When an LLM is making a bad design decision but the engineer doesn't have the experience to spot it AND the consequences don't become apparent until much later (which is often the case) -- it's kinda hard to learn.
Have you really found claude to much more more capable than eg deepseek? Anthropic has little to no chance of producing a competitive business model in the long term.
The cheap models are cost-competitive if you are running them in long-running agentive tasks.
But they take a lot longer to reach the same goal for complex tasks, so the difference is still very real, and the cost-savings are still very much a question of how well you manage to characterise the tasks they will do quickly and pick and choose what to use when.
I kind of agree that I think the cheap models will eat away at the moat very effectively, but if it doesn't seem more capable to you, you're not giving it complex enough tasks to see what they can do.
(FWIW, I've burned billions of tokens on each of Deepseek, Kimi, GLM5.2, GPT, Sonnet, Opus, Haiku using the same harness, and we've kept stats on cost per task)
absolutely, for me the tui, ultracode agentic workflows, and streaming logic are far superior. the closest model is minimax 3.0 imo and i ended up adding a custom tui, agentic workflows, streaming logic and implementing skills to that (in typed) in order to get to an acceptable claude fallback. on their own i haven’t found one model comparable to claude, not even chatgpt.
Yeah, using deepseek feels like shit and I spend hours steering deepseek in a direction versus opus-4.7 or 4.8 where I can just kinda let it ball out on some reverse engineering problems.
> Anthropic has little to no chance of producing a competitive business model in the long term.
Extraordinary thing to say about the fastest growing company in the history of capitalism. They will soon have access to public markets, essentially unlimited capital, and can build insanely large models that they don't have to make public... ever. They can just use those models to run their business, train better models, eat competitors, etc.
But maybe it's Anthropic that isn't thinking ahead enough - you clearly think you can see around corners with your proclamation. So why do you think they have "little to no chance" of surviving long term?
We both know that both revenue and especially valuation, especially for something with as little moat as software, can vanish overnight. This is a commodity market and anthropic is serving boutique software, and the market gets more crowded every day.
> Extraordinary thing to say about the fastest growing company in the history of capitalism. They will soon have access to public markets, essentially unlimited capital
There is no such thing as unlimited capital. The faster they grow the faster they burn capital. Eventually it will run out.
Forgive me, but the active repos all look like reimplementations of existing good open source code (which of course is ideal training data) - rm_safe has rip for example. Or prototypes. Is there anything that actually has a user base > 1?
Skills atrophying in terms of what? Remembering specific API's that you always had to look up anyway? You don't lose developer intuition, analytical thinking or technical inclination, and those are the things that matter, anyway.
I recently did a fleetwide upgrade to Zig 0.16. Do I remember every single change from 0.15? No. Do I have to? Also no. Both because I can look it up if I need to, but also because the LLM already does.
If I don't look at a codebase that I myself haven't looked at in a year, I will not recognize some things when I return to it. Is this sense of "atrophy" meaningful when this was a problem long before LLMs came on the scene?
How do you deal with the lack of cognitive engagement? I think it is the primary driver of developing and maintaining skills as well as generating new ideas. Letting an agent do it for you will just get you average but well formatted code. Not something new.
You very conveniently avoided skill atrophy, the biggest issue for existing engineers to deal with (not even going into topic of cancelling whole hiring-junior-and-raise-senior approach which is just shortsighted and retarded to be polite, but general greed has overtaken the field so its not shocking this went out of window). Everybody using llms excessively is measurably doing worse, ie students. I have hard time believing out industry is somehow immune to that and refuse to do the experiment on myself if avoidable, which it sort of is for me.
Its like drug that will give you few years of great high, and ruin rest of your life afterwards. Use it by all means, I don't care about your output, nobody here does, you do you.
I do care about my long term skills, which aren't about piping some llm outputs. My employer ain't dumb fuck who is pushing for llms at all costs as much as possible. Anyway, most of my day work are processes, discussions, pushing things through - llms can't do a shit here, its personal conversations, connections, often psychical contact to get things done on time. Startup world would be different but I am as far from that unstable environment as I am from say gaming industry, just not worth my time outside SV area.
So I just use llms to verify my coding results, they are fine for that, but I do the creativity. Its by far the best part of my software dev work, why the heck would I be automating that away? Its like automating sex away so you can have more time... reading HN or some other way to just waste time, dumb approach from all angles.
Of course this changes if one is working on personal projects, self-employed, small startup etc but most folks here are not in that category.
I am another skilled senior, have been coding since I was 7, although you have a few more years of experience on me, and am commenting here just for the goldilocks moment, as I have read and reflected on both of your comments and find my reality is somewhere in the middle.
On personal projects, where I am in charge of all the hats (product development, UI, UX, backend, security, server admin, etc) -- absolutely crazy force multiplier. You get a nice suite of backend and e2e tests running, with full business scenario layered on top of that, and constantly running agents to do the coding, another agent on a higher level of reasoning to review that work, and sometimes occasionally poping into another competitors model to review their work just for added comfort -- it feels like wizardry. I am not vibing it, but I wouldn't say I am carefully scrolling through every line. I review whats fundamentally important, especially when it comes to data, overall structure, and large, cross cutting concerns, but I would be lying if I say some code doesn't land that I don't read. But I have the security of the test suites and validations , so I pour more effort into that.
It's a nice self reinforceing loop.
All of this might sound like I agree with you, and to some extent I do, but I am realizing as the apps I have built out like a cannon shot out of hell with tremendous speed and polish right out of the gate are starting to slow down. Feature adds are getting more complex. My memory is not what it used to be. Each run and pass through the code consumes more of my tokens and limits. I am starting to do less in the same amount of time. Codex did a vertical slice of a feature for me (well defined and well planned). It contained functionality that has historically plagued us developers -- the dreaded time. I used xHigh GPT 5.5. It had obvious bugs, but I wanted the robots to catch it. I popped it in claude (on the new sonnet 5! heyo!) -- Claude caught the bugs. Even said they "immediately stood out" I wondered how this happened. Frontier model from company A was evaluated by workhorse model from company B. All of this again took massive amounts of usage. And time.
And this is -- best case scenario, perfect world, everything is in perfect alignment.
Now for the work reality.
Multiple product and experience owners. Multiple dev teams. Different enterprise teams support services you rely on. You don't have full unfettered access to frontier models. You have to use copilot, or some other enterprise harness, and you run out of credits for the month, you are SOL. It's not as good as your claude, you think to yourself, but hey, its familiar enough, and you have 5k credits left for the month for Opus 4.8, better make the best of it. But now you burned half of them working on that Transactional Bug that was mixing synchronous and asynchronous semantics that the other guy's model should have picked up on. What happened? Maybe he didn't use Opus, maybe he used Haiku, maybe his prompt was bad. Who knows. Gotta fix it. Oh, you gotta reach across the isle and put in a request to get the Enterprise team to look at this caching inconsistency on user data that you need and is really the source of your race conditions. Tick tick tick. Model limits approaching. You start wondering if you just did all this by hand like "in the old days" would you have got it done correctly faster? Or at least, cheaper. You'll never know.
It’s simultaneously simple and deceptively difficult to coax a growing system into staying sane. It reminds me of forcing a fractal into growth yet somehow letting it remain similar at all scales, manually.
Scaling in this sense is not operational (“servers”), but conceptual (“features”).
I don’t want to be a downer but I find many devs are not great at this. Very clever folks, but they tend to not see these issues clearly. They’ll nod and recognize when you talk about separating content from form and the importance of various design principles like high cohesion and loose coupling but completely disregard them once in contact with reality.
Part of the problem, as you nicely showed, is that technology is only a single slice of this problematic pie. Organizations in general are systems as well and they tend to be either badly architected, badly maintained or often both. Some technological issues are downstream from organizational issues and IME those can be become rather dominant variables in the equation and no amount of AI - save full AGI taking control of the company - is going to save us from those factors.
And you’ll never know because even if you could turn back time and do it from scratch, you’re likely to opt again not to do it all manually because the cognitive load is going to keep tempting you to reach for the agent again.
the distinction between personal projects and Enterprise development is a big one. A severe bug in my personal projects, i fix it on the fly. A bug in our products rolled out, nightmare.
We've just done an official evaluation at work, using extensive statistics on our gigantic monorepo in a company with ~2000 devs over the course of 2 years, everyone from hardware engineers to regular old frontend engineers. It's a highly profitable and mature public company, and has been for going on a decade at this point without missing a beat. We were given infinite access & budgets to basically any and all AI tooling we could imagine, and we have several "AI Native" teams (whatever the fuck that even means). We're doing agentic coding, we have harnesses of all kind, skills, we have many teams doing spec-driven development, designers using all the various things like Figma Make and access to tools like Devin/Factory Droid/Claude Code/Codex/etc.
This is all to say, we as a company are using AI a lot in all possible corners, but thankfully our leadership isn't schizophrenic and isn't mandating everyone hit token limits or whatever, it's more of a "Let's see what works and what doesn't" type of thing, and we measure a lot of statistics. Nobody here really cares whether LLMs are the next coming of Christ or not, as a company there are many people (even in SLT) that are indifferent to LLMs, and many who are reasonably hyped.
I wish I could link to the actual document we were all shown since it has a beautiful breakdown of the methodology and a fine-grained breakdown of the stats and the categories measured, but in the grand scheme of things, ALL the AI tooling we have implemented (at least on the engineering side of the equation) has contributed to a total of... drum roll please... 7 (seven) Percent overall productivity increase! The most productive teams saw a productivity increase of around 20%, while some teams actually saw drops in productivity into the negative percentage points. My team, none of us really give a shit about AI and we're somewhere in the 3-5% range on certain categories of tasks, which I'd say is a fairly good assessment.
Productivity here is measured in many ways, including but not limited to speed of MR review and merge times, feature/ticket/roadmap closure/delivery, rollback/revert incidence rate, how often people interact with the MR review bots and implement their suggestions/fixes, how many times people check back on AI transcriptions/meeting notes (hint: Nobody looks back on any of it, it's all just noise that gets generated and never actually referenced outside a few extremely rare cases) and many more things I'm forgetting. It is an imperfect number of course, because measuring productivity in engineering is a sisyphean task, but in my opinion it is accurate to the reality on the ground and outside of all the hype and marketing bullshit.
So, I remain thoroughly unconvinced of these personal anecdotes of people being "massively" more productive, especially once you factor in the fact that we now have a 2000EUR budget/month/dev for all the AI tooling, those productivity numbers start looking pathetic once you factor in the costs (which are only increasing as the AI companies need to start recouping the gazillions they've burned). Some teams have started begging to disable coderabbit and other similar tools in their MRs because they're producing nothing but walls of noise that makes reviewing any MR a nightmare of sludging through endless slop of useless bullshit, ours included.
> I'm surprised no one tried fudging numbers though.
Oh some of the "AI Native" teams tried in the past, before the large scale statistics came out. The funniest one was the team responsible for rolling out AI to the whole org citing numbers like 500% more productivity, compared to what? Who knows, since that team didn't exist until relatively recently.
It probably helps that we're in a highly regulated field (banking), so by default we don't chase trends, especially not ones that are potential regulatory nightmares :p Things go a bit slower, which has its pros and cons, one of the pros being that you're forced to consider the long term a bit more and you have room to plan things out and conduct reasonable experiments.
> I don't think they're a net gain if you're a skilled senior
I've had Claude Code running a /loop for the last week driving down complex crashing bugs in a prototype compiler entirely unilaterally. I occasionally glance over.
A few of those crashing test cases were ones I've spent more than a week trying to track down myself. I have 30 years of experience of doing this.
It's worked 24/7.
So far it has fixed over 500 of them.
Will there be technical debt? Yes. But nothing that remotely compares to the cost I'd have incurred of fixing all of those myself.
It is hard to reconcile those gains without thinking that if people are saying these are not a net gain, they haven't really tried learning how to get the full benefit. If you sit and watch a model work and keep intervening all the time, then sure, they're not going to be a net gain.
If you give Anthropic money they will make your life worse in another aspect, it's relevant to all their models. The best principle is to not give money to people who want to harm you.
Anthropic has gone past fearmongering and well into terrorism. I think people on Hacker News should not recommend working with terrorist orgs.
Should we work with companies that give the "Department of War" full access to their tools (OpenAI) or Chinese companies with completely opaque ownership and dev structures?
Sure about Dario (and all billionaire) weirdness, but no gains if you are a skilled senior is well, very far out in our experience (our company is 30 years old with mostly the original employees and founders): what we deliver now at the speed and quality we deliver it would have been impossible 10 years ago with our team size of skilled seniors. We replaced all the commercial products our clients and ourselves used with our own, giving us millions more revenue and profit with the upselling and efficiency benefits. We work for regulated clients: our code is reviewed, pentested and audited regularly by us and 3rd parties so its not slop either. You are definitely leaving money on the table. We do mostly use chinese models on our own hardware (we colocate cages of racks) so this is not about Anthropic but about AI in general.
Skill athrophy is a real thing though; we try to prevent this by have hackethons (for lack of a better word) without AI where I pick something extremely non trivial and we implement it for fun and profit without AI (with would not matter much as they are currently bad at these things); last one was flex paxos for our in house db with obvious metrics for the endresult: data integrity (duh) under failure and performance better or at least the same as our raft production version.
Sure, that is why you need to be early. I fully believe my company won't make it another 30 years (or 10), so we prepare for that. Also, I will be dead by then, but that is unrelated.
For now everyone is still sufficiently crap at using AI to need help. We had enough clients trying to build something themselves and then come crying to us.
Having a health problem that puts an end date on your effort must tint your business choices in a unique and interesting way. I find your ideas intriguing, and wish to subscribe to your newsletter.
He has also been telling bald-faced lies about open source/open weights models that are easily disproved. For example, he claimed that they lack the collaborative benefits of open source because "we can't see inside the model".
Open weights models are responsible for enabling reams of research on interpretability methods that do just that. And they have facilitated so much collaboration on architecture, inference optimizations, training and steering methods, and other topics that were completely out of reach with closed models like Anthropic's. It's really staggering to me.
The premise (that "powerful models" implicate "safety" concerns that must be controlled by "companies") does not seem true or self-evident to me. This tired fearmongering campaign centered on GPT-4 three years ago, a model now surpassed by open models that run on laptops.
Did fearmongers like Amodei say, "Oops, we were wrong! It wasn't that dangerous after all"? No. Of course they didn't.
1) the company has device-level control to the degree that they can not only restrict which API endpoints people can connect to but which accounts they use to do so (in which case this already isn't an issue); or
2) they don't, and all bets are off anyway, open weights or not.
You mean to tell me that anyone can own a nail-gun? We can't have people buying their own nail-guns, next thing you know they might build things that aren't up to code!
> "Once the weights of a model are public, they cannot be retrieved. If a model possesses dangerous capabilities, it is permanently out in the wild... We need to consider regulatory frameworks that account for the unique risks of open-source distribution of highly capable frontier models."
Curious if I'm being downvoted by people who think there will never be a model with dangerous capabilities? Or people who think when it happens we shouldn't add guard rails?
While I appreciate, they publish this information, it's increasingly hard to keep track of it all. I've lost the mental model of how different models at different effort levels perform and what tasks they are good at.
In practice, I tend to just use the default on Claude Code that works well enough. But I wonder to what degree other users really play around with these settings to optimize for their project.
I always use Opus 4.8 at max effort for everything. The $20 subscription didn't have enough tokens, but the $100 one had too many of them. So now I just max out Opus in order to maintain 100% weekly utilization.
I'm a senior skilled developer and I find Anthropic $20 + Open AI $20 + OpenCode Go $10 offers more value than $100 on any particular service.
Juggling between all different models/agents is quite simple with Zed.
A caution about OpenCode Go though, the entire company seems to be run by AI so there's lot of billing related issues with zero support. I subscribe new every month as I lost money due to double payment with automatic subscription.
For non coding related tasks I use local models.
P.S. If anyone is interested to read more about my setup, let me know I'll publish a blog post.
I've been running about the same stack for well over a year now, Anthropic cheapest + OpenAI cheapest + z.ai coder (black friday offer).
The Z.AI is a bit wonky, so now I'm moving to Openrouter for Qwen+Kimi+Deepseek?GLM
My summer project is to figure out a proper agentic system where a "big" model does the planning, but automatically uses a cheaper one for the grunt work. Having Opus to config edits is just stupid :)
> My summer project is to figure out a proper agentic system where a "big" model does the planning, but automatically uses a cheaper one for the grunt work. Having Opus to config edits is just stupid :)
My company pays for the tokens so I don’t care. Biggest model and max everything. The slight risk of a smaller model making a mistake is more expensive than just running the bigger model all the time.
Would love to read that blog post. I'm toying with running local AI model with Claude and GLM as well depending on a task. Pretty decent success but it could be better.
I'm a heavy enough user that I have both the OAI and Anth $200 plans. I always use at least 50% of my weekly Opus quota at Extra setting (meaning I use double the limit of the $100 plan, at minimum). Max I rarely touch because it is twice as slow and the incremental capability gain is minimal. Usually if Opus can't sort something well at Extra, the answer isn't to use Max but to hand the issue off to GPT-5.5 at XHigh.
I too have settled into a kind of dual Claude/GPT model setup. I will often use one to review the other's work, or critique the other's plan in some way. Sometimes I'll have Claude implement a feature one way, then have GPT do it the other way, then have them both review each other's implementation. Then synthesize a final plan from the previous implementations+reviews.
I might just be having fun with models, but I have actually noticed their capabilities vary somewhat, and so my (perhaps vain) hope is that by using both, one can catch each the other's blindspots. It's still unclear to me if that's consistently happening, but I am making substantial progress in my personal and professional projects, so something seems to be working.
> Sometimes I'll have Claude implement a feature one way, then have GPT do it the other way, then have them both review each other's implementation. Then synthesize a final plan from the previous implementations+reviews.
I've done variants of this a number of times, but feel like it was a generally waste of my time to then have to compare them and write up which parts I liked or disliked: if the output is something substantial, each will have its pros and cons. Clear-cut wins aren't very common. Of course it could work well if we automated the whole thing with an orchestrator; you just need a model with actual good taste (according to your own preferences) ... so we'll have to compare all the models to find that one
Yes, same, between the two of them I feel like results are just better because they have different priorities.
At the same time, I’ve invested in tooling that prints and lints architecture I want, so which model is less of an interesting decision, because the results tend to be very close.
This is actually very counterproductive with Opus 4.8 - you are wasting a lot of time.
For Opus 4.8 training with overblown internal dialogue and second opinions - Max effort burns just tokens and wastes time without much value. Spinning wheels.
Why are you optimising for token use rather than getting whatever tasks you're trying to do done? Max effort Opus will happily sit there talking to itself for five minutes fixating on every tiny detail while Sonnet or even Haiku will just blast through the problem and be ready for the next one in seconds.
Because I'm paying $100/month for this, so I absolutely want to use all of those dollars. Any leftover weekly usage is a gift to Anthropic.
I am getting things done. I've made major progress on my projects, and even started new ones. My most requested tasks are: code review, brainstorming and research. The fixation on tiny details is exactly what I'm paying for.
I've been plugging this perhaps too many times now, but I am trying to bootstrap a user-sourced corpus of exactly "what model is good at task X". So, not benchmarks, but high-level tasks. There's a bit of a ordering problem in that nobody wants to bother commenting on a site that has few comments - so PTAL and contribute if you can. https://model.reviews
In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it.
They are often used for reading code though.
To expand on this, while the "big model to write a plan, small model to write the specific code" idea is quite common it trips up on edge cases.
In theory the flow works like this:
- small fast models read lots of code, and pass details to the large model to write a plan
- large model takes those details and writes a detailed plan
- medium models write the code
The issue happens when the medium model hits something that the plan didn't take into account (which happens a lot - the big model didn't actually read the code). Then it has to either guess, or pass back to the large model.
If it guesses, the plan usually starts to fall to bits.
If it passes back to the large model, inevitable the large model has to start reading lots of code. In that case you are paying the expensive tokens to read so you might as well have it write the code too (many less tokens are written than are read)
It might be possible to get this to work, but I haven't seen anyone who has tried agentic work with frontier models be satisfied with this hybrid setup.
I'd note that Amp (mentioned above) is probably the leader in using multiple providers in a coding agent but still uses frontier models to write code.
> In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it.
That's not something I understand very well. The less expensive models will quite happily chug away at tasks, if the codebase is well-structured (small files help a lot) and your instructions are clear. In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked. You can run small models almost 10-100x as long for the same price in dollars, which covers a lot of correction and adjustment.
Why does everyone say the trade-offs are rarely worth it?
> In contrast, I've never seen a large model turn bad instructions (instructions that would cause a human to think before starting) into a result I liked
I think the distinction is here.
I expect my agent to build from product level descriptions. This might include specific special cases that I call out, but will rarely highlight existing special cases or edge cases - they already exist in the code, and I'd expect a programmer to make sure that behavior continues to work.
If a feature hits lots of these edge cases, the weaker model that is reading the code (aka Haiku) won't understand their significance, and will report back to the planning model incomplete or incorrect information.
The planning model (Opus - which hasn't actually seen the code remember!) will build a plan that is incorrect or incomplete and delegate coding to the mid level model (Sonnet) which will do it's best to make things work, without understanding the overall picture.
This is how you end up with slop - for example Sonnet reimplements things that already exist because it found one of the edge cases, but Opus had never known about it because Haiku didn't understand it.
It's possible that the new "agent teams" feature in Claude code can help with this. That keeps each agent alive with its context so they can ask each other things, but I haven't tried that enough to be sure - let alone with the specific model mix like this.
In your case, you are giving the Sonnet model specific instructions for what to implement mindlessly. I'd expect that to work well!
But that's not the same as the agentic workflow many other are using.
I appreciate the suggestion! But it isn't clear to me, from reading their marketing site, what they bring to the table from this perspective. Can you give me a more targeted pitch?
I haven't used them in a while so my info may be out of date, but they tended to track whatever models were the best and auto-use them for each task (eg, one for planning, subagent for a code search, other frontier for implementing). Their CLI seemed very well thought out to make you do things "the correct way" -- for instance, `/handoff` instead of `/clear`.
Sorry for the late answer and the missing context. usef- is right, the manual is probably the better page to share. Amp tries to give you a plug and play experience, where you can always see the actual costs and models/effort are autoselected for you. Some of my colleagues are big fans and use it a lot. I also like it, but prefer OpenCode.
I don't demand a customized compiler for my code even if such a compiler could outperform gcc. There is a lot of value in focusing on correctness to an extreme degree even if the outcome might be suboptimal to something more tailored - a tool with a large customer base can justify more resources going into its maintenance.
Ah I see! Yes, I was talking about a coding harness, not an enterprise agent. I entirely agree with you that your suggestion of driving it via evals is the right thing for that use case!
Same boat as you, and my answer is "... Except when I ask and overall or checkup task that is specifically heavy or overseeing in which case I use the maximum level" which lately meant ultracode.
I'm not going to play around with thinking level every request because the goal is to make me save time not spend it in a different setting menu.
I tend to run it on High and then step it up for problems where I'm noticing it struggles, bump it back down after. Sometimes I accidentally leave a session in Ultracode for a day and wonder why things are taking so long, but generally happy with the results.
Exactly this is my problem with all AI tools. I want someone else to create working tools for me so I can focus on my product. It is the same with other tools. I do not want to spent huge amounts of energy and time to setup my IDE, operating system or desk layout. I guess it is too early to have that now.
It's really not that much. It's a bit hard to make sense of it not because it's hard to keep track of, but because they are being deceptive and opaque about what you're actually buying, and the thing you're paying for is different from one day to the next, as they fuck around with the parameters to boost subjective performance during a launch, then quietly degrade the service to cut costs.
Just because it’s hard to keep track of doesn’t mean it’s not relevant.
Playing around with learning the differences is incredibly helpful to schedule on ones calendar weekly for an hour or two, while saving links throughout the week to try out.
Same advice as ever? We call it context engineering now, but prompt engineering still matters a lot. Most of the failures I run into are unspecified assumptions made by the model that derails the conversation, but usually updating the first prompt fixes it. Opus in my experience is a bit better about checking assumptions, while Sonnet will plow on ahead. An example is mentioning a file that doesn't exist: Sonnet will go ahead and try to grep your entire hard drive for it. Opus will say it's not local and request the path.
I trust neither for general knowledge and I still find Opus giving me answers that are completely BS. But the token spend for Q&A is nothing compared to coding, so I always use Opus + a lot of thinking. For coding, I find Opus to be better value/token but I haven't done any sort of rigorous test.
- For Claude.ai subscriptions I think Sonnet is much cheaper than Opus. This is why there was a "Sonnet only" usage bar for Max tier for the longest time.
- For some tasks the sheer amount of raw input tokens is the most important. For example multimodal computer use tasks. You can't make them any more efficient on Opus by turning down the reasoning, so a cheaper model like Sonnet is useful for them
Yeah, I was looking at the same chart and was very surprised at where the curve is relative to opus... Feels like sonnet 5 is "what if opus had an extra-low effort level"?
The arguable caveat is Sonnet may run faster (although this isn't known for sure, due to more tokens being used for the same task), so you can potentially get more done in a synchronous iterative workflow
I don't really believe this however, because so much time is spent fixing up after models, that a slower but more intelligent model is a net time saver in my experience.
From my benchmarks, sadly, it doesn't seem to be the case much. Surprisingly. I found Sonnet comparable in speed to Opus (sic), but perhaps I was testing it wrong?
Well, it is a Sonnet model, it is indeed better[0] than Sonnet 4.6 (smarter, faster, cheaper), but I don't see why would you use it as opposed to Opus 4.8 low or GLM-5.2...
You're referring to the Agentic search, but if you look at the Agentic computer use the cost is basically halved.
However, I am also confused about market positioning. Too expensive to perform daily tasks - open souce models are much cheaper - and not frontier model to address complex real world problems.
No, you are misunderstanding the graph. Draw a vertical line anywhere, that is a "constant cost" line. For any given cost, Opus 4.8 has a higher performance than Sonnet 5. Only where Sonnet 5 effort is at medium or low would it make any sense to use it, as there isn't even an equivalent Opus effort level to compare to.
Alternatively you can draw a horizontal "constant performance" line and see that Opus is cheaper for a given performance level.
Why are you comparing xhigh reasoning between Sonnet and Opus? Of course Sonnet xhigh is cheaper than Opus xhigh, but that isn't the point; the point is that at e.g. 80% accuracy on Opus costs ~$0.45 (medium reasoning) whereas on Sonnet it costs ~$0.52 (xhigh/max reasoning).
> Too expensive to perform daily tasks - open souce models are much cheaper
There is a real advantage, especially for businesses, in using an off the shelf solution from a corporate provider.
Personally, the advantage of not having to set up multiple solutions from multiple sources outweighs the cost of a $20 a month subscription. Think about why a lot of consumers prefer Apple devices over Linux. There are a lot of advantages to Linux, but "never having to think about my tools" is its own advantage.
The specific market positioning is... for me to use at my big tech company job, where we aren't allowed to use GLM and similar, but have fixed caps on how much token usage we're allowed to rack up a month.
That's just one benchmark, though. Tab to the next one and Sonnet 5 performs better as effort goes up just as you'd expect. I imagine the suggestion is that performance vs effort tradeoff is task dependent.
But they don't show "strictly better" performance at cost per task!
The graphs show parts of the cost/performance pareto frontier occupied by Opus 4.8 and others occupied by Sonnet 5.0. If Opus 4.8 was strictly better at cost per task like you say, by definition the entire frontier would be occupied by Opus.
So neither is pareto-dominant over the other. In contrast, Sonnet 5.0 is Pareto-dominent over Sonnet 4.6 on those graphs.
> by definition the entire frontier would be occupied by Opus.
But the entire frontier is occupied by Opus under any reasonable interpolation scheme (piecewise linear which is what they've done, and most reasonable spline or polynomial fits would also lead to the same result) over the overlapping x values for which both are defined.
Under that interpolation scheme, for x > ($ cost of Opus low effort), Opus is Pareto-dominant over Sonnet 5. You can see this by picking any point on Opus's interpolation and realizing that you get strictly worse by switching to Sonnet for the same x value or the same y value. Meaning if you want to pay the same $x then you get a worse y, or if you want the same y you pay more $x.
I really don't get what you're proposing. The cost ranges do not overlap at the low end. You can't (by definition!) interpolate outside of the range.
If you mean extrapolate, at that point you're just making up data. The available effort levels are discrete and covered totally by the benchmarks. You can draw on the monitor with a sharpie to show a "ultra-low" effort level for Opus that scores better than Sonnet "low" at the same price, but it doesn't magic the ultra-low effort into actual existence.
(Anyway, the blog post now has an errata and a graph that shows substantially better relative performance for Sonnet 5.0 than the original graph.)
That's why I said "over the shared frontier" in my first post and more precisely in my second post I said "over the overlapping x values for which both are defined."
It was a claim that applies to a range of x-values where both curves are defined.
Of course if you go beyond those x-values where only one of the two are defined, then trivially the one that is defined constitutes the Pareto frontier in that region. Which is what I understand to be your point?
The post I was replying to said "performs strictly better at the same cost per task". That claim was obviously not true, there are costs where Opus cannot do the task and Sonnet can, so Opus can't be performing strictly better that the same cost. It seems that you agree that it is not true.
You could make it true by artificially dropping some of the data points, but, like, why?
(Again, this is moot given the updated graph.)
> Of course if you go beyond those x-values where only one of the two are defined, then trivially the one that is defined constitutes the Pareto frontier in that region.
Not so! It's only sound to do that at the low end of the cost axis (x) or the high end of the performance axis (y). You can't do it at the low end of the performance axis or the high end of the cost axis.
They've absolutely both changed. The initial version I saw didn't include max effort data points on the first chart, and the plot itself was much less favorable to Sonnet at high/xhigh relative to Opus, but the new chart shows them as closer competitors. Weird.
I just re-wrote the /code-review skill anthropic ships to use Sonnet 4.6 for some tasks as it was using Opus for simple git diff commands and similarily mechanical tasks (launched 100+ agents for one of my diffs, cmon). I wonder how Sonnet 5 will impact my usage.
Does anyone else have any review token saving measures?
What is a "task" in real-world terms? If it will be $15/million output tokens, and high/xhigh is somewhere in the $7.50/task range. Does that mean a single task is using 500k tokens. That seems like it would start to add up fast.
Except for the fact that Opus 4.8 is not good. Constant hallucinations, doesn't use the web very intentionally until you explicitly ask it to and it nopes out rather quick on benign items. Anthropic has been very disappointing as of late. All of the gatekeeping is taking a toll on what should be some of the better models out there, but you can't trust 4.8 to go off on its own. It will burn down tokens doing what it deems correct as per its guidance. Truly painful to use.
The point is Anthropic has advertised their models in this way. There are plenty of models that can be used in long running situations that have proven to be more capable. Opus 4.8 is not that, and ironic given it's their top public model.
