It’s much easier to get your AI agents to do something consistently than it is to get your human colleagues to do it. Heck, personally we know these things are the right things to do, but we are just too busy and our minds wander to what we think our more productive uses of our time. That completely changes when it’s agents being instructed to do the work instead.
The crazy thing is that it s looking like AI might be able to write software better than humans eventually simply because they do not get bored of doing tedious tasks. Many things that we know work but don’t practice because humans aren’t very scriptable are now viable and can be easily applied with agents. For example, I’m finding a lot of success in using separate agents to write implementation and tests from a common specification and then using an auditor to run the tests so neither agent is contaminated by the other’s work. This is just part of the clean room engineering process that was developed for people by IBM in the 1980s, and it was shown useful then, but with AI it can be widely and consistently applied.
They'd be 10x better than us already if task tedium was the problem. It's design sense that they're missing.
I find that in the areas where people think that LLMs excel at coding and don't like doing manually it's usually because the human was inclined to slop out repetitive boilerplate and thought that was the only way. Tests are usually like this, sadly.
LLMs are definitely good at providing reams of duct tape (which is drudge work) to patch up those bits of the code base where the code sucks. The problem is that duct tape is not the most architecturally sound construction material.
Plenty of languages are rich with boilerplate (think Java, C++...). Expressing intent succinctly is non-trivial, and especially so when you want to provide a rich vocabulary like modern languages do.
While modern tooling helps, the form of expression still means cognitive load (in reading code, deciding what to copy-paste and refactoring later).
And sometimes, tooling which is succinct brings a whole can of worms with it (think pytest with assertion rewriting and unexpected behaviour of your .pyc files — if you are familiar with Python as your nickname seems to suggest :)).
> They'd be 10x better than us already if task tedium was the problem... the areas where people think that LLMs excel at coding and don't like doing manually it's usually because the human was inclined to slop out repetitive boilerplate and thought that was the only way.
There is so much work out there that is repetitive / boilerplate / tedium. If you get to personally work on interesting work more than 50% of the time (pre-LLM) I'd say your job is #blessed.
IME if a technical task is tedious and repetitive it is nearly always because the system was badly designed or because it wasnt automated properly.
My job is automation and system design, so if I can't fix or work around these things that reflects poorly upon my skills.
Some people treat writing tests as inherently boring because their test frameworks usually suck. If they don't suck and a test is a very close approximation of a spec, it's not boring at all, especially if you use it as a means of codifying a spec before implementation.
Don't disagree that that's the system not working well.
You're lucky to be in a system that works out for you. Not everyone is in such a lucky position. LLMs help automate the tedium out of their job, and hopefully, have a bit more extra energy to make a better system within their immediate locus of control.
I'm sure you're where you are in some part due to hard work, skill, decisions. But surely you don't believe that it's purely down to skill where one lives/works at?
To be clear, the guy who wrote this article is Martin Fowler. Fowler was the author of a book literally titled "Refactoring", which he wrote over 20 years ago (it didn't invent the term "refactoring", but it's widely credited as popularizing it).
In other words, this is not some new AI-bandwagon-jumping blogger trying to rebrand old practices as something new ... this is the guy who coined the very term "refactoring"! He's not pretending what's old is new, he's arguing that best practices from 20+ years ago remain relevant (and showing receipts).
It’s an easy mistake to make, given the name of the website, but Fowler frequently hosts guest authors. This article was written by Giles Edwards-Alexander, ThoughtWorks CTO for EMEA.
The thing is, humans make do. Given lacking, outdated documentation somewhere on sharepoint, big picture revealed somewhere in passing at some meetings or water cooler talks, underprioritizing refactoring humans still deliver something even if at reduced quality and timelines.
AIs either execute extremely poorly or don't work at all without these. Decent engineering practices are suddenly a hard prerequisite with AIs, not just some longer term improvement.
On a related note, this is unironically good argument to integrate AIs into the workflow even if there's zero net benefit in your use case. It's a perfect excuse to bring the proper engineering practices in.
There is a big difference pre and post AI, namely that humans have quite good long term context management.
But now that everyone is coding with AI, all agents need to acquire context every session. Thus the value of doing best practices is much higher and benefits are there immediately.
No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief!
Next up is demonstrating the AI is more productive with better programming languages.
> This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans?
> We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct.
> We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
(It's an interesting question though - my prediction would be that AI is best in the languages it consumed the most examples of, and I reckon the size of the StackOverflow site for each language would be a rough proxy for that).
> Next up is demonstrating the AI is more productive with better programming languages.
This is hard to do given the current reliance on model weights in its programming language knowledge. You’ll get your best results for python or some variation of python. But
I’m not really sure how to read those charts, but it’s completely possible that python did not work well for the approach they were trying.
I would be the first person to give up python for kotlin or some other more natively typed higher performing language. But the python bias is so far too big to ignore.
You might be surprised. I've had an AI start spitting out none functioning python code. I've rewritten it with Go. The requirements are that every change has to pass compilation, linting and tests. Same requirements as Python. Go keeps the AI on the rails better. The compilation failing provides quick and probably better feedback than a whole dictionary of tests.
As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
True, and to my point, "popular" isn't necessarily "better".
> because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances
Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.
Outlets benefit from standards, however. Every device (well kind of) uses the same few plug types, and a different plug would be a disadvantage.
On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
Boring: The documentation should be in code, not in external Word documents uploaded to the company SharePoint server.
Exciting: The documentation for the AI should be in code, not in external Word documents uploaded to the company SharePoint server.
Boring: You should give your developers the big picture of the project, not just micromanage them using Jira tasks.
Exciting: You should give your AI the big picture of the project in CLAUDE.md, not just micromanage it using prompts.
Boring: Refactoring makes your developers more productive in long term.
Exciting: Refactoring makes your AI more productive in long term.