You can tell from this that gwern has never spent any time climbing up the ranks of a large organization.
Almost all of the hard problems in the world are straightforward coordination problems. The few that aren’t come to mind much easier because they’re much more legible than coordination problems.
To paraphrase a saying, amateurs discuss effort and professionals discuss impact and in whatever field you’re in, with a few tiny notable exceptions, once you reach beyond a certain magnitude of impact it becomes 95% about solving coordination problems.
And throw in Conway's law [0] and the scope of the coordination becomes easier to understand.
My career trajectory bought it come for me, intermittently contracting for/with a UK govt acquisition organisation buying systems for Govt customers.
Task: make these disparate systems speak to each other
* Q: Why don't the systems use the same protocols?
* A: They are delivered by different contractors
* Q: Why don't the contractors use the same (existing) standard?
* A: They aren't incentivised to do contractually
* Q: Why do they have different types of contract?
* A: Because each one is contracted by a separate acquisition team
* Q: Why don't the acquisition teams use common requirements / contracting / technical / etc approaches?
* A: Because they are separately funded by different customers
* Q: Why don't the customers speak to each other?
* A: decades of institutionalised rivalries and differences of opinion regarding apparently common problems
It took me about ten years to work down this list, understanding each layer. One of my most satisfying career moments was when I was asked to brief a senior customer decision maker who owned the resulting mess, and drew them a diagram showing the layers. He then set in motion a top-down alignment of the bit he could directly control: requirements writing. The belief was that getting consistent requirements would be a good starting point that would hopefully flow down the organisations involved.
[Edit] Another problem with increasing coherence in this way is that it costs money (e.g. to adopt / set up the standards / protocols involved). Money that could be spent delivering customer requirements, and which PMs are reluctant to spend on 'under the bonnet' techie things the customers won't see let alone understand.
I retired three years later so have lost touch. But before I left a couple of new procurements were starting up, and I attended at least one meeting where the separate customers were in fact in the same room and playing nicely with each other, which was a very good sign. The acquisition authority had also established some cross-cutting functions (design, contractual support) that would in theory support the new projects coherently, when they reached that point.
As my career progressed, I got a better handle on the genuine cultural issues behind some of the customer-level differences. For example, one particular category of user had very different career progressions / longevity in different parts of the overall organisation (heroes in one part, zeros in the other, approximately). This affected the relative prioritisation of the relevant requirements by the respective customers, and the perceived value and affordability of solutions, in a way that overshadowed the tech-focussed arguments younger-techie-me would have made.
a) what a relatively mundane value-add via co-ordination looks like. Something like this is impossible to communicate without so much context but these stories are happening in every single pocket of a giant org.
b) What the artistry of it is. It's not easy and can take years to understand a single co-ordination problem well enough that you push single pebble and cause the right avalanche of change
c) Why anyone who is looking for impact inevitably gets drawn into it, independent of their feelings of whether they personally enjoy it or not (spoiler alert: I have never met anyone who has told me this was their true passion from birth. All of us wade into it reluctantly and eventually develop the internal motivation to make it satisfying).
I’ve said for a decade now that there’s a huge difference between programming and software engineering and you’ve managed to put it in a single not too complex sentence, which generalizes well to other domains.
If you’re reading this as a junior engineer anywhere and don’t quite understand it, just print it out and hang it by the bathroom mirror right next to your operator precedence table.
It’s sad because they’re all just systems. People in a particular circumstance behave in a statistically predictable way given a system of incentives. If a behavior is unexpected, you’ve encountered a bug in your mental model and you should just chase it down like any software bug. Making a change to the system requires the same set of programming and analytical skills of navigating any complex code base.
There’s a reason why nerds who grokked this became masters of the universe for a few solid decades.
I have this intuition, that we (people), even the really bright ones, are really bad at predicting emerging properties of systems.
This phenomenon might be seen almost everywhere, but I find it that using cellural automata (Game of Life anybody?) and abstract board games with limited rules is the easiest way to show people that they are actually really not that far from children if it comes to imagining consequences of few rules interacting with each other on scale greater than 10-20 moves ahead.
Go board game is my favourite example - few almost self explaining rules and if you show this to someone that did not play it before and he will be lost for days or weeks at a time (if it comes to predicting consequences of two players laying stones according to those few simple rules).
And then, after they learn anything about this game, change one rule (allow for two moves in a turn for example) and see how they are lost again...
It's almost like in the quote from Feynman (this is more a paraphrase because I do not remember correct phrasing, only the general idea) - "People do not understand what it really takes to know anything".
People, well rather any given individual, might react in a predictable manner. Guessing this manner woupd require to a) build a modelnfor said individual and b) put all the input into that model that the person has at any given time.
A) is close to impossible while b) is deeply in the realm of never ever not even close. Now imagine that for any number of individuals across a complex network of social connections.
People who optimize for effort look for truth. People who optimize for impact look for effectiveness. Unless your need is to predict to an individual level to high resolution, this is wasted effort.
People are largely a statistically predictable bag of attributes in many areas that matter for impact. The small amount of areas where humans are wildly unpredictable is what makes us beautiful and we focus on that as the "soul" of a human but they're also not usually the ones that intersect with how we care about impact.
There will be a statistical prediction of all the ways people will choose to get to a train station based on distance, weather, percentage car ownership, quality of sidewalk etc. Something as simple as improving the lighting of the right road to the train station can increase the ridership to a significant degree because it helps people feel safe walking home in the evening which means they also feel safe heading to work in the morning.
You spend enough time with a sensitive enough radar in a large organization and you see the same statistical patterns. People are milling around and impacted by their environment and things you want to make happen aren't happening until you notice the corporate equivalent of the unsafe road and you put down a couple of lights and rapid change occurs. You do it enough times consistently and you get promoted and the patterns become subtler.
Many things that look like coordination problems are really optimization problems.
It’s common for huge government programs like the Space Shuttle, F-35, California High-Speed Rail etc to try and satisfy a widest range of different groups and therefore be N increasingly poor solution for any one group. In such cases the root problem isn’t finding the optimal solution to make everyone happy, the root problem is picking which groups get what they want and are willing to pay for it such that a project works well.
It could easily be the correct solution is to abandon a project early rather than waste a great deal of time and money either directly or by to upsetting some groups who will block the project. The same applies inside of organizations, though it’s less obvious from the outside.
Should array indices start at 0 or 1? My compromise of 0.5 was rejected without, I thought, proper consideration. - Stan Kelly-Bootle
The examples you listed above are what I would call 0.5 solutions. To me, these are straightforward co-ordination problems that failed to find a solution and then devolved into optimization problems that failed to find the right local minima because the problem formulation was flawed in the first place.
Co-ordination problems aren't fuzzy and nice almost all the time. A lot of solving co-ordination problems is getting two people to agree to knife the third to make the problem go away.
Your example disproves your point. Array index works just fine if C programs use 0 indexing and Pascal programs use 1 indexing. There’s zero need for compromise between different groups here and the overhead of trying to do so is often just wasted effort.
Coordination only applies within some group, so your example is showing coordination bias. By assuming everyone needs to agree to something you’re failing to optimize.
Pulling the plug doesn’t look good appealing when this project is your livelihood, but looking after stakeholder’s interests isn’t always the same thing as continuing to collect a paycheck. Thus at the root of such projects should be an optimization problem not a coordination one.
There's 3 seperate aircrafts in the F35 program which, if we designed any 2 of them, could have been coherent, well designed, well executed single aircrafts. It doesn't matter which 2, pick any 2 of the 3 missions and we could have gotten 2 great aircrafts.
But planners looked at the costs and said we simultaneously can't justify 3 seperate programs and also we can't simplify it down to one program, hence the ungainly mess that resulted.
Like I said before, if someone could have convinced any 2 of the branches of the military to knife the other one, we could have reached a much better place than we are now. That we didn't was a co-ordination problem.
I wouldn’t call the F-35 a mess, all 3 are solid aircraft and the F-35A was both cheaper than the F-22 and can be exported.
Rather than having the branches pick 2 of 3 the real question IMO is if making more F-22 instead of trying to include the F-35A would have simplified the F-35B and F-35C enough to be a net benefit. That isn’t about one side losing the fight, it’s a question of which approach results in all sides being better off.
However, from a military contractor standpoint the F-35 was a gravy train and ultimately any talk about efficiency means reducing profit. In that context things worked wonderfully for Lockheed Martin and it’s many suppliers and subcontractors.
The F-35A was fine, it was mostly the B that clusterfucked the project. Again, if we're talking about incentive problems, those are just straightforwardly co-ordination problems.
It’s complicated, the weight reduction redesign spurred by the B significantly improved the other 2 variants. So arguably the B was a major benefit to the project even if it drove up prices and added significant delays.
The only way to quantify if it was a success or is in comparison to having the B end up as a separate project. Which comes down to a cost benefit analysis, though we don’t actually know as much as that alternate because it didn’t happen.
I think this is only partly true. I think it's more generally true that once hard technical problems are solved once, all that's left are coordination and prioritisation problems.
"The few that aren’t come to mind much easier because they’re much more legible than coordination problems."
How much of the world is spent working on truly greenfield R&D vs how much attention do we pay to it in the media, in our discussions and as examples that we use when we try and convey abstract concepts?
Almost all of the hard problems in the world are straightforward coordination problems. The few that aren’t come to mind much easier because they’re much more legible than coordination problems.
To paraphrase a saying, amateurs discuss effort and professionals discuss impact and in whatever field you’re in, with a few tiny notable exceptions, once you reach beyond a certain magnitude of impact it becomes 95% about solving coordination problems.