AMD ran into significant compatibility issues with AM4, with some motherboards not being able to supply the amount of power needed by the newer CPUs and PCI-E Gen 4 support being removed in the final BIOS release due to reliability issues. A lot of motherboards also didn't have enough flash space to contain the firmware needed for the whole CPU range, so the update to support the newer gen had to remove support for the oldest gen.
Turns out it's really hard to guarantee compatibility with several years of third-party products which weren't designed with your new fancy features in mind.
Newer doesn't usually mean more power, though. You can just have a power limit, it's fine. And I don't expect PCIe to get faster with a CPU upgrade anyway.
The flash space was pretty forseeable and they dealt with it, more of an excuse than anything.
I'd bet customer perception is also a factor; there's a risk the old boards (not even made by Intel) die and cause problems, and Intel wouldn't want to deal with press/comments like "This CPU stopped working after 2 months" or "I installed this new CPU, and within 2 months it killed my motherboard that had been working fine for 7 years".
They've released CPU upgrades with the same socket before, I'm sure they have the sales data to know how that performs vs new socket.
Laptops have outsold desktops for well over a decade, and their CPUs pretty much non-upgradable. I can't easily find a nice chart to reference, but intuition tells me the desktop industry is similarly trending towards complete "system" sales vs individual parts. In other words: Most people don't upgrade their CPU, they upgrade by replacing the entire system. If true, this also means the socket would be almost entirely irrelevant to sales performance.
