> I am baffled as to why academic institutions aren't failing to recruit
Because academics (hi!) aren't in it for the money. If you value making lots of money, or having work-life balance, then academia isn't for you (nor is social work, teaching, nursing, fine art, etc.). But this probably makes it hard to hire RSEs who aren't academics masquerading as software engineers...
I agree, but there has to be a balance between doing it for love and getting a living wage. One of my colleagues, an Associate Professor, works in Halfords (shop/garage chain in the UK) on the weekends. That isn't right. And many of these folk are incredibly intelligent. They are Ricks amongst Mortys. They don't deserve to be paid not-quite-enough to afford a house and kids.
Because academics (hi!) aren't in it for the money. If you value making lots of money, or having work-life balance, then academia isn't for you (nor is social work, teaching, nursing, fine art, etc.). But this probably makes it hard to hire RSEs who aren't academics masquerading as software engineers...