I will note I ended up at an institution (admittedly an academic one) that for my field is usually met with "Where?" because the funding allowed for some very nice quality of life considerations and more academic freedom/less pressure to support yourself with grants.

It's not impossible, but you do have to figure out ways to offset the risk and prestige hit.