This has been my experience too. And that conclusion, I've found that if I really don't want to deal with such quirks, I can indeed literally just run the same docker compose file in rootful mode, and it works.

Making the changes required to run under rootless is often very simple.