I think there’s a very simple explaination actually.

Most people psychologically don’t like wearing or carrying technology unless there’s a really good reason. Most people also don’t like psychologically feeling isolated.

VR doesn’t have a good reason, and makes you feel isolated. No further rationale is necessary.