Because the code that knows how to call the VDSOs is in libc, and how VDSOs work is very architecture dependent. Not saying you couldn't in theory do it. But at that point you might as well just load libc.