In my experience, users who genuinely don't want English will most definitely have their browser language set to the language they do want.

I think what you might be seeing is that many users are OK with English even if it's not their native language.