Given that that phrase is a snippet I've heard over and over in news coverage, it's inevitable that it will be repeatedly applied as other than yourself. It plays into a widespead mythology that success is purely a matter of virtute/good-attitude (anti-straw-man: I didn't say attitude has no effect at all), and a concommitant view that for those who did not succeed it's due to their unvirtuousness/bad-attitude. How could it not be used to "look down on those who tried and failed", given the part about "... stop the people who don't want something badly enough. They are there to keep out the other people." - that wording invites anyone who succeeds to think that the "other people" are lesser, the ones the wall is there to keep out.

And critically, this doesn't all exist in a vacuum. His intent might have been entirely benign, but that doesn't mean the overall effect will be entirely benign.