I've worn my Vibrams exclusively (even to 2 weddings) for the past 4 years, and it's been great for my social life. It's probably the best ice breaker ever invented. Strangers regularly approach and start conversations.
That is a great reason to wear them. My wife and I both wear Vibram's (we even did a Tough Mudder event in them), and they're great for starting conversations on the trail. Of course, if you don't like small-talk with strangers, they will be a nightmare :)