Yeah I take them out too. The only exception is ear protectors (loops) which I wear in discoteques/parties. When I talk to people I don't usually take them out because I wear them for protection. If they notice I will explain and they don't mind.

In fact it's easier to hear them with those in anyway. I'm just very sensitive to sound and I have already damaged my hearing a bit when I was young.