In my experience it's okay to have some troll or odd users. If they do targeted campaigns against single users or small groups we ban them.