Oh, interesting. "!" is token id 0 in the Qwen tokenizer; I wonder if there's some tokenizer shenanigans either in inference or training that end up causing this specific behavior.