I don't get the argument; how do you decide between using 'a' or 'an'?

You use 'an' when the word that comes after it begins with a vowel.

They pick random top-k next token based on their amazing 4chan/reddit training data, duh.

So you you think a model if asked.

"There is an animal very similar to a crocodile but I cannot remember it's name"

and the model responds with

"I believe the animal you are thinking of might be " ("a" / "an")

Are you saying that it would pick the result fairly randomly and then based on it's choice pick an animal that starts with a consonant or a vowel?