Maybe we're getting close to what makes me doubt the utility of LLMs. Much like humans, they are quick to employ System 1 instead of System 2. Unlike humans, their System 1 is so well trained it has a response for almost everything, so it doesn't have a useful heuristic for engaging System 2.
For me the utility is here today. Even if I have to carefully probe and check the responses there are still plenty of things where it's worth it to me to use it right now.
