I mean humans do that too, and I don't think it's very unjustified. The "we deduce from a deep base premise P down a chain of inferences" picture is extremely incomplete and has been challenged all over the place - by normal people, by analytic and continental philosophers, by science itself, etc.
Not trying to say that LLM's are equivalent to humans but that the concept of reasoning is undefined.
And the fact that their performance does increase when using test-time compute is empirical evidence that they're doing something that increases their performance on tasks that we consider would require reasoning. As to what that is, we don't know.
But humans verify things. AI just fools you and I would say it is the biggest problem.I have with AIs.
They give me stuff that I do not know whether to trust or not and what surprises I will find down the way later.
So now my task is to review everything, remove cruft. It starts to compete against investing my time to deep-think and do it thoughtfully from the get go and come up with something simpler, with less code and/or that I understand better.
Not trying to say that LLM's are equivalent to humans but that the concept of reasoning is undefined.
And the fact that their performance does increase when using test-time compute is empirical evidence that they're doing something that increases their performance on tasks that we consider would require reasoning. As to what that is, we don't know.