It's totally possible that humans don't do reason. It's possible that the parrot in our brain makes the decision, and then the "frontend" of our mind makes fake reasons that sound logical enough.
But it's just a possibility, and I don't find it's particularly convincing.
Look up split brain experiments [1]. Basically, in patients where the corpus callosum is severed to some degree, the two halves of the brain have limited communication. Since the two halves control different parts of the sensory system, you can provide information selectively to one or the other half of the brain, and ask that part of the brain to make choices. If you then provide the other half of the brain the wrong information, and ask it to reason about why it made a choice, the other brain will happily pull a ChatGPT and "hallucinate" a reason for a choice neither that half nor the other half of the brain ever made.
While that does not prove that we never apply reasoning ahead of time, it is a pretty compelling indication that we can not trust that reasoning we give isn't a post-rationalisation rather than an indication of our actual reasoning, if any.
No but that does happen a lot of the time. The difference is that we can choose to engage the deductive engine to verify what the parrot says. Sometimes it's easier to do so (what's 17×9?) and sometimes it's harder (a ball and a bat costs $1.1 and the bat costs $1 more than the ball, what does the ball cost?)
You can ask ChatGPT to show it's working too. It's likely, for many things, (as it is with humans a lot of the time) that the way it does it when it walks you through the process is entirely different to how it does it when it just spits out an answer, but it does work. E.g. I asked it a lot of questions about the effects of artificial gravity using a rotating habitat a few days ago, and had it elaborate a lot of the answers, and it was able to set out the calculations step by step. It wasn't perfect - it made the occasional mistakes - but it also corrected them when I asked follow-up questions about inconsistencies etc. the same way a human would.
(Funnily enough, while both your example questions are easy enough, I feel I was marginally slower on your "easy" question than your "harder" one)
Maybe we're getting close to what makes me doubt the utility of LLMs. Much like humans, they are quick to employ System 1 instead of System 2. Unlike humans, their System 1 is so well trained it has a response for almost everything, so it doesn't have a useful heuristic for engaging System 2.
For me the utility is here today. Even if I have to carefully probe and check the responses there are still plenty of things where it's worth it to me to use it right now.
Fast thinking/slow thinking by Daniel Kannerman (and the Unlocking Project) is a book people need to read and understand.
When we slow think, we work things out, when we fast think we are parrots.
Hmm. If you want to argue that ChatGPT has half of what an AI needs, I could buy that a lot more than I buy "ChatGPT is the road to AGI".
Do inference engines have the other half? Or Cyc plus an inference engine? Can that be coupled to ChatGPT?
My own (completely uninformed) take is that such a coupled AI would be very formidable (far more than ChatGPT), but that it will be very hard to do so, because the representations are totally different. Like, really totally - there is no common ground at all.
I just meant "half" in the sense that there are two major chunks that are needed. I did not mean that each "half" was used as much as the other, or was as much work to implement.
But it's just a possibility, and I don't find it's particularly convincing.