But it did learn some basic arithmetic. If GPT4 can multiple 2 4-digit numbers will you change your mind?

GPT saw a bunch of arithmetic and can repeat it. That’s the joke behind the Reddit usernames and /r/counting.

The four digit number thing is just the current lower bound of where it gets confused because a lack of training data.

Once you teach a patient 8 year old the rules of multiplication once, they can multiple any two numbers (that they’ve never seen before) with an arbitrary number of digits. An LLM cannot and will not ever be able to do that because it is a specific tool and it is not designed to do that (doing that would be a bad outcome for an LLM since we have different tools that can do multiplication much more efficiently).

So yes, if a LLM learns rules based math (which it is not intended to do) I’ll eat not only my, but every hat in existence.

They probably mean that if you test it on common NLP benchmarks it performs better.