Yeah I'm not sure what the exact context of the statement is.
I am absolutely certain that we have not already discovered let alone implemented the best possible learning algorithms. Humans have had more time to evolve, there's a great chance that we do learn more efficiently, and have developed specialized brains that are primed to learning things like how to navigate the physical world on planet Earth as bipeds.
That said, to say that we operate with less training data is just ignoring the reality of all the data we're training on at all times.
If we were to model in lossless fidelity what humans are capable of seeing, hearing, smelling, tasting, feeling, thinking consciously and subconsciously etc. essentially all the data flowing through our minds that we are constantly training on every moment of every day, even while we sleep/are unconscious, what sort of bitrate do you think would be required?
Modern LLMs train on datasets in the what, tens of terabytes in size? Let's call it 100 TB.
I would imagine that to losslessly reproduce the full suite of human sensory data (whatever that means for things like taste, touch, smell) would require a bitrate that hits that 100 TB total relatively quickly?
Let's stick to comparing language skills to language skills: at least in my experience with my two kids, they learn word formation patterns before they turn 2 — easy to notice because you see them make mistakes on exceptions.
LLMs needed how much training data to be able to do so?
FWIW, I still see them make up wrong words not following any grammatical pattern, esp in Serbian with less training data.
"...we're optimized for having not many experiences. You only live for about a billion seconds—that's assuming you don't learn anything after you're 30, which is pretty much true. So you live for about a billion seconds and you've got a 100 trillion connections. So [you've] got crazily more parameters than you have experiences. So our brains [are] optimized for making the best use of not very many experiences."
I think this is disingenuous comparison. When we read a book we can estimate the amount of data we're taking in based on the character count (each character being represented by some fixed amount of bits).
What you're suggesting on the other hand is something akin to counting the number of pixels on each page we look at. That's absurd overestimate of the amount of data a person reading is actually taking in.
I believe there is a point: we simulataneously ingest words, but also glyph shapes and learn acceptable variations between them (eg. serif vs non-serif, large x-height vs small, curlier or more elegant, playful letters...) — all of these contribute to our multi-faceted learning, but ultimately, we do seem to need less of the data to learn (how long it takes for us to learn to recognize letters vs OCR based on ML).
I am absolutely certain that we have not already discovered let alone implemented the best possible learning algorithms. Humans have had more time to evolve, there's a great chance that we do learn more efficiently, and have developed specialized brains that are primed to learning things like how to navigate the physical world on planet Earth as bipeds.
That said, to say that we operate with less training data is just ignoring the reality of all the data we're training on at all times.
If we were to model in lossless fidelity what humans are capable of seeing, hearing, smelling, tasting, feeling, thinking consciously and subconsciously etc. essentially all the data flowing through our minds that we are constantly training on every moment of every day, even while we sleep/are unconscious, what sort of bitrate do you think would be required?
Modern LLMs train on datasets in the what, tens of terabytes in size? Let's call it 100 TB.
I would imagine that to losslessly reproduce the full suite of human sensory data (whatever that means for things like taste, touch, smell) would require a bitrate that hits that 100 TB total relatively quickly?