Well, that didn't take long, did it? 50% on ARC public test set [1] less than a week after the announcement of the prize. Though I have to say, the solution, at least superficially, does look like what Chollet alluded to: hybrid of LLM with "discreet program search/synthesis". Again, I'm not familiar with that field, so perhaps it's not at all what he had in mind, but it's intriguing. What do you think? Do you understand Chollet's idea enough to explain whether this solution is on the right track?
if scale is all you need then why don't we have AGI now?
Well, it's my turn to use the "dumb question" card :) We don't have enough scale, obviously! I don't know if scale is all we need for AGI to emerge, but clearly we haven't reached the end of benefits from scaling up. Until we do, it seems like the easiest and the most promising approach. Considering the size of Youtube as a training corpus, we are pretty far from that end. Are there reasons to think otherwise?
LLM architecture has been rapidly changing
Aside from a mixture of experts architecture, which has its pros and cons vs a single large monolithic model, I'm not sure what has fundamentally changed in the architecture of the original transformer proposed in 2017. Minor tweaks here and there, sure, but it's pretty much the same model, no?
if scale was all you needed then shouldn't Google be leading the way?
Oh, a lot of people have been asking how could Google drop the ball so bad, for so long. There are reasons, both well known, and hidden from outsiders, but compute is not all you need to scale, you also need vision, clear direction, and effective coordination of efforts from multiple teams. Something that OpenAI has (or at least had), and which is rare at large corporations.
Re: academics - good ideas get noticed. Today, if someone discovers something good they don't even need to publish. Post a github link on r/MachineLearning, together with benchmark results, and let people test it.
I'm worried. More worried about displacement
This is very interesting - I haven't even thought about it. It's very possible that in the beginning after the mass layoffs, GPT-5 will screw some things up, in subtle ways, and only GPT-6, some time later, will be able to fix them. People need to be ready for that. The period between GPT-5 and GPT-6 will be rough in more ways than I imagined.
> Well, that didn't take long, did it? 50% on ARC public test set [1] less than a week after the announcement of the prize.
I think you also misunderstand the challenge and very clearly the author misunderstands neurosymbolic AI, as he implements it... He has it generate programs and then search over those programs. He also tries to challenge Francois's claims (What it means about current LLMs) while he actively performs "claim 1" and misunderstands the context of "claim 3" (model weights are frozen, so there is no online learning. This is distinct from what's going on here, since he is updating the model's priors before answering. But whatever insights the model has gained from this exercise do not persist after execution. i.e. there is no continual learning). "claim 2" is just irrelevant.
A key part that is concerning to me is this
> In addition to iterating on the training set, I also did a small amount of iteration on a 100 problem subset of the public test set.
Potentially the confusion is that each data file has a pair where one has "train" and "test" which is your sample and then your actual input/output pair. So you're only supposed to train from ARC-AGI/data/training, but you cannot use ARC-AGI/data/evaluation for anything other than... evaluation.
Not to mention that we don't know what data is in GPT. It would not be surprising if this was in it. Maybe they filtered out the official repo but there are plenty of examples around the web. Did they take check for all such examples? If not, then the result is entirely invalidated.
There's a lot of reason to believe information leakage exists here.
So I'll wait for an open solution before I start to
> Re: academics - good ideas get noticed.
I also need to stress that ARC has been tested in LLMs for quite some time now. You can go see it in both the GPT2 and GPT3 papers. Though these are different versions than the one in the current competition. That version has ARC-e and ARC-c for easy and challenge. GPT2 gets 68.8/51.4 with "zero-shot" (I'm not confident) and the original LLaMA gets 78.9/56.0. So really, if people aren't aware of ARC (prior to the video) then it really demonstrates that they are not doing this kind of research or even reading the papers.
And I think we need to be clear that we need to differentiate academics and normal people. And I'm including anyone with a "machine learning researcher" and "machine learning engineer" title in "academics." This is where all the building is happening and these people all should be very aware of ARC. The public not knowing, well, that's a whole different story and isn't really all that important now is it. They're not the ones improving these systems (for the most part. There are of course always exceptions to the rule).
if scale is all you need then why don't we have AGI now?
Well, it's my turn to use the "dumb question" card :) We don't have enough scale, obviously! I don't know if scale is all we need for AGI to emerge, but clearly we haven't reached the end of benefits from scaling up. Until we do, it seems like the easiest and the most promising approach. Considering the size of Youtube as a training corpus, we are pretty far from that end. Are there reasons to think otherwise?
LLM architecture has been rapidly changing
Aside from a mixture of experts architecture, which has its pros and cons vs a single large monolithic model, I'm not sure what has fundamentally changed in the architecture of the original transformer proposed in 2017. Minor tweaks here and there, sure, but it's pretty much the same model, no?
if scale was all you needed then shouldn't Google be leading the way?
Oh, a lot of people have been asking how could Google drop the ball so bad, for so long. There are reasons, both well known, and hidden from outsiders, but compute is not all you need to scale, you also need vision, clear direction, and effective coordination of efforts from multiple teams. Something that OpenAI has (or at least had), and which is rare at large corporations.
Re: academics - good ideas get noticed. Today, if someone discovers something good they don't even need to publish. Post a github link on r/MachineLearning, together with benchmark results, and let people test it.
I'm worried. More worried about displacement
This is very interesting - I haven't even thought about it. It's very possible that in the beginning after the mass layoffs, GPT-5 will screw some things up, in subtle ways, and only GPT-6, some time later, will be able to fix them. People need to be ready for that. The period between GPT-5 and GPT-6 will be rough in more ways than I imagined.
[1] https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50...