Dumb idea --- how about if we limit local models to specific domains --- medicine for example.

Most doctors don't care much about engineering or accounting or software development or 10000 other things that big vendor models address.

This area is yet to be really explored. Nvidia aims to provide the hardware to do so.

That's a fairly obvious idea, not dumb at all, but unfortunately it doesn't seem to pan out. Trying to specialize an LLM in one area harms its 'cognition' in all areas. For instance, if you train a coding model without all the Shakespeare and soap operas and Wikipedia and pirated Stephen King books and ancient Roman history and whatever, you end up with a worse coding model.

I'm not sure anyone really understands why.

The article is not backed up by reality. Why would use anything but a domain-specific LLM, if they actually worked?

The author is probably confusing RAG with pretraining. You can RAG on PubMed but you can't arrive at a competitive model by pretraining solely on it.