A fast, low power ROM is the key ingredient to near term local inference with large models at low power. If I could offer you a $500 ROM that provided the model data for frontier inference on power similar to a desktop GPU, you would buy it, and consider it a bargain, even when it came time to pay another $500 for the upgrade.
Surely it is clear to you that Read-Only /Memory/ does not /compute/, and our need is to compute through the data in the memory... That is CiM - a technology not that similar to ROM... Because a plain ROM does not solve problems in this area...
In other words,
> If I could offer you a $500 ROM that provided the model data
Then I would have a physical token containing what I already had as a file, and the problem of running that file into something efficient would remain... Because the ROM does not "run" its contents...
> Surely it is clear to you that Read-Only /Memory/ does not /compute/
> Because the ROM does not "run" its contents...
Conventional GDDR/HBM don't compute either, yet inference is implemented using these.
Compute isn't the inference bottleneck. Inference requires high bandwidth, high capacity memory. The compute resources necessary are fungible, comparatively cheap and already available, at least for a small number of concurrent loads, such as in most local inference use cases.
> Then I would have a physical token containing what I already had as a file
I suspect you are not grasping what I mean by ROM. Dense, high performance ROM would not be the hardware equivalent of a "file", with performance bottlenecked by low bandwidth, high latency storage media, serialized for RW coherence reasons. It would have extremely high bandwidth, on par with GDDR, low latency due to a dedicated high performance bus, high concurrency due to a lack of any RW coherence obligations, and operate at low power (no gate leakage, no dynamic refresh,) and low cost compared to equivalent GDDR/HBM capacity.
Essentially what high performance ROM would provide is high capacity, low power HBM, albeit read-only. At that point all you need is sufficient TOPS to run the inference algorithm. The compute part is already available, affordable and readily scales up and down as per performance/cost/power budgets.
But ROM has a massive disadvantage being static. So, either it is cheap and practical "like a CD", or decision making will be forced to do its evaluations.
We have a von-Neumann architecture RAM<->CPU, which is really suboptimal for running current relevant Neural Networks ("RAM<----...---->CPU"). Advantage: flexible.
We have a CiM with Taalas HC1 which has the massive and enabling advantages of running NNs very fast and very energy efficiently.
What could high-speed ROM bring? It must be a good combination of "fast" and "cheap" to to be "interesting" for the market, between those two contenders.
I believe that "practical" as in "replaceable" is also a fundamental property of what we desire in this field: the Processing units are not all there is, also the side-RAM (for context, kv-cache etc.) is a necessary part of the system, so the NN-container is just a piece (which needs expensive co-parts). Whether the NN-container is CiM or not, it will be critical if it can be replaced (like a cartridge, disk, etc.) so that the other parts will not need replacement with it.
> I believe that "practical" as in "replaceable" is also a fundamental property of what we desire in this field
I suspect that there is a important frequency factor in in the "replaceable" calculus. Already I see people dragging their feet about adopting newer models once they've found familiarity with some older model: "good enough" is a thing. I know there are industries where "validated" is a concept, and they do not ride wave crests. So, if we imagine that as all this eventually shakes out and we're not replacing models every few months, but instead with about the same frequency as our cell phones or similar, the ROM model works. If the performance and price make this pattern highly appealing, then that's what will win, certainly for local inference. If some datacenter operator could, today, adopt a ROM approach that cut their power budget by a large factor, but had to suffer 2-3x longer model update cycles, they'd likely consider it.
For better or worse.
I have no problem with CiM as a concept. If it can reduce power/size/cost then it's another avenue that inference will probably incentivize, where incentive has previously been insufficient. As we both agreed long ago in this thread this new era is motivating things that were previously neglected, and CiM is possibly a part of that. My dream is that all of these get a hard look as people try to figure out how to run all of this without enormous gigawatt sucking datacenters that rival DOD program budgets.
You missed the whole point of Taalas HC1: that it is Compute-in-Memory.
> 2. Merging storage and computation // Modern inference hardware is constrained by an artificial divide: memory on one side, compute on the other, operating at fundamentally different speeds. // This separation arises from a longstanding paradox. DRAM is far denser, and therefore cheaper, than the types of memory compatible with standard chip processes. However, accessing off-chip DRAM is thousands of times slower than on-chip memory. Conversely, compute chips cannot be built using DRAM processes. // This divide underpins much of the complexity in modern inference hardware, creating the need for advanced packaging, HBM stacks, massive I/O bandwidth, soaring per-chip power consumption, and liquid cooling. // Taalas eliminates this boundary. By unifying storage and compute on a single chip, at DRAM-level density, our architecture far surpasses what was previously possible.
A fast, low power ROM is the key ingredient to near term local inference with large models at low power. If I could offer you a $500 ROM that provided the model data for frontier inference on power similar to a desktop GPU, you would buy it, and consider it a bargain, even when it came time to pay another $500 for the upgrade.