> Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers.
I'm not sure what you mean by this. Memory bandwidth is the main bottleneck for single-user decode. The bottleneck is actually more severe for end-user inference than cloud inference, because end users don't have the option to increase arithmetic intensity by computing tokens for multiple clients in the same pass.
One thing we've learned from Apple is the viability of spamming more LPDDR5X channels (up to 1024-bit total bus width on M3U) as a means of achieving high bandwidth while keeping the cost/capacity reasonable.
I'm not sure what you mean by this. Memory bandwidth is the main bottleneck for single-user decode. The bottleneck is actually more severe for end-user inference than cloud inference, because end users don't have the option to increase arithmetic intensity by computing tokens for multiple clients in the same pass.
One thing we've learned from Apple is the viability of spamming more LPDDR5X channels (up to 1024-bit total bus width on M3U) as a means of achieving high bandwidth while keeping the cost/capacity reasonable.