Does not matter much in this case. GPU bound.

Seems to be true, but also seems hard t obenchmark with max having more GPU cores too.

My understanding is that that tends to be more critical with LLMs than image/video gen models, which are relatively more compute vs. memory transfer intensive than LLMs

Performance might still end up being bounded by data transfer speed if SSD streaming is heavily used to make up for limited RAM. By comparison, it doesn't take many parallel-batched sessions to make LLM decode compute-bound on typical hardware (hence seeing very limited gains from even wider batching), but this just doesn't apply when streaming weights from disk, the setting is completely different.