Haha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have.

Yeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.

I don't know if that's, well, a rule of thumb, it might be, well, straight multiplication.

Well, yeah, the straight multiplication, er, well, "follows" from the rule of thumb. Hope that helps!

Saying, well a rule of thumb is, well, 100 billion bytes is a 100 gigabytes, is well, not a rule of thumb. It is, well, just the common definition.

Right. The rule of thumb is that the overhead size of the model that's not the weights is so vastly outweighed by the actual number of weights that it can be disregarded. My shorthand for that was to write "the weights take up ~100% of the size of the model". What then "follows", both in the sense that the explanation is written after the rule as well as that it logically follows, is that, well, 100 billion bytes is, you know, 100 GB. I don't see why you're, like, paying so much attention to this?

Right. That's, well, not a rule of thumb. Asking how much of a bottle of water is, well, water, and someone says "well, a good rule of thumb is that it's all water", is just an answer. You don't need an estimate when, well, there is nothing to estimate.

Yeah, well, that's just, like, your opinion, man.

Anyway, more seriously, I hope it's obvious by now that I don't particularly care that my means of communication is so offensive to you. I think you should, like, cry a river, build a bridge, then, well... get over it, you know?

And on the, er, "topic"? A rule of thumb for me does not have to be one for you, even if it's explicitly presented as such a rule. I thought that would be obvious but, well, here we are. Anyway, how's life been treating you?

A rule of thumb is, well, an estimation. 1 = 1 isn't an estimation, it's just, well, the same number.

At this point, what isn't much of an estimation anymore is that you are here to explain things that nobody's asked for, and like to assume people around you have been waiting for your pearls of wisdom. You're unable to realize when this isn't so. You're so full of yourself that you don't notice.

Also, in this particular thread, you started by wrongly making a correction of something that was clearly not an error, nor a wrong use of words, not even a misspelling. The problem is you started to post a correction before realizing that you didn't read right. That happens when one is more eager to boast one's own greatness than one is interested in the topic at hand. The result is that in this thread, you were clearly, as a rule of thumb, well, 100% wrong.

I understand that's you're frustrated from some deeply held software beliefs not holding up to examination, but making things up in a different thread is not a healthy way to work towards acceptance.

Oh, that is a useful rule to know! Thanks!

That's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.

And the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.

Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.

brb, going to see if 2nd hand mac studios are available!

Strix Halo, 128GB RAM. I got a refurbished Corsair AI Workstation for a smoking price ($2100) about two months ago. Lucky timing that it was in stock.

Strix Halo as well. Bought it for $1,800 new on sale and shoved an extra 4tb drive into it. Been amazing for local AI. Maybe not the absolute fastest thing (usually around 30t/s depending on the task) but has been awesome for a local AI box that I can solar power.

Nice. Mind sharing the solar side of your setup?

Couple of rack mount batteries and roughly 5kw of solar panels. Feeds into a subpanel so I can flip it when I want a couple rooms of solar on the house, or hook a generator up if needed. Can't power the entire house, but works well for thinks like computers, lighting, etc. And if I want to expand, just throw on more panels, or realistically, just throw on more batteries to store the juice.

Thanks - that’s what I want to do!

Usually theyre quantized. Also, there was a window where AMD 395+ W/128GB was just a high end $2500 hardware with unified gpu memory.

Yeah, here I am sitting deeply deeply deeply regretting not buying couple CMP 170HX at $200 or $350, knowing I could just flip them ethically at purchase price if nothing came of it... I could have just casually built a 128GB dual A100 local AI monster

I'm working with a lab that has a few Ampere GPUs on infiniband and they are just not compatible with the latest quants and vLLM updates. FP8 is about as low as you can go.

But they're reportedly a soft nerfed GA100 64GB/40GB at $1200, that's not more expensive and certainly can't be slower than a Mac Studio.

quantized + offload

I have an RX 6700 XT with 12gb vram and 64gb system ram. running dense models like 27b is difficult, but i can run IQ4/IQ5 qwen 122b-a10b or 35b-a3b at ~20tok/s

IQ4 qwen 122b-a10b would mean 61GB total size and 5GB active, so about 5GB of the model loaded into GPURAM plus any generated context, and 61GB of weights loaded into system RAM? I don't know if that math is correct, but does that run well? Wouldn't that only leave 3GB of system RAM?

MoE models can use system memory along with a GPU.

and get high token bandwidth?

Not badly so because MoE models(identifiable by "CoolName-xxxB-AxxB" naming scheme) have bunch of branches in the middle that only one out of all gets non-zero values. Each of branches aka "Experts" as well as top/bottom parts are significantly smaller than the whole, and so CPU emulation of CUDA operations mixed with GPU taking as much as possible become not so out of question, unlike for dense models("CoolName-xxxB" without "-AxxB")

Similar to a spark, which isn't blazing fast but usable.

dgx spark, nvfp4 so I have spare room for KV cache (context)