That's a reasonable option, just be aware that you get about 1/3 as much memory ...

jbellis 36 days ago | parent | context | favorite | on: Jamesob's guide to running SOTA LLMs locally

That's a reasonable option, just be aware that you get about 1/3 as much memory bandwidth with the M5 Pro, or 2/3 with the M5 Max [now you're at $4100 for the lowest-end]. So both your prefill (flops-bound, M5 has a lot less) and decode (bw-bound) will be slower.