There is a bug in llama-cpp for qwen/gemma models, use vLLM instead | Hacker News

Hacker Timesnew | past | comments | ask | show | jobs | submit

verdverm 59 days ago | parent | context | favorite | on: Ask HN: Has anyone replaced Claude/GPT with a loca...

There is a bug in llama-cpp for qwen/gemma models, use vLLM instead

pdyc 59 days ago [–]

what bug and it affects what?

verdverm 59 days ago | [–]

it's a prompt cache invalidation bug that causes all input to be reprocessed instead of getting preloaded

There are other reasons to prefer vllm to llama-cpp as well

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact