Hacker News new | past | comments | ask | show | jobs | submit login

You should be looking at 7-8b sized models, then. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct is considered pretty strong for its size. That said, you shouldn't expect more than glorified autocompletion at that point.



I'm not familiar with the practical memory requirements on Macs but I suspect that with 16gb of integrated ram you won't have issues running 14B models even at q6_k, certainly would be fine at q4 and it's definitely going to be capable of writing code based on instruction as well as minor refactoring, generating docstrings etc.


The model itself will fit just fine, of course, but you'll also want a large context for coding. And then since it's integrated RAM, it's also used by everything else running on your system - like, say, your IDE, your compiler etc, which are all fairly memory hungry.

Also keep in mind that, even though it's "unified memory", the OS enforces a certain quota for the GPU. If I remember correctly, it's something like 2/3 of the overall RAM.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: