> Will a model that small suffer heavily from a 4bit quantization?

From my experience with quantized 7B llama models, avoid 3B if you can. Without benchmarks, I think this is a decent rule of thumb.