Two bytes per pixel for, say, a single 640x256 pixel screen would already use more RAM than available on a 256k Amiga 1000.
RAM size only ha to double a few times for this to become an entirely outdated concern, but that's what they had to work with at the time they designed the chipset. Still no excuse for AGA!
But you can do RGB 565, which is exactly what mobile GPUs do (or used to do, back when memory cost and bandwidth were more limited).