i’m also (seem apparently to be) aphantastic and draw quite well. it’s just like you say - i “reason” about a remembered shape and light based on facts and physics, not “observation”. i reconstruct how the thing should look based on what i know of it’s geometry, not what it looks like.
interesting to hear someone else say this! i’ve never quite put it into words.