Oh it's worse than that just pupil shrinking in dark. Basically, cone cells see color and detail but need more light, rods see in low light but mostly serve peripheral vision. So your vision system is just wholly optimized to only doing detail work in fairly bright light; nighttime is for predator evasion.
Another way to look at that: We likely evolved decent color vision to identify edible fruits. That wasn't doable with vision in the dark (and we opted out of the olfactory tech tree), so now your accurate-vision tasks are optimized for daylight.
I just assumed more photons results in a stronger signal to my rods and cones. But maybe it's making my eye shrink the aperture instead (or both).