In Europe (and anywhere else north of the Tropic of Cancer), the sun is always approximately due south at noon. That’s the reason for the connection, and “midi” indeed means both south (in some contexts) and noon in French.