I don't understand copyright law particularly well, but I'd expect it does matter here.

The TOS only applies to the person extracting the data. So if person A extracts the data, violating TOS, then puts it up on a website where it is downloaded by person B (who did not necessarily ever agree to Google Maps' TOS), then only A has broken TOS, B hasn't. But B may have violated Google's copyright.

Is that right or am I missing something?

It's against Genius' TOS to copy their lyrics.

Luckily for Google, Genius doesn't "own" any lyrics.

Seems like the law should be written so Google doesn't "own" any of their mapping data either.

They didn't create the roads, and there's no human actively taking the photos.

It's not like the law cares about effort when it sided with Google against Genius.

And Google doesn't own those buildings nor the architectural design.

Why do you say that, seems like a good comparison to make.