You are worried about someone who physically has access to your phone and knows your passcode tracking you? That is not the problem being discussed here.
I know it's not being discussed, I am introducing it as an adjucent attack vector that is already an issue with Google Maps' location sharing. Google counters it by sending periodic emails to remind you that you're sharing your location, and with whom.
Being snooped on by someone with physical access to your device, as well as the opportunity to use it while unlocked (passcode or not) happens more times than one might think! It could be anything from a creepy colleague, or an insecure romantic partner.
