Yeah, it's a big flaw in the setup. The instructions I linked have users add it temporarily, to unlock the rest of the options, and remove it afterwards.
The problem isn't that Google knows your phone number but that they will use it (or more precisely the attacker will use it after redirecting your texts) for account recovery which you can prevent by removing the number.
