Recovery numbers should be stored somewhere safe, like your house. If your attackers have access to your house they could beat you up with a baseball bat instead of going through the trouble with finding your recovery codes and phishing your passwords.
Recovery numbers are recovery telephone numbers. If they sim-swap you, they have your telephone number. The end. Virtually all email accounts and online services tie themselves to your damn telephone number, and there are tens of thousands of people in each phone companies who can move your number.
