I think some countries (such as the UK) recognize religious and ethical reasons. For example you can say that you're a vegan and can't take the vaccine because they're tested on animals.
But I'd rather be injected with a litre of smallpox than go vegan.
If you have a suppressed immune system from other medications, then other illnesses like the common cold and the flu can kill you.
This is why people who take immunosuppressants typically avoid going out in public, and when they do they wear serious masks intended to protect them from others.
Immuno-compromised people aren't some isolated hobbit species, they're people like you and me that have jobs, run errands, learn in schools, consume food, often live with other humans who have to go out in public, etc. And new folks with sub-par immune systems are born or created every day.
Many vaccine-resistant folks will throw this population away from the entire risk analysis equation, treating them as a mere error term. At least, this has been my observation. In my opinion they should just admit they don't care about the fate of others beyond the 'thoughts and prayers' passive level.
You're just making things up. Lots of drugs screw with your immunity to various degrees and for the most part they live normal lives. This difference now is we have a very contagious and much more deadly disease.