Your time is valuable. Never spend it with people who bring less value into your life than you would have got if you spent this time alone. Never ever (exception: if you are responsible for someone, like a child or elderly person). It's that simple. The hard part is estimating how much value somone/going out brings into your life.

Either way, you can call that rude and selfish, I call it a wise way to spend my limited resources.