You can have both. I know folk who work in healthcare doing SW as they are passionate about health things. You can have it all if you step away from companies that are SW or IT type shops first and look for companies whose core business appeals to you. E.g. a white water rafting company needs SW people too.