Personally I don't have a problem with poor people living right next to rich people. I think the more we isolate ourselves from each other, the more we lack empathy for each other. Having been homeless, my personal experience is that people who haven't been homeless don't get it at all. The first night I had to think about where to put my stuff, I realized I was not prepared to deal with that.
The problem is not poor people living next to rich people. SROs would be fine if they were simply people with very little income who take care of themselves and their community. But you know what actually goes on there.