IMO, it's rational economic thinking (if perhaps a little selfish): 90% of your net worth is in a single asset (either due to poor financial planning on your part, or because one asset has ballooned in value), so you will use your municipal voting power to protect that asset.
Unfortunately this is a negative feedback loop. Essentially you are saying that because housing prices are expensive and people are over-leveraged, they will vote for any legislation that will protect and increase housing prices. This legislation increases prices further, which means even more over-leveraged housing owners.
We need something to break out of this negative feedback loop.
This comes up a lot. Here's the thing - if you own a SFH in San Francisco, and suddenly you had the right to tear it down and a build a 3 unit apartment building, would that increase or decrease the value of your property?
I'm not sure, but I lean toward increase, because you do own the land beneath your house. If you were the only one who did this, then I'd say yes, almost certainly. However, if everyone did this, supply would increase, so the question is, would that result in a decrease in prices, offsetting the value of converting a single family residence into three units?
Personally, I doubt it. In my neighborhood in SF, nice 3br flats sell for about 85% of the cost of a 3br house. There's a lot of slack here to make up for the reduction in prices that could result from more units, and honestly, I'm not eve sure that increasing density in SF would lower prices in the first place.
I know that last one may seem counter-intuitive, but keep in mind, Manhattan generally has housing costs than SF, so greater density certainly doesn't necessarily mean lower housing costs. A lot of people have pointed out that SF's high housing prices may be impeding further economic growth in the region. It's entirely possible that greater density would attract even more high paying workers to the region, with even greater economies of scale, higher levels of economic activity, and so forth. It's not out of the question that greater density could result in higher, not lower, housing costs.
I'm not convinced that SF nimby-ism is driven by a desire to maximize the dollar value of real estate assets. I'm also not really convinced that it's unique to San Francisco. My guess is that most medium-density neighborhoods that have been in a steady-state for 75+ years are resistant to change, regardless of whether they're located in SF or elsewhere.
The break out should be that, in terms of metrics of the physical world, SF is a horrible city to establish a business in. There are plenty of bubbles going around, but the "Bay Area or Bust" in tech is certainly one of them. The cost of living is so preposterously high, and the infrastructure so bad, in a rational world businesses would be throwing money at moving thousands of people out of the area to much cheaper cities instead of sucking up the ludicrous costs.
But what about the second cohort of people (those who's houses have ballooned in price) who aren't over-leveraged (because they've owned it for a very long time) and have every reason in the world to vote to block things that may cause their greatest asset to lose value? To them, I would think there is no negative feedback loop.
A rational but liberal approach would be "I need to protect my home value by ensuring the largest risks to continued success of SFBA economy are handled responsibly."
Both are rational (if selfish), but there is definitely a prevailing economic conservatism. I've felt it myself since levering up a few years ago (at what was then the top).
Another similar rational but liberal approach would be "I want my non-homeowning friends and co-workers to continue to be able to afford the Bay Area, rather than get priced out over time."
As a young SF homeowner, that's more important to my quality of life than the price of my home going up.
