That makes sense for undeveloped land (e.g. the Balboa Reservoir project). But there's very little of that in San Francisco. Realistically, adding to San Francisco's housing supply means replacing a lot of existing stock with higher density housing. If that higher-density housing is mostly or completely occupied by locals (by which I mean people who will be domiciled at that location), then it's a net win. But if not, then San Francisco residents end up worse off.
For example, say that a developer tore down housing for 1,000 people on some property, and then replaced it with housing for 3,000 people. That adds capacity for 2,000 people. But if 950 people were domiciled at the old property, but only 750 people are domiciled at the new property, then it's actually a net decrease in supply for people who reside in San Francisco. If the original property was old and run-down (which is what usually gets replaced in these situations), and the new property consists of units that have great appeal to wealthy people looking for a San Francisco pied-à-terre, then a situation like this is certainly possible.
Fortunately, that situation can be handled effectively through scale.
There is a limited number of wealthy people looking for a San Francisco pied-à-terre, and that is a market that can be exhausted without driving all other residents out. At which point standard market forces mean construction for the less wealthy will occur.
I generally agree with you from a policy point of view, but I think you're oversimplifying:
The number of wealthy people looking for a pied-à-terre in SF is not a fixed quantity. It seems to me that that number has grown significantly compared to, say, 10 years ago. And that that's mostly due to the increased prestige of the city (as perceived by that wealthy set). That prestige comes from economic growth, but also from a sense of how friendly the city is to pied-à-terres for wealthy people. So building more of them might just end up increasing the demand for them.
Sure, the process won't go on literally forever until the city is nothing but vacant luxury condos, but the total demand after that compounding effect has gone on a while may be much more than it appeared at first.
Over time, the number of people looking for a pied-à-terre is not fixed. This is true. This is also true for every other possible category of people.
Thankfully, this not-fixed number is also not infinite. Just as the housing market also does not need to operate over a fixed quantity of housing. Building more increases that not-fixed quantity so that the not-fixed quantity of demand might be met.
When you have an economic boom, you need to grow your infrastructure to match. Not sit around bemoaning that that might enable future growth that will, to a great degree, happen even if you don't grow infrastructure.
Look no further than the current housing crisis - accurately predicted in 2000 - to see the results of the notion that growing infrastructure is bad because it encourages use of infrastructure.
You just switched the subject of the conversation from "luxury condos for wealthy people" to "infrastructure". That's a tricky debate tactic. Of course you have to "grow infrastructure," nobody is arguing that in the abstract, but there's a lot of choices to make about what and how. Luxury condos are one type of thing that can be built, middle-income housing is another. They have different effects on the city. The current incentives that lead developers to building only luxury condos are not a law of nature or economics, they're a result of political choices by the city and state, and they can be changed.
Housing is infrastructure. Not building housing is not building infrastructure. If you don't build "luxury condos", then what was intended to be middle-income housing will become "luxury condos" because you failed to build sufficient infrastructure. This is not a hypothesis - this is an empirical observation from every city that's ever tried rent control.
There are exactly zero tricky debate tactics going on here.
For example, say that a developer tore down housing for 1,000 people on some property, and then replaced it with housing for 3,000 people. That adds capacity for 2,000 people. But if 950 people were domiciled at the old property, but only 750 people are domiciled at the new property, then it's actually a net decrease in supply for people who reside in San Francisco. If the original property was old and run-down (which is what usually gets replaced in these situations), and the new property consists of units that have great appeal to wealthy people looking for a San Francisco pied-à-terre, then a situation like this is certainly possible.