Quite simply, housing has to be either affordable, or an investment. Those two things are literally, by definition, mutually exclusive. If something is an investment it means that it must appreciate in excess of inflation over time. That means, by definition, that it will become unaffordable over time. This is why, whenever you hear terms like "affordable housing" you can safely shut your brain off. There will never be affordable housing, the whole idea is preposterous.
Saying "we're going to build affordable housing" is putting the cart before the horse: it is the building that causes the affordable, not the affordable that causes the building!
So to understand this we look to the supply-side. Basically, scarcity causes the investment mindset. This is very very complex and interlocks with much broader structural changes to the economy that have taken place since houses were considered affordable, structural changes that are much larger even than housing.
Used to be, you worked for a company for 10+ years. A company would move into a town (by building a big factory, say), and work with the town to build housing for its employees. Think the Kodak plant in Rochester, for example. In the 20s and 30s, Kodak's employees were having trouble finding places to live. So, they formed the Kodak Employees Realty Corporation and built a couple thousand houses for their employees in Rochester. There are many examples of this in cities all across America.
Nobody would think to do this any more, because the economy has changed structurally. There are less people involved in building stuff and generally we have all decided that changing our physical environment is simply a thing that we don't do any more. Our entire economy has moved to one that's based on physical things (building things, producing things, moving things, in the physical world), to one that is based on information. You can move bits around and make profit without ever needing to touch the physical world *.
We created financial capital to accomplish the actual important thing, moving atoms around in the real world. Now, financial capital has in some sense "taken over" -- moving bits is no longer a thing that you do to make moving atoms around easier, but a goal in and of itself. Financial capital has taken the head of the table, pushing out physical capital. In essence what has happened is that the United States has gone from being a complete organism (lots of muscle, a little bit of brain to move that muscle around) to being a subset of a much larger organism (all brain). That we're "all brain" is why we can't move atoms here; our inability to move atoms here causes scarcity in housing; scarcity in housing causes investment mindset in housing; housing appreciates.
Causality of all of this, I think, is far more complex than some simple economic explanation. It's deeply cultural and involves the reorganization of the entire global economy. This is why you're never going to lose money on your house.
* this, of course, is _not actually true_, but rather a geographic illusion. The physical world is touched more than ever, its just that the changes to the physical world (which are often horrific) happen in other places besides the USA. Southeast Asia, China, Africa etc.
I think it's possible to housing to be affordable and for it to behave more similarly to to something like gold than something like a bond or stock. That said, it's quite tricky because of property taxes - if they applied to gold imagine what it would do to gold bugs. Fortunately, real estate can be rented, which isn't really a thing for gold.
But to complete the analogy, it seems like the goal should be for residential values in terms of build price per some typical target like ~1500sqft/3br (regardless of form) to remain relatively constant adjusted for inflation and relative to % of median wage while allowing land value to adjust density to match. This allows people to gain from selling to convert from low density to high density, but keeps affordability relatively constant. This likely means that some kind of alteration in tax or sale regime to promote transition to higher density and not just transition to higher value should be present in the market. (IE in places where 1acre is going for ~$1million, it should make more sense for someone to buy and convert to multiple units than for someone to just buy.) This would lead to expensive areas starting to convert to higher density leaving lower value areas to the main option for single family inflows. (Of course this will have loopholes - obviously it incentivizes the "rich enclaves using a COOP" type of model if the tax pressure were assessed at sale - so it need to be thought through more clearly.)
Edit: That said, I do think there's something to your organism analogy. For example, all of what I wrote above given, it explains why it'll still be impossible to find a good plumber and that does interact with the price of building.
I lean reflexively libertarian on housing (mostly just my psychological NIMBY backlash). Therefore my take is that the market would naturally accomplish the scheme you've described, no need for policy, if we simply remove constraints on supply. Basically, the "stop hitting yourself" policy of housing :)
Unfortunately it doesn't work that way and never will, not just because of NIMBY but because of the reasons I outlined above.
Saying "we're going to build affordable housing" is putting the cart before the horse: it is the building that causes the affordable, not the affordable that causes the building!
So to understand this we look to the supply-side. Basically, scarcity causes the investment mindset. This is very very complex and interlocks with much broader structural changes to the economy that have taken place since houses were considered affordable, structural changes that are much larger even than housing.
Used to be, you worked for a company for 10+ years. A company would move into a town (by building a big factory, say), and work with the town to build housing for its employees. Think the Kodak plant in Rochester, for example. In the 20s and 30s, Kodak's employees were having trouble finding places to live. So, they formed the Kodak Employees Realty Corporation and built a couple thousand houses for their employees in Rochester. There are many examples of this in cities all across America.
Nobody would think to do this any more, because the economy has changed structurally. There are less people involved in building stuff and generally we have all decided that changing our physical environment is simply a thing that we don't do any more. Our entire economy has moved to one that's based on physical things (building things, producing things, moving things, in the physical world), to one that is based on information. You can move bits around and make profit without ever needing to touch the physical world *.
We created financial capital to accomplish the actual important thing, moving atoms around in the real world. Now, financial capital has in some sense "taken over" -- moving bits is no longer a thing that you do to make moving atoms around easier, but a goal in and of itself. Financial capital has taken the head of the table, pushing out physical capital. In essence what has happened is that the United States has gone from being a complete organism (lots of muscle, a little bit of brain to move that muscle around) to being a subset of a much larger organism (all brain). That we're "all brain" is why we can't move atoms here; our inability to move atoms here causes scarcity in housing; scarcity in housing causes investment mindset in housing; housing appreciates.
Causality of all of this, I think, is far more complex than some simple economic explanation. It's deeply cultural and involves the reorganization of the entire global economy. This is why you're never going to lose money on your house.
* this, of course, is _not actually true_, but rather a geographic illusion. The physical world is touched more than ever, its just that the changes to the physical world (which are often horrific) happen in other places besides the USA. Southeast Asia, China, Africa etc.