The people of the United States are broadly free to build a home wherever they can afford to, comrade, including on land that would otherwise be used for farming.
(Actual answer: I know a bunch of people who live in houses in the middle of seemingly-nowhere in rural Ohio, and almost none of them farm anything at all. They just seem to like the space and the quiet and the desolate isolation.
The only farmer who I know is my parents' neighbor, who has a house few miles away from their place.)
Generally you are allowed on resident per 40 acres or something similar - farms are getting larger and that leaves plenty of land that doesn't have a house that could.
I like your version of America. Sadly, California's not that free. Some billionaire can't just buy up some land and just put in apartment/office/factory tower as they please, the local government and residents just aren't going to stand for that.
(Actual answer: I know a bunch of people who live in houses in the middle of seemingly-nowhere in rural Ohio, and almost none of them farm anything at all. They just seem to like the space and the quiet and the desolate isolation.
The only farmer who I know is my parents' neighbor, who has a house few miles away from their place.)