Why worry about something that hasn't happened? Why not worry about a careless teenager wanting to get a sick Camera angle of the inside of your fridge? It's as valid a concern.
Normally I would agree with you, I often chide people for inventing problems to solve. But it is a little different when it isn't a hypothetical as much as an inevitable future occurrence.
As for refrigerators, I am just as worried about people trying to fly drones around my house as I am about people flying drones around planes, yes. I have it on good authority that anyone with the funds can purchase access to 3D drone-generated map of my city, my house included. A gentleman here in town who replaces windows doesn't bother to measure them anymore; he just uses the drone photos.