I gave up - family of 5, but I do the following: my iPhone auto-silences unknown callers (system setting) and I only share my Google Voice account publicly (ie, only friends/family have my direct line). I also have scam-block from my carrier.

It seems to work quite well. I miss an occasional call but have progressively whitelisted (add as contacts) all numbers from school/doc/programs as well.

It does take some diligence but it feels better than getting sideswiped by some rando.