I think that's debatable. The quality of your measurement is then simply a function of how well your tests are in the first place. Do you really think giving a person FizzBuzz is any kind of accurate predictor of future success at a job?
(Switching back to my old account because rate limits.)
I wouldn't ever use something like FizzBuzz to assess a candidate. It would be more of "here's a mostly finished sample application with a corresponding SQL file, add this feature (e.g. a search bar for a blog) and fix any (intentionally introduced) security bugs you find".
They would be evaluated based on how successfully they complete the main task, and if they have an eye for finding/patching vulnerabilities, that's a bonus that can be used as a secondary selector if a lot of candidates pass. If no one does, it won't be used against them.
That's how I'd approach it, personally. Something specific to the kind of work we're doing, but abstract enough to be approachable without a lot of insider knowledge.
FizzBuzz isn't a test of whether a candidate will succeed, but of whether they will definitely fail. Someone who passes may be good or bad, but someone who can't do it is definitely not qualified.
Someone who can't do it is definitely not qualified to do fizzbuzz level programming off the cuff in a stressful interview situation.
Whether that tells you anything about their ability to do fizzbuzz level programming in a more normal work environment is an open question.
There are a few HN readers who've had experience of "choking" on fizzbuzz level programming tests during interviews - even though it's trivial there's something about the interview situation that causes some people to sometimes freeze.
I think that's debatable. The quality of your measurement is then simply a function of how well your tests are in the first place. Do you really think giving a person FizzBuzz is any kind of accurate predictor of future success at a job?