I'd like to see an incident where caffeine has caused such severe cutaneous and dermatologic reactions as modafinil has. If such drugs were deregulated, we would certainly see the number of victims of these horrible reactions go up.
Can't really call them victims if they go into it with knowledge of the drug and it's effects.
Now if deregulation also resulted in drug companies actively concealing negative side effects, and thus duping consumers, then they would have been victimized.