- Have checked their spam folder at least once every month over the course of the past two years, and
- Have marked a set of emails as spam on at least 1 day every three months over the course of the past two years.
I'm very confident that, within the next 2 years, we will not have an accurate enough classifier that would cause > 80% of these people to simultaneously (a) check their spam folders less than one day each year, and (b) mark emails as spam on less than 1 day every year.
Oh, and to help prevent cheating:
- Let's say the classifier must not have been trained with any emails in the world that are sent more recently than 1 month prior to the beginning of its trial.
I'm also very tempted to say we don't need to worry about AGI before we can achieve the above for 30% of these people, but I'm less confident in this one.
[1] I suppose, as a practical matter, even if we had such a classifier, this would be untestable without access to everyones' email accounts. So, for the sake of argument, let's say our population under study itself is a simple random sample of 100M accounts from the set of providers who service > 50M accounts each and who are willing to run such a test at the time of the trial.
That's cheating because improving AI is working on both sides - it's making better classifiers but also better spam generators :)
In the future with general AI classifying spam will be a hard problem even for people.
Imagine you get a message looking like it's from your facebook friend with his catchphrases telling you about his last trip and saying how great that travel agency was :) Spam or not?
Quite an interesting point I hadn't considered at all. On the one hand I'm wondering: what's your suggestion on how to address this with minimal changes to my criteria? On the other hand I'm wondering: well, if the analog of this is that AGI might get solved with more AGI, then that's only going to make me less likely to be worried in the first place!
I had issues with just taking "current spam data, not trained on that particular data":
- It allows trainers to hard-code future rules based on their experience of what has passed through past filters, even if their model isn't technically trained on this dataset
- You might get similar emails sent to different mailboxes and the instances not included would still be allowed (and I don't really want to go down the rabbithole of defining a similarity metric between emails)
- I think I want to allow spammers to evolve their capabilities at least using current techniques, which we all presumably agree is "less than AGI". After all, intelligence implies adapting to a dynamic environment. It's not really going to feel like AGI (and certainly not going to make me worry) if it looks like AGI is trivial to outsmart by humans or less-than-AGI techniques.
- Have had an email account for at least 5 years,
- Have checked their spam folder at least once every month over the course of the past two years, and
- Have marked a set of emails as spam on at least 1 day every three months over the course of the past two years.
I'm very confident that, within the next 2 years, we will not have an accurate enough classifier that would cause > 80% of these people to simultaneously (a) check their spam folders less than one day each year, and (b) mark emails as spam on less than 1 day every year.
Oh, and to help prevent cheating:
- Let's say the classifier must not have been trained with any emails in the world that are sent more recently than 1 month prior to the beginning of its trial.
I'm also very tempted to say we don't need to worry about AGI before we can achieve the above for 30% of these people, but I'm less confident in this one.
[1] I suppose, as a practical matter, even if we had such a classifier, this would be untestable without access to everyones' email accounts. So, for the sake of argument, let's say our population under study itself is a simple random sample of 100M accounts from the set of providers who service > 50M accounts each and who are willing to run such a test at the time of the trial.