My take on this is that the system is by and large useless.
It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system).
So what's left when all the criminals this is supposed to catch have figured it out?
False positives. Only false positives.
Is it really worth turning personal devices into snitches that don't even do a good job of protecting children?
Also, numbers about false positives must be taken with a grain of salt because of the non-uniform distribution of perceptual hashes. It might be that your random vacation photos and kitty pics have a 1-in-a-million chance of a fapo, but someone who happens to (say) live in an apartment that has been laid out very similarly to a scene in pictures appearing in the CSAM database may have a massively higher chance of fapos for photos taken in their home.
> It won't catch anything but the dumbest of dumb criminals
Dumb is a pretty accurate description of a large fraction of criminals. For the most part you only get smart criminals when you are talking about crimes where you have to be smart to even plan and carry out the crime.
Can you give source for that number?
Regardless, browser is like any other app. If that amount of people don’t know how to install apps on their computers, then we have a either real dump people (or just lack of motivation) or great UX design failure in general.
It's a dauntins realisation that sinks in once you have to do support for a web site or app catering to the general population instead of a niche.
They don't read, don't know the diff between an app and a web site, don't know right click or drag and drop, think google or chrome is the internet and overall their startegy to solve any problem is as follow:
- look for something obvious that seems like the answer but is not scary
- click
- wait for it
- repeat 3 times until ok or give up and call someone or get angry or both
Working on a streaming video site really opened my eyes on this one. Most tickets we received were insults, some were incomprehensible garbage, a few were actionable request from someone not understanding anything about their computer.
This is nothing like your github ticket. Your parent number are being generous IMO.
Apple seems to have completely botched this PR stunt/feature.
Reading your comment, I realize how these… ‘criminals’ could use phone number networks to share illegal sexual content peer to peer.
In other words, Apple doesn’t need to analyze your images to find these criminals. They only need to analyze the frequency or quantity of flagged images.
In other words not one image correctly/falsely tagged, but individuals and networks of individuals who are *collecting* and *storing* mass quantities of these images. And, they’re using Apple privacy and security to hide from law enforcement.
Yes, but when you admit that the target is just the dumb criminals, then why adopt a scheme that has false positives?
Decompress and downsample. Drop the least significant bit or two, maybe do it in the dct domain instead. SHA256. It'll preserve matching for at least some cases of recompression and downsampling. But finding an unrelated image that matches is as hard as attacking SHA256, the only false positives that could be found would be from erroneous database entries.
> Is it really worth turning personal devices into snitches that don't even do a good job of protecting children?
Yes, because the point is not to protect children. It's to get everyone used to the idea that their content is being monitored. Once that is accomplished, other forms of monitoring can and will be added.
Exactly. It's a Trojan Horse (https://en.wikipedia.org/wiki/Trojan_Horse) to make more pervasive individual control the new normality. The current motivations are just a pretext.
Perceptual hashes are only used to reduce the search space for human review. Apple doesn’t have images in the CSAM database to do a comparison, but if it’s just a picture of a door their going to reject it. Also, because human review is an expense Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions.
I don't really want my family photos reviewed by strangers. "Reducing the search space" of photos on my phone isn't an outcome I want to live with. At the time someone is looking at photos of my, my wife/husband/girlfriend/boyfriend, and my kids, they'd better have a darned good reason (e.g. a search warrant).
I'd also appreciate if Apple let me know if my false positives were reviewed and found to not be CASM.
I saw a story on here yesterday about iphones resetting to default settings after restarting. So people were turning off backups to the cloud, and then finding that their device turned the feature on after sometime.
The system as described only submits its safety vouchers when photos are uploaded to iCloud.
Not saying it will stay that way, but there are three distinct realms of objection to this system, and it's probably useful to separate them:
1. Objections that in the future, something different will happen with the technology, system, or companies; so that even if the system is unobjectionable now, we should object because of what it might be used for in the future; or how it might change.
2. Objections that Apple can't be trusted to do what they say they are doing, so that even if they say they will only refer cases after careful manual review, or that they will submit images for review that were not uploaded to iCloud, we can't believe them, so we should object.
3. Objections that hold for the system as designed and promised; in other words, even if all the actors do what they say they are doing in good faith and this monitoring never expands, it's still bad.
People who have the third kind of objection need to deal with the fact that Apple is basically putting in a system with more careful safeguards than are already in place in many Internet services, even for their "private" media storage or exchange. You likely don't know how the services you use are scanning for CSAM but if the service is at all sizeable (chat, mail, cloud storage) it's likely using PhotoDNA or something similar.
I think there are valid objections on all three bases. But there's a difference in saying "this is bad because of something that might happen" and "this is bad because of what is actually happening".
> Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions.
How can we be sure they won’t cut costs by increasing worker load? I could see them giving each reviewer less time to review individual pictures before passing it on to law enforcement.
This is the threat model I am looking at. It is number one with a bullet. We have already had a court case where an adult actress had to show up in court and prove that she was adult when experts testified that the images were of a non-adult woman.
Baby in the sink? No. But a bunch of the aforementioned? Yeah.
> Perceptual hashes are only used to reduce the search space for human review.
False. The Apple proposed system leaks the cryptographic keys needed to decode the images conditional on the match (threshold of matches) of the faulty neuralhash perceptual hash.
Matching these hashes results in otherwise encrypted highly confidential data being decodable by apple, accessable on their servers to the relevant staff along with anyone who compromises them or coerces them.
Apple can decode the data either way. Their the ones doing the encryption on their servers.
There are two basic reasons for this first it’s a backup service which makes end to end encryption risky, but second they also let users share access to their baked up photos. iCloud > photos > shared album.
“ Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images. Apple then manually reviews each report to confirm there is a match”
The design goal was no human review for individual matches.
I knew a probation officer for sex offenders. They told me that most of them were quite dumb. What the repeat offenders were, though, is dedicated. They had all day to try to avoid getting caught, and the PO had a few minutes per week per offender.
It's true that in any arms race, a given advance gets adapted to. This will surely catch a bunch of people up front and then a pretty small number thereafter as the remainder learn to avoid iPhones. But that's how arms races work. You could say that about almost any advance in fighting CSAM.
Probably some of both. One point of the criminal justice system is to shift incentives such that people with their acts together satisfy their desires without criming. There are plenty of smart, greedy people who just go get an MBA and siphon off value in ways that are technically legal. The risk-adjusted ROI is better.
Yeah, Facebook's blog post makes me wonder what all the stuff they report actually is. When people say CSAM, I think "kids getting raped" but apparently there's stuff that people find humorous or outrageous and spread it like a meme (and not like pornography).
"We found that more than 90% of this content was the same as or visually similar to previously reported content. And copies of just six videos were responsible for more than half of the child exploitative content we reported in that time period."
"we evaluated 150 accounts that we reported to NCMEC for uploading child exploitative content in July and August of 2020 and January 2021, and we estimate that more than 75% of these people did not exhibit malicious intent (i.e. did not intend to harm a child). Instead, they appeared to share for other reasons, such as outrage or in poor humor (i.e. a child’s genitals being bitten by an animal)."
Based on this, I wouldn't conclude that FB is the platform where people pedos go share their stash of child porn.
Their numbers also include Instagram, which I believe is quite popular among teenagers? I wonder how likely it is for teens' own selfies and group pics get flagged and reported to NCMEC.
> It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system).
Given the reported numbers of illegal images detected by similar systems within Facebook and Google, I think it is very clear that this will catch a lot of illegal content.
The false positive rate reported in the blogpost for imagenet was 1 in a trillion, and the author concludes that this algorithm is better than they expected.
"After running the hashes against 100 million non-CSAM images, Apple found three false positives"
So closer to 1/10M. The reporting threshold is made artificially higher by requiring more than one positive.
But anyway, that's beside the point.
A perceptual hash is not uniformly distributed; it's not a random number. Likewise for photos taken in a specific setting; they do not approach the randomness of a set of random images.
So someone snapping a photos in a setting that has features similar to a set of photos in the CSAM database may risk a massively higher false positive rate. It's no longer a million sided dice, it could be a thousand sided dice when your outputs happen to be clustered around similar values due to similar setting.
But I can't say I care about false positives. To me the system is bad either way.
"After running the hashes against 100 million non-CSAM images"
They don't say what kind/distribution of non-CSAM images. Landscapes? Parent pix of kids in the bathtub? Cat memes? Porn of young adults? Photos from real estate listings?
I suspect some pools of image types would have a much higher hit rate.
Edit: And, well "hot dog / not hot dog" is impressive on a set of random landscapes too.
Well the same article also claims zero false positives for "a collection of adult pornography." I don't know if the size of that collection is mentioned anywhere.
Anyway, I suspect that the algo is more likely to pick defining features of the scene and overall composition (furniture, horizon, lighting, position & shape of subject and other objects) more than the subject matter itself.
Sometimes the best way to catch the really smart or sophisticated criminals is to exploit their less smart and less sophisticated accomplices, co-conspirators, peers, acquaintances, or even their victims.
The point of these innovations is never the stated purposes. To catch criminals is an excuse. I would bet a great deal that this system is by and large pressured by state actors for the purpose of creating a new political surveillance tool.
I really doubt this. In the long term, a few people Apple wants to frame will surely slip into the mix. If Apple didn't want Trump to win, a CASM flag a week before the election might do it.
There is an interesting constitutional quirk which arises from the scanning being done client side, specifically for US citizens. If the US Government forced Apple to add other entries to the hash table, this would constitute a warrantless Government search of the private physical property of US citizens. This is a clear-cut, unambiguous breach of the 4th Amendment.
Whereas if the CSAM scanning was performed exclusively in the cloud, protection under the 4th Amendment does not exist as it would likely fall under the third party doctrine.
Now I'm not saying the US Government would let mere unconstitutionality get in the way of any surveillance program. But Apple would. You don't think Apple wouldn't be itching for another opportunity to flex in public? Especially now, with their reputation on the line? Apple would love nothing more than to have more opportunities like they got with the San Bernardino iPhone.
Apple could also encrypt every upload to iCloud, and not have any scanning on the client, and still be able to say to the government "sure, you can have the files; we can't read them and neither can you". Apple wants to reduce your privacy from the government above and beyond what the law requires. The questions is: why?
1. Leaked documents show Pegasus software exploits all iphones using an exploit in iMessage
2. Apple releases security update (doesn’t patch imessage. exploit)
3. Apple announces CSAM client scanning coming soon
4. Apple releases another security update (still leaves iMessage exploit unpatched and used by Pegasus)
….
Perhaps Apple is under pressure to provide a back door prior to patching a tool that may be widely used by governments around the world.
Some, but not all. And if you still have access to a Mac or iOS device which remains associated to this iCloud account, the amount that is lost can be even less.
That's not technically possible, though. Most tech companies don't take on impossible tasks, because it will take an infinite amount of money to realize it, and the shareholders will be bankrupt before the product is delivered.
Every piece of data is CSAM encrypted with a one-time pad. It's just that nobody knows the one-time pad.
I think you're implying that scanning of private personal property by a corporation without a warrant protects users from searches of their content in the cloud that is authorized by a warrant or national security letter. I don't understand the mechanism if there isn't end to end encryption, and I don't understand the mechanism if there is end to end encryption.
Scanning makes phones a greater threat, and also erodes the expectation of privacy that is a legal barrier to surveillance.
I did not intentionally imply anything here, so if anything I wrote appears to include an implied component, that was not intended and may not represent my opinion.
All I'm saying is that the implementation Apple has described would be constitutionally blocked from being co-opted by US law enforcement. Obviously if there's no end-to-end encryption, any cloud operator could still be coerced into searching for material server side, as that falls under the so-called third party doctrine.
What kind of non-CSAM crime could be detected with just a couple of hashes? Wouldn’t Apple need to reduce the similarity score in order to even get something close?
Also Fifth Amendment. People are being compelled to testify against themselves by running this CSAM scanner. Apple's end-to-end encryption was sold to users as exactly that.
To all of a sudden introduce this scanner doesn't negate the expectation of privacy as that is how it was sold and marketed. There is an implied warranty of merchantability of how this service functions.
Are you a lawyer or legal scholar, or just guessing?
The government can't compel warrantless searches of Apple. 3rd party doctrine means Apple can search your iCloud, and can give it away if they choose. Same as how Apple can search your phone if you run their software, and can give away whatever they find if they choose.
> If the US Government forced Apple to add other entries to the hash table, this would constitute a warrantless Government search of the private physical property of US citizens. This is a clear-cut, unambiguous breach of the 4th Amendment.
There's no reason not to assume this isn't already happening, being closed source and proprietary. The question to ask is, what are we going to do about it?
If you take that line of argument, you must also accept that you have no reason not to assume that binary distributions of Android and Windows haven't been doing similar things for the past decade.
I agree with you in that I don't think the problem is the closed-source aspect. Closed source software can still be audited (with difficulty). The problem is that the source material for the hashes can't be audited, even when we know exactly how the system works.
This is already a warrantless search that’s effectively controlled by the government. Obviously there’s enough chaff in the air to prevent that from being legally useful in any way.
So far the courts have determined that since providers invade their customer privacy of their own free will with no incentive or coersion by any government agency, that it is not a search by the goverment.
It's just your friendly trillion dollar tech company putting on a mask and cape and engaging in a bit of vigilante fun. You know? Like batman! ( https://www.youtube.com/watch?v=Kr7AONv3FSg )
Since the govt. supplies the hashes in an unauditable way, it absolutely is controlled by the government. What's to stop them from using hashes of non CSAM material?
The government doesn’t supply the hashes in an unauditable way, that is a totally false statement.
The hashes are supplied by NCMEC, a non-profit which is auditable, not a secret government agency.
In any case, even if a non-CSAM hash were somehow in the database, Apple reviews the images before making reports, and those reports are used in normal criminal prosecutions.
Courts have determined that for this purpose the NCMEC is an agent of the government. NEMEC is 99% funded by the government and its ability to handle child porn is directly deprived from a explicit legislative carveout for them by name. What they do would be a felony for you or I to do. The fact that they are technically non-profit rather than an agency makes them significantly less accountable to the public. We cannot FOIA their communications, their composition isn't subject to public review, we cannot vote them out. And we have no way to tell what their database contains, nor is there any avenue for redress should we somehow learn of an inappropriate listing.
To the extent that you can say that they're not exactly a government agency, they absolutely have been deputized by the government.
> And we have no way to tell what their database contains, nor is there any avenue for redress should we somehow learn of an inappropriate listing.
Yes there is. It’s called legal liability. They are not immune to being held accountable for their actions just like any other non-profit. They may be immune from prosecution for possessing CSAM, but they don’t have any kind of immunity for damages they cause through their own actions.
So you would agree that Apple's use of private set intersection serves the purpose of shielding themselves and their data sources for the purposes of mitigating their legal liability from the harms created by false listings by concealing the content of the databases?
I don't think it's entirely false. NCMEC is not a regular non-profit. They have special clearance to do things that regular citizens and non-profits are not allowed to do.
Presumably, it’s done this way so they can say computers other than your personal device do not scan photos and “look” at decrypted and potentially innocent photos. And technically the original image is never decrypted in iCloud by Apple - if 30 images are flagged they are then able to decrypt the CSAM scan meta data which contains resized thumbnails, for confirmation.
In summary, I’m guessing they tried to invent a way where their server software never has to decrypt and analyze original photos, so they stay encrypted at rest.
Apple frequently decrypts icloud data including photos based on a valid warrant. This new local scanning method does not stop apple from complying and decrypting images like they have for years.
(Note: I have worked with law enforcement in the past specifically on a case involving Apple and two iCloud accounts. You submit a PDF of the valid warrant to Apple. Apple sends two emails one with the iCloud data encrypted. A second email with the decryption key.)
Of course, but it's a kind of last resort thing to support a valid legal process they cannot (and probably don't want to) skirt around. They also publish data on warrant requests.
To me it's pretty clear they are doing the absolute minimum possible to keep congress from regulating them into a corner, where they lose decision making control around their own privacy standards. The system they came up with is their answer for doing it in the most privacy conscious way (e.g. not decrypting user data in icloud) while balancing a lot of other threat model details, like what if CSAM-hash-providing organizations provide img hashes for a burning American flag, and lots of other scenarios outlined in the white paper.
Yes I agree, bit of a stretch. Based on their whitepaper, it's a smaller version of the original image, I guess just large enough to support the human verification step.
But I'm unsure that the thumbnail is included with every CSAM "voucher" -- it's likely only included when you pass the 30 image limit. Need to read that section more clearly.
A thumbnail is included with every safety voucher.
However, it is encrypted with a key that resides on your hardware and is unknown to Apple. So Apple doesn't have enough information to decrypt your thumbnails at will.
A secret sharing scheme is used to drip-feed Apple the key: each time a positive match occurs, Apple learns a bit more about your key. Once the threshold is reached, Apple will have learned enough to recover your encryption key, and will be able to use it to decrypt all your matching thumbnails at once.
Most people feel that things that happen on your device are safer than things in the cloud, you have probably noticed how Apple constantly stress that this or that happens "on device".
And for the suspicious, it's of course much easier to notice if Apple would change their algorithms if they happen on device.
Apple basically never announces things before they are ready to be released, so them not announcing this means very little. They may be working on it, and their usual secrecy is biting them in the ass very hard.
If it’s going to really save a significant amount of data center resources, then it’s also probably going to reduce the battery lifespan of all these devices significantly. That may probably be good for Apple’s bottom line temporarily, but it will hurt in the long run. I’d imagine it’d be a lot easier to optimize the data center compute resources than optimizing the scanning on individual devices and not trashing battery lifespan.
> I’d imagine it’d be a lot easier to optimize the data center compute resources than optimizing the scanning on individual devices and not trashing battery lifespan.
No amount of data center optimization will beat running computations on hundreds of millions of devices other people have to pay for.
If that’s a valid reason to steal electricity and compute resources from your customers, then why not go the whole way and use all the Mac’s as storage and compute for iCloud?
> If this is the case there is zero reason to scan locally and you can just scan the uploaded image once it is on the server.
You’re having a house party. Because of the pandemic, you’d rather people who have COVID not attend. You can’t trust everyone to get vaccinated or get tested beforehand. So, you decided to set up a rapid-test system, just to be sure.
Would you rather test in your kitchen or your driveway?
I don't see how this scanning reduces the likelihood of a government searching their servers. Seems to me like this can only result in more court orders than they had before scanning.
If you’re Apple, and you’re throwing your weight around in the US bread and butter market to convince the FBI to not scan your servers for CSAM, which is more compelling: we check for it when it gets here, or we keep it off our servers using crypto hash magic?
Sigh, for the last time, it doesn't actually matter if the NeuralHash is identical. You need multiple images matching, and then the images are compared by another system on Apple's end, which you don't know anything about.
The system is specifically designed so that colliding images does not pose a threat to the user.
NeuralHash and the CSAM scanning is grotesque, but please, criticize it for what it is, not some bullshit that is easily dismissed as technical ignorance.
Then let's get rid of the NeuralHash entirely, if it doesn't matter, right?
If it's a critical part of the system, then it should be inspected thoroughly. If Apple claims a minuscule chance of a hash collision, and the reality is that collisions are relatively common, that significantly changes the requirements for the backend system, which Apple keeps secret. We have every right to believe, bbased oon ppublic info, that Apple was expecting that NeuralHash would be almost fool-proof, leaving the backend system to be a rubber stamp. This would be tragic.
The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device.
Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the original post here. In fact, I've not seen anybody look at the hash distribution over natural images, which would be an actual argument against the system.
Let's not forget what the alternative is: this is about images that are uploaded on icloud anyway. The alternative is to upload the image in clear (or with ane encryption key that apple controls), and let apple run the CSAM filter on their servers.
Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about)
This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)
They already have the decryption keys for iCloud. Undoubtedly they've already been running a similar CSAM filter server-side for ages. The only thing doing this stuff client side has done is reduce privacy and erode trust.
What should they do if the neuralhash matches CSAM? Should they trust that the nerualhash actually matches real CSAM and you will should be reported to the police? That's clearly wrong, since there are going to be false positives, by design.
The whole point for this is to be a probabilistic filter, so that they need to run the real CSAM scanner on a subset of files.
You can fall into two camps:
a) apple should never ever scan my private images I upload on their cloud.
b) apple can scan the images once they reach their servers.
If you pick (a), then clearly neuralhash shouldn't exist and you can argue against that on the ground that you want utter privacy. But you have to be consistent:every other cloud service that does scan the images server side should receive the same critique.
If you pick (b), then you must recognize that this additional machinery doesn't increase their reach to your private data, but quite the opposite, it allows them to implement e2e encryption for 99.9% of your content. You may argue that it's unnecessary and confusing and spooky and be afraid of the slippery slope precendent for other uses.
But if Apple really cares about children why they did not done this scans in iCloud like all the others? Did not care as much as Google or Facebook? Seems to me like Apple does not care at all and seems more like a dev with big ego wanted to add neural hashes to his CV but if you can explain how Apple cared for children all this years but only now are doing something I really want to see the explanation
Because Apple is the one company that actively tries not to know anything about you.
FB and Google will exhaustively analyse every single facet of your online presence and use your pictures to train their ML models for face detection and object detection.
Apple, on the other hand, even explicitly splits Map directions to segments so that they can't know where you left from and where you are going to.
Apple was sending unencrypted on the network what application you started, if they would "actively tries not to know anything about you" they could have implemented this better.
Anyway how is your assumption make sense , Apple cares about children and about your privacy so scanning your images in iCloud was wrong until 2021 when something changed, what changed? does Apple cares more about children starting from now or they care less about privacy? or are they forced to do it?
Apple's claims are based on the statistical likelihood that there would be 30 collisions with CSAM hashes within one user account.
Just because someone has found an image of a nearly featureless diagonal thing which collides with another image of a nearly featureless diagonal thing, that doesn't disprove Apple's claims.
I find this an unconvincing argument as well, you're saying that because Apple made a false claim, any claim may be valid. This is obviously not the case, what they did was to, albeit likely knowingly, calculate the hash collision probability /if each bit is a coin flip/, which comes out to pow(2, -k) for k bits. It's tiny. Of course, each bit is /not/ an independent coin flip under the NeuralHash function.
So again the actual argument becomes: what is that distribution like?
> what they did was to, albeit likely knowingly, calculate the hash collision probability /if each bit is a coin flip/, which comes out to pow(2, -k) for k bits. It's tiny.
I doubt that's what they did. I think they ran tests on huge numbers of pictures, got an estimate, put in a safety factor, and determined the threshold to hit their target (and put in another safety buffer then).
Naturally occurring collisions are not going to be an issue, and adversarial ones neither, I predict. Just as with current cloud providers.
Apple never made a false claim. They have never anywhere stated that neuralhash makes false positives at 1 in a trillion. Only that that is the rate for the system as a whole to flag accounts for review. The explicitly mention that they will vary the number of matches needed to maintain this if it turns out to be higher or lower based on images in the wild.
There are good arguments against this system but most of the technical debate seems to have devolved into amplifying lies now.
Fine, let’s say they didn’t lie, they made misleading claims.
Still. Why trust them after that?
If a company can make my own smartphone report me to the police, and they want my business, they better prove I can trust them. Apple has plainly done the opposite.
Sure, but a NeuralHash system with a collision chance of 100% would obviously not be abused to say NeuralHash collision implies CSAM - the secondary validation system will definitely be the last bastion.
In contrast, a NH system believed to have a collision chance of 1 in a trillion trillion may well be considered infallible, and any detection be directly reported as CSAM, with the 'backend verification' amounting to nothing more than a rubber stamp.
Of course, if you implicitly trust Apple not to do the second, than you're right, the NH collision rate doesn't matter too much.
The conclusion section of the article associated with the GitHub repo linked here is that collisions are not common and Apple’s published collision probability matches their findings. Furthermore the thresholding scheme requires 30+ independent collisions which is astronomically improbable.
You (or at least Apple's customers) trust in and rely on Apple's proprietary software to do its job all the time. How is this different? I find this argument very weak.
There is a big difference, if say Apple tracking of what you run is problematic you get a bad user experience like apps starting after 1 minute of waiting, or if Apple App Store contains malware you will probably get some annoying issue while Apple will try to silently cleanup their mess BUT with this system there is a difference, this is designed not to serve you or protect you, it is designed to do some checks then if some specific rules match send some guys on you to destroy your life, today is FBI tomorrow other authoritarians.
Issues in any other Apple software will not send the police on you. Why would you install a software on your desktop/laptop that is designed to snitch on you, you would need to get some advantage or be forced by some law.
For now I see only disadvantages but please let me know of any real advantage and not speculation
Disadvantages:
- closed software with hidden db can't be trusted, so as a user you will always have a doubt that some non CP images are in the db(Apple always collaborates with governments)
- bugs in this stuff will cause you big problems(we seen in the past how false accusation destroyed peoples life) and we also seen bad actors abusing this kind of stuff.
- this is also clearly a beginning, now that Apple has the capability then even if they were saints a judge could force them to add new hashes, change the configs etc.
1. You can at least somewhat audit the software running on an iPhone, for example by means of reverse engineering. You can’t audit the server side.
2. It’s one thing to rely on proprietary services like Find My or Siri.
It’s another thing to rely on a secret server-side app that has the power to destroy your life.
What I somehow fail to grasp in the first argument is that this whole system is designed specifically so that it runs client-side. AFAIK all the alternatives (as in « cloud photo services ») have been doing the exact same thing on the server side for decades. If you upload your photos to the cloud, a lot of service actually already have the power to destroy your life.
Tell this to the victims of Pegasus. If anyone were able to get their hands on the "secret backend system" we wouldn't be talking about spy games, we'd be talking about people's lives being ruined
Apple still has not patched the security exploit in iMessage used by Pegasus. Apple has released two ios security updates since the Pegasus revelations but still has not patched it most widely used exploit… hmmmm.
Now apple is getting a local client side scanning tool ready. Interesting timing.
afaik none of Apples other "proprietary software" is designed to pass my personal images to a human for visual inspection if it mistakenly outputs 2 high-enough numbers after a handful of convolution operations and matrix multiplications.
They pass a "visual derivative" to "a human", but only after some matrix multiplications etc. that result in extremely low probability false positives.
It could also happen that you lose your phone and "a human" finds it and randomly puts in the correct passcode on the first try and visually inspects your personal images. In fact, that seems vastly more likely [1].
[1] About 4% of smartphones are lost or stolen every year [https://www.mcafee.com/blogs/consumer/family-safety/almost-5... ], but make it just 1/1000, so 1e-3. Then a 6 digit passcode, 1e-6, so we're at 1e-9 per year, or 1000x as likely as being falsely flagged, assuming Apple's numbers (which can easily be achieved by calibrating the threshold).
Discussing the preimage attack on NeuralHash is not technical ignorance. Dismissing the preimage attack as irrelevant is.
0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever.
1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image.
2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No.
First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens.
Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs).
The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security.
Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly".
3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not.
That's a lot of words to say that you could sufficiently mangle an image that it could pass through all of Apple's algorithmic hurdles while not actually being CSAM. Of that I have no doubt. You could definitely generate a mangled image that fools multiple perceptual hash algorithms.
Let's set aside the questions of where you got all these hashes to generate collisions with, how you got 30 of these mangled images into your victim's camera roll without them noticing. And let's also set aside whether your victim's device is an iPhone with iCloud Photo Library enabled (and has sufficient storage). I still don't get what these mangled images have achieved, other than giving the manual review team something other than child porn to look at.
Seems to me like it'd be easier to just find actual child porn, print it out, place it somewhere in the victim's house and then report it to the police.
I think you missed the point of the first paragraph. The point is that you can now hide child porn by making its hash collide with innocent images. They won't ever make it to manual review. Ergo, NeuralHash is now useless.
Why would anyone go to the effort of technical concealment[1] of CSAM when they could just resist the urge to import child porn in their phone's photo library in the first place? I've managed to resist the urge to import regular porn into my photo library, and being caught with regular porn is (at most) embarrassing. It's not potentially life-destroying.
It's inconceivable that anyone could desire possession of NCMEC-catalogued CSAM images without being aware that they're risking serious consequences if they're caught. Who wants their deepest, darkest, potentially life-ruining secrets just milling about with photos of the dog and last night's dinner?
[1] ...which is all but impossible for an average user to prove was effective; it's not like the Photos app has a "Not Child Porn!" checkmark.
I agree with you in that I do not understand why anybody doing something illegal would upload related data to a cloud storage.
But if nobody would import CSAM into their icloud library why do all the pictures need to be scanned in the first place? I would imagine anybody doing major illegal stuff being informed about important measures in order to not be caught.
I agree, why Apple is doing this is an interesting and pertinent question. I don't think it's actually because they think this will put a big dent into CSAM. So the question is: what is motivating Apple?
Perhaps it's a prerequisite for deploying end-to-end encryption of iCloud Photo Library and/or iCloud Backups. The latter in particular has remained decryptable by Apple supposedly due to pressure from the FBI. Perhaps CSAM is what the FBI are using to justify their pressure.
Perhaps it's because Apple's team of lobbyists are seeing ahead to future anti-privacy, anti-encryption legislation being justified under the guise of CSAM. If Apple can show that the CSAM problem is already "solved" then such justifications disintegrate.
So, the presumed attack (not against individuals, but to defeat the system) is
1. Identify some innocuous pictures that many many people have (memes, Beyoncé, whatever).
2. Produce CSAM.
3. Mangle it such that it is still CSAM visually, but NeuralHash-collides with the innocuous pictures from step 1.
4. Distribute.
5. Wait until they are (via some other mechanism) a) identified as CSAM, b) added to the NCMEC database, c) added to the Apple on-device database of blinded hashes in some iOS update.
6. Millions of people are suddenly incorrectly flagged for exceeding the threshold by NeuralHash (since they have the innocuous pictures in their library), and the review teams are flooded and can't pick out the small number of actual CSAM holders.
That is not without a certain elegance. However, it seems to me that
A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.
B) it would be quickly defeated by amending the 2nd tier algorithm (between NeuralHash and human review), though, as you highlight, that might be tricky given that the team working on this presumably only has access to the innocuous false positive collision image, not the (purposefully mangled) CSAM.
> A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.
Note that this requires no single "desired" target picture. There are millions of popular, innocuous pictures. As long as you can make your CSAM match any one of them without significant mangling, you're good to go. Not having to choose one specific target makes this much easier to accomplish.
> it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.
I'm so tired of people suggesting that you can't. Please explain to me why you posted suggesting otherwise.
I've contemplated making some that are also photodna matches, I expect that it's possible. But access to photodna is only through some awful windows tools, and AFAICT people would just keep posting denials even after an example was posted-- so it's not worth the effort at least not worth it just to further the public discussion.
> Please explain to me why you posted suggesting otherwise.
a) I didn't suggest otherwise, I said that it is predicated on that assumption, about which I was undecided, largely because b) I didn't know better.
I read that thread 7 days ago, when the collisions were a gray blob or a clearly modified dog (to Lena) or clearly modified Lena (to dog). I hadn't re-read the thread in the last 4 to 5 days, when you demonstrated the natural looking collisions (second-preimage images).
> allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images
That’s a really interesting attack vector I hadn’t seen mentioned previously.
Most people are talking about the potential for adversarial images to be sent to users. If they were instead injected into the database itself (either by poisoning real CSAM or social engineering) that would have far wider ramifications.
I wonder what the most widely-saved pornographic images are across iCloud users.
If actual CSAM were perturbed to match the hash of, say, images from the celebrity nude leak a few years back and added to the database then thousands of users could be sent to “human review”. Since the images are actually explicit how would the human reviewers know not to flag them to authorities?
People don't seem to grasp what kind of images end up in the CSAM databases. They are most definitely not "leaked celebrity nude selfie" level stuff.
Think of the most vile sexual thing you could do to a child and then times that by two and halve the child's age in your mind. That's the shit that gets in there.
It's not something even 4chan weebaboos share. It's stuff that makes Liveleak regulars go "ewwwww, gross".
The issue is not that a celebrity photo might get into a CSAM database. It's that a true CSAM photo, which has been modified to have the same hash as a "leaked celebrity nude selfie", will probably get into a CSAM database.
(A threshold of) matchings result in the private keys for the images being leaked to Apple, where they're vulnerable to:
(1) Review by apple staff
(2) Access and leaking by other apple staff
(3) Access by hackers who have compromised their system
(4) Access by parties coercing apple/staff, including via national security letters.
All of which compromise the privacy of the user. This matters or the neuralhash comparison wouldn't exist in the first place.
Totally agree that the whole system is grotesque-- but that doesn't stop it also being grotesque in every detail as well. The fact that there are false positives when they easily could have designed a system that had none (at the expense of increased false negatives) shows that Apple doesn't especially value customer privacy even if you accept their vigilante privacy invasion. The fact that it's possible to construct adversarial false positives and that their reports didn't disclose this fact shows they either don't know what they're doing or they're not being honest about the risks (or both).
Note: Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors.
The hash is 96 bits long. When hashing 1 billion pictures, that gives a collision probability of 6e-12. If it were uniformly distributed. There's no way people have hashed billions of images already. It just shows that it's pretty probably there will be collisions, and on visual inspection, it looks as if the collisions will happen on visually similar images. So if there's a naked baby pic in the CSAM database, quite a few of you 100s of child pictures can be flagged.
Clearly this is not a cryptographic hash, and hence it's known hashes are not uniformly distributed.
Apple explained in their technical summary [0] that they'll only consider this an offence if a certain number of hashes match. They estimated the likelihood of false positives there (they don't explain which dataset was used, but it was non-CSAM naturally) is 1 out of a trillion [1]
In the very unlikely event where that 1 in a trillion occurrence happens, they have manual operators to check each of these photos. They also have a private model (unavailable to the public) to double-check these perceptual hashes which also used before alerting authorities.
> they have manual operators to check each of these photos
For now. But what will happen when there are thousands of false positives per day? Will they increase the staff? Or will they add another algorithmic layer? Or just up the threshold a bit? There's no guarantee. The only thing that's certain is that the NeuralHash doesn't inspire confidence.
Regardless of what Apple does, law enforcement must always manually review suspected CSAM before requesting a warrant based on it. So the idea that you could SWAT someone with some hash collisions on innocent images is just not possible.
At most you could maybe temporarily lock someone’s iCloud account. But again, the collisions would need to be multiple and all look like CSAM at reduced resolution.
In general, it seems not correct to think about NeuralHash like SHA or RSA. It’s not a cryptographic system and collisions are not a one-step endgame.
> Regardless of what Apple does, law enforcement must always manually review suspected CSAM before requesting a warrant based on it. So the idea that you could SWAT someone with some hash collisions on innocent images is just not possible.
A quick search found four clear cases where law enforcement has favored technological false positives over evidence:
The one on Ousmane Bah really frustrates me- not only was he on a date at prom during the theft, he was in another state! "Nothing to hide" does not mean "nothing to fear" and allegations (even false ones) of possessing CSAM will ruin lives.
At the end of the day, what it really comes down to is trust; personally, I do not have enough faith in due process to not ruin innocent people. But I'm just some guy online, I will readily admit I don't know anything about anything.
Afiu, Apple's NeuralHash uses exact collisions when they do their Private Set Intersection.
The main advantage of using exact collision is that you can then blind the perceptual hash with a cryptographic hash and avoid any leak of information. (Taking for example sha256 of this perceptual hash won't allow any attacker to get any information on the features from the hash, but if the perceptual hash are the same then the input of the sha256 is the same and therefore the output of the sha256 is the same).
This is important because it alleviates the risks of an eventual leaking the database as Apple never touched and compared sensitive content but only cryptographic hashes of the perceptual hashes.
Some other system like PhotoDNA, rely on a euclidean distance between features being less than a threshold to register a match, which allows to quantify how far the image is from CSAM, but mean that the hash leak some information about the original content.
But they can't use exact matches because their algorithm doesn't create same hash for same image on their various platforms due to hardware differences in floating point arithmetic on their different devices. Unless they have on their servers floating point arithmetic emulators that can calculate exact hash for each of their different devices for each offending image then they can't only match exactly.
Per ATP: Apple will compare hashes of local photos with a national registry of child pornography photos. Once a certain (unknown) threshold is reached, let's say 20 hits, some kind of escalation occurs, with some kind of manual (human) review steps.
I still have zero opinion on this photo scanning kerfuffle. I just don't know enough. Of all the "hot takes" on this issue, ATP's has been the most comprehensive. So appreciated.
I'm interested in the details of the manual review. Does Apple have access to the database of original images and will they use it compare? If not, I can imagine a scenario where a photo of a naked child is flagged as matching the database, the human reviewer sees that it is in fact a naked child and assumes this must mean the image has been legally determined to be child pornography. The case gets referred to authorities and the innocent victim's life is upended for potentially years until the case plays out.
Law enforcement must manually review all suspected CSAM before seeking a warrant based on it. There is a whole process there, beyond whatever Apple has implemented, before a prosecution begins.
Understand that matching a file in the NCMEC database is not itself a crime. The whole CSAM-detecting ecosystem is just a tool for surfacing potential crimes. Having a few pics of your own child naked is not illegal and it’s pretty easy for law enforcement to figure out if that’s the case.
>Law enforcement must manually review all suspected CSAM before seeking a warrant based on it.
Is this a legal statute or simply convention due to the ways things have historically worked (i.e. pre-hash matching at scale)? If warrants are granted based on probable cause, it seems easy to convince a judge that a hash match is sufficiently unlikely that it would exceed the threshold for probable cause. In the context of cryptographic hashes, this is accurate. But if law enforcement doesn't distinguish between cryptographic and perceptual hashes, then there is the real possibility for cases opened and warrants issued unjustifiably.
Sure, matching a hash isn't a crime and you will eventually be exonerated. But as they say, you can beat the rap but you can't beat the ride.
Ad.2 If hashes are to be matched approximately, not exactly, for example will be considered a match if they differ in less than 3 bits out of 96 then the most interesting thing should be how many collisions you can find if you compare them like that.
Apple's private set intersection which leaks the keys to decrypt the images coniditional a neuralhash match requires an exact match.
They probably didn't realize they got different results on different toolchains/devices, since they target a mono-culture and the whole subsystem shows fairly little careful thought went into it. They could easily make an exact integerized version which would be consistent.
If you can get exact collisions, this can be gamed. For example, suppose there are two rival gangsters. One wants to set the police on his rival. He knows that a certain (innocuous) image is on his rival's phone. So he pays someone to generate a fake child-porn image with the same neuralhash, and ensure that it gets into the child porn DB. Then, apple reports the rival to the police, and they come and investigate him. OF course, they may notice that the image isn't the right one, but by that time they may have found other incriminating evidence.
Not sympathetic to a rival gangster? Ok lets find an innocent victim: not a rival criminal, but an innocent witness who our protag wants to intimidate. Gangster wants to intimidate the witness, but can't get at them, so cooks up a scheme to convince the witness that the police are in his pocket. Exactly as above, causing the police to investigate the witnesses phone.
Another one might be, a certain government wants to identify opposition groups using images associated with them . Apple is not keen to be associated with that, but the government can simply generate fake child-porn (remember, programmatically generated CP is just as illegal) for each image of interest.
> a human verifying the images would dismiss the false positive
How is a human supposed to distinguish that a visual derivative (a low res sobel filtered image, presumably) of ordinary, lawful, adult pornography isn't child porn when the system has already identified it as such?
I agree that using real child porn is an attack too, but at least in that case you could say the system was doing as designed (even though what its doing shouldn't be something that we want) ... but it's not even guaranteed to do as designed.
Are you saying humans cannot visually inspect the actual image that the flag is set against? If not, then how on earth do you prosecute someone if you can't demonstrate they had the actual image, and not just the derivation used to flag it?
The way I see it working is Apple scans a ton of shit, some of it shows up as possible child porn, human intervenes and looks at the source images, if there is indeed child porn they report to the police.
Apple's review has no access to the matched images in the database, just a 'derivative' of your image. Hopefully the prosecutors will check against the real images. But by that point your privacy has been damaged and your reputation might be trashed.
So you think an underpaid worker in India is interested in the upkeeping of innocence of a person they don't know in the US? They'll just flag the images as fast as possible, they are probably paid by pictures-reviewed.
"Amazon has people transcribing audio in Costa Rica, India and Romania." according to Engadget. So it would be safe to assume Apple does something along those lines, I can't remember where Siri transcripts have been sent when there was the Alexa controversy.
From there it's an entry with law enforcement and you need to find a way to convince them that the images are your children.
And if data isn't cleared correctly and you have another run in with law enforcement, there will always be this picture-stuff.
you think the US government would respect the law before initiation of actions to take down a criminal?
they rarely do that when dealing with people that dont have criminal backgrounds
If your enemy can get an image on your phone and in the “child porn DB”, I think they can easily get you in trouble without Apple’s help.
they can either just send the police an anonymous message or set up a child porn web site and have it ‘accidentally’ leak its password database, and make sure your email address is in it.
A very relevant point on this entire discourse about Apple’s on-device CSAM scanning:
According to the U.S. law, key snippets of which are quoted on the Stratechery blog (by Ben Thompson), Apple isn’t obligated to scan for CSAM. It’s only obligated to act on CSAM if it finds them.
While it’s good for Apple to scan on its systems (iCloud) like Facebook, Google and other companies do on their servers, it’s inappropriate to do it on individual devices, which starts with the assumption that anyone who has iCloud photos enabled is a potential CSAM hoarder and needs to pay with their device’s battery life and time for the scanning to happen and report back. It’s a sort of micro-robbery that Apple is doing on the devices when there is no legal compulsion to do so.
Everything else on trusting Apple’s NeuralHash or the sanctity of the NCMEC hashes come later, IMO.
I sincerely hope Apple realizes that it’s got a dud solution on hand, eats humble pie (which it’s usually not capable of) and ditches this whole thing. I know a lot of egos at Apple are at stake here. But doing the right thing matters for a company that claims that “privacy is a fundamental human right” and has a CEO who’s a member of a marginalized/discriminated community and understands the risks of these efforts.
"While it’s good for Apple to scan on its systems (iCloud) like Facebook, Google and other companies do on their servers, it’s inappropriate to do it on individual devices"
I would even challenge the justification to do this on servers, unless the data is public. If it's behind a personal login, you might as well consider it personal property/data. I find the distinction of where data is stored not very meaningful.
Allowing things to be searched for criminal content just because it's not in your immediate physical sphere makes no sense. It doesn't work like that in the physical world either. When I send a letter, and it leaves my house, no authority has the right to check its contents without a legitimate reason. Likewise, if I put stuff in a storage box in some warehouse, no authority can search it without a warrant.
Note that I'm talking about personal storage (iCloud, Gmail), not public social networks like Facebook.
Absolutely. If I own my data, someone processing this data on my behalf has no right or obligation to scan it for illegal content. The fact that this data sometimes sits on hard drives owned by another party just isn't a relevant factor. Presumably I still own my car when it sits in the garage at the shop. They have no right or obligation to rummage around looking for evidence of a crime. I don't see abstract data as any different.
Our major privacy blunder was accepting scanning of private data in any context. The fight should be for the absolute privacy of personal data. Where the scanning happens is mostly irrelevant.
Fully agree. Only this week did I learn that companies were already doing this on servers since at least 2018-2019. Can't say I ever read anything about it before.
I find it quite shocking that a foundational element of criminal justice, innocent until proven guilty and needing a reason to search individual property, is tossed aside like it's nothing.
They say only if they find 30 matching images, they'd act. So if they find 20 or 29 and don't report them, they are actually breaking the law!! I am wondering why they chose that magical number!
Just getting a match on the neural hash is not sufficient to declare an image CSAM. It just flags it as a candidate for review. Apple is not required to report it unless it can be verified to be CSAM. there is a multi-step process involving multiple processes and organizations to do so.
They are opening the door for a lot of questions like this. They didn't say (as far as I know) if this 30-number cover multiply upload sessions or it gets reset with every upload. I feel like this number gets reset to 0 each for each upload session. Somebody can deliberately safely upload 29 actual CSAM photos each time. I know Apple is not stupid, but you know the government and lawmakers will ask it for a smaller magical number...
They think they've outsmarted the law on that one. The system is setup to generate a bunch of false positives on its own, so a "match" which is below threshold may not actually be a match-- apple can plausibly deny it.
It's unclear if their claimed threshold of 30 is before or after the false positives they intentionally introduce. I'm going to guess it's before.
Keep in mind that Apple's claimed false positive rate (one in a trillion chance of an account being flagged innocently), and the collision rate determined by Dwyer in the blog post linked from the repo [2], are both derived without making any adversarial assumptions. Given that NeuralHash collider and similar tools already exist, the practical false positive rate is now expected to be much much higher.
Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar).
A casino that makes no adversarial assumptions about the clientele could argue as follows: the probability that you accidentally win the game is much less than one in one trillion, so this game is very safe, and the House Edge is excellent [3]. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. Some of the clientele will cheat. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play [4], and the relevant probability is no longer less than 1/1000000000000. In fact, the probability that the client will win is closer to 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino.
(reposted based on my comment on last week's thread [1])
Themselves? No. Other people and communities that they dislike? You bet.
Just look at political memes. What is the chance that both sides will have a few people try to create memes for their opponents which are adversarial. They'll take care to make sure those images aren't on their own apple devices before spreading it to areas where the other side likes to share memes.
Another example are the very people trying to be caught. Someone who is against current laws about the subject might seek to create as many false positives as possible to overwhelm the system. They might even specifically target otherwise legal baby photos to manipulate in this way as it would make it more likely they get past any sort of second tier manual review and result in law enforcement wasting resources.
Lastly, this is nothing new. Planting such material and flooding websites with it has long been a tactic used by some. Up until now it has been limited because doing so requires violating the law yourself and few people hate others enough to go through that level risk to self. But this is mostly risk free because creating adversarial images like this isn't outright illegal and even in cases where there are laws against it violating those laws is extremely different than violating actual laws against CSAM in every way that factors into a person's willingness to break laws.
The existence of a preimage attack makes Apple's system completely useless for its nominal purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images.
If these new images never make it to the NCMEC database, then new CSAM content will be completely NeuralHash-proof. However, if these images eventually make their way into the NCMEC database, then everybody who has the perfectly innocent originals will be dealing with an adversarial environment.
> If these new images never make it to the NCMEC database
The remark that an image might not be in the database applies equally to all CSAM pictures, and so is not actually dependent on any technical aspect of the hash at all.
The fact that you can modify an image to the point that it gets a different hash isn't important compared to the (separate) issue of how you keep your database up to date. And detecting old CSAM is in any case not as important as tracking and interdicting production of new images.
The second remark is good, but mainly because it reminds us to demand Apple ensures that additions are scanned for malicious images before being added to the central database. If the operator determines that a certain images collides with some known public image they can notify Apple. Apple in turn modifies the hasher to dissolve the collision, roll out a patch, and can then update the database.
> ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images.
Even then, for that to affect _John Doe_, they would have to make 30+ images whose hash matches that of images in _John Doe’s_ iCloud account.
I think that means they could target individuals, but only if they knew or could guess what photos they have in their account.
They also might be able to target groups of individuals, say people who went on holiday to Paris. It would be interesting to see whether such people have enough overlap in the sets of Neuralhashes of photos they took there.
This argument is utterly incoherent. Of course they don't include into the false positive rate images that are intentionally designed to generate positive hits, what would be the point of that? The only interesting metric is the false positive rate for normal images.
It's not an argument. It's an explanation of what the number means.
> The only interesting metric is the false positive rate for normal images.
Wrong. The only interesting metric is the false positive rate _in practice_: i.e. how likely are false positives to affect innocents. Indeed, Apple is presenting their "one in a trillion" number as reassurance, as if it was the probability that an account that doesn't distribute CSAM gets flagged by their system. But that probability depends strongly on adversarial questions, and cannot be calculated using optimistic assumptions about all images being "normal".
2 collisions out of a million images. I'm not sure how big the CSAM database is but if it's a tens of thousands and there are millions of photos uploaded a day then Apple could have a problem on their hands. This is all extrapolating from a study that doesn't use photos representative of what people actually upload. I would suspect when most photos being uploaded are of humans the actual collision rate will be much higher.
It could happen on purpose if I intentionally send you 30 colliding images. I don’t know how iMessage handles images, but WhatsApp for example will put them directly into your photo library (and from there directly into iCloud if you’ve got syncing enabled).
Perhaps I could even do that without revealing my motives to you.
I mean, you could send them actual illegal content too.
And human reviewers are in the process. If you've got 30 matches and they are all pictures of bridges or whatever do you think the FBI is going to show up at your house?
The technology is not why the Apple system is unwanted. It's just extra fuel for the fire.
This system is unwanted because it puts a spy literally in your house and in your hands. It's bad enough that cloud everything blurs the line between what's yours and what's mine. Placing any law enforcement tech on a user's own device takes that line between "public" and "private" and completely erases it.
Absolutely. The problem is Apple introducing a spy into your home.
This alone should be bad enough, but some people are rather trusting. Showing that the spy is also tripping balls both exposes additional risks and emphasizes that Apple neither has their best interest at heart nor is putting adequate care into their actions. The latter gives people reason to question apple's claims of additional protection mechanisms that are non-falsifiable.
Please help me understand. Isn't this the reason why the process involved a final manual review? If so, isn't the point of having identical hashes moot? Or is the point that having more identical hashes means reviewing more personal pictures manually, leading to a privacy issue?
I don't think I would trust a huge corporation with this. Plus, leaving the review to some internal classified process where some poor faceless guy needs to reach an unrealistically high quota of reviewed images per day to get his bonus, might be a bit of a risk.
it's not just internal policy - the safety vouchers will not decrypt (technically impossible) unless there are ~30 matches. It is a policy encoded in cryptography.
I don't really get what this repository is trying to achieve and what's the point of collecting collisions. Collisions will happen, that's just how it is with hashes.
It's already a public knowledge that Apple has 2 more systems (some server-side verification and a manual check later) to prevent false-positives. So what's the point of researching collisions in NeuralHash?
Are you fine with an apple employee looking at all your private pictures just because some hashing algorithm decided you're a pedophile? Personally, I'm not.
First, this doesn't change my opinion on CSAM, I still consider the thing way too intrusive until Apple announces E2E for iCloud.
Now, I can't really call something I voluntarily uploaded to Apple's servers a "private picture". But that's just a matter of perspective, and I understand that many people would disagree with me on this.
I'd argue that the hash collisions (both natural and synthetic) that I've seen give me more confidence in the system, not less.
On the natural hash collisions (of which there are two), we have objects of similar shape against a solid background. It seems that a natural hash collision of a CSAM image would be unlikely (or if it does occur, it would be something that perhaps is also an infringing image).
As for the synthetic hash collisions, there are visible artifacts in the picture that, if you compare with the original picture, make the overlay of the original picture on the synthetically generated hash collision obvious. Could people get tricked into downloading memes¹ with synthetically generated hash collisions? Sure, people are idiots. But I'm guessing the majority of folks will look at the picture and say, this is a sh*t picture in this meme and download something else.
1. And that, of course, assumes that meme hosters don't apply similar scanning techniques to what they serve up.
I saw two examples. The dog/girl one has obvious artifacts in the picture of the girl. The directly linked image doesn't have artifacts, but putting them side by side, I can see what got matched up and they're still very approximately the same picture in that they're both pictures of women and the eyes are in the same part of the picture which gives support to your perspective. I do still wonder whether it would be possible to take, say, a (legal) nude and turn it into an innocent-looking image that still matches the hashes or not. I'm more inclined to believe it now than before, but theoretical possibilities don't usually map to realistic concerns.¹
1. One example of this would be that theoretically, LaTeX's cross-reference mechanism can get caught in a cyclic state. This can only happen with page references and the most likely scenario is a reference to a roman-numeraled page number where if the page reference is output as ix the referenced location moves to page x and when the page reference is updated to x the referenced location moves to page ix (in practice, functioning examples required a shift between xcix and c, but either way, the probability of this happening in a real document is vanishingly small).
No. Most proper cryptographic hash systems (e.g. used for verifying files, rather than data structures) never have collisions.
Try to find a SHA256 collision.
Anywhere, ever, in the history of mankind.
This isn't for lack of looking. A lot of very smart people have looked for them. If you find one, I bet you'll be eligible for a tenured faculty slot at a good university, if not more. A whole world of secure systems would need to be re-engineered.
Hypothetical collisions of course exist, by the pigeonhole principle, just not in the real world.
Apple is claiming to have a visual equivalent to a cryptographic hash -- one which won't change with a single byte, but only if the image is substantially different.
At least their security analysis relies on that.
From their whitepaper: "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account"
If your claim is that their hash algorithm isn't cryptographic, their security analysis is incorrect.
It's trivial to cause a neural hash non-match-- either imperceptibly with a little noise, or by adding an overlay on the image.
If you downsample and quantize an image before sha256ing it you get a bit of robustness to accidental false-negatives. While both schemes are trivially bypassable.
Yes, absolutely. It's technically possible to find SHA256 collisions accidentally, but it's so unlikely, that if you found one, it would merit serious investigation. People would not believe your statement that you found them accidentally, and "oh, I guess mlajtos really just found the colliding pair by chance" wouldn't be declared until after a very thorough investigation. In the meantime, major stakeholders (e.g. Bitcoin) would probably move away to another hash function, just in case.
Dwyer calculated 1431168 NeuralHashes and found two collisions. Humanity collectively calculates over 120000000000000000000 SHA-256 hashes every second. Still, we're reasonably sure that this immense brute-force search will not lead to any collisions in any reasonable amount of time.
And a story may start with the first word, but if I present the word "Octopus" and say check out my story, you're going to be well within bounds to question me on it.
Well to be fair there were two collisions, so your story should be "Octopus imploded" and now everyone is captivated by the plight of this unlucky cephalopod.
I’m glad that people are trying to figure out any technical flaws in the system as best they can, but if I’m being honest I do trust Apple’s engineers to have built something that is solid from a technical stand point.
Am I correct in that the primary reason folks are so upset is that the system could (probably) be easily modified such that -any- content could invoke legal action? That the main problem is really the scanning at all, and not the chances that it could be attacked by an individual actor but instead by a government?
I can’t speak for everyone, but that’s certainly a technical part of it. Another big part of the problem is that it’s insulting to presume everyone guilty, and make them to use their own resources (own phone, own battery cycles) to investigate them as if they were suspects. But that’s been discussed plenty on other threads here at HN.
It might be solid from a technical standpoint.
Once you built it, governments will be coming and asking for more.
Are you aware that the Chinese government already has been granted access to the infrastructure holding the keys to iCloud in China?
Exactly that. The tech seems fine, but I live in a country with a government that has strong censorship laws, and I do not trust Apple to not bend to countries like China in extending this to political content.
I still think the biggest problem is that at some point a human is going to look at a false positive, this may be picture of my naked children and this human may not have the best intentions with my picture.
That said, Nextcloud is my backend and I do not upload anything to iCloud (except for MS authenticator 2fa backups), so I'm safe right?
So your threat model here is that the person at Apple tasked to check for Child Porn is an actual Paedophile and might accidentally see a false positive of your child's naked photo?
You do know that they don't see the whole photo at full megapixel resolution? They're just given "a visual derivative" of the photo for checking.
Also, you really think that the persons tasked with this process are just randos off the street and not vetted specifically?
And where do you get the "visual derivative" information? Apple sure didn't communicate that to me. All I know is some person may look at my pics at some point.
We are not speaking about a situation where not a "arbitrary" picture is miss-classifieds.
We are speaking about a situation where a innocent picture involving a naked or not fully clothed child is deemed similar to a non innocent picture of a naked or not fully clothed child.
Now you might argue that there should not be a picture a a naked or not fully clothed child of any form ever on any phone, but IMHO that is short sighed, discriminating and at best shows you don't know to much about the world and other cultures.
Let's list some simple reasons such a think could happen first:
- Photos meant for a doctor, or living partner to ask if something is normal or a problem. In many different ways.
- Photos of little children bathing or similar a e.g. dad sends to their mom who is currently on a business trip.
- etc.
Reasons people are less aware of is that not all countries are as stuck up about nakedness especially in the family. So it's totally normal for families that e.g. before or after taking a shower family member independent of age and gender walk through the apartment naked. Similar if you didn't got any shame about the naked body indoctrinated you might totally do thinks like visiting a "naked-beach" with your family (meeting other families and taking advantage of it often being less crowded) and in turn normal innocent beach family pictures contain naked children. And on itself that's not a problem. But with Apples approach stuff like this is like to trigger both systems Apple announced and wrongly label your while family as pedophiles...
Where I live some kids even play naked on normal beaches or near lakes or in their backyard, whatever. Up to 2 years old it's very normal. Within families kids and parents see each other naked up until puberty of the kids and the kids don't feel comfortable anymore. If you go to a sauna, everybody is naked.
Or what if a kid gets her/his hands on a phone and takes some pics (doesn't even require unlocking the phone) by accident?
The US may be one of the most up-tight countries about nakedness in general if you ask me. At the same time it's hyper sexualized and produces people like Nicky Minaj, but I guess there is still some fabric over their most "special" parts so it's ok. But oh god, what if Justin Timberlake rips it off... Pandemonium.
Up to 2 years? Hold my beer, in Scandinavia 5 year olds routinely run naked on public beaches. You're not even very shocked if you see nude grownups bathing.
> Now you might argue that there should not be a picture a a naked or not fully clothed child of any form ever on any phone
No I'm definitely not arguing that. I'm not American, where I live you'll sometimes see nude bathers in the city centre, and most definitely nude children on the beach.
1 in a trillion is derived from a dataset of 100 million photos, presumably a representable proportion of these were "similar images" like bathing kids.
Given the problematic aspects of a "globally" available datasets and given that in many countries people are more stuck up about such thinks I highly doubt that this dataset contained a "representative proportion" given that you are more open about such thinks and take picture (in general).
I.e. there can be a massive difference between a probability "over all humans" and a probability "over people of a given culture" as long as either the given culture is in a minority or underrepresented in given data.
Given that people normally don't (knowingly) give out their private family photos when they know they culture is seen as "bad" by some people and this picture might be abused I think we can at least assume such culture(s) are underrepresented.
Through we can't say how much that changes the probability.
1 in a trillion with a billion devices with 10k pics is not a small chance. But what do we know? It's not like Apple is communicating any numbers anywhere so we can make a reasonable guess as to the number of false positives, and as to whether we may think it is actually worth it.
It’s not per photo, it’s per photo library. But it’s per year, so on average once every 1000 years there will be a false positive for someone.
They are communicating numbers. For example, they tested with 100 million photos and got 3 false positives. They also tested with 100 k normal porn photos and got 0 false positives.
I think their numbers are completely irrelevant, now that we know the visual hash can be gamed. Since it can be, it will be.
Basically, that 1 in a trillion number has an implicit "assuming people aren't cheating", as most mathematical models do. But it's already evident people can cheat this system.
I don't know what the odds will end up being, 1 in a trillion or 1 in 100, but they will not be based on statistical analysis. The odds will be based on cultural and social factors... how quickly do Apple reviewers get overwhelmed? How easily can script kiddies use the tools to fake hashes? Are there consequences for false reports?
How many people want to get you in trouble?
As it is explained in the "readme" part, in this specific context, "naturally occurring" means that no one has purposefully manipulated any of the images to make them collide: that the images were already published and "out there" and happen to collide. In other words, it does not necessarily imply that the images correspond to natural photographic scenes (which seems to be your interpretation of it).
Besides, you could probably "naturally" obtain such type of colliding images by photographing similar-looking objects against a white (or generally featureless) background. Furthermore, it suggests/demonstrates that similar-looking images with similar backgrounds can lead to unexpected collisions in practice (i.e. "naturally"), even if you do not assume an adversarial scenario.
Are you sure that, if you take a picture of a naked body part, it won't collide with anything that looks similar in their database?
It is unlikely unless you manage to capture some position and happen to have some background. This whole thing is a nothingburger. This is one of those weird things were many people have baseless gut reactions and then try to go and prove if flawed even though they don't have a complete picture.
It is unlikely that there is a collision of benign image with the database and even if that happens it is not some automatic process that just sends cops to your house to raid it.
Of course we can get bunch of collitions with essentially same images, I don't get why this is so magical just squint your eyes and I'm sure you have two objects with in your reach that could be made to collide, but that isn't a gotcha on any level
Isn't a hash collision from similar images the point of the whole thing?
At any rate, IANAL, but I'm pretty sure you can't be convicted based on a hash alone. If you get busted for possession of a picture of a nematode and you can show the jury it's just a picture of an axe that has the same value when run through this algorithm, you'll be fine. And there's a decent chance prosecutors won't chase down individuals who will just have a single collision in their photo library with this tech in the first place - people who have dozens or hundreds will be much more interesting.
Technically speaking, this does not prove that an adversarial attack is possible on the CSAM system of apple, Given that apple has another not released neural hash system on their servers which is potentially larger and works better than the one on device.
The more interesting technical question for me is: do collisions transfer across models? or how to find collisions that transfer across models?
Is it possible for the courts to use this system to search a defendant's phone for leaked documents say? Like if NSA learns that one of a small group leaked document X, can they get a court to force Apple to add the hash of Document X to the database on that group of people's phones? If so, I bet this becomes the new norm for investigating leaks.
I think no one is anymore afraid of 40 accidental natural image collisions.
But un-natural image collisions or bad images in the database and similar are a different matter and had been the main critique point from the get to go as far as I can tell.
Also given how many people use IPhones, how many pictures they have and how often they have many similar pictures, thinks are not necessary that simple.
I wouldn't be surprised if some flat, small height fully adult (e.g. 30) woman does some sexting and goes from 0 to >40 collisions in a month. Not because of arbitrary collisions but because the similarity some of here sexting pictures might have with the ones from a 14y old but older looking girl (which e.g. where forced and ended up in the database).
As a non-apple user you could be impacted indirectly by people you know being directly impacted or by Apple's practices being imported into the law. E.g. laws that attempt to outlaw encryption lacking apple-like backdoors.
This is of course entertaining, but since Apple has already tested for this, with 100 million images, and adjusted the rules accordingly, it has no practical implications.
Is it really a catalogue when there only are two of them?
I find it amusing that they probably ran this tool against a set of millions or even billions of images and this is the best they could come up with. They are practically praising Apple here lmao
It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system).
So what's left when all the criminals this is supposed to catch have figured it out?
False positives. Only false positives.
Is it really worth turning personal devices into snitches that don't even do a good job of protecting children?
Also, numbers about false positives must be taken with a grain of salt because of the non-uniform distribution of perceptual hashes. It might be that your random vacation photos and kitty pics have a 1-in-a-million chance of a fapo, but someone who happens to (say) live in an apartment that has been laid out very similarly to a scene in pictures appearing in the CSAM database may have a massively higher chance of fapos for photos taken in their home.