It's true that Apple never _sees_ a hash that doesn't match, but the encrypted hash is included in the safety voucher. That is to say, all hashes are uploaded, but only the matches can ever be decrypted, and that's only if there are enough matches.
From the technical summary [0]:
The device creates a cryptographic safety voucher that encodes the match result. It also encrypts the image’s NeuralHash and a visual derivative. This voucher is uploaded to iCloud Photos along with the image.
From the technical summary [0]:
The device creates a cryptographic safety voucher that encodes the match result. It also encrypts the image’s NeuralHash and a visual derivative. This voucher is uploaded to iCloud Photos along with the image.
[0]: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...