If you were messing around with a gun that had the safety on, and the gun malfunctioned and shot a child while you were waving it around, would it be a valid defense to say that you took all the necessary precautions (since the safety was on, after all) and it's just a silly oopsie that the gun did that?
You should have a responsibility to put the necessary safeguards on a potentially dangerous tool that you're using and to do your due-diligence if you're using that tool. It would have been impossible to accidentally produce porn of a non-consenting individual if they had not been using an AI image manipulation pipeline at all. If you do choose to use an AI image manipulation pipeline, you should take on the responsibility to make sure that it's not going to do illegal things you didn't tell it to do. If there's a risk of causing harm with it, just don't use it! You don't need to be producing cute images of people in Vegas or whatever, and if doing so necessarily implies a risk of accidentally creating porn of them instead, you could (and in fact, have a responsibility to) just choose to do neither.
A gun malfunction has a potentially lethal consequence. This bug caused a person to see a fake photo of herself with fake boobs out. Perhaps it's ok that the safety standards for the latter are lighter, no?
You also didn't address the main point of the earlier comment. Is every programmer responsible - even criminally, some suggested - for potential bugs or vulnerabilities in one of their products?
You should have a responsibility to put the necessary safeguards on a potentially dangerous tool that you're using and to do your due-diligence if you're using that tool. It would have been impossible to accidentally produce porn of a non-consenting individual if they had not been using an AI image manipulation pipeline at all. If you do choose to use an AI image manipulation pipeline, you should take on the responsibility to make sure that it's not going to do illegal things you didn't tell it to do. If there's a risk of causing harm with it, just don't use it! You don't need to be producing cute images of people in Vegas or whatever, and if doing so necessarily implies a risk of accidentally creating porn of them instead, you could (and in fact, have a responsibility to) just choose to do neither.