You assume a perfect implementation of the backdoor. Even if the cryptographic part were well-implemented, someone will accidentally ship a release build with a poorly safeguarded test key, or with a disabled safety that they normally use to test it.

It's an unnecessary moving part that can break, except that this particular part breaking defeats the whole purpose of the system.