Getting to it might be very, very hard. But once it's in your hand you've got it. Diamonds can found as you're leaving the building, or you can be stopped at the airport, or the big pile in your house can be found, etc.
A signing key is like, 4096 bits at most? 4 sms messages will do it.
So the plan is to ... what ... break the physical security precautions of the HSM during a burglary and then leave without the operator noticing? When I say "break the physical security precautions" that typically involves things like grinding open the casing, depotting the innards, avoiding trigger the self-zeroing mechanism etc etc. and doing so while avoiding the anti-tamper sensors.
If the key is located only in the HSM, and there are no known flaws in the HSM, yep, that's a hard nut to crack, I'm not going to claim I could do it. But given enough resources, with time and planning and insiders? Take a look at https://cryptosense.com/blog/how-ledger-hacked-an-hsm - and that's a remote attack.
I think their point is that it's easier because you don't need to do the "leave without the operator noticing" that you need to for physical items. Once you have access to those 4096 bits you don't have to escape with them, you just need to transmit 4 sms messages.
An HSM is going to make that difficult. The whole point of an HSM is that you can’t get the key from the HSM, you can only take the entire HSM with you.
Of course it is possible to read a key from an HSM, it’s just designed to be incredibly difficult. I don’t think I would be able to do it, at all, even if I could bring the HSM home with me for a couple weeks.
And in the case of the HSMs I've dealt with, just taking it home is enough to render it useless without some incredibly specific and difficult precautions.
A signing key is like, 4096 bits at most? 4 sms messages will do it.