The worse problem is that once you get the private key you can MITM the traffic or, if forward security was not used, decrypt a sniffed one -- this will limit the casual attacker to a directed attacks, though.