You can get around this. A biased HRNG is still an RNG, and there are other sources of entropy on a system anyways, so a decent OS can make exploiting a bias in an RNG infeasible.
I assume that HSM as discrete hardware devices certainly use their hardware RNG only as one input to some /dev/random like CSPRNG. On the other hand I would also assume that single-chip "HSMs" (smartcards...) do not although I vaguely remember that for TPM (which is hardware-wise a smartcard with weird host interface) the RNG output is somehow dependent on state of attestation registers.
I'm probably biased because I spend good part of quarter by designing reasonably secure CSPRNG for smartcard chip without hardware RNG (and ended up exploiting essentially any cross-clockdomain communication as entropy source) and thus I assume that typical smardcard vendors don't care about that (too much work) while HSM vendors simply leverage infrastructure of whatever (RT)OS they use and probably harden that somewhat.
