Hypothetical example pseudocode:

    if is_pointer(pt):
        // do pointer-based stuff
    else:
        raise error

If you train the branch predictor to expect a pointer, it will speculatively treat arbitrary values as pointers until it can determine that they are not. So you can pass in any value and get it treated like a pointer for the duration of the window of speculative execution.

Any conditional branch is potentially vulnerable, an attacker just needs some sort of side effect from speculative execution that persists after rollback.