My TV already has higher latency than that and I manage on Rock Band just fine - anything that depends on A/V synchronization is generally aware of latency being a problem and has knobs for the users to adjust for it.

That being said, waiting multiple frames for video playback to start while audio buffers fill and transmit is a shitty experience - so it’s always good to reduce it.