I've experienced what you describe. I think the problem is most severe when you're on a high-latency connection, such as a VPN to an endpoint far away, because there are a few round trips in the process with youtube-dl. I don't really have a solution to that, but using --force-window=immediate can help a little (it will make a blank mpv window appear immediately, before the video is ready, so at least you aren't left wondering whether you misclicked prior to the video starting.)