I think that is Apple's mistake. I have to go into settings and turn the two finger pref/next pages to three, three finger spaces swipe to four just to avoid the problem you described.

And why the hell don't they enable "tab to all controls" by default?

Yes, still, I only "use" this feature by mistake.

especially because it's too easy to make it when scrolling up