That's not great but it would be acceptable to switch focus between screens (or big enough tiles in a tiled desktop). Together with some clever mechanisms one could define task/tab/window groups per area and make switching much more manageable for large amounts of elements without having to surrender to "pick the mouse"-"select"-"back to the kb".