True, but also a myopic understanding of what actually goes on. Who do you think actually serve on the boards? The answer is other executives. And so there's the all too common situation where pay raises and golden parachutes are essentially quid pro quos.
That situation would require 2 people to have alternate roles at the same companies which is incredibly rare, if not already prevented in the corporate bylaws.
Many board members are in entirely different industries and sectors and some only serve exclusively on boards. Many others are investors whose money is being used to pay those executives. There's also a layer of shareholders above the board and voting is required for all major decisions.