I agree with you, but as far as I know there is a large gender imbalance (Female to Male is much more common than Male to Female - almost twice as common). How would you explain that?
Society is largely still more accepting of Female to Male transition, than Male to Female. Plus the gendered power balance might leave more people adverse to making the shift in public.
I would also 'guess' that those who transition and dress moving from female to male, can 'pass' more easily in society. Even if someone still has female features, if they dress like a guy, no one cares.