As you observe, workers talking to each other to agree on prices for labor is the flip side of the coin from CEOs talking to each other to agree on prices for labor. But the latter is in fact illegal. CEOs can't talk to each other and agree to pay $10/hour for warehouse workers. Coordinating with others to set prices for labor or goods isn't viewed as within the scope of freedom of association. Union activities are in fact specifically exempted from anti-trust laws because otherwise they would fall within the scope of them. Workers can agree with each other not to accept less than $15/hour for warehouse workers.
You need something more than freedom of association to justify unions, rooted in the recognition of bargaining disparities between employers and employees.
The way I'm thinking about it, you don't need anything extra to justify unions, but you do need something extra to prohibit wage fixing. I agree that freedom of association, when considered alone, would allow wage fixing and price fixing and all sorts of anti-competitive behavior.
We're moving away from areas where I'm confident I know what I'm talking about, but I think as a society we've decided that while anti-competitive laws do infringe on the rights of business leaders, we're trying to balance their rights with those of everyone else, and the laws are necessary to prevent a permanent class divide between business leaders who cannot be challenged, and workers under them. In the long run, allowing complete free association among CEOs would limit the freedoms of the rest of society.
Society is a constant project of balancing various conflicting rights, and this is one of many cases where we limit the rights of a few to defend the rights of many.
You've got it the wrong way round. You start by assuming everything is permitted and then you selectively (and ideally reluctantly) bring the weight of law to bear when you discover that it's a net benefit to society. It's a correction for gross inequalities of power.
That's something the libertarian left and libertarian right surely agree on - minimising the application of the monopoly of force and all that.
So you assume both employers and employees can freely organize and associate. That's your starting point.
Now - we've decided that the right to join a union should be protected (it evens out an existing power imbalance) and that collusion to force down wages should be illegal (because it amplifies an existing power imbalance to the detriment of society at large)
You need something more than freedom of association to justify unions, rooted in the recognition of bargaining disparities between employers and employees.