The 1958 Supreme Court ruling struck down a law that required groups to provide their membership lists to the State of Alabama. They don't have to "prevent" people from being able to assemble, but building lists of which groups people belongs to still violates their "right to pursue their private interests privately". So in large part, the right to assemble also includes the right to be part of any group (even communists!) without the government needing to know.
But again what evidence is there that the NSA is creating these list of groups ? They could just be simply looking at which individuals a person is speaking to. It is a subtle distinction but legally a significant one.
I just seem to see a lot of conjecture but no actual evidence.
It might be legally significant, but technically, there is no real difference.
Given your social network, and training data consisting of the public members of a group you can use a semi-supervised learning algorithm to determine group membership. Since social networks have a small diameter and the public members are typically well connected, you would expect this approach to be fairly accurate. Or rather, accurate enough to act on, if you do not need to provide legal justification for your decision.
What's the required legal justification for putting someone on a no-fly list?
If you want to give some real world basis to the legal distinction you need oversight and transparency. In particular, global analysis of the whole dataset must be illegal, since it pretty much corresponds to total surveillance. Your results might not be completely accurate, but they don't need to be if you can act once you have something like 51% confidence.
