I was homeschooled, and I personally did a lot of organized sports with other homeschoolers. Not team sports per say, but rather, something kinda similar to gym classes/track-and-field. The biggest events held among homeschoolers in that area probably had around 200+ kids attending, with a mix of religious and non-religious families and a wide variety of ages.

edit: FWIW this was in Toronto, Canada.