Why would you believe genetics plays a part? There is minimal evidence for that. You have actual evidence for things like your higher than average time engagement, nutritional indicators, as well as health and dental care. You probably live somewhere with decent air and water quality. Then of course the likely fact that parents have relatively prodigious vocabularies, fluency and articulation. This is why your kids are smart.
Lots of evidence that genetics have a large influence on early language development, especially speech, if you care to look. E.g. this study finds genetics contributing over 60% of variance: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3851292/
No doubt when we start looking at impairments and deficits that you can find plenty of situations where a high proportion of the phenotypic variance is heritable. But when you restrict the study to the top 60th to 95th percentile (which we are discussing here) I don't think you will find particularly strong relationships.
Genetics are a marginal element approaching none.