It's worth saying that all of Al-Aly's papers so far have had the same basic structure, and the same basic problems: they take a population of older, sicker people (veteran's administration data), compare to some synthetic, non-randomized control group (this is where the "study of 15M" comes from), then extrapolate from that comparison to everyone. This is problematic for a few reasons:
* the older, sicker population is simply not representative of the population at large. No amount of normalization or adjustment can truly fix this.
* Moreover, the normalization process creates a "modeling problem" -- it's too easy for the researcher to find whatever outcome he's seeking by mucking with the weightings. With methods like this, you have to be very careful interpreting the results, because tiny details in the methods matter a lot.
* the signals in the data often are smaller in magnitude than the covariates in the study population (e.g. finding an X% difference in thing Y, when the population is more than X% sicker, overall)
* Use of ICD (medical billing) codes for a study like this is fraught. Choice of codes can create/remove a small signal in the data, and yet, it's not always clear that certain codes should be used (basic example: do you use the ICD code for a bypass graft as a "cardiac event" in a study like this? Almost certainly, a heart bypass surgery reflects a life-long problem, not an acute event due to Covid.) Again, methods matter a great deal here.
* Al-Aly is...less than transparent...about the size of the observed effect and its dependence on age and health. Often these details are buried in supplemental materials, and then he goes in the news media and makes bombastic claims about something "even happening in younger populations", based on a relatively weak effect size extrapolated in younger, healthy people.
* Relatively strong claims are made about small differences in outcome. For example, I just saw one last night from Al-Aly that turned a 13 per 1000 difference in outcome (1.3%), into a breathless claim of "over 40% increased risk". Mathematically true, perhaps, but utterly misleading (also, the difference was observed primarily in the oldest people, so the citing mean difference was simply wrong).
I could go on. Point is, please don't take this study as definitive. This is one paper, and Al-Aly has an agenda.
* the older, sicker population is simply not representative of the population at large. No amount of normalization or adjustment can truly fix this.
* Moreover, the normalization process creates a "modeling problem" -- it's too easy for the researcher to find whatever outcome he's seeking by mucking with the weightings. With methods like this, you have to be very careful interpreting the results, because tiny details in the methods matter a lot.
* the signals in the data often are smaller in magnitude than the covariates in the study population (e.g. finding an X% difference in thing Y, when the population is more than X% sicker, overall)
* Use of ICD (medical billing) codes for a study like this is fraught. Choice of codes can create/remove a small signal in the data, and yet, it's not always clear that certain codes should be used (basic example: do you use the ICD code for a bypass graft as a "cardiac event" in a study like this? Almost certainly, a heart bypass surgery reflects a life-long problem, not an acute event due to Covid.) Again, methods matter a great deal here.
* Al-Aly is...less than transparent...about the size of the observed effect and its dependence on age and health. Often these details are buried in supplemental materials, and then he goes in the news media and makes bombastic claims about something "even happening in younger populations", based on a relatively weak effect size extrapolated in younger, healthy people.
* Relatively strong claims are made about small differences in outcome. For example, I just saw one last night from Al-Aly that turned a 13 per 1000 difference in outcome (1.3%), into a breathless claim of "over 40% increased risk". Mathematically true, perhaps, but utterly misleading (also, the difference was observed primarily in the oldest people, so the citing mean difference was simply wrong).
I could go on. Point is, please don't take this study as definitive. This is one paper, and Al-Aly has an agenda.