"Results for all cause mortality were similar. Prior to 2000, 24 trials reported all cause-mortality and 5 reported significant reductions in total mortality (25%), 18 were null (71%) and one (CAST) reported significant harm (Table 3). Following the year 2000, no study showed a significant benefit for total mortality."
So 24/30 pre-2000 trials they looked at reported all cause mortality. This is important because all cause mortality is the gold standard endpoint in a clinical trial. If you were cherry picking an outcome measure, this is the last one you would choose as it's the hardest one to get a positive result for.
This casts doubt on the paper's central thesis. How to explain the difference in pre and post registration all cause mortality? The possibilities are:
1. Trials performed after 2000 are genuinely less likely to be positive because of altered funding, lack of low hanging fruit etc. It is a well known problem that estimating the survival of the control group in a randomised trial based on historical data is usually an underestimate, which messes up the power calculation and the chance of a positive result. Beyond a certain point, it becomes infeasible to do a clinical trial to show a benefit over a very healthy control group, because you would need 100,000 patients.
2. After registration became mandatory, fraudulent investigators who make up their results stopped doing clinical trials.
The other point is that in most areas, there are only a few acceptable clinical trial endpoints. For example in cancer studies, there are really only 3: overall survival (time alive), event free survival (time until the cancer comes back or starts growing) and response rate (% of cases where the tumour shrinks by 20% or more). Cardiology is a bit different because there is less consensus about valid endpoints. Nevertheless, clinicians and regulatory bodies are pretty strict about which endpoints they consider meaningful.
So in my opinion the authors chose the subject area that would give them the most 'shocking' result (Cardiology, due to having more options for endpoints), glossed over the interpretation (why such a big difference in the gold standard endpoint that is supposed to be immune to manipulation?), and over-hyped the significance of the result.
"Results for all cause mortality were similar. Prior to 2000, 24 trials reported all cause-mortality and 5 reported significant reductions in total mortality (25%), 18 were null (71%) and one (CAST) reported significant harm (Table 3). Following the year 2000, no study showed a significant benefit for total mortality."
So 24/30 pre-2000 trials they looked at reported all cause mortality. This is important because all cause mortality is the gold standard endpoint in a clinical trial. If you were cherry picking an outcome measure, this is the last one you would choose as it's the hardest one to get a positive result for.
This casts doubt on the paper's central thesis. How to explain the difference in pre and post registration all cause mortality? The possibilities are: 1. Trials performed after 2000 are genuinely less likely to be positive because of altered funding, lack of low hanging fruit etc. It is a well known problem that estimating the survival of the control group in a randomised trial based on historical data is usually an underestimate, which messes up the power calculation and the chance of a positive result. Beyond a certain point, it becomes infeasible to do a clinical trial to show a benefit over a very healthy control group, because you would need 100,000 patients. 2. After registration became mandatory, fraudulent investigators who make up their results stopped doing clinical trials.
The other point is that in most areas, there are only a few acceptable clinical trial endpoints. For example in cancer studies, there are really only 3: overall survival (time alive), event free survival (time until the cancer comes back or starts growing) and response rate (% of cases where the tumour shrinks by 20% or more). Cardiology is a bit different because there is less consensus about valid endpoints. Nevertheless, clinicians and regulatory bodies are pretty strict about which endpoints they consider meaningful.
So in my opinion the authors chose the subject area that would give them the most 'shocking' result (Cardiology, due to having more options for endpoints), glossed over the interpretation (why such a big difference in the gold standard endpoint that is supposed to be immune to manipulation?), and over-hyped the significance of the result.