Many investigators in parapsychology who were sincere and intelligent appear to have based their career on the incorrect use of frequentist statistics.
And it's not just them. Ernerst Rutherford, who discovered the atomic nucleus, "If your experiment needs statistics, you ought to do a better experiment." In the 1990s I was a physics grad student and I think none of the professors had ever heard of the idea of a parameter estimator so we had a bunch of ad-hoc ways to fit power law coefficients that gave different answers and no way to judge goodness of fit that was thought out at all.
One postdoc in my lab suffered through a difficult job market before finally, after a decade of anxiety and uncertainty, got a tenure track position and eventually wrote a paper on how to fit power law curves... in a statistics journal.
And this was in a good department with people in which I was proud of both the teaching and research going on.
I'm a little confused. Are you saying that a tenure track prof wrote a paper on how to evauluate fitted power law curves? Was it something else besides least squares? Because I can't possibly see this getting accepted to a statistics journal.
Fitting distributions is a little bit different than the usual model fitting scenario where least squares is appropriate. Sometimes people do things like construct a histogram and then do a least squares fit to the bin heights, but that procedure doesn't satisfy the usual assumptions that justify least squares (observations with independent, equal variance, Gaussian errors).
Cosma Shalizi has written some interesting posts on this subject, and also published papers in statistics journals:
It is far beyond least squares as it was and is practiced
There already was stuff in the stats literature in the 1990s that was much better but people in the physics community (such as myself) were not aware of that literature. On the other hand, stats people were not particular aware of the way power laws were occuring in physics.
I saw things that did not add up ten years earlier and Mark Newman did too but we were both so caught up in the rat race, consensus reality, collective delusion, whatever you call it that I left physics before I could address the problem and he suffered through years of bullshit before he could find the time to do something about it.
Watching Mark write great papers, write great book chapters and suffer from tremendous anxiety over his career was a big reason why I left.
If you sample 100 values out of a very large pool and add them up then divide by 100 what you get is not the mean of the distribution, but an estimator of the mean of the distribution as you would get a slightly different answer if you picked a different 100.
Often estimators are simple formulas like that (they are for power laws) but there are subtle details, for instance to get the standard deviation in that case you might think you divide by N (100) but you really should divide by N-1 (99).
Back in the 1990s myself and the people around knew some popular statistic formulas but not the concept of estimating a parameter.
And of course its not physics. Social scientists and life science people tend to take a course on statistics but it is fair to say that the median paper in those fields has some mistake in how they do statistics.
Many investigators in parapsychology who were sincere and intelligent appear to have based their career on the incorrect use of frequentist statistics.
And it's not just them. Ernerst Rutherford, who discovered the atomic nucleus, "If your experiment needs statistics, you ought to do a better experiment." In the 1990s I was a physics grad student and I think none of the professors had ever heard of the idea of a parameter estimator so we had a bunch of ad-hoc ways to fit power law coefficients that gave different answers and no way to judge goodness of fit that was thought out at all.
One postdoc in my lab suffered through a difficult job market before finally, after a decade of anxiety and uncertainty, got a tenure track position and eventually wrote a paper on how to fit power law curves... in a statistics journal.
And this was in a good department with people in which I was proud of both the teaching and research going on.