Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I'm a little confused. Are you saying that a tenure track prof wrote a paper on how to evauluate fitted power law curves? Was it something else besides least squares? Because I can't possibly see this getting accepted to a statistics journal.


Fitting distributions is a little bit different than the usual model fitting scenario where least squares is appropriate. Sometimes people do things like construct a histogram and then do a least squares fit to the bin heights, but that procedure doesn't satisfy the usual assumptions that justify least squares (observations with independent, equal variance, Gaussian errors).

Cosma Shalizi has written some interesting posts on this subject, and also published papers in statistics journals:

http://bactra.org/weblog/491.html


It is far beyond least squares as it was and is practiced

There already was stuff in the stats literature in the 1990s that was much better but people in the physics community (such as myself) were not aware of that literature. On the other hand, stats people were not particular aware of the way power laws were occuring in physics.

I saw things that did not add up ten years earlier and Mark Newman did too but we were both so caught up in the rat race, consensus reality, collective delusion, whatever you call it that I left physics before I could address the problem and he suffered through years of bullshit before he could find the time to do something about it.

Watching Mark write great papers, write great book chapters and suffer from tremendous anxiety over his career was a big reason why I left.


Remember what a parameter estimator is.

If you sample 100 values out of a very large pool and add them up then divide by 100 what you get is not the mean of the distribution, but an estimator of the mean of the distribution as you would get a slightly different answer if you picked a different 100.

Often estimators are simple formulas like that (they are for power laws) but there are subtle details, for instance to get the standard deviation in that case you might think you divide by N (100) but you really should divide by N-1 (99).

Back in the 1990s myself and the people around knew some popular statistic formulas but not the concept of estimating a parameter.

And of course its not physics. Social scientists and life science people tend to take a course on statistics but it is fair to say that the median paper in those fields has some mistake in how they do statistics.




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: