Statcheck uses regular expressions to find statistical results in APA format. When a statistical result deviates from APA format, statcheck will not find it. The APA formats that statcheck uses are: t(df) = value, p = value; F(df1,df2) = value, p = value; r(df) = value, p = value; [chi]2 (df, N = value) = value, p = value (N is optional, delta G is also included); Z = value, p = value. All regular expressions take into account that test statistics and p values may be exactly (=) or inexactly (< or >) reported. Different spacing has also been taken into account.
Very helpful -- this points out that this is a very focused tool on simple calculations of test statistics. They appear in papers formatted very strictly, like this:
t(37) = −4.93, p <.001
or:
χ2(1, N = 226) = 6.90, p <.01.
Because of the strict format and the limited scope of the tool, one would suspect the false positive rate is very low. And because it's just a simple calculation (not a matter of interpretation), authors should not (in theory) be offended by getting a notice.
Pretty sure APA format is required for APA journals, but either way, fortunately, the desire on the part of authors to get papers published and on the part of reviewers to have standards and clear presentation outweighs anyone's desire to game statcheck.
But you do make a great point in that statcheck could and should red-line papers that present no discernible stats, and provide links to the APA style guide!
If you are trying to publish a fraudulent paper, you aren't going to do it by including subtle math errors to fudge your data in the direction you want. You would just falsify the data itself; way harder to find, and much more effective.
https://github.com/MicheleNuijten/statcheck
And if you're curious how it works, as I was:
Statcheck uses regular expressions to find statistical results in APA format. When a statistical result deviates from APA format, statcheck will not find it. The APA formats that statcheck uses are: t(df) = value, p = value; F(df1,df2) = value, p = value; r(df) = value, p = value; [chi]2 (df, N = value) = value, p = value (N is optional, delta G is also included); Z = value, p = value. All regular expressions take into account that test statistics and p values may be exactly (=) or inexactly (< or >) reported. Different spacing has also been taken into account.