So basically this study is “we looked at various publicly available data sets of information, and here is a thing we notice that has previously gone unnoticed”.
The noticed it in multiple datasets.
What would replication look like here besides someone else looking at the dataset and agreeing that they also see the curiosities?
Or do you mean someone measuring a new fresh set of data and looking at it?
Asking for replication in this case is surprising, because seemingly the entire value is to prompt other research to go figure out what’s going on with these things.
My reading is that the GP moreso takes issue with how social sciences are discussed, and is using an arbitrary replication-less finding in the 'hard sciences' (this one) as a podium to speak from--regardless of why replication doesn't exist for it, or whether that reason makes sense.
Rather obviously, I'd think, if someone else looks at similar samples and sees no "Obelisks" at all. Or looks at their samples.
> What is H_0 in this study?
"The stuff we're seeing here doesn't exist"?
Or, IOW, "All the electron microscopes used have the same weird bug, showing obelisk-shaped pieces of RNA material where there really isn't anything at all"?
Feels a bit like what you want is really "We were all high on funny mushrooms when we saw that".
Look, I see from your other comments that you're actually not talking about this, but something else entirely. And you may well have a point there, but in order to make that point here, you would have had to come up with something a lot better than this. Because I'm fairly sure they weren't all high on funny mushrooms. Aren't you too, really?
It's more that it's a category error. Replication has a context for where it's important, e.g. a treatment and cause & effect relationship. It doesn't If you say "we did this to people and it caused them to respond this way X% of the time," that's something that needs to replicate for us to know if we should take it seriously. If you look at a bunch of separate data sources and find out that a bunch of them show that a certain thing verifiably exists, that doesn't need to replicate to be taken seriously. The data should be checked to make sure the thing actually does exist, but that's verification.
So most findings in research psychology definitely need replication. But the idea that the existence of something is verifiably found across a bunch of different datasets doesn't need a new set of experiments to show -- you just check the data.
One thing that makes soft sciences softer is that it is inherently more difficult to achieve the same statistical significance, due to both higher underlying variances and logistics of obtaining large samples. Publishing with higher p-values results in lower reproducibility by definition.
The problem with psychology is that despite the fact that the subject matter is inherently more difficult to study, researchers are forced into the same publish-or-perish system as biologists and mathematicians. A higher degree of skepticism towards new studies might technically be prejudice, but it’s certainly justified.
