The definition of “proper experimental design” varies widely by the field, and DL’s methodology looks quite solid, as far as social sciences and the task at hand are concerned.
The generally poor quality of research in the social sciences (proven so by the lack of reproducibility) is not really a good defense for the poor quality of any individual study.
Yes, it is? Science does not guarantee knowledge in any field, it's only supposed to increase the odds. Are the odds of getting true knowledge in this field increased compared to the alternatives (I.E. people just arbitrarily claiming stuff based on personal anecdotes?) Then it's the best we have in that field and rationally you have no reason to believe the claims of knowledge acquired from worse methods over the better alternatives - just like it works in other fields.
Why do you assume research in the social sciences even increases the odds of a true result? Combining low standards with a publish or perish culture actually encourages the publication of spurious results (and worse behavior) and hinders the careers of researchers with higher standards.
I didn't say low standard of evidence has higher odds than high standard of evidence.
Low standard has higher odds than no standard of evidence or no research at all. High > low > zero. By what logic could you claim that low = zero or zero > low?
All else being equal, I'd agree. The low standard might at least filter out some of the false results.
But all else is not equal. The pressure to publish something interesting, even if false, greatly outweighs the incentive to be accurate, and overwhelms whatever filter the low standard might provide.
Even the author of this article acknowledges the problem:
> When I write a paper, I have to deal with co-authors who push for putting in false or misleading material that makes the paper look good and my ability to push back against this has been fairly limited.
To their credit, the author makes some effort to point out the flaws in this study, and acknowledges that this is just "a preliminary result that is, at best, weak evidence."
That's a start, but obviously not as good as actually putting in the work to correct those flaws.
