> I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper
Same. Now, when I review manuscripts, I pay much more attention to whether there is enough information to replicate the experiment or simulation. We can put out a paper with wrong interpretations and that’s fine because other people will realise that when doing their own work. We cannot let papers get published if their results cannot be replicated.
> The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review
Things that take too much space to go in the experimental section should go to a electronic supplementary information document. But then it would be nice if the ESI were appended to the article when we download a PDF because tracking them is a pain in the backside. Some fields are better than others about this, for example in materials characterisation studies it’s very common to have ESI with a whole bunch of data and details.
Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. It’s also a nice way of doubling a student’s papers count by the end of their PhD.
> Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2]
> Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use.
This may be possible in some sciences, but not in epidemiology or biomed. Often the study is based on tissue samples owned by some entity, with permission granted only to some certain entity.
Datasets in epidemiology are often full of PII, and cannot be shared publicly for many reasons.
Same. Now, when I review manuscripts, I pay much more attention to whether there is enough information to replicate the experiment or simulation. We can put out a paper with wrong interpretations and that’s fine because other people will realise that when doing their own work. We cannot let papers get published if their results cannot be replicated.
> The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review
Things that take too much space to go in the experimental section should go to a electronic supplementary information document. But then it would be nice if the ESI were appended to the article when we download a PDF because tracking them is a pain in the backside. Some fields are better than others about this, for example in materials characterisation studies it’s very common to have ESI with a whole bunch of data and details.
Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. It’s also a nice way of doubling a student’s papers count by the end of their PhD.
> Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2]
That’s just evil!