Part of working as a scientist involves reading published material and using the results. You may be synthesising the results from other papers as part of planning your own study or to make overall recommendations to another group. You may look at the published material to interpret your own data, and to decide what to do next. If you continue in science, you may also find yourself reviewing manuscripts for publication and assessing proposals for funding. In these cases, you need to look at the claims made by authors, generally in abstracts, media releases, etc., and decide whether the data support those claims.
How closely you scrutinize those claims will depend on why you are reading the material. If it’s very close to your own research, the results may influence the direction your research takes, and you’ll want to be very sure that you should trust the results. If, on the other hand, you’re making a broad synthesis, your conclusion may be based on many published papers, and you’ll pay less attention to individual ones.
This astivity asks you to look closely at some data analyses, decide whether you accept them, and whether your examination leads you to a different conclusion about the results. It is designed to get you thinking about how to review Results and Methods sections of papers. We have provided a guide to the kinds of questions you should ask about the treatment of the data.
We have asked a series of questions as a guide to your dissection of the paper, and they are provided at the back of this document. Things to pay particular attention to:
The biological question. Is clear, and stated in a way that links to a statistical model?
The statistical model. The authors should give you enough information for you to be satisfied, but it’s often useful for you to write out your own model based on their description. Is everything that should be in the model there? If not, why not?
The experimental (or observational) units. What are they, and have they been used appropriately?
If it’s a complex model, be on the lookout for mixed models, i.e. the presence of random and fixed effects. Remember that the correct hypothesis tests in mixed models are not the same as when all factors are fixed. You can often see mismatches by looking closely at the degrees of freedom for particular tests.
As an example of how to do this, we’ll work through a single paper, and you’ll be asked to answer the following questions:
What (biological) question(s) were the data designed to answer?
What kind of statistical model(s) were fitted to the data?
What are your preliminary conclusion, based purely on what the authors said about their results?
What assumptions are associated with the statistical model(s) used?
Did the authors provide you with enough information to determine whether the data analysis is appropriate?
If not, what additional information would you like to see presented?
What changes would you make to the data analysis?
What would you conclude after your assessment of the data analysis?