Analyst Choices Cause Wildly Different Wildlife Study
A large-scale study in ecology and evolutionary biology reveals that different researchers analyzing the same data can produce dramatically different
Different research teams analyzing identical wildlife datasets can produce wildly different results. A study published in BMC Biology found substantial variation in effect sizes and predictions solely due to analysts' differing statistical decisions.
The research, led by a project team that recruited 174 analyst groups comprising 246 researchers, used two unpublished ecological datasets. One dataset came from evolutionary ecology, focusing on blue tits (Cyanistes caeruleus), to compare sibling number and nestling growth. The other was from conservation ecology, studying Eucalyptus, to examine the relationship between grass cover and tree seedling recruitment. Each analyst team investigated prespecified research questions using the same raw information.
Variation in Blue Tit Growth Analysis
For the blue tit data, 141 usable effect sizes were generated. The average effect was convincingly negative, indicating less growth for nestlings with more siblings. However, the range of reported effects was vast. Results showed near-continuous variation from large negative effects to effects near zero. Some analyses even crossed the traditional threshold of statistical significance in the opposite direction, suggesting a positive relationship.
Contrasting Results for Eucalyptus Seedlings
The Eucalyptus dataset produced 85 usable effects. Here, the average relationship between grass cover and seedling number was only slightly negative and not convincingly different from zero. Most effects ranged from weakly negative to weakly positive. About one third of the effects crossed the traditional significance threshold in one direction or the other. The study authors also noted, "there were also several striking outliers in the Eucalyptus dataset, with effects far from zero."
No Clear Link to Method Choices or Reviews
The researchers examined whether specific analytical decisions explained why some results were far from the average. They found substantial variation in variable selection and random effects structures among the different teams' models. Peer reviews of the analytical methods also varied. Despite this, the study found no strong relationship between any of these factors and a result's deviation from the meta-analytic mean. Analyses with outlier results were no more or less likely to have dissimilar variable sets, use random effects, or receive poor peer reviews than analyses finding results near the mean.
Implications for Wildlife Science
The existence of this substantial variability raises critical questions for ecology and evolutionary biology. It challenges how scientists should interpret published results, given that analytical flexibility can lead to divergent conclusions from the same evidence. The findings also prompt reflection on how future analyses should be conducted to improve consistency and reliability. The study shows that variation in scientific outcomes often exceeds what sampling error alone would produce.
