Manchester warning: ‘No difference’ may mislead

beehiveweb.co.uk editorial team 2026-08-18 18:44 2 min read

A laboratory pipette drops a clear liquid into a row of glass test tubes.

By the Beehive Web editorial desk | Source published 17 August 2026

A p-value above 0.05 does not prove that an effect is absent. Researchers from the universities of Manchester, Oxford and Arkansas warn that treating this statistical result as evidence of “no difference” can turn an unresolved question into an unjustified conclusion.

Their paper, published in PNAS on 10 August 2026, examines how equivalence testing could produce more defensible research interpretations.

An above-0.05 p-value does not prove absence

A p-value measures how surprising the observed data would be if there were genuinely no effect. When it exceeds the commonly used 0.05 threshold, researchers lack sufficient evidence to declare a statistically significant difference—but they have not demonstrated that the compared outcomes are equivalent.

Manchester warning: ‘No difference’ may mislead

The University of Manchester account says the mistaken interpretation appears in about 50% of research papers and conference presentations. It does not identify the evidence behind that estimate, so its prevalence cannot be assessed from the supplied material.

Equivalence testing changes the question

Rather than looking only for evidence of a difference, equivalence testing asks whether any possible difference is too small to matter scientifically, clinically or practically. The team focuses on the two one-sided tests procedure, known as TOST.

Researchers must decide before collecting data how small an effect would be considered unimportant. That boundary is a scientific judgement rather than an answer produced automatically by statistical software.

Small samples can hide meaningful effects

A small or highly variable study may return a non-significant result even when a meaningful effect exists. In a therapy study, for example, reporting “no difference” could prematurely divert attention from a promising treatment; the same mistake could conceal a genuine risk.

Equivalence testing is intended to leave weak evidence classified as inconclusive while separating it from evidence that an effect is genuinely negligible.

The source leaves key limits unresolved

The University summary presents a methodological warning, not results from a new clinical experiment. It provides no study population, sample size or measured health outcome, and it does not publish quantitative validation results for the team’s free online calculator. The supported implication is therefore limited to research interpretation: some non-significant findings may be uncertain, but the paper does not establish that any particular previous result was wrong.

Source: University of Manchester News

Context & actions About this article

Source check Source trail

The report preserves the University’s methodological warning while identifying claims that cannot be evaluated from its summary alone.

Checks

Confirmed the paper title, named institutions and PNAS publication date against the suppli...
Kept a p-value above 0.05 distinct from evidence that an effect is absent.
Flagged that the summary provides no experiment, sample or supporting citation for its app...
Separated the paper’s research-methodology implications from clinical guidance.

University of Manchester News Check wider coverage