Original Research
Part of Original research for digital PR
Checking whether the sample supports the headline
Audit a research headline against the analysed sample, measure, denominator and selection method before making a public claim.
Read a proposed research headline as a claim about a group, a measure and a level of certainty. Then check whether the analysed cases, selection method and calculation support that claim. If the headline implies more people, a stronger conclusion or a cause the study did not establish, revise it before release.
Translate the headline into claims
“Australians prefer…” makes a claim about Australians and preference. A voluntary survey of one business’s customers may show only what its participating customers selected when asked a particular question. A description of the narrower sample at the end of the article cannot reliably fix an overbroad opening.
Check the wording against four records:
- Target population:Who was the study intended to describe?
- Analysed cases:Who contributed to this figure, and how were they selected?
- Measure:What question or recorded event produced the result?
- Calculation:What denominator, comparison and handling of missing values produced it?
Use the base behind a subgroup figure, not the full study count. Include the relevant place and period in the headline or opening when leaving them out would broaden the claim.
Before Publishing a Research Headline: Verify These Four Elements
- 1. Target populationIs the headline clearly about the intended group? (e.g., ‘Australian households’ vs. ‘one provider’s customers’)
- 2. Analysed casesWere participants selected using a transparent and appropriate method?
- 3. MeasureWas the data collected via a clear, consistent and valid question or event?
- 4. CalculationIs the denominator correct? Are missing values handled appropriately?
Check inference and precision
A probability design can support estimates for a defined population when coverage, non-response, weighting where used and analysis are properly addressed. For a volunteer or other non-probability sample, ordinary design-based sampling error cannot be estimated from known selection probabilities. A large volunteer sample can still miss people unlikely to encounter or answer the invitation.
A margin of error addresses a specified source of uncertainty under its assumptions; it does not account for every possible error. Do not attach a conventional probability-sample margin of error to a non-probability survey. A model-based precision measure needs its model, assumptions and calculation explained and justified.
Inspect each comparison group’s actual base. Before calling one group “more likely”, check whether the apparent difference is supportable under the study design and uncertainty. If a subgroup or pattern was selected after looking at results, say it was exploratory. A striking chart alone does not settle statistical or practical importance.
Key Statistics to Evaluate Research Claims
- Sampling Method
- Voluntary (non-probability)
- Margin of Error
- Not applicable (non-probability sample)
- Response Rate
- Unknown (no data on non-response)
Make a claim decision
Evidence position / Headline decision
- A result for a defined group using a suitable measure
- Name that group and what was measured.
- A selected group presented as a wider population
- Narrow the population wording or obtain stronger evidence.
- An association presented as a cause
- Remove the causal wording unless the design supports it.
- An unstable or uncheckable calculation
- Hold the claim until it is resolved.
For example, “Australian households are abandoning service A” would require evidence about Australian households, a defined measure of abandonment and an appropriate comparison. Responses from one provider’s participating customers could not, on their own, carry that headline. A narrower account of those responses may still be useful if the question, selection and base are clear.
Ask someone outside the study team to read only the headline and opening, then explain what they think was measured. Compare that reading with the sample record. If it is materially broader, revise the first impression.
