The sample must answer the population question
A large sample is not automatically representative. A survey of people who volunteer after a late-night event may miss those who cannot attend at that time. Increasing the number of respondents from the same biased process can make the estimate more stable without making it more accurate for the target population.
Check how people were selected, who failed to respond, how questions were worded, and whether answers measure behavior or intention. “Would you support a useful service?” invites a different response from a question that states its cost and alternatives. An intention to attend is not an attendance count.
Keep counts and proportions separate
A rate compares a count with a denominator. Ten incidents among one hundred visits is a higher incident rate than twenty among one thousand visits, despite the smaller incident count. An answer that gives only a numerator cannot settle a rate comparison. Always ask “out of how many?”
Relative and absolute changes also differ. A rise from 2% to 3% is one percentage point and a 50% relative increase. Both descriptions are mathematically correct, but they convey scale differently. A high percentage increase from a tiny baseline can still be a small absolute change.
Check what an average hides
A mean can change because unusually large values change, while most cases remain similar. The median tracks the middle observation but does not describe the full distribution either. Comparing average outcomes without considering group composition may also mislead: an overall rate can move as the sizes of subgroups change.
Comparisons need a relevant baseline. “The program is expensive” might mean expensive relative to last year, to alternatives, or to its benefits. Different baselines support different conclusions. In a reasoning exercise, use the comparison actually offered rather than supplying favorable missing figures.
Worked example
Last year a workshop had 20 returning attendees out of 50. This year it had 30 returning attendees out of 100. The organizer says the return rate improved.
- Last year
- 20 ÷ 50 = 40%.
- This year
- 30 ÷ 100 = 30%.
- What increased
- The count of returning attendees increased from 20 to 30.
- What decreased
- Their share of all attendees decreased from 40% to 30%. More returning people and a lower return rate can both be true.
Check your understanding
Take it into practice
Write the denominator and baseline next to every number before accepting the comparison.