Selection bias

From RationalWiki
Jump to: navigation, search
Part of the series on

Logic and rhetoric

Icon logic.svg
Key articles
General logic
Bad logic

Selection bias is the bias that occurs in a survey or experimental data when the selection of data points isn't sufficiently random to draw a general conclusion. If selection bias isn't accounted for or acknowledged, and the results are claimed to be widely applicable, it is known as the spotlight fallacy.


Examples of selection bias are rife amongst online polls. These polls are completed only by visitors to the site, and so those with an interest in the website's mission or ones that agree with its POV are the only ones to be surveyed. While the informality of online polls is often clear and admitted to, it's important to not draw generalised conclusions from them.

A website devoted to preventing harassment of women unsurprisingly concluded that nearly all women were victims of harassment at some point,[1] meanwhile a poll on alternative medicine managed to receive more votes than people who read the associated article, a sure sign that something very odd was happening.[2] Polls by the right-wing news site WND are as famous for showing that their readership have right-wing views as much as they are famous for having ridiculous numbers of available options.[3] Similarly, Facebook's recent "Questions" feature spreads via friends in similar circles who are likely to be in a similar area – so asking "what town is the best?" will be easily slanted. Reviews, of restaurants or films for example, may also be highly biased towards the negative because people are more likely to rant on the internet after having a bad time, than they are to rave about something after a good or adequate time.

In science[edit]

Scientific selection bias is a massive problem and often great pains are taken to avoid it. In Andrew Wakefield's heavily discredited paper on the MMR vaccine, he took 12 children who had behavioural disorders and attempted to link them to vaccines. No attempt was made, in the paper, to look at children without behavioural disorders to see if their exposure to the MMR vaccine was greater or less. This would also be an example of confirmation bias.

Selection bias doesn't just apply to people. With meta-studies of existing research and scientific literature, the cited literature may be subjected to a selection bias based on several factors. Only certain types of studies might be included or only ones indexed in certain journals – certainly meta-analysis is restricted to data that is published, and publication bias is a huge problem in itself. This is why good meta-analyses clearly state their search terms and criteria and include all papers that fit these terms, so selection bias is either minimised or at least acknowledged and controlled objectively. Often in pseudoscience the literature is chosen specifically to support a point of view. In this case it is referred to as cherry picking.

Time scale is also a possible biasing factor. Surveys done only at certain times or for too short a period of time will bias a result so that it is not applicable to other times. This is often seen in political approval ratings, where approval during a specific time frame (such as during a crisis or scandal) is not representative of public approval over the course of several years. A clear bias can be made if sample data is restricted to a certain time frame that emphasises a change. RationalWiki's pageviews between March 15th and March 21st 2011 went down significantly according to,[4] but this isn't representative of a longer trend that actually shows an increase. Again, intentional massaging of data by cutting off certain time periods is cherry picking, and greatly frowned up on science and statistics but is often the way global warming denialism presents data.

See also[edit]


Articles on Cognitive biases
  Clustering illusion  -  Cognitive biases-related stubs  -  Confabulation  -  Confirmation bias  -  Demonization  -  Dunning-Kruger effect  -  Experimenter bias  -  Framing  -  Frequency illusion  -  Gambler's fallacy  -  Hindsight bias  -  Hostile media effect  -  Just world hypothesis  -  Outcome bias  -  Overconfidence effect  -  Overgeneralization  -  Response bias  -  Survivorship bias  -  Ultimate attribution error  -  Walled garden