Threats to scientific validity
The threats to scientific validity were laid out over 40 years ago in the seminal work Experimental and quasi-experimental designs for research by Campbell and Stanley.[1] In science, validity is important in two ways. Internal validity refers to the degree to which the data addresses the original tested hypotheses. External validity describes the generalizability of specific research findings to phenomena outside of the research project.
The goal of science is to construct theories and gather data to support or falsify these theories in a manner that maximizes both internal and external validity. The scientific method attempts to limit and control the number of threats to validity faced by an experiment. Cogent application of statistics and good experimental design can greatly increase validity. Meticulous design, method and analysis of science result in the validity lacking in pseudoscience. Proponents of pseudoscience, woo, and quackery either ignore threats to validity or use them to generate false data to back up their crazy claims.
The eight most frequently cited threats to internal validity are:
- History - the specific events occurring between the first and second measurements in addition to the experimental variables
- Maturation - natural changes within the participants over time (not specific to particular events), e.g., growing older, hungrier, more tired, and so on.
- Testing - the effects of taking a test upon subsequent retests.
- Instrumentation - changes in calibration of a measurement tool or changes in the observers or scorers may produce changes in the obtained measurements.
- Statistical regression - operating where groups have been selected on the basis of their extreme scores.
- Selection biases - findings resulting from differential selection of respondents for the comparison groups.
- Experimental mortality - or differential dropout between groups.
- Selection-maturation interaction - etc. e.g., in multiple-group quasi-experimental designs
Four factors jeopardizing external validity or representativeness are:
- Reactive or interaction effect of testing, a pretest might increase
- Interaction effects of selection biases and the experimental variable.
- Reactive effects of experimental arrangements, which would preclude generalization about the effect of the experimental variable upon persons being exposed to it in non-experimental settings
- Multiple-treatment interference, where effects of earlier treatments are not erasable.
[edit] Footnotes
- ↑ Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research.