Threats to scientific validity
The threats to scientific validity were laid out over 40 years ago in the seminal work Experimental and quasi-experimental designs for research by Campbell and Stanley. In science validity is important in two ways. The first is how valid the experiment and the data are in addressing the original hypotheses being tested. This is called internal validity. The second is how valid it is to extend the results of the experiment to wider populations. This is called external validity.
The goal of science is to construct theories and gather data to back up these theories that maximizes both internal and external validity. The scientific method is an attempt to limit and control the number of threats to validity an experiment faces. Cogent application of statistics and good experimental design can greatly increase validity. The meticulous design, method and analysis of science is what separates it out from many pseudosciences that attempt to claim some form of validity. Proponents of pseudoscience, woo, and quackery either ignore threats to validity or use them to generate false data to back up their crazy claims.
The eight most frequently cited threats to internal validity are:
- History, the specific events occurring between the first and second measurements in addition to the experimental variables
- Maturation, processes within the participants as a function of the passage of time (not specific to particular events), e.g., growing older, hungrier, more tired, and so on.
- Testing, the effects of taking a test upon the scores of a second testing.
- Instrumentation, changes in calibration of a measurement tool or changes in the observers or scorers may produce changes in the obtained measurements.
- Statistical regression, operating where groups have been selected on the basis of their extreme scores.
- Selection, biases resulting from differential selection of respondents for the comparison groups.
- Experimental mortality, or differential loss of respondents from the comparison groups.
- Selection-maturation interaction, etc. e.g., in multiple-group quasi-experimental designs
Four factors jeopardizing external validity or representativeness are:
- Reactive or interaction effect of testing, a pretest might increase
- Interaction effects of selection biases and the experimental variable.
- Reactive effects of experimental arrangements, which would preclude generalization about the effect of the experimental variable upon persons being exposed to it in non-experimental settings
- Multiple-treatment interference, where effects of earlier treatments are not erasable.
Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research.