Validity
The extent to which a study's design, execution, and analysis prevents bias. Scientific validity allows researchers to judge how much confidence they have in their research findings being true.
Click here to download a fact sheet on Validity at the bench.
What is validity?
Validity is the extent to which a study’s design, execution, and analysis prevents bias. Scientific validity allows researchers to judge the level of confidence they have in their research findings being true. There are three types of validity to consider:
- Internal validity – achieved through methodological rigor when designing and conducting the research project. The internal validity of a study can be affected by systematic error and random error. Items like randomization and blinding decrease systematic error and can increase internal validity. Adequate sample sizes can reduce random error increase precision of a study’s results
- External validity – achieved when the study findings can be replicated a different time or onto other populations (i.e. other ages, sex, species of animals), or by other researchers, in another lab.
- Construct/Translational validity – achieved when the animal disease model accurately reflects the clinical disease that it is meant to represent.
Who should concern themselves with validity?
All researchers should consider all three types of validity in their studies.
When should validity be taken into consideration?
Validity should be considered throughout the study, from the design phase to the analysis. Methods used to address validity should be planned in advance. Additionally, validity should be considered when reporting a study’s findings.
Why is validity important?
Without considering validity throughout the study, researchers are at risk of systematically biasing the research findings and may result in the model lacking clinical impact, thereby contributing to the current reproducibility crisis.
Optimal Practices
To achieve good internal validity:
Refer to the NIH preclinical reporting guidelines to ensure that all 7 items are met within the study design. For example, appropriate randomization and blinding techniques are planned.
Implement randomization in your experimental design.
Implement blinding in your experimental design.
Do a sample size calculation to help ensure your study is adequately powered.
Account for biological and technical replicates for each outcome.
Record all procedure details and protocol deviations.
More information on how to implement these methods to increase internal validity can be found on this website, along with the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines, and the SYRCLE Risk of Bias tool.
To achieve good external validity:
Publish accurate and detailed standard operating procedures (SOPs) so that another individual can easily replicate your experiments in an attempt to reproduce your findings. Given space restrictions in most journals, consider publishing your protocols as supplemental methods or on an online repository (e.g. Open Science Framework).
Assess whether you can replicate your findings in other strains or species (e.g. C57BL/6 mice as well as BALB/c mice; rats as well as mice).
Assess whether you can replicate your findings with differing experimental conditions (e.g. different doses, different routes of administration, male and female animals, etc.).
Consider teaming up with another lab and assessing whether they can replicate your findings.
To achieve good construct/translational validity:
Design your preclinical disease model to reflect the disease characteristics of patients.
Choose procedures that are relevant to potential downstream clinical implementation such as route of administration, formulation of therapy, timing and dose of treatment.
Measure outcomes that are clinically relevant or may be important to the patient
Perform the experiment in various backgrounds that reflect the baseline health status of the downstream target population. This might include common co-morbidities, age or sex as a factor.
For example, if your disease is common in patients that have type 1 diabetes, you might perform your experiment in a NOD mouse strain or if your disease is common in children you may choose neonatal or young mice to study. It is also considered good practice to perform your experiments in male and female mice to account for sex-based differences.
- Consider conducting large animal studies if findings from small animal studies demonstrate favorable outcomes.
Suboptimal Practices
To achieve suboptimal internal validity:
- Do not implement procedures such as randomization, blinding, and sample size calculations into the design of your experiment
- Fail to account for and report protocol deviations
-
Perform the experiment multiple times, but only publishing results from the studies that show favorable outcomes
To achieve suboptimal external validity:
Do not publish accurate and detailed procedures.
Only use healthy, wild-type animals for all experiments.
Do not explore whether your results are generalizable to other experimental conditions (e.g. both male and female animals, different routes of administration, etc.
- Choose procedure details and time points that are exclusively due to convenience.
To achieve suboptimal construct/translational validity:
- Use a preclinical model that does not reflect the disease characteristics of patients.
- Fail to measure clinically relevant outcomes.
Resources and Tools
Publications outlining importance:
- Annette M. O’Connor, Jan M. Sargeant, Critical Appraisal of Studies Using Laboratory Animal Models, ILAR Journal, Volume 55, Issue 3, 2014, Pages 405–417.
- Henderson, V. C., Kimmelman, J., Fergusson, D., Grimshaw, J. M., & Hackam, D. G. (2013). Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS medicine, 10(7), e1001489.