Stat Plan
Mind
“The Blueprint of Scientific Truth”
The Vision
The Vision
“A Statistical Analysis Plan (SAP) is the ultimate binding contract between a researcher, their data, and the scientific method; it ensures that findings emerge from pre-specified logic rather than post-hoc statistical creativity. The core principle of pre-specification—'Decide Before You See'—forces investigators to make all analytical decisions prior to viewing outcome data, protecting against both conscious and unconscious biases that systematically erode research credibility. In modern empirical science, clarity in analysis is the only reliable safeguard against the reproducibility crisis, where unchecked analytical flexibility can cause 50–90% of published findings to be false, exaggerated, or irreproducible. A rigorously designed SAP transforms statistics from a tool of persuasion into a tool of truth.”
Systemic Failure Audit
Systemic Failure Audit
Status
Active Critical Scanning
50-90% of published research findings may be false or exaggerated due to high levels of analytical flexibility and undisclosed researcher degrees of freedom.
Researcher degrees of freedom—choosing between multiple models, covariates, or outcomes—can inflate false positive rates from a nominal 5% to over 60% when multiple approaches are tested without correction.
Only 36% of 100 landmark psychology studies could be successfully replicated in major open science initiatives, exposing the fragility of post-hoc analytic practices.
31% of clinical trials are found to switch primary outcomes after seeing non-significant results, a practice that invalidates statistical inference and misleads clinicians.
Scientific Retraction: Discovery of post-hoc 'outcome switching' or 'p-hacking' is a primary driver of high-impact journal retractions and institutional investigations.
Statistical Dilution: Hunting for 'significant' subgroups when the overall effect is null creates false clinical hope and fuels wasteful downstream research.
Regulatory Rejection: Regulatory bodies like the FDA or EMA will reject trial data if the final analysis deviates from a pre-registered SAP without transparent, rigorous justification.
The Deadly Sins
The Deadly Sins
Detection & Mitigation ProtocolOutcome Switching
"Changing the primary outcome to a secondary one because the primary failed to reach significance, thereby retrofitting success."
Explicitly pre-register the one and only primary outcome before data collection begins.
Subgroup Fishing
"Hunting for specific groups (e.g., 'men over 50') that show significance when the overall trial failed."
Define all subgroup analyses a priori and strictly limit their number to maintain statistical power and credibility.
Covariate Fishing
"Testing multiple demographic or clinical adjustments to find the 'right' model that produces p < 0.05."
List exactly which covariates will be included in the final model before unblinding.
Missing Data Imputation Post-hoc
"Choosing an imputation method (like LOCF vs Multiple Imputation) based on which one improves the results."
Declare the missing data handling strategy in the SAP before viewing dropout patterns.
Ignoring Assumptions
"Failing to plan for what happens if data is non-parametric or violates test assumptions."
Define a decision framework for switching from parametric to non-parametric tests based on pre-specified normality criteria.
Multiple Comparison Error
"Testing dozens of endpoints without adjusting for the increased risk of false positives."
Apply Bonferroni or False Discovery Rate (FDR) corrections to all secondary analyses.
Analysis After Unblinding
"Writing the analysis code after knowing which group is which, allowing for subtle 'interpretive drift'."
Analysts should remain masked (Group A vs Group B) until the final analysis code is locked and executed.
Technical Standards
Technical Standards
- Draft the SAP before data collection (Prospective) or at minimum before unblinding (Retrospective).
- Register the SAP on Clinical Trials.gov or OSF to create a permanent, public timestamp.
- Submit the SAP as an appendix to the IRB/Ethics Committee during the initial application.
- Use strict version control with PI and Statistician signatures if public registration is restricted.
Readiness Checklist
Readiness Checklist
Implementation Playbook
Implementation Playbook
phase 1 pre specification
Define the primary outcome, timepoint, and estimand before any data access. List all planned covariates with clinical justification. Specify handling of missing data (MI vs complete case).
phase 2 protocol lock
Upload SAP to Clinical Trials.gov or OSF. Obtain PI and Statistician signatures. Freeze analytic decisions before unblinding.
phase 3 analysis design
Predefine primary statistical model (e.g., linear mixed model, Cox regression). Define criteria for assumption checks (normality, proportional hazards). Specify adjustment for multiple comparisons.
phase 4 masked analysis
Analyst works with Group A vs Group B labels only. Code is finalized without seeing treatment assignments. Independent statistician reviews the analytic code.
phase 5 unblinding and execution
Unblind only after analysis code is locked. Run final analysis exactly as pre-specified. Document any deviations with justification.
phase 6 reporting
Report both pre-specified and exploratory analyses clearly labeled. Include sensitivity analyses for robustness. Publish SAP alongside manuscript as supplementary material.
phase 7 audit and reproducibility
Archive raw code, data dictionary, and version history. Enable independent replication where possible. Maintain transparent documentation for regulatory review.
Foundational Methodology
Foundational Methodology
Canonical Foundations
Canonical Foundations
Authority & Lineage Audit"Altman — Practical Statistics for Medical Research"
"Gelman et al. — Bayesian Data Analysis"
"Hernán & Robins — Causal Inference: What If"
"Harrell — Regression Modeling Strategies"
"Moher et al. — CONSORT 2010 & Extensions"
"Ioannidis — Why Most Published Research Findings Are False"
"Rubin — Multiple Imputation for Nonresponse in Surveys"
"Little & Rubin — Statistical Analysis with Missing Data"
"Cochrane Handbook for Systematic Reviews of Interventions"
"Senn — Statistical Issues in Drug Development"
The Final Truth
The Final Truth
“The statistical analysis plan is not paperwork; it is the contract you make with the scientific method to ensure your findings are discoveries, not inventions. When rigor is sacred, every p-value represents a step toward legitimate medical truth.”