VALIDATION CHECKLIST HOLDOUT v0 10 6
# v0.10.6 — Independent Holdout Validation Checklist
## Do not retune on the original 600
The 600-university benchmark was used for calibration. Use a new cohort for validation and keep v0.10.6 unchanged while testing it.
## 1. Integrity
- Scoring Engine reports v0.10.6.
- 10,000 maximum points.
- 5 pillars / 24 KPIs.
- Pillar maxima = 5,000 / 1,800 / 1,500 / 700 / 1,000.
- KPI8 = 600 points.
## 2. External validation
Preferred External Spearman band: 0.60–0.70.
Do not tune merely to hit a single number; also inspect tier separation and outliers.
## 3. Strong/medium/lower separation
- Strong anchors should concentrate in the upper HE tiers.
- Strong-vs-lower pairwise discrimination should preferably remain >=95%.
- Investigate extreme reversals via KPI_Audit before changing methodology.
## 4. Size bias
Review Total Points correlations with:
- Research Outputs
- Lifetime Citations
- Researcher Base
A sustained citation correlation substantially above ~0.70–0.72 deserves review.
## 5. Redundancy
Review every absolute KPI Spearman >=0.85.
Pay particular attention to whether the v0.10.6 de-correlation changes continue to separate KPI10, KPI21 and KPI22 from their baseline measures.
## 6. Data quality
- Open `Primitive_Data_Audit`.
- Distinguish MISSING, OBSERVED_ZERO and STRUCTURAL_ZERO.
- Confirm Groupings are present in the raw master so KPI20 is computed natively.
- Review `Sparse_Metric_Diagnostics`, especially OpenDOAR coverage.
## 7. Freeze decision
If the holdout cohort preserves plausible top-tier behavior, strong tier separation, acceptable size bias, and external convergence around the preferred band, freeze v0.10.6 as the basis for v1.0.
## Do not retune on the original 600
The 600-university benchmark was used for calibration. Use a new cohort for validation and keep v0.10.6 unchanged while testing it.
## 1. Integrity
- Scoring Engine reports v0.10.6.
- 10,000 maximum points.
- 5 pillars / 24 KPIs.
- Pillar maxima = 5,000 / 1,800 / 1,500 / 700 / 1,000.
- KPI8 = 600 points.
## 2. External validation
Preferred External Spearman band: 0.60–0.70.
Do not tune merely to hit a single number; also inspect tier separation and outliers.
## 3. Strong/medium/lower separation
- Strong anchors should concentrate in the upper HE tiers.
- Strong-vs-lower pairwise discrimination should preferably remain >=95%.
- Investigate extreme reversals via KPI_Audit before changing methodology.
## 4. Size bias
Review Total Points correlations with:
- Research Outputs
- Lifetime Citations
- Researcher Base
A sustained citation correlation substantially above ~0.70–0.72 deserves review.
## 5. Redundancy
Review every absolute KPI Spearman >=0.85.
Pay particular attention to whether the v0.10.6 de-correlation changes continue to separate KPI10, KPI21 and KPI22 from their baseline measures.
## 6. Data quality
- Open `Primitive_Data_Audit`.
- Distinguish MISSING, OBSERVED_ZERO and STRUCTURAL_ZERO.
- Confirm Groupings are present in the raw master so KPI20 is computed natively.
- Review `Sparse_Metric_Diagnostics`, especially OpenDOAR coverage.
## 7. Freeze decision
If the holdout cohort preserves plausible top-tier behavior, strong tier separation, acceptable size bias, and external convergence around the preferred band, freeze v0.10.6 as the basis for v1.0.