Checkpoint Promotion
Unverified●32/40Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add checkpoint-promotionWho is stuck, and on what
Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a tuned model ships, or when a promoted model needs re-gating against updated goldens.
The whole source
Frontmatter — 2 properties
| name | checkpoint-promotion |
|---|---|
| description | Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a tuned model ships, or when a promoted model needs re-gating against updated goldens. |
| 1 | --- |
| 2 | name: checkpoint-promotion |
| 3 | description: Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a tuned model ships, or when a promoted model needs re-gating against updated goldens. |
| 4 | ---A5 — No allowed-tools declared — no way to tell what this skill may touch |
| 5 | |
| 6 | # Checkpoint Promotion |
| 7 | |
| 8 | The Phase 5 gate for the whole |
| 9 | plugin: a checkpoint that trains |
| 10 | cleanly and beats its task metric |
| 11 | still doesn't ship without |
| 12 | clearing all four stages below. |
| 13 | `eval-harness-first` built the |
| 14 | suite re-run here — this skill is |
| 15 | where that suite's baseline |
| 16 | decides something. |
| 17 | |
| 18 | **Input:** a trained checkpoint, |
| 19 | `eval/baseline-<model>.json` from |
| 20 | `eval-harness-first`, and the |
| 21 | frozen `eval/drift-suite.yaml`. |
| 22 | **Output format:** |
| 23 | `promotion-report.md` — the |
| 24 | four-stage evidence plus a |
| 25 | terminal `PROMOTE` or `REJECT` |
| 26 | verdict that `/finetune` Phase 5 |
| 27 | and `/promote-checkpoint` consume |
| 28 | directly. |
| 29 | |
| 30 | ## The Four-Stage Gate |
| 31 | |
| 32 | Each stage gates the next — a |
| 33 | failure at stage 2 means stage 3 |
| 34 | doesn't run. Stages 2 and 3 share |
| 35 | one expensive inference pass, so |
| 36 | running them concurrently and |
| 37 | applying gate order at verdict |
| 38 | time is licensed on a |
| 39 | **deterministic** arena (nothing |
| 40 | saved by serializing); a |
| 41 | judge-based arena should still |
| 42 | wait for stage 2 first — that's |
| 43 | where the real savings are. |
| 44 | |
| 45 | 1. **Data-quality gate.** Before |
| 46 | any eval touches the |
| 47 | checkpoint: dedup the training |
| 48 | set, check for eval-goldens |
| 49 | leakage (the exact failure |
| 50 | `trace-to-training-data`'s |
| 51 | Hygiene section exists to |
| 52 | prevent), and scan for label |
| 53 | noise. A checkpoint trained on |
| 54 | leaked goldens invalidates |
| 55 | every later stage. |
| 56 | 2. **Held-out + frozen |
| 57 | capability-drift suite.** |
| 58 | Re-run `eval-harness-first`'s |
| 59 | `eval/drift-suite.yaml` — |
| 60 | MMLU/GSM8K/IFEval plus 200–500 |
| 61 | domain-adjacent items — against |
| 62 | the checkpoint and diff against |
| 63 | `baseline-<model>.json` per |
| 64 | benchmark against the Drift |
| 65 | Budget table below. |
| 66 | 3. **Paired arena vs. base.** |
| 67 | Position-randomized judge, |
| 68 | checkpoint vs. base model, same |
| 69 | prompts — or the deterministic |
| 70 | paired-comparison variant in |
| 71 | `references/gate-templates.md` |
| 72 | when every grader in the |
| 73 | harness is deterministic (no |
| 74 | LLM-judge; position |
| 75 | randomization N/A there). |
| 76 | **A holdout win that |
| 77 | loses the live arena does not |
| 78 | ship** — stage-2 numbers and |
| 79 | stage-3 judgments must agree; a |
| 80 | win on frozen goldens and a |
| 81 | loss in paired comparison is a |
| 82 | real signal, not a discrepancy |
| 83 | to explain away. |
| 84 | 4. **Canary.** 5–10% stratified |
| 85 | rollout with auto-rollback for |
| 86 | any checkpoint reaching |
| 87 | production traffic. **Local-only |
| 88 | users stop at stage 3** — |
| 89 | skipping stage 4 for a local |
| 90 | deployment is the correct |
| 91 | stopping point, not a shortcut. |
| 92 | |
| 93 | ### Drift Budget |
| 94 | |
| 95 | | Drift (pts) | Verdict | |
| 96 | |---|---| |
| 97 | | ≤1 | Noise — proceed | |
| 98 | | 2–5 | Rerun with seed variation before deciding | |
| 99 | | >5 | **HARD FAIL** — no exception for task gains | |
| 100 | |
| 101 | The >5pt row governs regardless |
| 102 | of the others: a checkpoint that |
| 103 | gained 8 points on the target |
| 104 | task and lost 6 points of general |
| 105 | capability still fails here — |
| 106 | task improvement never buys back |
| 107 | a drift-budget breach. |
| 108 | |
| 109 | **Item count derives from the |
| 110 | budget, not convenience:** the |
| 111 | strict n for a half-width under |
| 112 | half the 5pt hard-fail threshold |
| 113 | is ~1,300 at typical accuracy |
| 114 | (p≈0.7); n=200 is a pragmatic |
| 115 | floor (±6pt half-width at that |
| 116 | same p, n=50 ±13pt) — report the |
| 117 | half-width with every verdict, |
| 118 | and treat a margin smaller than |
| 119 | it as `REJECT (uncertain)`, not |
| 120 | PASS/HARD FAIL. Full math and a |
| 121 | 5-run cautionary example: |
| 122 | `references/gate-templates.md`. |
| 123 | |
| 124 | **RERUN is not a verdict.** A |
| 125 | 2–5pt drift only ever produces a |
| 126 | `PROMOTE` or `REJECT` after the |
| 127 | seed-variation rerun completes — |
| 128 | `PROMOTE` requires landing back |
| 129 | at ≤1pt (noise); any rerun still |
| 130 | >1pt — 2–5pt band or >5pt breach |
| 131 | alike — resolves stage 2 to a |
| 132 | hard `REJECT`. No report may |
| 133 | reach the Verdict section with |
| 134 | stage 2 still showing `RERUN`. |
| 135 | |
| 136 | ## Catastrophic Forgetting |
| 137 | |
| 138 | Unmanaged LoRA fine-tuning loses |
| 139 | real general capability, and |
| 140 | stage 2 is what catches it: |
| 141 | |
| 142 | - **~43% knowledge loss |
| 143 | unmanaged** — no replay, no |
| 144 | regularization. |
| 145 | - **~10% with basic management** |
| 146 | — some replay or a conservative |
| 147 | LR. |
| 148 | - **~3% with replay + EWC** — the |
| 149 | disciplined case. |
| 150 | - **10–30% general-data replay |
| 151 | mix is the standard |
| 152 | mitigation** — blend general- |
| 153 | domain data into training |
| 154 | rather than target-task data |
| 155 | alone. |
| 156 | |
| 157 | If a checkpoint hits the >5pt |
| 158 | hard fail in stage 2, work this |
| 159 | escalation ladder in order — the |
| 160 | one canonical order this skill |
| 161 | and `references/gate-templates.md` |
| 162 | both point to: |
| 163 | |
| 164 | 1. **Adjust the replay-mix |
| 165 | fraction — swap rows, don't |
| 166 | add them** (adding confounds |
| 167 | fraction with total optimizer |
| 168 | steps). Dose is not monotonic |
| 169 | at small-run scale (<~100 |
| 170 | steps) — re-check drift after |
| 171 | any swap. |
| 172 | 2. **Lower the learning rate.** |
| 173 | 3. **Fewer epochs.** |
| 174 | 4. **A smaller LoRA rank** — the |
| 175 | same rank/LR levers |
| 176 | `lora-qlora-recipes` and |
| 177 | `preference-optimization` tune |
| 178 | for the training run, applied |
| 179 | here in reverse. |
| 180 | |
| 181 | This order is a default, not a |
| 182 | law: **remediation guidance from |
| 183 | a single before/after run pair |
| 184 | is a hypothesis** — label it |
| 185 | low-confidence once any lever |
| 186 | produces a reversal, and prefer |
| 187 | a seed-variation repeat over |
| 188 | trusting the next rung blindly. |
| 189 | A lever that clears the drift |
| 190 | breach but drops a |
| 191 | success-criterion metric below |
| 192 | target is a two-sided tradeoff |
| 193 | for a human, not a reason to |
| 194 | keep descending the ladder. Full |
| 195 | reasoning and the 5-run |
| 196 | trajectory behind both caveats: |
| 197 | `references/gate-templates.md`. |
| 198 | |
| 199 | **Disclose drift-suite |
| 200 | instruction reuse.** A replay row |
| 201 | copying the drift harness's exact |
| 202 | instruction phrasing (not just |
| 203 | disjoint source items) makes that |
| 204 | benchmark's post-replay score an |
| 205 | upper bound — flag it |
| 206 | instruction-familiar, or re-probe |
| 207 | with a paraphrase, before |
| 208 | treating a near-budget pass as |
| 209 | clean. |
| 210 | |
| 211 | ## The Verdict |
| 212 | |
| 213 | `promotion-report.md` covers all |
| 214 | four stages as sections and |
| 215 | **must end with a terminal |
| 216 | verdict: `PROMOTE` or `REJECT`**, |
| 217 | the evidence that produced it, |
| 218 | and exactly one top remediation |
| 219 | when the verdict is `REJECT`. |
| 220 | Template: `references/gate-templates.md`. |
| 221 | The terminal contract other |
| 222 | skills parse: |
| 223 | |
| 224 | ``` |
| 225 | ## Verdict |
| 226 | |
| 227 | REJECT |
| 228 | |
| 229 | Evidence: domain-adjacent drift |
| 230 | suite dropped 6.2pt (threshold: |
| 231 | >5pt hard fail) despite +8pt on |
| 232 | the target task. |
| 233 | |
| 234 | Top remediation: swap the |
| 235 | replay-mix fraction from 10% |
| 236 | toward 20%, holding step count |
| 237 | constant. |
| 238 | ``` |
| 239 | |
| 240 | - **REJECT is a result, not an |
| 241 | error.** A checkpoint that |
| 242 | fails stage 2's drift budget or |
| 243 | stage 3's arena comparison did |
| 244 | its job. Don't treat a REJECT |
| 245 | as a failed run needing a rerun |
| 246 | of this skill; it's the correct |
| 247 | output of a working gate. |
| 248 | - **One remediation, not a |
| 249 | menu.** Evidence sections may |
| 250 | list everything observed; the |
| 251 | verdict section names the |
| 252 | single highest-leverage fix per |
| 253 | the escalation ladder above. A |
| 254 | report that hedges across three |
| 255 | possible fixes hasn't done the |
| 256 | prioritization this skill |
| 257 | exists to do. |
| 258 | - **No auto-retraining.** This |
| 259 | skill produces a verdict and a |
| 260 | report, not a re-triggered |
| 261 | training run. A `REJECT` hands |
| 262 | the remediation back to a human |
| 263 | decision at |
| 264 | `finetuning-method-selection` or |
| 265 | the relevant training skill. |
| 266 | |
| 267 | ## Related Skills |
| 268 | |
| 269 | - `eval-harness-first` — owns the |
| 270 | drift suite and baseline this |
| 271 | skill re-runs and diffs |
| 272 | against; no `baseline-<model>.json` |
| 273 | means nothing to gate against. |
| 274 | - `quantized-export` — the only |
| 275 | valid next step after a |
| 276 | `PROMOTE` verdict. |
| 277 | - `preference-optimization` and |
| 278 | `lora-qlora-recipes` — own the |
| 279 | LR and rank levers in the |
| 280 | Catastrophic Forgetting |
| 281 | escalation path; this skill |
| 282 | diagnoses the breach, those |
| 283 | skills own the config that |
| 284 | caused it. |
| 285 | - `dataset-curation` — owns the |
| 286 | replay-mix construction recipe |
| 287 | the escalation ladder's first |
| 288 | rung applies. |
| 289 | |
| 290 | Complete `promotion-report.md` |
| 291 | template with all four stages, |
| 292 | the drift-suite scoring table, |
| 293 | the paired-arena protocol (item |
| 294 | count, position randomization, |
| 295 | win-rate threshold), and a |
| 296 | replay-mix configuration example: |
| 297 | `references/gate-templates.md`. |
| 298 |
Reviews
Installed this one?Write the first review and take the Trailblazer badge.
Alternatives
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.◐◐◐◐◐●35/40Ebay Seller Tools·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.●····●32/40