AgentAlley
Submit

Phase 3 — Grade → Iterate (the bounded loop)

Phase 3 of building a Claude Managed Agent — the bounded grade→iterate loop.

How to use it

Claude Code
  1. Run the line below. It pulls the whole folder into ~/.claude/skills/grade-iterate, including the files SKILL.md points to.
  2. Describe your job in plain words. Claude Code follows the skill from there.
Claude Code — installs the whole folder, not just SKILL.md
npx degit alirezarezvani/claude-skills/agent-launcher/skills/grade-iterate#main ~/.claude/skills/grade-iterate

For one project only, change the path to .claude/skills/grade-iterate. This skill also uses outcome_builder.py, verdict_reader.py, eval_scaffold.py, goal_state.py, cma-primitives.md, loops-and-workflows.md — copying SKILL.md alone won't be enough. See the folder on GitHub.

Claude (web or desktop app)
  1. On this page open ⋯ → Download .md.
  2. Save it as SKILL.md in a folder, zip the folder, then Customize → Skills → + → Create skill → Upload a skill.
  3. Pick the file and Save. Claude shows the name and description and runs a security scan.
  4. Check the skill is switched on.
  5. Start a new chat and describe your job in plain words. The AI follows the skill from there.
ChatGPT or another app
  1. ChatGPT: make a Project and paste it into Instructions.
  2. Neither? Paste it at the top of a new chat — it works for that chat.
Not working?
  • Check which app you pasted it into — the steps above name the right one.
  • Some skills need the paid tier of Claude or ChatGPT.
Step-by-step guide with screenshots · Ask in the forum

Paste into Claude, ChatGPT or Cursor.

Source of Phase 3 — Grade → Iterate (the bounded loop)

Show the full text74 lines
grade-iterate/SKILL.md74 lines · 4.1 KB
Outline
RawView on GitHub
namedescriptionversionauthorlicensetagscompatible_tools
grade-iteratePhase 3 of building a Claude Managed Agent — the bounded grade→iterate loop. Define a CMA outcome (a required markdown rubric graded by an isolated grader), read each verdict, decide the next move (sharpen / re-run / promote to schedule), and once a version passes, run held-back eval cases in parallel. Use when the user says "grade my agent", "make it pass the rubric", "iterate until it's good", "is it good enough", or when the orchestrator routes phase=grade-iterate. outcome_builder.py builds the user.define_outcome payload (rubric required, max_iterations clamped 1..20 — never unbounded); verdict_reader.py reads the grader result and recommends the next move; eval_scaffold.py generates held-back cases + a parallel run plan (capped at the 25-thread CMA ceiling). Distinct from stage-launch (first launch) and run-without-you (scheduling).2.11.2Alireza RezvaniMIT[cma, outcome, rubric, grader, grade-iterate, loop, max-iterations, eval, held-back][claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]

Phase 3 — Grade → Iterate (the bounded loop)

This is the plugin's loop: CMA's outcome primitive self-grades the agent's work in an isolated context and feeds failing verdicts back for the next attempt. It is always bounded by max_iterations (1..20) — never "improve forever".

See ../../references/loops-and-workflows.md and the outcome section of ../../references/cma-primitives.md.

Workflow

  1. Define the outcome.
    python3 scripts/outcome_builder.py \
  --sheet ./my-agent/build-sheet.json --max-iterations 5 \
  --out ./my-agent/payloads/outcome.json
    The rubric is required; max_iterations is clamped to 1..20. Send the payload as a user.define_outcome event (append to the running session).
  2. Read every verdict first.
    python3 scripts/verdict_reader.py --result ./my-agent/last-verdict.json
    Tables the rubric outcome and recommends: SHIP (satisfied), SHARPEN then re-run (needs_revision), ESCALATE (max_iterations_reached / failed), RESUME (interrupted). With ≤1 iteration left it flips to "make the single highest-value fix or escalate now".
  3. Loop invariant. Each iteration must move ≥1 rubric line fail→pass, or the run halts at the cap and escalates. Don't burn the budget on cosmetic edits.
  4. Once a version passes, run held-back eval.
    python3 scripts/eval_scaffold.py \
  --sheet ./my-agent/build-sheet.json --out ./my-agent/eval.json --concurrency 5
    Held-back cases (never seen during iteration) run in parallel, capped at the 25-thread CMA ceiling, each graded against the same rubric.
  5. Decide. SHIP as v0, or promote to a scheduled deployment (Phase 4). Record the verdict on the goal: goal_state.py set --phase run-without-you.

Hard rules

  • Bounded, always. No outcome without a max_iterations cap.
  • Read the verdict before acting. The grader's explanation drives the next move.
  • Held-back cases are held back. Never grade generalization on cases the agent already iterated against.

Forcing-question library (recommend + cite)

  1. "What are the 3–5 rubric lines?" Recommend: grounded, checkable criteria. Cite: cma-primitives.md (rubric required).
  2. "How many iterations before you'd rather look yourself?" Recommend: 3–5. Cite: loops-and-workflows.md (bounded loop).
  3. "On a fail, sharpen the prompt or the tools?" Recommend: whichever rubric line failed points to. Cite: verdict_reader next-move table.
  4. "Which cases did the agent NOT see?" Recommend: hold back ≥3 for generalization. Cite: this SKILL (held-back eval).

Tools

  • scripts/outcome_builder.py — user.define_outcome payload (rubric required, cap 1..20).
  • scripts/verdict_reader.py — grader result → next move.
  • scripts/eval_scaffold.py — held-back cases + parallel run plan (≤25 threads).
1---
2name: grade-iterate
3description: Phase 3 of building a Claude Managed Agent — the bounded grade→iterate loop. Define a CMA outcome (a required markdown rubric graded by an isolated grader), read each verdict, decide the next move (sharpen / re-run / promote to schedule), and once a version passes, run held-back eval cases in parallel. Use when the user says "grade my agent", "make it pass the rubric", "iterate until it's good", "is it good enough", or when the orchestrator routes phase=grade-iterate. outcome_builder.py builds the user.define_outcome payload (rubric required, max_iterations clamped 1..20 — never unbounded); verdict_reader.py reads the grader result and recommends the next move; eval_scaffold.py generates held-back cases + a parallel run plan (capped at the 25-thread CMA ceiling). Distinct from stage-launch (first launch) and run-without-you (scheduling).
4version: 2.11.2
5author: Alireza Rezvani
6license: MIT
7tags: [cma, outcome, rubric, grader, grade-iterate, loop, max-iterations, eval, held-back]
8compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
9---
10 
11# Phase 3 — Grade → Iterate (the bounded loop)
12 
13This is the plugin's **loop**: CMA's `outcome` primitive self-grades the agent's
14work in an isolated context and feeds failing verdicts back for the next attempt.
15It is **always bounded** by `max_iterations` (1..20) — never "improve forever".
16 
17See [`../../references/loops-and-workflows.md`](../../references/loops-and-workflows.md)
18and the outcome section of
19[`../../references/cma-primitives.md`](../../references/cma-primitives.md).
20 
21## Workflow
22 
231. **Define the outcome.**
24 ```bash
25 python3 scripts/outcome_builder.py \
26 --sheet ./my-agent/build-sheet.json --max-iterations 5 \
27 --out ./my-agent/payloads/outcome.json
28 ```
29 The **rubric is required**; `max_iterations` is clamped to 1..20. Send the
30 payload as a `user.define_outcome` event (append to the running session).
312. **Read every verdict first.**
32 ```bash
33 python3 scripts/verdict_reader.py --result ./my-agent/last-verdict.json
34 ```
35 Tables the rubric outcome and recommends: **SHIP** (`satisfied`), **SHARPEN**
36 then re-run (`needs_revision`), **ESCALATE** (`max_iterations_reached` /
37 `failed`), **RESUME** (`interrupted`). With ≤1 iteration left it flips to
38 "make the single highest-value fix or escalate now".
393. **Loop invariant.** Each iteration must move ≥1 rubric line fail→pass, or the
40 run halts at the cap and escalates. Don't burn the budget on cosmetic edits.
414. **Once a version passes, run held-back eval.**
42 ```bash
43 python3 scripts/eval_scaffold.py \
44 --sheet ./my-agent/build-sheet.json --out ./my-agent/eval.json --concurrency 5
45 ```
46 Held-back cases (never seen during iteration) run in parallel, capped at the
47 25-thread CMA ceiling, each graded against the same rubric.
485. **Decide.** SHIP as v0, or promote to a scheduled deployment (Phase 4). Record
49 the verdict on the goal: `goal_state.py set --phase run-without-you`.
50 
51## Hard rules
52 
53- **Bounded, always.** No outcome without a `max_iterations` cap.
54- **Read the verdict before acting.** The grader's explanation drives the next move.
55- **Held-back cases are held back.** Never grade generalization on cases the agent
56 already iterated against.
57 
58## Forcing-question library (recommend + cite)
59 
601. "What are the 3–5 rubric lines?" *Recommend:* grounded, checkable criteria.
61 *Cite:* cma-primitives.md (rubric required).
622. "How many iterations before you'd rather look yourself?" *Recommend:* 3–5.
63 *Cite:* loops-and-workflows.md (bounded loop).
643. "On a fail, sharpen the prompt or the tools?" *Recommend:* whichever rubric line
65 failed points to. *Cite:* verdict_reader next-move table.
664. "Which cases did the agent NOT see?" *Recommend:* hold back ≥3 for generalization.
67 *Cite:* this SKILL (held-back eval).
68 
69## Tools
70 
71- `scripts/outcome_builder.py` — user.define_outcome payload (rubric required, cap 1..20).
72- `scripts/verdict_reader.py` — grader result → next move.
73- `scripts/eval_scaffold.py` — held-back cases + parallel run plan (≤25 threads).
74 

Discussion

Alternatives

Also in Models & evalsSee all 533 in Development →
AI engineerAct as an expert AI engineer specializing in practical machine learning implementation and AI integration for production applications, ensuring efficient and robust AI solutions.Coding · CC0-1.0OneKGPd: Individual-Level Queries over the 1000 Genomes ProjectQuery the 1000 Genomes Project dataset (3,202 whole-genome-sequenced individuals, GRCh38) at the level of individual participants. Use when a question is about individuals or variants in the 1000 Genomes Project cohort: which individuals carry variants matching specific criteria in a gene or region, which individuals are homozygous-reference at a position, which variants exist in the dataset or carried by specified individuals in a gene or region, the relatedness between two specified individuals. Variants are returned with 1000 Genomes allele frequencies (AF), gnomAD v4.1 exome and genome AF, AlphaMissense score, and HGVSp annotations.Science · MITPyMC Bayesian ModelingBayesian modeling with PyMC. Build hierarchical models, MCMC (NUTS), variational inference, LOO/WAIC comparison, posterior checks, for probabilistic programming and inference.Science · MITStatsmodels: Statistical Modeling and EconometricsStatistical models library for Python. Use when you need specific model classes (OLS, GLM, mixed models, ARIMA) with detailed diagnostics, residuals, and inference. Best for econometrics, time series, rigorous inference with coefficient tables. For guided statistical test selection with APA reporting use statistical-analysis.Science · MIT

Browse more free Claude skills or everything in Development.