Phase 3 of building a Claude Managed Agent

Phase 3 — Grade → Iterate (the bounded loop)

by alirezarezvani · MIT license · ★ 26,349 on GitHub · v2.11.2 · updated Aug 30

Phase 3 of building a Claude Managed Agent — the bounded grade→iterate loop. Define a CMA outcome (a required markdown rubric graded by an isolated grader), read each verdict, decide the next move (sharpen / re-run / promote to…

---
name: grade-iterate
description: Phase 3 of building a Claude Managed Agent — the bounded grade→iterate loop. Define a CMA outcome (a required markdown rubric graded by an isolated grader), read each verdict, decide the next move (sharpen / re-run / promote to schedule), and once a version passes, run held-back eval cases in parallel. Use when the user says "grade my agent", "make it pass the rubric", "iterate until it's good", "is it good enough", or when the orchestrator routes phase=grade-iterate. outcome_builder.py builds the user.define_outcome payload (rubric required, max_iterations clamped 1..20 — never unbounded); verdict_reader.py reads the grader result and recommends the next move; eval_scaffold.py generates held-back cases + a parallel run plan (capped at the 25-thread CMA ceiling). Distinct from stage-launch (first launch) and run-without-you (scheduling).
version: 2.11.2
author: Alireza Rezvani
license: MIT
tags: [cma, outcome, rubric, grader, grade-iterate, loop, max-iterations, eval, held-back]
compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
---

# Phase 3 — Grade → Iterate (the bounded loop)

This is the plugin's **loop**: CMA's `outcome` primitive self-grades the agent's
work in an isolated context and feeds failing verdicts back for the next attempt.
It is **always bounded** by `max_iterations` (1..20) — never "improve forever".

See [`../../references/loops-and-workflows.md`](../../references/loops-and-workflows.md)
and the outcome section of

Overview

This is the plugin's loop: CMA's outcome primitive self-grades the agent's work in an isolated context and feeds failing verdicts back for the next attempt. It is always bounded by max_iterations (1..20) — never "improve forever".

Manual install (npx, ZIP)

Claude Codenpx skills add alirezarezvani/claude-skills --skill grade-iterate -g -a claude-code

Claude Code (manual)

mkdir -p ~/.claude/skills && curl -sL https://codeload.github.com/alirezarezvani/claude-skills/tar.gz/19392f7a0826 \
  | tar -xz -C ~/.claude/skills --strip-components=3 claude-skills-19392f7a0826/agent-launcher/skills/grade-iterate

Windows (PowerShell)

New-Item -ItemType Directory -Force "$HOME\.claude\skills" | Out-Null; Invoke-WebRequest "https://m.agentalley.io/zip/1a5e48b92de68c5750d0/grade-iterate.zip" -OutFile "$env:TEMP\grade-iterate.zip"; Expand-Archive "$env:TEMP\grade-iterate.zip" -DestinationPath "$HOME\.claude\skills" -Force

Claude.ai / desktopDownload the ZIP and upload it under Settings → Capabilities → Skills.

Codex, Cursor, other agentsPoint them at SKILL.md. The other files are plain text.

Connects with Agent Alley MCP · agentalley.io/mcp

Files included

SKILL.md4.1 KB
README.md997 B
scripts/3 files · 11.6 KB

Read the files here

alirezarezvani/19392f75 files

Shared references2SKILL.md reads these from elsewhere in the repo. Not in the ZIP.

Get them on GitHub ↗

SKILL.md

Show the full text74 lines

grade-iterate/SKILL.md74 lines · 4.1 KB

PreviewCode

Outline

RawView on GitHub

Phase 3 — Grade → Iterate (the bounded loop)

This is the plugin's loop: CMA's outcome primitive self-grades the agent's work in an isolated context and feeds failing verdicts back for the next attempt. It is always bounded by max_iterations (1..20) — never "improve forever".

See ../../references/loops-and-workflows.md and the outcome section of ../../references/cma-primitives.md.

Workflow

Define the outcome.
```
python3 scripts/outcome_builder.py \
  --sheet ./my-agent/build-sheet.json --max-iterations 5 \
  --out ./my-agent/payloads/outcome.json
```
The rubric is required; max_iterations is clamped to 1..20. Send the payload as a user.define_outcome event (append to the running session).
Read every verdict first.
```
python3 scripts/verdict_reader.py --result ./my-agent/last-verdict.json
```
Tables the rubric outcome and recommends: SHIP (satisfied), SHARPEN then re-run (needs_revision), ESCALATE (max_iterations_reached / failed), RESUME (interrupted). With ≤1 iteration left it flips to "make the single highest-value fix or escalate now".
Loop invariant. Each iteration must move ≥1 rubric line fail→pass, or the run halts at the cap and escalates. Don't burn the budget on cosmetic edits.
Once a version passes, run held-back eval.
```
python3 scripts/eval_scaffold.py \
  --sheet ./my-agent/build-sheet.json --out ./my-agent/eval.json --concurrency 5
```
Held-back cases (never seen during iteration) run in parallel, capped at the 25-thread CMA ceiling, each graded against the same rubric.
Decide. SHIP as v0, or promote to a scheduled deployment (Phase 4). Record the verdict on the goal: goal_state.py set --phase run-without-you.

Hard rules

Bounded, always. No outcome without a max_iterations cap.
Read the verdict before acting. The grader's explanation drives the next move.
Held-back cases are held back. Never grade generalization on cases the agent already iterated against.

"What are the 3–5 rubric lines?" Recommend: grounded, checkable criteria. Cite: cma-primitives.md (rubric required).
"How many iterations before you'd rather look yourself?" Recommend: 3–5. Cite: loops-and-workflows.md (bounded loop).
"On a fail, sharpen the prompt or the tools?" Recommend: whichever rubric line failed points to. Cite: verdict_reader next-move table.
"Which cases did the agent NOT see?" Recommend: hold back ≥3 for generalization. Cite: this SKILL (held-back eval).

Tools

scripts/outcome_builder.py — user.define_outcome payload (rubric required, cap 1..20).
scripts/verdict_reader.py — grader result → next move.
scripts/eval_scaffold.py — held-back cases + parallel run plan (≤25 threads).

1	---
2	name: grade-iterate
3	description: Phase 3 of building a Claude Managed Agent — the bounded grade→iterate loop. Define a CMA outcome (a required markdown rubric graded by an isolated grader), read each verdict, decide the next move (sharpen / re-run / promote to schedule), and once a version passes, run held-back eval cases in parallel. Use when the user says "grade my agent", "make it pass the rubric", "iterate until it's good", "is it good enough", or when the orchestrator routes phase=grade-iterate. outcome_builder.py builds the user.define_outcome payload (rubric required, max_iterations clamped 1..20 — never unbounded); verdict_reader.py reads the grader result and recommends the next move; eval_scaffold.py generates held-back cases + a parallel run plan (capped at the 25-thread CMA ceiling). Distinct from stage-launch (first launch) and run-without-you (scheduling).
4	version: 2.11.2
5	author: Alireza Rezvani
6	license: MIT
7	tags: [cma, outcome, rubric, grader, grade-iterate, loop, max-iterations, eval, held-back]
8	compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
9	---
10
11	# Phase 3 — Grade → Iterate (the bounded loop)
12
13	This is the plugin's loop: CMA's `outcome` primitive self-grades the agent's
14	work in an isolated context and feeds failing verdicts back for the next attempt.
15	It is always bounded by `max_iterations` (1..20) — never "improve forever".
16
17	See [`../../references/loops-and-workflows.md`](../../references/loops-and-workflows.md)
18	and the outcome section of
19	[`../../references/cma-primitives.md`](../../references/cma-primitives.md).
20
21	## Workflow
22
23	1. Define the outcome.
24	```bash
25	python3 scripts/outcome_builder.py \
26	--sheet ./my-agent/build-sheet.json --max-iterations 5 \
27	--out ./my-agent/payloads/outcome.json
28	```
29	The rubric is required; `max_iterations` is clamped to 1..20. Send the
30	payload as a `user.define_outcome` event (append to the running session).
31	2. Read every verdict first.
32	```bash
33	python3 scripts/verdict_reader.py --result ./my-agent/last-verdict.json
34	```
35	Tables the rubric outcome and recommends: SHIP (`satisfied`), SHARPEN
36	then re-run (`needs_revision`), ESCALATE (`max_iterations_reached` /
37	`failed`), RESUME (`interrupted`). With ≤1 iteration left it flips to
38	"make the single highest-value fix or escalate now".
39	3. Loop invariant. Each iteration must move ≥1 rubric line fail→pass, or the
40	run halts at the cap and escalates. Don't burn the budget on cosmetic edits.
41	4. Once a version passes, run held-back eval.
42	```bash
43	python3 scripts/eval_scaffold.py \
44	--sheet ./my-agent/build-sheet.json --out ./my-agent/eval.json --concurrency 5
45	```
46	Held-back cases (never seen during iteration) run in parallel, capped at the
47	25-thread CMA ceiling, each graded against the same rubric.
48	5. Decide. SHIP as v0, or promote to a scheduled deployment (Phase 4). Record
49	the verdict on the goal: `goal_state.py set --phase run-without-you`.
50
51	## Hard rules
52
53	- Bounded, always. No outcome without a `max_iterations` cap.
54	- Read the verdict before acting. The grader's explanation drives the next move.
55	- Held-back cases are held back. Never grade generalization on cases the agent
56	already iterated against.
57
58	## Forcing-question library (recommend + cite)
59
60	1. "What are the 3–5 rubric lines?" Recommend: grounded, checkable criteria.
61	Cite: cma-primitives.md (rubric required).
62	2. "How many iterations before you'd rather look yourself?" Recommend: 3–5.
63	Cite: loops-and-workflows.md (bounded loop).
64	3. "On a fail, sharpen the prompt or the tools?" Recommend: whichever rubric line
65	failed points to. Cite: verdict_reader next-move table.
66	4. "Which cases did the agent NOT see?" Recommend: hold back ≥3 for generalization.
67	Cite: this SKILL (held-back eval).
68
69	## Tools
70
71	- `scripts/outcome_builder.py` — user.define_outcome payload (rubric required, cap 1..20).
72	- `scripts/verdict_reader.py` — grader result → next move.
73	- `scripts/eval_scaffold.py` — held-back cases + parallel run plan (≤25 threads).
74