/ar:setup — Create New Experiment
Set up a new autoresearch experiment interactively.
How to use it
Claude Code
- Run the line below. It pulls the whole folder into
~/.claude/skills/setup, including the files SKILL.md points to. - Describe your job in plain words. Claude Code follows the skill from there.
Claude Code — installs the whole folder, not just SKILL.md
npx degit alirezarezvani/claude-skills/engineering/autoresearch-agent/skills/setup#main ~/.claude/skills/setupFor one project only, change the path to .claude/skills/setup. This skill also uses bench.py, evaluate.py, setup_experiment.py — copying SKILL.md alone won't be enough. See the folder on GitHub.
Claude (web or desktop app)
- On this page open ⋯ → Download .md.
- Save it as SKILL.md in a folder, zip the folder, then Customize → Skills → + → Create skill → Upload a skill.
- Pick the file and Save. Claude shows the name and description and runs a security scan.
- Check the skill is switched on.
- Start a new chat and describe your job in plain words. The AI follows the skill from there.
ChatGPT or another app
- ChatGPT: make a Project and paste it into Instructions.
- Neither? Paste it at the top of a new chat — it works for that chat.
Not working?
- Check which app you pasted it into — the steps above name the right one.
- Some skills need the paid tier of Claude or ChatGPT.
Paste into Claude, ChatGPT or Cursor.
Source of /ar:setup — Create New Experiment
Show the full text78 lines
| name | description | command |
|---|---|---|
| setup | Set up a new autoresearch experiment interactively. Collects domain, target file, eval command, metric, direction, and evaluator. Use when the user runs /ar:setup or asks to start optimizing a file with the autoresearch loop. | /ar:setup |
/ar:setup — Create New Experiment
Set up a new autoresearch experiment with all required configuration.
Usage
/ar:setup # Interactive mode
/ar:setup engineering api-speed src/api.py "pytest bench.py" p50_ms lower
/ar:setup --list # Show existing experiments
/ar:setup --list-evaluators # Show available evaluators
What It Does
If arguments provided
Pass them directly to the setup script:
python {skill_path}/scripts/setup_experiment.py \
--domain {domain} --name {name} \
--target {target} --eval "{eval_cmd}" \
--metric {metric} --direction {direction} \
[--evaluator {evaluator}] [--scope {scope}]
If no arguments (interactive mode)
Collect each parameter one at a time:
- Domain — Ask: "What domain? (engineering, marketing, content, prompts, custom)"
- Name — Ask: "Experiment name? (e.g., api-speed, blog-titles)"
- Target file — Ask: "Which file to optimize?" Verify it exists.
- Eval command — Ask: "How to measure it? (e.g., pytest bench.py, python evaluate.py)"
- Metric — Ask: "What metric does the eval output? (e.g., p50_ms, ctr_score)"
- Direction — Ask: "Is lower or higher better?"
- Evaluator (optional) — Show built-in evaluators. Ask: "Use a built-in evaluator, or your own?"
- Scope — Ask: "Store in project (.autoresearch/) or user (~/.autoresearch/)?"
Then run setup_experiment.py with the collected parameters.
Listing
# Show existing experiments
python {skill_path}/scripts/setup_experiment.py --list
# Show available evaluators
python {skill_path}/scripts/setup_experiment.py --list-evaluators
Built-in Evaluators
| Name | Metric | Use Case |
|---|---|---|
benchmark_speed |
p50_ms (lower) |
Function/API execution time |
benchmark_size |
size_bytes (lower) |
File, bundle, Docker image size |
test_pass_rate |
pass_rate (higher) |
Test suite pass percentage |
build_speed |
build_seconds (lower) |
Build/compile/Docker build time |
memory_usage |
peak_mb (lower) |
Peak memory during execution |
llm_judge_content |
ctr_score (higher) |
Headlines, titles, descriptions |
llm_judge_prompt |
quality_score (higher) |
System prompts, agent instructions |
llm_judge_copy |
engagement_score (higher) |
Social posts, ad copy, emails |
After Setup
Report to the user:
- Experiment path and branch name
- Whether the eval command worked and the baseline metric
- Suggest: "Run
/ar:run {domain}/{name}to start iterating, or/ar:loop {domain}/{name}for autonomous mode."
| 1 | |
| 2 | name "setup" |
| 3 | description "Set up a new autoresearch experiment interactively. Collects domain, target file, eval command, metric, direction, and evaluator. Use when the user runs /ar:setup or asks to start optimizing a file with the autoresearch loop." |
| 4 | command /ar:setup |
| 5 | |
| 6 | |
| 7 | # /ar:setup — Create New Experiment |
| 8 | |
| 9 | Set up a new autoresearch experiment with all required configuration. |
| 10 | |
| 11 | ## Usage |
| 12 | |
| 13 | |
| 14 | /ar:setup # Interactive mode |
| 15 | /ar:setup engineering api-speed src/api.py "pytest bench.py" p50_ms lower |
| 16 | /ar:setup --list # Show existing experiments |
| 17 | /ar:setup --list-evaluators # Show available evaluators |
| 18 | |
| 19 | |
| 20 | ## What It Does |
| 21 | |
| 22 | ### If arguments provided |
| 23 | |
| 24 | Pass them directly to the setup script: |
| 25 | |
| 26 | |
| 27 | python {skill_path}/scripts/setup_experiment.py \ |
| 28 | --domain {domain} --name {name} \ |
| 29 | --target {target} --eval "{eval_cmd}" \ |
| 30 | --metric {metric} --direction {direction} \ |
| 31 | [--evaluator {evaluator}] [--scope {scope}] |
| 32 | |
| 33 | |
| 34 | ### If no arguments (interactive mode) |
| 35 | |
| 36 | Collect each parameter one at a time: |
| 37 | |
| 38 | **Domain** — Ask: "What domain? (engineering, marketing, content, prompts, custom)" |
| 39 | **Name** — Ask: "Experiment name? (e.g., api-speed, blog-titles)" |
| 40 | **Target file** — Ask: "Which file to optimize?" Verify it exists. |
| 41 | **Eval command** — Ask: "How to measure it? (e.g., pytest bench.py, python evaluate.py)" |
| 42 | **Metric** — Ask: "What metric does the eval output? (e.g., p50_ms, ctr_score)" |
| 43 | **Direction** — Ask: "Is lower or higher better?" |
| 44 | **Evaluator** (optional) — Show built-in evaluators. Ask: "Use a built-in evaluator, or your own?" |
| 45 | **Scope** — Ask: "Store in project (.autoresearch/) or user (~/.autoresearch/)?" |
| 46 | |
| 47 | Then run `setup_experiment.py` with the collected parameters. |
| 48 | |
| 49 | ### Listing |
| 50 | |
| 51 | |
| 52 | # Show existing experiments |
| 53 | python {skill_path}/scripts/setup_experiment.py --list |
| 54 | |
| 55 | # Show available evaluators |
| 56 | python {skill_path}/scripts/setup_experiment.py --list-evaluators |
| 57 | |
| 58 | |
| 59 | ## Built-in Evaluators |
| 60 | |
| 61 | | Name | Metric | Use Case | |
| 62 | |------|--------|----------| |
| 63 | | `benchmark_speed` | `p50_ms` (lower) | Function/API execution time | |
| 64 | | `benchmark_size` | `size_bytes` (lower) | File, bundle, Docker image size | |
| 65 | | `test_pass_rate` | `pass_rate` (higher) | Test suite pass percentage | |
| 66 | | `build_speed` | `build_seconds` (lower) | Build/compile/Docker build time | |
| 67 | | `memory_usage` | `peak_mb` (lower) | Peak memory during execution | |
| 68 | | `llm_judge_content` | `ctr_score` (higher) | Headlines, titles, descriptions | |
| 69 | | `llm_judge_prompt` | `quality_score` (higher) | System prompts, agent instructions | |
| 70 | | `llm_judge_copy` | `engagement_score` (higher) | Social posts, ad copy, emails | |
| 71 | |
| 72 | ## After Setup |
| 73 | |
| 74 | Report to the user: |
| 75 | Experiment path and branch name |
| 76 | Whether the eval command worked and the baseline metric |
| 77 | Suggest: "Run `/ar:run {domain}/{name}` to start iterating, or `/ar:loop {domain}/{name}` for autonomous mode." |
| 78 |
Discussion
Browse more free Claude skills.