Planning product experiments, writing testable hypotheses

Experiment designer

by alirezarezvani · MIT license · ★ 26,349 on GitHub · updated Aug 30

Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.

---
name: experiment-designer
description: Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
---

# Experiment Designer

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

## When To Use

Use this skill for:
- A/B and multivariate experiment planning
- Hypothesis writing and success criteria definition
- Sample size and minimum detectable effect planning
- Experiment prioritization with ICE scoring
- Reading statistical output for product decisions

Overview

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

Manual install (npx, ZIP)

Claude Codenpx skills add alirezarezvani/claude-skills --skill experiment-designer -g -a claude-code

Claude Code (manual)

mkdir -p ~/.claude/skills && curl -sL https://codeload.github.com/alirezarezvani/claude-skills/tar.gz/19392f7a0826 \
  | tar -xz -C ~/.claude/skills --strip-components=3 claude-skills-19392f7a0826/product-team/skills/experiment-designer

Windows (PowerShell)

New-Item -ItemType Directory -Force "$HOME\.claude\skills" | Out-Null; Invoke-WebRequest "https://m.agentalley.io/zip/b9d185f974e4b1538876/experiment-designer.zip" -OutFile "$env:TEMP\experiment-designer.zip"; Expand-Archive "$env:TEMP\experiment-designer.zip" -DestinationPath "$HOME\.claude\skills" -Force

Claude.ai / desktopDownload the ZIP and upload it under Settings → Capabilities → Skills.

Codex, Cursor, other agentsPoint them at SKILL.md. The other files are plain text.

Connects with Agent Alley MCP · agentalley.io/mcp

Files included

SKILL.md3.1 KB
references/2 files · 3.4 KB
scripts/1 file · 3.0 KB

Read the files here

alirezarezvani/19392f74 files

SKILL.md3.1 KB

SKILL.md

Show the full text105 lines

experiment-designer-2/SKILL.md105 lines · 3.1 KB

PreviewCode

Outline

RawView on GitHub

Experiment Designer

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

When To Use

Use this skill for:

A/B and multivariate experiment planning
Hypothesis writing and success criteria definition
Sample size and minimum detectable effect planning
Experiment prioritization with ICE scoring
Reading statistical output for product decisions

Core Workflow

Write hypothesis in If/Then/Because format

If we change [intervention]
Then [metric] will change by [expected direction/magnitude]
Because [behavioral mechanism]

Define metrics before running test

Primary metric: single decision metric
Guardrail metrics: quality/risk protection
Secondary metrics: diagnostics only

Estimate sample size

Baseline conversion or baseline mean
Minimum detectable effect (MDE)
Significance level (alpha) and power

Use:

python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute

Prioritize experiments with ICE

Impact: potential upside
Confidence: evidence quality
Ease: cost/speed/complexity

ICE Score = (Impact * Confidence * Ease) / 10

Launch with stopping rules

Decide fixed sample size or fixed duration in advance
Avoid repeated peeking without proper method
Monitor guardrails continuously

Interpret results

Statistical significance is not business significance
Compare point estimate + confidence interval to decision threshold
Investigate novelty effects and segment heterogeneity

Hypothesis Quality Checklist

Contains explicit intervention and audience
Specifies measurable metric change
States plausible causal reason
Includes expected minimum effect
Defines failure condition

Common Experiment Pitfalls

Underpowered tests leading to false negatives
Running too many simultaneous changes without isolation
Changing targeting or implementation mid-test
Stopping early on random spikes
Ignoring sample ratio mismatch and instrumentation drift
Declaring success from p-value without effect-size context

Statistical Interpretation Guardrails

p-value < alpha indicates evidence against null, not guaranteed truth.
Confidence interval crossing zero/no-effect means uncertain directional claim.
Wide intervals imply low precision even when significant.
Use practical significance thresholds tied to business impact.

See:

references/experiment-playbook.md
references/statistics-reference.md

Tooling

`scripts/sample_size_calculator.py`

Computes required sample size (per variant and total) from:

baseline rate
MDE (absolute or relative)
significance level (alpha)
statistical power

Example:

python3 scripts/sample_size_calculator.py \
  --baseline-rate 0.10 \
  --mde 0.015 \
  --mde-type absolute \
  --alpha 0.05 \
  --power 0.8

1	---
2	name: experiment-designer
3	description: Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
4	---
5
6	# Experiment Designer
7
8	Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.
9
10	## When To Use
11
12	Use this skill for:
13	- A/B and multivariate experiment planning
14	- Hypothesis writing and success criteria definition
15	- Sample size and minimum detectable effect planning
16	- Experiment prioritization with ICE scoring
17	- Reading statistical output for product decisions
18
19	## Core Workflow
20
21	1. Write hypothesis in If/Then/Because format
22	- If we change `[intervention]`
23	- Then `[metric]` will change by `[expected direction/magnitude]`
24	- Because `[behavioral mechanism]`
25
26	2. Define metrics before running test
27	- Primary metric: single decision metric
28	- Guardrail metrics: quality/risk protection
29	- Secondary metrics: diagnostics only
30
31	3. Estimate sample size
32	- Baseline conversion or baseline mean
33	- Minimum detectable effect (MDE)
34	- Significance level (alpha) and power
35
36	Use:
37	```bash
38	python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute
39	```
40
41	4. Prioritize experiments with ICE
42	- Impact: potential upside
43	- Confidence: evidence quality
44	- Ease: cost/speed/complexity
45
46	ICE Score = (Impact * Confidence * Ease) / 10
47
48	5. Launch with stopping rules
49	- Decide fixed sample size or fixed duration in advance
50	- Avoid repeated peeking without proper method
51	- Monitor guardrails continuously
52
53	6. Interpret results
54	- Statistical significance is not business significance
55	- Compare point estimate + confidence interval to decision threshold
56	- Investigate novelty effects and segment heterogeneity
57
58	## Hypothesis Quality Checklist
59
60	- [ ] Contains explicit intervention and audience
61	- [ ] Specifies measurable metric change
62	- [ ] States plausible causal reason
63	- [ ] Includes expected minimum effect
64	- [ ] Defines failure condition
65
66	## Common Experiment Pitfalls
67
68	- Underpowered tests leading to false negatives
69	- Running too many simultaneous changes without isolation
70	- Changing targeting or implementation mid-test
71	- Stopping early on random spikes
72	- Ignoring sample ratio mismatch and instrumentation drift
73	- Declaring success from p-value without effect-size context
74
75	## Statistical Interpretation Guardrails
76
77	- p-value < alpha indicates evidence against null, not guaranteed truth.
78	- Confidence interval crossing zero/no-effect means uncertain directional claim.
79	- Wide intervals imply low precision even when significant.
80	- Use practical significance thresholds tied to business impact.
81
82	See:
83	- `references/experiment-playbook.md`
84	- `references/statistics-reference.md`
85
86	## Tooling
87
88	### `scripts/sample_size_calculator.py`
89
90	Computes required sample size (per variant and total) from:
91	- baseline rate
92	- MDE (absolute or relative)
93	- significance level (alpha)
94	- statistical power
95
96	Example:
97	```bash
98	python3 scripts/sample_size_calculator.py \
99	--baseline-rate 0.10 \
100	--mde 0.015 \
101	--mde-type absolute \
102	--alpha 0.05 \
103	--power 0.8
104	```
105