Structure a product data analysis, metric deep-dive

Data Analysis Standard Skill

Turn raw numbers into product decisions. Structure every analysis with a clear question, methodology, finding, and recommended action.

Analysis Framework: The 4-Question Method

Every analysis starts here:

1. What changed? (describe the metric and its movement)
2. Why did it change? (root cause — segment, funnel step, cohort, channel)
3. So what? (business or product impact)
4. Now what? (recommended action with confidence level)

Never deliver data without answering all four. A chart with no narrative is not an analysis.

---

Metric Triage Template

Use when a metric has moved unexpectedly:

METRIC: [Name]
MOVEMENT: [X% change over Y period]
BASELINE: [What was normal]

SEGMENTATION CHECK:
- By platform (iOS / Android / Web)?
- By user cohort (new / returning / power users)?
- By acquisition channel?
- By geography?
- By plan/tier?

ROOT CAUSE HYPOTHESIS:
1. [Most likely explanation] — Evidence: [data point]
2. [Alternative explanation] — Evidence: [data point]
3. [Ruling out] — Eliminated because: [reason]

CONCLUSION: [Single sentence answer to "why did this change?"]
CONFIDENCE: [High / Medium / Low] — based on [data available]

---

Funnel Analysis Structure

Stage	Metric	Current	Benchmark/Target	Drop-off %	Notes
[Top of funnel]	[Users]	[N]	[N]	—
[Step 2]	[Users]	[N]	[N]	[X%]
[Step 3]	[Users]	[N]	[N]	[X%]
[Conversion]	[Users]	[N]	[N]	[X%]

Biggest drop-off: [Step X → Step Y] — Hypothesis: [reason] Recommended investigation: [specific query or test]

---

Cohort Analysis Guidelines

Always define:

• Cohort definition: [What groups users — signup week, first action, plan type]
• Retention metric: [What counts as retained — login, core action, revenue]
• Retention window: [D1, D7, D30, W4, M3, etc.]

Output a cohort retention table and annotate:

• Baseline retention for each cohort
• Cohorts that over/underperform and why (feature launch? campaign? seasonal?)
• Trend direction across cohorts (improving / declining / stable)

---

Stakeholder Analysis Output Format

[Analysis Title] — [Date]

Question being answered: [Specific question in plain English] Time period: [Date range] Data source: [Where data comes from]

Finding:

[1–2 sentence plain-English summary of what the data shows]

Key chart / table: [Include or describe]

Root cause: [Best explanation with evidence]

Confidence level: [High / Medium / Low] — [reason]

Recommended action:

1. [Immediate action — owner, timeline]
2. [Investigation needed — what to check next]
3. [Monitoring — what metric to watch and at what cadence]

What this analysis does NOT tell us: [Important caveat — what data is missing or what can't be concluded]

---

Required Inputs

Ask the user for these if not provided:

• Metric or question being investigated
• Time period (what changed, from when to when)
• Data available (which segments, sources, or queries you have access to)
• Business context (what decision this analysis informs)
• Audience (who will read this — exec / team / data team)

Deeper Materials

This skill ships with support files — use them when they are available:

• references/analysis-integrity.md — Analysis Integrity: the Checks Between Query and Conclusion. Apply it while producing the output; it carries the calibration and judgment calls the method summary above compresses.
• templates/analysis-writeup.md — a fill-in version of the deliverable with the quality gates inline. Offer it when the user wants to work the document themselves rather than have it generated.

Scoring Rubric (0–40)

Score any output of this skill before handing it over; 32+ is ship-quality.

Dimension	0	5	10
Four-question completeness	Describes what changed and stops	Covers what/why but "so what / now what" are thin	All four answered with proportionate depth; the "now what" is decision-ready
Evidence behind the root cause	Root cause asserted from intuition	One supporting data point, alternatives unexamined	Root cause tested against at least one rival explanation, with the discriminating evidence shown
Uncertainty honesty	Reads as certain; no confidence statement	Confidence stated but not justified	Confidence level justified, and "what the data cannot tell us" names the real blind spots, not token ones
Actionability	Findings with no action	Action named but ownerless or dateless	Recommended action has an owner, a timeline, and a stated expected effect worth checking later

Quality Checks

• Analysis answers all 4 questions: what changed, why, so what, now what
• Root cause has evidence (not just hypothesis)
• Confidence level is stated and justified
• What the data cannot tell us is explicitly named
• Recommended action includes an owner and timeline

Anti-Patterns

• Do not present correlations as causation — always state the distinction explicitly
• Do not report a metric movement without stating the time window and comparison baseline
• Do not skip the "so what" — raw observations without recommended actions are incomplete analysis
• Do not overstate confidence — label hypotheses clearly and note what data would be needed to confirm them
• Do not ignore segment breakdowns — aggregate metrics can mask opposing trends in sub-segments

Guidelines

• Always state what the data cannot tell you — never oversell confidence
• Correlations are not causation — flag this every time
• If the user has no baseline, recommend establishing one before drawing conclusions
• Recommend the simplest chart for each finding: bar for comparison, line for trends, scatter for correlation, table for detailed breakdowns
• Always specify the time window — "conversion dropped" is meaningless without "from X to Y over Z period"