AgentAlley

Skills · Data & AI

Llm Evaluation

Unverified●31/40

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add llm-evaluation

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

The whole source

No sign-in, no blur, nothing truncated
llm-evaluation/SKILL.md146 lines3.8 KBRawView on GitHub
Frontmatter — 2 properties
namellm-evaluation
descriptionImplement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
1---
2name: llm-evaluation
3description: Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# LLM Evaluation
7 
8Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
9 
10## When to Use This Skill
11 
12- Measuring LLM application performance systematically
13- Comparing different models or prompts
14- Detecting performance regressions before deployment
15- Validating improvements from prompt changes
16- Building confidence in production systems
17- Establishing baselines and tracking progress over time
18- Debugging unexpected model behavior
19 
20## Core Evaluation Types
21 
22### 1. Automated Metrics
23 
24Fast, repeatable, scalable evaluation using computed scores.
25 
26**Text Generation:**
27 
28- **BLEU**: N-gram overlap (translation)
29- **ROUGE**: Recall-oriented (summarization)
30- **METEOR**: Semantic similarity
31- **BERTScore**: Embedding-based similarity
32- **Perplexity**: Language model confidence
33 
34**Classification:**
35 
36- **Accuracy**: Percentage correct
37- **Precision/Recall/F1**: Class-specific performance
38- **Confusion Matrix**: Error patterns
39- **AUC-ROC**: Ranking quality
40 
41**Retrieval (RAG):**
42 
43- **MRR**: Mean Reciprocal Rank
44- **NDCG**: Normalized Discounted Cumulative Gain
45- **Precision@K**: Relevant in top K
46- **Recall@K**: Coverage in top K
47 
48### 2. Human Evaluation
49 
50Manual assessment for quality aspects difficult to automate.
51 
52**Dimensions:**
53 
54- **Accuracy**: Factual correctness
55- **Coherence**: Logical flow
56- **Relevance**: Answers the question
57- **Fluency**: Natural language quality
58- **Safety**: No harmful content
59- **Helpfulness**: Useful to the user
60 
61### 3. LLM-as-Judge
62 
63Use stronger LLMs to evaluate weaker model outputs.
64 
65**Approaches:**
66 
67- **Pointwise**: Score individual responses
68- **Pairwise**: Compare two responses
69- **Reference-based**: Compare to gold standard
70- **Reference-free**: Judge without ground truth
71 
72## Quick Start
73 
74```python
75from dataclasses import dataclass
76from typing import Callable
77import numpy as np
78 
79@dataclass
80class Metric:
81 name: str
82 fn: Callable
83 
84 @staticmethod
85 def accuracy():
86 return Metric("accuracy", calculate_accuracy)
87 
88 @staticmethod
89 def bleu():
90 return Metric("bleu", calculate_bleu)
91 
92 @staticmethod
93 def bertscore():
94 return Metric("bertscore", calculate_bertscore)
95 
96 @staticmethod
97 def custom(name: str, fn: Callable):
98 return Metric(name, fn)
99 
100class EvaluationSuite:
101 def __init__(self, metrics: list[Metric]):
102 self.metrics = metrics
103 
104 async def evaluate(self, model, test_cases: list[dict]) -> dict:
105 results = {m.name: [] for m in self.metrics}
106 
107 for test in test_cases:
108 prediction = await model.predict(test["input"])
109 
110 for metric in self.metrics:
111 score = metric.fn(
112 prediction=prediction,
113 reference=test.get("expected"),
114 context=test.get("context")
115 )
116 results[metric.name].append(score)
117 
118 return {
119 "metrics": {k: np.mean(v) for k, v in results.items()},
120 "raw_scores": results
121 }
122 
123# Usage
124suite = EvaluationSuite([
125 Metric.accuracy(),
126 Metric.bleu(),
127 Metric.bertscore(),
128 Metric.custom("groundedness", check_groundedness)
129])
130 
131test_cases = [
132 {
133 "input": "What is the capital of France?",
134 "expected": "Paris",
135 "context": "France is a country in Europe. Paris is its capital."
136 },
137]
138 
139results = await suite.evaluate(model=your_model, test_cases=test_cases)
140```
141 
142## Detailed patterns and worked examples
143 
144Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
145 
146 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40