AgentAlley

Skills · Data & AI

Dataset Curation

Unverified●31/40

Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor◐PartialPlain prose you can paste in — but no Cursor rules file
Codex◐PartialPlain prose you can paste in — but no AGENTS.md
Gemini CLI◐PartialPlain prose you can paste in
Copilot◐PartialPlain prose you can paste in — but no Copilot instructions file
npx agentalley add dataset-curation

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.

The whole source

No sign-in, no blur, nothing truncated
dataset-curation/SKILL.md204 lines7.8 KBRawView on GitHub
Frontmatter — 2 properties
namedataset-curation
descriptionPrepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
1---
2name: dataset-curation
3description: Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Dataset Curation
7 
8This skill assumes `finetuning-method-selection`
9already routed here — the next step is preparing
10data, not choosing a method. What follows: format
11selection by target method, the template/packing
12mechanics behind the most common silent training
13failures, rules for mixing in synthetic data
14without collapse, and the dataset card that closes
15out Phase 2 before a run starts.
16 
17**Input:** raw examples (demonstrations, preference
18judgments, or task prompts) plus a routing decision
19from `finetuning-method-selection`.
20**Output format:** a formatted, packed, validated
21JSONL dataset plus a completed dataset card — the
22Phase 2 artifact `/finetune` checks before launching
23training.
24 
25## Format Selection
26 
27| Method | Shape | Rows |
28|---|---|---|
29| SFT, single-turn | Instruct (`instruction`/`response` or `prompt`/`completion`) | ~1,000+ floor |
30| SFT, multi-turn | Conversation / ChatML `messages` list | ~1,000+ floor |
31| DPO / ORPO | Preference pair (`prompt`, `chosen`, `rejected`) | Method-dependent, see `preference-optimization` |
32| KTO | Unpaired (`prompt`, `completion`, `label`) | Method-dependent, see `preference-optimization` |
33| GRPO / RLVR | Prompt-only (`prompt` + verifier metadata) | Method-dependent, see `grpo-rlvr-training` |
34 
35- **~1,000+ rows is the recommended floor for SFT**,
36 not a target. Below it, a handful of low-quality
37 or duplicate examples can dominate the gradient;
38 above it, **quality over quantity** — a smaller
39 verified, deduplicated set beats a larger noisy one.
40- The ChatML shape, for orientation; the other four
41 formats plus a ShareGPT conversion note live in
42 `references/formats-and-templates.md`:
43 
44 ```json
45 {"messages": [
46 {"role": "user", "content": "..."},
47 {"role": "assistant", "content": "..."}
48 ]}
49 ```
50 
51## Chat Templates and Loss Masking
52 
53Apply the target model's chat template **before**
54any concatenation or packing, never after — packing
55raw text and templating the packed blob afterward
56corrupts turn boundaries, landing role markers in
57the wrong place relative to each example.
58 
59- **Train on assistant responses only.** Mask the
60 loss (`-100` in the labels tensor) over system/user
61 turns and the template's own role markers — only
62 assistant-turn content tokens contribute to loss.
63- **Template/tokenizer mismatches are a top silent
64 failure mode.** A model trained against one chat
65 template but served or evaluated with a different
66 one degrades without erroring. Verify the same
67 template string used in training is applied at
68 inference and eval time.
69- **Keep the dataset in `messages` shape** and let
70 the trainer template and mask it
71 (`assistant_only_loss=True` in current TRL) —
72 pre-rendering to a flat text field destroys the
73 turn boundaries masking needs. Full code sketch:
74 `references/formats-and-templates.md`. Sanity-check
75 before training — decode only unmasked positions;
76 expect only assistant text:
77 
78 ```python
79 keep = batch["labels"][0] != -100
80 print(tokenizer.decode(batch["input_ids"][0][keep]))
81 ```
82 
83## Packing
84 
85**Without packing, 40–70% of compute is spent on
86padding** — variable-length examples batched at a
87fixed sequence length waste the gap between each
88example's length and the batch's max. Packing
89concatenates multiple examples into one sequence
90up to the max length, cutting most of that waste.
91 
92- **Packing changes batch semantics.** A packed
93 sequence can contain several original examples, so
94 "steps per epoch" and any LR schedule keyed to
95 example count shift once packing is on — recompute
96 schedule milestones against packed-sequence count.
97- **MANDATORY: decode and manually inspect 5–10
98 packed sequences before scaling to a full run.**
99 Confirm example boundaries land where expected,
100 template markers are intact per sub-example, and
101 the loss mask is still assistant-only within each
102 packed sequence. Not optional — packing bugs are
103 silent (the loss curve looks normal) and only
104 surface in eval quality, hours later:
105 
106 ```python
107 for seq in packed_dataset.select(range(10)):
108 print(tokenizer.decode(seq["input_ids"]))
109 ```
110 
111## Synthetic Data Rules
112 
113- **Keep ≥25% real data as a collapse guard.**
114 Training on a growing share of model-generated
115 data without a real-data floor drives measurable
116 quality collapse over successive generations —
117 25% real is the minimum that holds the line.
118 **General-domain replay rows
119 count toward this floor** —
120 "real" means "not generated
121 for this task from this
122 student," not "human-authored."
123 An all-synthetic-by-construction
124 dataset can meet the ≥25% floor
125 through replay alone (see
126 `references/synthetic-data.md`'s
127 Replay-Mix Construction recipe);
128 state which rows count as "real"
129 in the dataset card rather than
130 leaving the floor structurally
131 unmeetable.
132- **Magpie and rejection sampling are the
133 workhorses.** Magpie extracts prompts from the
134 model's own template prior; rejection sampling
135 generates several candidates per prompt and keeps
136 only the ones a filter passes. Both beat naive
137 single-shot generation.
138- **Targeted, student-aware generation beats static
139 generation by 1.3–2x sample efficiency** — aiming
140 at the student's actual failure modes hits a
141 quality bar with fewer filtered examples.
142- **Typical accept rates after filtering run
143 10–30%.** Plan volume accordingly — a 10,000-row
144 target at 15% accept needs ~65,000+ raw generations.
145- Generation-method ranking, filter funnel, replay-
146 mix construction, and distillation pattern:
147 `references/synthetic-data.md`.
148 
149## The Dataset Card
150 
151Every dataset that reaches training gets a card —
152the required Phase 2 artifact `/finetune` checks
153before launching. The card is not free-form
154documentation; it MUST carry these fields:
155 
156- **Provenance** — where every row came from (real
157 source(s), synthetic method(s), or both),
158 traceable to `trace-to-training-data` output.
159- **Counts** — total rows, and rows per split
160 (train/eval/held-out) if split.
161- **Synthetic/real ratio** — the measured ratio,
162 checked against the ≥25% real floor above.
163- **Dedup method** — exact-match, semantic
164 (embedding threshold), or both; see the filter
165 funnel in `references/synthetic-data.md`.
166- **Template used** — the exact chat template
167 string/identifier, kept consistent through
168 inference and eval — this is what ties an
169 `eval-harness-first` run back to the checkpoint.
170- **Packing config** — whether packing was used,
171 max sequence length, and confirmation the
172 5–10-sequence manual inspection above was done.
173 
174A dataset missing any of these six fields isn't
175ready for `/finetune` — the card is a gate, not a
176summary written after the fact.
177 
178### Phase 2 Exit Checklist
179 
180Before handing off to `/finetune`, confirm:
181 
1821. Format matches the method (table above).
1832. Template applied before concatenation.
1843. Loss masked to assistant turns only.
1854. 5–10 packed sequences decoded and read.
1865. ≥25% real data in the final mix.
1876. Dataset card complete — all six fields.
188 
189## References
190 
191- `references/formats-and-templates.md` — JSONL
192 examples per format, current-TRL masking code,
193 and the ShareGPT conversion note.
194- `references/synthetic-data.md` — generation-method
195 ranking, filter funnel, replay-mix construction,
196 and teacher→student distillation pattern.
197 
198Related skills: `finetuning-method-selection` routes
199here; `lora-qlora-recipes`, `vision-sft`, and
200`preference-optimization` consume the datasets this
201skill produces; `trace-to-training-data` is the
202provenance source for graded-trajectory datasets;
203`eval-harness-first` grades the resulting checkpoint.
204 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40