AgentAlley

Skills · Infrastructure & ops

Spark Training Gotchas

Unverified●29/40

Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add spark-training-gotchas

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.

The whole source

No sign-in, no blur, nothing truncated
spark-training-gotchas/SKILL.md201 lines7.8 KBRawView on GitHub
Frontmatter — 2 properties
namespark-training-gotchas
descriptionPreflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
1---
2name: spark-training-gotchas
3description: Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Spark Training Gotchas
7 
8DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified
9memory, aarch64) has ten recurring failure modes across
10launch, memory, thermals, bandwidth, and precision. Each is
11named G1–G10 so it can be checked by number — the numbering
12is load-bearing for tooling that runs these checks. Read this
13before a long run, not after hour six.
14 
15## When to Use This Skill
16 
17- A training run fails to start, with an import error or a
18 segfault that doesn't point at the real cause.
19- A run OOMs while `nvidia-smi` still shows headroom.
20- Throughput degrades partway through a run that started fine.
21- Before any multi-hour or multi-epoch job on GB10.
22- Wiring two Sparks together, before picking a parallelism
23 strategy.
24- Choosing between FP8 and NVFP4 for a Spark-hosted run.
25 
26## Common Issues Quick Reference
27 
28| # | Symptom | Fix |
29|---|---|---|
30| G1 | undefined symbol / segfault | cu130 wheel or container |
31| G2 | flash-attn wrong backend used | skip pip build; monkeypatch on NGC |
32| G3 | OOM despite headroom | drop page cache |
33| G4 | throughput drop / reboot | expect ~100W sustained cap |
34| G5 | memory-bound step slow | budget 180–192 GB/s |
35| G6 | cache evicted mid-run | one GPU server at a time |
36| G7 | NVFP4 slower than FP8 | stay FP8 unless `sm_121a` |
37| G8 | playbook fails outright | check upstream issues |
38| G9 | env breaks after install | use a container |
39| G10 | 2-Spark TP hangs | DDP/FSDP only, never TP |
40 
41## The Ten Gotchas
42 
43### G1: CUDA 12/13 ABI Mismatch
44 
45- **SYMPTOM:** `ImportError: undefined symbol` naming a CUDA
46 function, or a segfault on the first `.cuda()` call.
47- **CAUSE:** most PyPI wheels link `libcudart.so.12`; Spark
48 ships CUDA 13. pip never checks CUDA ABI, so it surfaces
49 only at import or first kernel launch.
50- **CHECK:** `references/gotcha-checks.md` G1 — the wheel's
51 CUDA build tag.
52- **FIX:** reinstall from `download.pytorch.org/whl/cu130` orA4 — This skill pulls in web or user content but never says to treat that content as data. A signal, not proof.
53 use a matched container.
54 
55### G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect
56 
57- **SYMPTOM:** `pip install flash-attn` still fails/hangs.
58 Unsloth may also silently train flash-attn over an
59 explicitly requested SDPA.
60- **CAUSE:** no aarch64/sm_121 wheel for bare pip — but NGC
61 containers ship a working SM121 flash-attn, and Unsloth
62 auto-prefers it, dropping `attn_implementation="sdpa"`.
63- **CHECK:** `references/gotcha-checks.md` G2 — is flash-attn
64 already present and working.
65- **FIX:** bare pip — skip flash-attn, use SDPA (unchanged). On
66 NGC — the only reliable override is the monkeypatch in
67 `references/gotcha-checks.md` G2.
68 
69### G3: UMA OOM Below 128GB
70 
71- **SYMPTOM:** OOM during model load/training while
72 `nvidia-smi` still reports free memory under the 128GB cap
73 — or, on some setups, `[N/A]` outright instead of a number.
74- **CAUSE:** mmap and the CUDA allocator double-count pages
75 during safetensors load; QLoRA can OOM *earlier* than bf16
76 since dequantization adds transient allocs.
77- **CHECK:** `references/gotcha-checks.md` G3 — read `free -g`
78 and `/proc/meminfo`, not `nvidia-smi`.
79- **FIX:** drop the page cache with
80 `sync; echo 3 > /proc/sys/vm/drop_caches` — needs root, a
81 between-run reset, not a mid-training step.
82 
83### G4: Thermal Throttling
84 
85- **SYMPTOM:** throughput drops partway through a multi-hour
86 run, or the box spontaneously reboots under sustained load.
87- **CAUSE:** sustained power draw caps around 100W versus the
88 240W rated figure; long runs push into that ceiling and
89 throttle or, sometimes, reboot.
90- **CHECK:** `references/gotcha-checks.md` G4 — sample
91 `nvidia-smi --query-gpu=temperature.gpu,power.draw`.
92- **FIX:** if power plateaus under 240W while temperature
93 climbs, treat throttling as the cause; improve cooling or
94 cap run length.
95 
96### G5: Bandwidth Ceiling
97 
98- **SYMPTOM:** memory-bound workloads, decode-heavy RL loops
99 especially, plateau well below expected throughput.
100- **CAUSE:** 273 GB/s is a spec ceiling, not sustained;
101 measured bandwidth runs 180–192 GB/s.
102- **CHECK:** `references/gotcha-checks.md` G5 — observed step
103 time vs. the measured range, not spec.
104- **FIX:** budget throughput from 180–192 GB/s; revise a plan
105 built on the 273 GB/s figure.
106 
107### G6: Global UMA Resource Contention
108 
109- **SYMPTOM:** a process's KV cache/weights get evicted
110 mid-run silently, no OOM in its own logs.
111- **CAUSE:** unified memory is
112 one global pool; an uncapped
113 or near-capacity process
114 competes with anything else
115 and can evict it. A small,
116 bounded workload doesn't — a
117 <4GB LoRA coexists fine
118 alongside vLLM capped at
119 `gpu-memory-utilization<=0.5`.
120- **CHECK:** `references/gotcha-checks.md`
121 G6 — other GPU-resident
122 processes and whether
123 capped.
124- **FIX:** the one-heavy-job
125 rule applies to **uncapped or
126 near-capacity** workloads —
127 cap or stop unrelated servers
128 first. A small, capped
129 workload need not
130 stop.
131 
132### G7: NVFP4 Slower Than FP8 on SM121
133 
134- **SYMPTOM:** switching an inference workload from FP8 to
135 NVFP4 on Spark makes it slower, not faster.
136- **CAUSE:** SM121 lacks `cvt.e2m1x2` unless kernels target
137 `sm_121a`; NVFP4 runs ~32% slower without it.
138- **CHECK:** `references/gotcha-checks.md` G7 — capability
139 reports `(12, 1)`; does the build target `sm_121a`?
140- **FIX:** stay on FP8 unless the build targets `sm_121a`.
141 
142### G8: Stale Official Playbooks
143 
144- **SYMPTOM:** following an official DGX Spark playbook still
145 fails, with no local misconfiguration explaining it.
146- **CAUSE:** official playbooks have shipped broken before;
147 the stack moves faster than the docs.
148- **CHECK:** `references/gotcha-checks.md` G8 — the playbook
149 repo's recent issues.
150- **FIX:** check `github.com/NVIDIA/dgx-spark-playbooks` issues
151 before trusting a recipe for an expensive run.
152 
153### G9: Container-First, Not Bare Pip
154 
155- **SYMPTOM:** a bare-pip environment that worked yesterday
156 breaks after an unrelated `pip install`, or two "identical"
157 environments behave differently.
158- **CAUSE:** bare pip lets Triton, xformers, and transformers
159 drift independently; nothing pins them to GB10's SM121
160 target.
161- **CHECK:** `references/gotcha-checks.md`
162 G9 — container or bare pip?
163- **FIX:** prefer an NGC container (see `spark-environment-setup`
164 for tag guidance) or Unsloth's container. If bare pip is
165 unavoidable, follow the NVIDIA install order, including
166 `--no-deps` on Unsloth.
167 
168### G10: Dual-Spark Is DDP/FSDP Only
169 
170- **SYMPTOM:** a tensor-parallel launch across two Sparks
171 hangs, runs far slower than single-Spark, or errors out.
172- **CAUSE:** ConnectX-7 is fast enough for gradient/parameter
173 sync (DDP, FSDP) but too thin for TP's fine-grained traffic.
174- **CHECK:** `references/gotcha-checks.md` G10 — the
175 configured parallelism strategy.
176- **FIX:** on a two-Spark setup, choose DDP or FSDP, never
177 tensor parallelism — TP is single-node only here.
178 
179## Fast Triage
180 
181The cheapest checks to run before anything else:
182 
183```bash
184python3 -c "import torch; print(torch.version.cuda)" # expect 13.x (G1); NGC builds have no +cu130 tag — that's not a failure
185```
186 
187```python
188import torch; print(torch.cuda.get_device_capability()) # expect (12, 1) (G7)
189```
190 
191```bash
192{ [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown # G9
193```
194 
195`assets/preflight.sh` runs G1, G3, G4, G7, G9 and produces one
196output line per gotcha in a fixed format: G-number first, then
197PASS/FAIL/WARN where automatable, SKIP when unavailable, or
198`INFO:` for a raw reading (G3, G4). Full commands:
199`references/gotcha-checks.md`. See also
200`spark-environment-setup` for the environment assumed working.
201 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Infrastructure & ops
Paper Poster (HTML): measurement-gated poster generationDEFAULT poster pipeline — build an academic conference poster (ICML/NeurIPS/ICLR/CVPR/...) as a single HTML/CSS file with measurement-driven hard gates, real paper figures, a two-hue design-token system, and print-ready PDF via headless Chromium. Use when theInfrastructure & ops · MIT●····●36/40Brand Monitoring 📡Brand monitoring tool for tracking mentions across social media platforms. Monitor Reddit, Google News, YouTube, and DuckDuckGo for brand mentions. Includes sentiment analysis, trend tracking, crisis detection, and competitor comparison. No API key required foInfrastructure & ops · MIT◐····●34/40Spark Memory & Thermal OpsManage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.Infrastructure & ops · MIT◐····●32/40Secrets ManagementImplement secure secrets management for CI/CD pipelines using Vault, AWS Secrets Manager, or native platform solutions. Use when handling sensitive credentials, rotating secrets, or securing CI/CD environments.Infrastructure & ops · MIT◐····●32/40