Spark Environment Setup
Unverified●30/40Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add spark-environment-setupWho is stuck, and on what
Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs.
The whole source
Frontmatter — 2 properties
| name | spark-environment-setup |
|---|---|
| description | Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs. |
| 1 | --- |
| 2 | name: spark-environment-setup |
| 3 | description: Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs. |
| 4 | ---A5 — No allowed-tools declared — no way to tell what this skill may touch |
| 5 | |
| 6 | # Spark Environment Setup |
| 7 | |
| 8 | DGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121 |
| 9 | GPU, 128GB unified memory, CUDA 13. This is a narrower and |
| 10 | younger platform than a standard x86 CUDA 12 box, so package |
| 11 | selection and ABI matching matter more than usual — the wheel |
| 12 | ecosystem for aarch64 + CUDA 13 is still filling in. |
| 13 | |
| 14 | ## When to Use This Skill |
| 15 | |
| 16 | - Setting up a fresh Spark box for training or inference. |
| 17 | - Hitting an import error mentioning `libcudart`, a missing |
| 18 | symbol, or a wheel that "installed fine but won't load." |
| 19 | - A framework install (PyTorch, Unsloth, TRL, vLLM, xformers) |
| 20 | fails, hangs, or silently falls back to CPU. |
| 21 | - Deciding whether to use an NGC container or bare pip. |
| 22 | - Restoring a working setup after an OS reinstall or a |
| 23 | base-image update, needing to re-verify from scratch. |
| 24 | |
| 25 | Each of these accepts the same general fix: match the |
| 26 | container/wheel combination to CUDA 13 and SM121, don't fight |
| 27 | the ABI. |
| 28 | |
| 29 | ## Container-First Rule |
| 30 | |
| 31 | Quick decision, before the detail below: |
| 32 | |
| 33 | - Standard training/inference work → NGC PyTorch container. |
| 34 | - Unsloth-centric fine-tuning → Unsloth container (it ships |
| 35 | the pinned Triton/xformers/transformers combination already |
| 36 | validated for that path). |
| 37 | - Neither fits (custom system package, local IDE interpreter) |
| 38 | → bare pip, following the exact sequence further down. |
| 39 | |
| 40 | Default to a container. Use `nvcr.io/nvidia/pytorch:25.09-py3` |
| 41 | as the base for general work — the newest tag confirmed working |
| 42 | on this hardware; pull a newer blessed tag if locally available |
| 43 | rather than hard-blocking on `25.11-py3`. NGC's tag is dated, so |
| 44 | running it directly is fine: |
| 45 | |
| 46 | ```bash |
| 47 | docker run --runtime=nvidia --gpus all -it --rm \ |
| 48 | nvcr.io/nvidia/pytorch:25.09-py3 |
| 49 | ``` |
| 50 | |
| 51 | `unsloth/unsloth:dgxspark-latest` is a *moving* tag by |
| 52 | contrast — resolve and pin its digest before running it for |
| 53 | anything reproducible; the bare tag is a discovery step only, |
| 54 | not the default invocation. Full pull-inspect-pin sequence and |
| 55 | flag rationale/volume mounts for `finetuning/` run dirs: |
| 56 | `references/container-workflow.md`. Treat bare pip as the exception. |
| 57 | |
| 58 | The reason for the container-first stance is pinning, not |
| 59 | convenience. Triton, xformers, and transformers versions |
| 60 | interact narrowly with GB10's SM121 target and CUDA 13; a |
| 61 | container locks all of them together against a combination |
| 62 | already validated on this hardware. Bare pip leaves that |
| 63 | resolution to you, one broken import at a time. |
| 64 | |
| 65 | When bare pip is warranted, follow the NVIDIA playbook's |
| 66 | install sequence verbatim and in order: |
| 67 | |
| 68 | ```bash |
| 69 | pip install "transformers==5.13.1" "peft==0.19.1" "hf_transfer==0.1.9" "datasets==4.3.0" "trl==1.8.0" |
| 70 | pip install --no-deps "unsloth==2026.7.2" "unsloth_zoo==2026.7.2" "bitsandbytes==0.49.2" |
| 71 | pip install -U "torchao==0.17.0" |
| 72 | ``` |
| 73 | |
| 74 | The second command's `--no-deps` flag is not optional — |
| 75 | letting pip re-resolve Unsloth's dependency tree on aarch64 is |
| 76 | a common way to pull in an incompatible torch or triton build. |
| 77 | The third line is not optional either: the NGC base image's |
| 78 | bundled `torchao` is too old for current `peft`'s LoRA-attach |
| 79 | path (`ImportError: ... torchao ... only versions above 0.16.0 |
| 80 | are supported`) — a hard blocker, not a warning. Every `==` pin |
| 81 | above is load-bearing, taken from the dated known-good version |
| 82 | matrix in `references/stack-matrix.md` (its `Last verified` date |
| 83 | governs staleness) — an unpinned install resolves current PyPI |
| 84 | versions well outside what this Unsloth release supports. |
| 85 | |
| 86 | Pull a fresh tag when a new blessed release is announced. |
| 87 | Rebuild locally from one of the two bases only when a project |
| 88 | needs an extra system package layered in — not to "upgrade" a |
| 89 | component the image already pins. Details on both paths: |
| 90 | `references/container-workflow.md`. |
| 91 | |
| 92 | One more preflight: official DGX Spark playbooks have shipped |
| 93 | broken before. Check recent issues on |
| 94 | `github.com/NVIDIA/dgx-spark-playbooks` (and the other |
| 95 | resources in `references/stack-matrix.md`) before trusting a |
| 96 | recipe verbatim for a long run. |
| 97 | |
| 98 | ## The ABI Rule |
| 99 | |
| 100 | The single most common failure on Spark is a CUDA 12/13 ABI |
| 101 | mismatch: a wheel built against `libcudart.so.12` loaded on a |
| 102 | system that only has `libcudart.so.13`. The install usually |
| 103 | succeeds; the failure surfaces later as a missing-symbol error |
| 104 | or a segfault that doesn't obviously point at CUDA. |
| 105 | |
| 106 | Fix: pull wheels from `download.pytorch.org/whl/cu130` (theA4 — This skill pulls in web or user content but never says to treat that content as data. A signal, not proof. |
| 107 | cu130-tagged aarch64 builds), or use one of the containers |
| 108 | above, which already carry a matched build. Before chasing a |
| 109 | stack trace that mentions a CUDA symbol, check which CUDA tag |
| 110 | the installed wheel was built against: |
| 111 | |
| 112 | ```bash |
| 113 | python3 -c "import torch; print(torch.version.cuda)" |
| 114 | ``` |
| 115 | |
| 116 | If that output doesn't start with `13`, the ABI mismatch is the |
| 117 | first thing to fix. NGC container builds (e.g. |
| 118 | `nvcr.io/nvidia/pytorch:25.09-py3`) build torch internally |
| 119 | against CUDA 13 with no `+cu130` wheel tag — `pip show torch` |
| 120 | won't say `cu130` there, and that absence alone is not a failure. |
| 121 | |
| 122 | Typical symptoms: |
| 123 | |
| 124 | - `ImportError: undefined symbol` referencing a CUDA runtime |
| 125 | function. |
| 126 | - A segfault on the first `.cuda()` call, no useful traceback. |
| 127 | - A wheel that installs cleanly, then fails at import time — |
| 128 | pip's resolver doesn't check CUDA ABI, only version constraints. |
| 129 | - Two "identical" environments behaving differently — usually one |
| 130 | has a cu130 wheel, the other a cu121/cu124 leftover. |
| 131 | |
| 132 | The fix is the same regardless of symptom: match the wheel's |
| 133 | CUDA tag to the system, or use a container that already does. |
| 134 | |
| 135 | ## Component Quick Table |
| 136 | |
| 137 | Condensed status for the components most likely to come up. |
| 138 | Full table with wheel URLs, build flags, the sm_121 vs sm_121a |
| 139 | distinction, and the dated known-good version matrix: |
| 140 | `references/stack-matrix.md`. |
| 141 | |
| 142 | | Component | Status | |
| 143 | |---|---| |
| 144 | | PyTorch | ✅ official cu130 aarch64 wheels | |
| 145 | | bitsandbytes | ✅ works out of the box | |
| 146 | | Triton | ✅ needs the `TRITON_PTXAS_PATH` parameter set | |
| 147 | | flash-attn | ❌ skip pip build; NGC bundles a working one — see `spark-training-gotchas` G2 | |
| 148 | | xformers | source build only (`TORCH_CUDA_ARCH_LIST=12.1`) | |
| 149 | | vLLM | nightly wheels only | |
| 150 | | TransformerEngine / NVFP4 train | container-only | |
| 151 | |
| 152 | Everything else — Unsloth, Axolotl, TRL, PEFT — installs |
| 153 | cleanly through the container-first path above. LLaMA-Factory |
| 154 | and NeMo are fragile on Spark; check upstream issues first. |
| 155 | |
| 156 | ## Verification Commands |
| 157 | |
| 158 | Confirm the environment can actually see the GPU before |
| 159 | running anything expensive: |
| 160 | |
| 161 | ```python |
| 162 | import torch |
| 163 | print(torch.cuda.is_available(), torch.version.cuda) |
| 164 | ``` |
| 165 | |
| 166 | This call returns two values; the exact output format is one |
| 167 | line, `<bool> <cuda-version>`: |
| 168 | |
| 169 | ```text |
| 170 | True 13.0 |
| 171 | ``` |
| 172 | |
| 173 | If it prints `False` instead, don't jump straight to a wheel |
| 174 | reinstall — ABI mismatch is one cause among several: |
| 175 | |
| 176 | | Hypothesis | Quick check | |
| 177 | |---|---| |
| 178 | | Runtime/flags | `nvidia-smi` fails in-container too | |
| 179 | | Device visibility | `echo $CUDA_VISIBLE_DEVICES` | |
| 180 | | Permissions | `ls -l /dev/nvidia*` | |
| 181 | | CUDA init state | wedged process; retry fresh shell/container | |
| 182 | | ABI mismatch (usual culprit) | `torch.version.cuda` not `13.x` | |
| 183 | |
| 184 | Check `nvidia-smi` first — if it doesn't show the GPU, it's one |
| 185 | of the first three, not ABI. Reinstall a wheel only once ABI is |
| 186 | confirmed. Per-hypothesis detail: `references/stack-matrix.md`. |
| 187 | Run right after the container starts, before installing |
| 188 | project-specific packages. |
| 189 | |
| 190 | One more check: if Triton kernel compilation fails once |
| 191 | training starts, set |
| 192 | `TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas` and retry — see |
| 193 | `references/stack-matrix.md` for the full workaround list. |
| 194 | |
| 195 | ## Next Steps |
| 196 | |
| 197 | A verified environment is only the starting point. See also: |
| 198 | `spark-training-gotchas` for failure preflights before a |
| 199 | training run, and `spark-memory-thermal-ops` for unified-memory |
| 200 | OOMs and thermal throttling during long ones. |
| 201 |
Reviews
Installed this one?Write the first review and take the Trailblazer badge.
Alternatives
Paper Poster (HTML): measurement-gated poster generationDEFAULT poster pipeline — build an academic conference poster (ICML/NeurIPS/ICLR/CVPR/...) as a single HTML/CSS file with measurement-driven hard gates, real paper figures, a two-hue design-token system, and print-ready PDF via headless Chromium. Use when the●····●36/40Brand Monitoring 📡Brand monitoring tool for tracking mentions across social media platforms. Monitor Reddit, Google News, YouTube, and DuckDuckGo for brand mentions. Includes sentiment analysis, trend tracking, crisis detection, and competitor comparison. No API key required fo◐····●34/40Spark Memory & Thermal OpsManage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.◐····●32/40Secrets ManagementImplement secure secrets management for CI/CD pipelines using Vault, AWS Secrets Manager, or native platform solutions. Use when handling sensitive credentials, rotating secrets, or securing CI/CD environments.◐····●32/40