Inference Optimisation
Inference optimisation (hill-climbing a model server)

Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s,…
--- name: inference-optimisation description: > Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap and with full-context stability. Computes the theoretical ceiling first, runs noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), to find the maximum possible tok/s on a machine, or to publish a tuned recipe. Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, speculative decoding, KV cache, TTFT, startup time, theoretical maximum. --- # Inference optimisation (hill-climbing a model server)
Overview
You improve how fast a model server runs on one machine, with evidence. The loop is:
Manual install (npx, ZIP)
npx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -g -a claude-codenpx degit sybil-solutions/inference-optimisation-skill/skills/inference-optimisation#main ~/.claude/skills/inference-optimisationConnects with Agent Alley MCP · agentalley.io/mcp
Files included
SKILL.md9.8 KB
Read SKILL.md here
Show the full text167 lines
Inference optimisation (hill-climbing a model server)
You improve how fast a model server runs on one machine, with evidence. The loop is:
- Measure the baseline.
- Calculate the ceiling.
- Change one thing.
- Measure it against controls.
- Keep it or reject it, and write down why.
- Repeat until the remaining gap needs work out of scope, then qualify and publish.
Detailed references, read when needed:
references/ceilings.md: how to compute theoretical maxima. It includes a worked example.references/methodology.md: measurement pitfalls and statistics.references/levers.md: levers by area, ordered by typical payoff, with measured evidence.references/gotchas.md: engine- and hardware-specific traps.references/publishing.md: qualification gates, pinning, poster and report.scripts/: an engine-agnostic harness. It has a bench, ceiling calculator, memory watchdog, serialized queue and ledger.
0. Before touching anything
- Write down the targets and constraints the user gave. Ask only if one is missing and it changes what you do.
- Metrics that matter: c1 decode, cN decode, prefill at several lengths, TTFT, startup.
- Memory cap.
- Context length that must stay stable.
- Which quality changes are allowed: lossless only, or labelled lossy trades such as int8 prefill.
- Inventory the machine:
- accelerator, memory type and size, and whether memory is unified;
- driver, engine version, model, quantization, current launch command.
- Measure, don't trust spec sheets:
- real memory bandwidth (device copy);
- real matmul throughput (large GEMM);
- clocks under load.
- Read any existing experiment log for this machine and model. Never rerun a rejected experiment without a new reason.
- Create the working directory:
PLAN.md: a timestamped log of every experiment, with numbers, verdict and reason.THEORY.md: the ceilings.ledger.jsonl: one line per run, from the harness.configs/: one JSON per run.best-config.json.
1. Ceilings first
Compute the theoretical maximum for each metric before optimising (scripts/ceiling.py, references/ceilings.md):
- Decode is memory-bandwidth bound. Count the bytes read per emitted token:
- dense weights;
- active experts: top-k ÷ experts × expert bytes;
- LM head;
- KV read at the working context.
- Divide by measured bandwidth.
- Speculative decoding divides the target's bytes by the accepted length. It also adds the draft's bytes, and the draft's LM head is often the biggest part.
- Prefill is compute bound. FLOPs per token ≈ 2 × active params, plus attention at about 4 × layers × hidden × position. Divide by ~50% of measured peak for a practical ceiling.
- TTFT floor: one forward pass plus tokenize and schedule.
- Startup floor: weight read from disk or page cache plus allocation, assuming JIT and graph caches persist.
- KV: bytes per token per layer × the layers that actually have KV (hybrid models have few). That gives how many full-length sequences fit under the memory cap.
Report progress as a percentage of the practical ceiling. If the user's target is above the ceiling, say so early, with the arithmetic.
2. Ground rules (non-negotiable)
- One change per experiment. Give each its own config file. Derive new configs from the current best.
- Control for noise. Run-to-run noise is typically ±5–10%.
- Take ≥6 decode samples per run and use the median.
- Run controls on both sides of each candidate: control, candidate, control.
- Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%.
- Repeat any single surprising win.
- Keep the environment fixed. Stay on AC power, on a fixed performance profile, with an idle host and no other GPU jobs.
- Make the harness refuse to run otherwise.
- Record the state at the end of each run, and void the run if it changed.
- Restore any profile you changed when you finish, and tell the user.
- Run one GPU job at a time. Serialize them with a lock (
scripts/queue.sh).- Never use
pkill -f/pgrep -fpatterns that can match your own shell. - Detach long queues (
setsid nohup … &) so they survive agent restarts.
- Never use
- Enforce the memory cap with a watchdog that kills the run.
- On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g.
drm-resident-*in/proc/<pid>/fdinfo,nvidia-smi, orioreg/vm_staton Apple) plus process anonymous memory.
- On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g.
- Stability gates. A config is only "best" after it passes:
- a long-context retrieval run, with needles in a ~50% then ~100% context document, found exactly;
- a soak with concurrent full-length requests, if the KV pool is sized for N users;
- a greedy output equivalence check, if the change claims to be lossless.
- Never force-kill a server mid-batch. Abort requests, wait for the drain, then stop with a timeout. A forced stop mid-prefill has caused GPU driver faults.
- Verify outputs, not just speed. Flag degenerate or looping output, failed requests and an idle GPU. A suspiciously fast result is usually broken output.
- Log everything in
PLAN.md, including failures and why they failed. Negative results are the most reusable output. - Secrets: never ask twice for a credential the user already saved. Never print it, and never copy it or its path into anything published.
3. The loop
baseline → profile → pick the lever with the largest (ceiling gap × likelihood) → config → A/B with controls
↑ ↓
└──────── update best-config + PLAN.md + ledger ←──── accept / reject with reason ←───────┘
Lever order
Cheapest first; see references/levers.md for evidence.
- Persistent JIT, compile and graph caches (startup −50–70%). Also use eager safetensors loading instead of mmap where it is slow.
- Decode graphs (CUDA/XPU graphs): often 2× c1 on their own.
- Speculative decoding:
- Choose the method (built-in MTP, EAGLE, a draft model, n-gram).
- Sweep draft length per concurrency and context; the best length differs.
- Prune the draft LM head's vocabulary.
- Keep sampling on the GPU.
- Make sure verify runs with decode-style attention.
- Concurrency and KV sizing:
- Set max running requests and the KV pool to N × context.
- For hybrid SSM/GDN models, size the state slots. They, not KV, often cap concurrency.
- Explicitly enable prefix caching and confirm
cached_tokens> 0.
- Prefill:
- Tune chunk size; it trades throughput against decode stalls.
- Use a faster attention library.
- Use int8/fp8 prefill GEMMs as a labelled quality trade.
- Avoid padding cliffs in grouped MoE GEMMs.
- Kernel thread and tile sizes: defaults are often tuned for a bigger card.
- Kernel work: profile, read the ISA, fix the dominant kernel. Do this only after config levers plateau.
When a lever fails
- If kernel-count reductions (fusion) give nothing under graphs, the time is inside the main kernels. Stop fusing.
- If higher acceptance doesn't give higher tok/s, the draft costs too much. Measure per-step time.
- If a weird result appears, suspect the measurement first: tokens counted per streamed chunk, prompt length not measured with the real tokenizer, cache hits, thermal state, or the battery.
Plateau
Declare a plateau when three consecutive levers land inside noise. Write a STATUS section in PLAN.md with:
- best numbers vs baseline vs ceiling;
- what limits further gains, with evidence;
- a ranked list of the remaining levers and their estimated cost.
Then keep the loop alive at a slower cadence if the user said "don't stop", or hand back.
4. Qualify and publish
See references/publishing.md.
- Pin weights by revision, engine by image digest or source commit, and kernels by commit. Build the published artifact from those pins in a clean checkout, never from a dirty tree.
- Run the acceptance gates on the clean build: load, chat, reasoning, tools round-trip, a long-context needle, and speed.
- Then A/B the clean build against your best image to prove they match.
- Report which numbers come from which harness. Gate numbers (e.g. thinking on, first 30 s) and hill-climb numbers (thinking off, medians) are not comparable.
5. Reporting to the user
- Start with a table: metric, baseline, now, ceiling, how it was measured (median of N, cold or warm, which harness).
- Then: what got the gains, why it stops here, the known limitations, and the system state you changed (power profile, running servers, uncommitted code, branches that must not be deleted).
- Say plainly what was not verified.
- When asked to free the GPU: drain, stop the server, confirm no processes and an idle clock, and end any loop.
| 1 | |
| 2 | name inference-optimisation |
| 3 | description > |
| 4 | Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, |
| 5 | Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, |
| 6 | AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative |
| 7 | decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap |
| 8 | and with full-context stability. Computes the theoretical ceiling first, runs |
| 9 | noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before |
| 10 | publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb |
| 11 | a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...), |
| 12 | to find the maximum possible tok/s on a machine, or to publish a tuned recipe. |
| 13 | Triggers hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model, |
| 14 | speculative decoding, KV cache, TTFT, startup time, theoretical maximum. |
| 15 | |
| 16 | |
| 17 | # Inference optimisation (hill-climbing a model server) |
| 18 | |
| 19 | You improve how fast a model server runs on one machine, with evidence. |
| 20 | The loop is: |
| 21 | |
| 22 | Measure the baseline. |
| 23 | Calculate the ceiling. |
| 24 | Change one thing. |
| 25 | Measure it against controls. |
| 26 | Keep it or reject it, and write down why. |
| 27 | Repeat until the remaining gap needs work out of scope, then qualify and publish. |
| 28 | |
| 29 | Detailed references, read when needed: |
| 30 | |
| 31 | `references/ceilings.md`: how to compute theoretical maxima. It includes a worked example. |
| 32 | `references/methodology.md`: measurement pitfalls and statistics. |
| 33 | `references/levers.md`: levers by area, ordered by typical payoff, with measured evidence. |
| 34 | `references/gotchas.md`: engine- and hardware-specific traps. |
| 35 | `references/publishing.md`: qualification gates, pinning, poster and report. |
| 36 | `scripts/`: an engine-agnostic harness. It has a bench, ceiling calculator, memory watchdog, serialized queue and ledger. |
| 37 | |
| 38 | ## 0. Before touching anything |
| 39 | |
| 40 | **Write down the targets and constraints** the user gave. Ask only if one is missing and it changes what you do. |
| 41 | Metrics that matter: c1 decode, cN decode, prefill at several lengths, TTFT, startup. |
| 42 | Memory cap. |
| 43 | Context length that must stay stable. |
| 44 | Which quality changes are allowed: lossless only, or labelled lossy trades such as int8 prefill. |
| 45 | **Inventory the machine:** |
| 46 | accelerator, memory type and size, and whether memory is unified; |
| 47 | driver, engine version, model, quantization, current launch command. |
| 48 | **Measure, don't trust spec sheets:** |
| 49 | real memory bandwidth (device copy); |
| 50 | real matmul throughput (large GEMM); |
| 51 | clocks under load. |
| 52 | **Read any existing experiment log for this machine and model.** Never rerun a rejected experiment without a new reason. |
| 53 | **Create the working directory:** |
| 54 | `PLAN.md`: a timestamped log of every experiment, with numbers, verdict and reason. |
| 55 | `THEORY.md`: the ceilings. |
| 56 | `ledger.jsonl`: one line per run, from the harness. |
| 57 | `configs/`: one JSON per run. |
| 58 | `best-config.json`. |
| 59 | |
| 60 | ## 1. Ceilings first |
| 61 | |
| 62 | Compute the theoretical maximum for each metric before optimising (`scripts/ceiling.py`, `references/ceilings.md`): |
| 63 | |
| 64 | **Decode is memory-bandwidth bound.** Count the bytes read per emitted token: |
| 65 | dense weights; |
| 66 | active experts: top-k ÷ experts × expert bytes; |
| 67 | LM head; |
| 68 | KV read at the working context. |
| 69 | Divide by measured bandwidth. |
| 70 | Speculative decoding divides the target's bytes by the accepted length. It also adds the draft's bytes, and the draft's LM head is often the biggest part. |
| 71 | **Prefill is compute bound.** FLOPs per token ≈ 2 × active params, plus attention at about 4 × layers × hidden × position. Divide by ~50% of measured peak for a practical ceiling. |
| 72 | **TTFT floor:** one forward pass plus tokenize and schedule. |
| 73 | **Startup floor:** weight read from disk or page cache plus allocation, assuming JIT and graph caches persist. |
| 74 | **KV:** bytes per token per layer × the layers that actually have KV (hybrid models have few). That gives how many full-length sequences fit under the memory cap. |
| 75 | |
| 76 | Report progress as a percentage of the practical ceiling. If the user's target is above the ceiling, say so early, with the arithmetic. |
| 77 | |
| 78 | ## 2. Ground rules (non-negotiable) |
| 79 | |
| 80 | **One change per experiment.** Give each its own config file. Derive new configs from the current best. |
| 81 | **Control for noise.** Run-to-run noise is typically ±5–10%. |
| 82 | Take ≥6 decode samples per run and use the median. |
| 83 | Run controls on both sides of each candidate: control, candidate, control. |
| 84 | Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%. |
| 85 | Repeat any single surprising win. |
| 86 | **Keep the environment fixed.** Stay on AC power, on a fixed performance profile, with an idle host and no other GPU jobs. |
| 87 | Make the harness refuse to run otherwise. |
| 88 | Record the state at the end of each run, and void the run if it changed. |
| 89 | Restore any profile you changed when you finish, and tell the user. |
| 90 | **Run one GPU job at a time.** Serialize them with a lock (`scripts/queue.sh`). |
| 91 | Never use `pkill -f`/`pgrep -f` patterns that can match your own shell. |
| 92 | Detach long queues (`setsid nohup … &`) so they survive agent restarts. |
| 93 | **Enforce the memory cap with a watchdog** that kills the run. |
| 94 | On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g. `drm-resident-*` in `/proc/<pid>/fdinfo`, `nvidia-smi`, or `ioreg`/`vm_stat` on Apple) plus process anonymous memory. |
| 95 | **Stability gates.** A config is only "best" after it passes: |
| 96 | a long-context retrieval run, with needles in a ~50% then ~100% context document, found exactly; |
| 97 | a soak with concurrent full-length requests, if the KV pool is sized for N users; |
| 98 | a greedy output equivalence check, if the change claims to be lossless. |
| 99 | **Never force-kill a server mid-batch.** Abort requests, wait for the drain, then stop with a timeout. A forced stop mid-prefill has caused GPU driver faults. |
| 100 | **Verify outputs, not just speed.** Flag degenerate or looping output, failed requests and an idle GPU. A suspiciously fast result is usually broken output. |
| 101 | **Log everything in `PLAN.md`,** including failures and why they failed. Negative results are the most reusable output. |
| 102 | **Secrets:** never ask twice for a credential the user already saved. Never print it, and never copy it or its path into anything published. |
| 103 | |
| 104 | ## 3. The loop |
| 105 | |
| 106 | |
| 107 | baseline → profile → pick the lever with the largest (ceiling gap × likelihood) → config → A/B with controls |
| 108 | ↑ ↓ |
| 109 | └──────── update best-config + PLAN.md + ledger ←──── accept / reject with reason ←───────┘ |
| 110 | |
| 111 | |
| 112 | ### Lever order |
| 113 | |
| 114 | Cheapest first; see `references/levers.md` for evidence. |
| 115 | |
| 116 | **Persistent JIT, compile and graph caches** (startup −50–70%). Also use eager safetensors loading instead of mmap where it is slow. |
| 117 | **Decode graphs** (CUDA/XPU graphs): often 2× c1 on their own. |
| 118 | **Speculative decoding:** |
| 119 | Choose the method (built-in MTP, EAGLE, a draft model, n-gram). |
| 120 | Sweep draft length per concurrency and context; the best length differs. |
| 121 | Prune the draft LM head's vocabulary. |
| 122 | Keep sampling on the GPU. |
| 123 | Make sure verify runs with decode-style attention. |
| 124 | **Concurrency and KV sizing:** |
| 125 | Set max running requests and the KV pool to N × context. |
| 126 | For hybrid SSM/GDN models, size the state slots. They, not KV, often cap concurrency. |
| 127 | Explicitly enable prefix caching and confirm `cached_tokens` > 0. |
| 128 | **Prefill:** |
| 129 | Tune chunk size; it trades throughput against decode stalls. |
| 130 | Use a faster attention library. |
| 131 | Use int8/fp8 prefill GEMMs as a labelled quality trade. |
| 132 | Avoid padding cliffs in grouped MoE GEMMs. |
| 133 | **Kernel thread and tile sizes:** defaults are often tuned for a bigger card. |
| 134 | **Kernel work:** profile, read the ISA, fix the dominant kernel. Do this only after config levers plateau. |
| 135 | |
| 136 | ### When a lever fails |
| 137 | |
| 138 | **If kernel-count reductions (fusion) give nothing** under graphs, the time is inside the main kernels. Stop fusing. |
| 139 | **If higher acceptance doesn't give higher tok/s,** the draft costs too much. Measure per-step time. |
| 140 | **If a weird result appears,** suspect the measurement first: tokens counted per streamed chunk, prompt length not measured with the real tokenizer, cache hits, thermal state, or the battery. |
| 141 | |
| 142 | ### Plateau |
| 143 | |
| 144 | Declare a plateau when three consecutive levers land inside noise. Write a STATUS section in `PLAN.md` with: |
| 145 | |
| 146 | best numbers vs baseline vs ceiling; |
| 147 | what limits further gains, with evidence; |
| 148 | a ranked list of the remaining levers and their estimated cost. |
| 149 | |
| 150 | Then keep the loop alive at a slower cadence if the user said "don't stop", or hand back. |
| 151 | |
| 152 | ## 4. Qualify and publish |
| 153 | |
| 154 | See `references/publishing.md`. |
| 155 | |
| 156 | Pin weights by revision, engine by image digest or source commit, and kernels by commit. Build the published artifact from those pins in a clean checkout, never from a dirty tree. |
| 157 | Run the acceptance gates on the clean build: load, chat, reasoning, tools round-trip, a long-context needle, and speed. |
| 158 | Then A/B the clean build against your best image to prove they match. |
| 159 | Report which numbers come from which harness. Gate numbers (e.g. thinking on, first 30 s) and hill-climb numbers (thinking off, medians) are not comparable. |
| 160 | |
| 161 | ## 5. Reporting to the user |
| 162 | |
| 163 | **Start with a table:** metric, baseline, now, ceiling, how it was measured (median of N, cold or warm, which harness). |
| 164 | **Then:** what got the gains, why it stops here, the known limitations, and the system state you changed (power profile, running servers, uncommitted code, branches that must not be deleted). |
| 165 | **Say plainly** what was not verified. |
| 166 | **When asked to free the GPU:** drain, stop the server, confirm no processes and an idle clock, and end any loop. |
| 167 |
Details
- License
- MIT
- Stars on the repo
- ★ 974
- Checked
- Source
- sybil-solutions/inference-optimisation-skill ↗
- Creator
- sybil-solutions · 1 listing
Similar skills
See all →
Genome Query: 1000 Genomes
Bayesian Modeling Framework
Statistical Modeling Library
Hugging Face Transformers
Waypoint Microbiome Models
Trading Backtest Framework