New: Earn 20% as an AgentAlley affiliate. Share your link and earn on every sale you refer. Join now New: Earn 20% as an affiliate. Join now

Inference Optimisation

Inference optimisation (hill-climbing a model server)

by sybil-solutions · MIT license · ★ 974 on GitHub

Inference optimisation (hill-climbing a model server) cover

Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel, Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3, AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s,…

---
name: inference-optimisation
description: >
  Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel,
  Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3,
  AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative
  decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap
  and with full-context stability. Computes the theoretical ceiling first, runs
  noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before
  publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb
  a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...),
  to find the maximum possible tok/s on a machine, or to publish a tuned recipe.
  Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model,
  speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
---

# Inference optimisation (hill-climbing a model server)

Overview

You improve how fast a model server runs on one machine, with evidence. The loop is:

Manual install (npx, ZIP)
Claude Codenpx skills add sybil-solutions/inference-optimisation-skill --skill inference-optimisation -g -a claude-code
Claude Code (degit)npx degit sybil-solutions/inference-optimisation-skill/skills/inference-optimisation#main ~/.claude/skills/inference-optimisation
Claude.ai / desktopZip the folder and upload it under Settings → Capabilities → Skills.
Codex, Cursor, other agentsPoint them at SKILL.md. The other files are plain text.

Connects with Agent Alley MCP · agentalley.io/mcp

Files included

  • SKILL.md9.8 KB
Read SKILL.md here
sybil-solutions/main1 file shown
SKILL.md
Show the full text167 lines

Inference optimisation (hill-climbing a model server)

You improve how fast a model server runs on one machine, with evidence. The loop is:

  1. Measure the baseline.
  2. Calculate the ceiling.
  3. Change one thing.
  4. Measure it against controls.
  5. Keep it or reject it, and write down why.
  6. Repeat until the remaining gap needs work out of scope, then qualify and publish.

Detailed references, read when needed:

  • references/ceilings.md: how to compute theoretical maxima. It includes a worked example.
  • references/methodology.md: measurement pitfalls and statistics.
  • references/levers.md: levers by area, ordered by typical payoff, with measured evidence.
  • references/gotchas.md: engine- and hardware-specific traps.
  • references/publishing.md: qualification gates, pinning, poster and report.
  • scripts/: an engine-agnostic harness. It has a bench, ceiling calculator, memory watchdog, serialized queue and ledger.

0. Before touching anything

  • Write down the targets and constraints the user gave. Ask only if one is missing and it changes what you do.
    • Metrics that matter: c1 decode, cN decode, prefill at several lengths, TTFT, startup.
    • Memory cap.
    • Context length that must stay stable.
    • Which quality changes are allowed: lossless only, or labelled lossy trades such as int8 prefill.
  • Inventory the machine:
    • accelerator, memory type and size, and whether memory is unified;
    • driver, engine version, model, quantization, current launch command.
  • Measure, don't trust spec sheets:
    • real memory bandwidth (device copy);
    • real matmul throughput (large GEMM);
    • clocks under load.
  • Read any existing experiment log for this machine and model. Never rerun a rejected experiment without a new reason.
  • Create the working directory:
    • PLAN.md: a timestamped log of every experiment, with numbers, verdict and reason.
    • THEORY.md: the ceilings.
    • ledger.jsonl: one line per run, from the harness.
    • configs/: one JSON per run.
    • best-config.json.

1. Ceilings first

Compute the theoretical maximum for each metric before optimising (scripts/ceiling.py, references/ceilings.md):

  • Decode is memory-bandwidth bound. Count the bytes read per emitted token:
    • dense weights;
    • active experts: top-k ÷ experts × expert bytes;
    • LM head;
    • KV read at the working context.
    • Divide by measured bandwidth.
    • Speculative decoding divides the target's bytes by the accepted length. It also adds the draft's bytes, and the draft's LM head is often the biggest part.
  • Prefill is compute bound. FLOPs per token ≈ 2 × active params, plus attention at about 4 × layers × hidden × position. Divide by ~50% of measured peak for a practical ceiling.
  • TTFT floor: one forward pass plus tokenize and schedule.
  • Startup floor: weight read from disk or page cache plus allocation, assuming JIT and graph caches persist.
  • KV: bytes per token per layer × the layers that actually have KV (hybrid models have few). That gives how many full-length sequences fit under the memory cap.

Report progress as a percentage of the practical ceiling. If the user's target is above the ceiling, say so early, with the arithmetic.

2. Ground rules (non-negotiable)

  1. One change per experiment. Give each its own config file. Derive new configs from the current best.
  2. Control for noise. Run-to-run noise is typically ±5–10%.
    • Take ≥6 decode samples per run and use the median.
    • Run controls on both sides of each candidate: control, candidate, control.
    • Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%.
    • Repeat any single surprising win.
  3. Keep the environment fixed. Stay on AC power, on a fixed performance profile, with an idle host and no other GPU jobs.
    • Make the harness refuse to run otherwise.
    • Record the state at the end of each run, and void the run if it changed.
    • Restore any profile you changed when you finish, and tell the user.
  4. Run one GPU job at a time. Serialize them with a lock (scripts/queue.sh).
    • Never use pkill -f/pgrep -f patterns that can match your own shell.
    • Detach long queues (setsid nohup … &) so they survive agent restarts.
  5. Enforce the memory cap with a watchdog that kills the run.
    • On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g. drm-resident-* in /proc/<pid>/fdinfo, nvidia-smi, or ioreg/vm_stat on Apple) plus process anonymous memory.
  6. Stability gates. A config is only "best" after it passes:
    • a long-context retrieval run, with needles in a ~50% then ~100% context document, found exactly;
    • a soak with concurrent full-length requests, if the KV pool is sized for N users;
    • a greedy output equivalence check, if the change claims to be lossless.
  7. Never force-kill a server mid-batch. Abort requests, wait for the drain, then stop with a timeout. A forced stop mid-prefill has caused GPU driver faults.
  8. Verify outputs, not just speed. Flag degenerate or looping output, failed requests and an idle GPU. A suspiciously fast result is usually broken output.
  9. Log everything in PLAN.md, including failures and why they failed. Negative results are the most reusable output.
  10. Secrets: never ask twice for a credential the user already saved. Never print it, and never copy it or its path into anything published.

3. The loop

baseline → profile → pick the lever with the largest (ceiling gap × likelihood) → config → A/B with controls
   ↑                                                                                         ↓
   └──────── update best-config + PLAN.md + ledger ←──── accept / reject with reason ←───────┘
Lever order

Cheapest first; see references/levers.md for evidence.

  1. Persistent JIT, compile and graph caches (startup −50–70%). Also use eager safetensors loading instead of mmap where it is slow.
  2. Decode graphs (CUDA/XPU graphs): often 2× c1 on their own.
  3. Speculative decoding:
    • Choose the method (built-in MTP, EAGLE, a draft model, n-gram).
    • Sweep draft length per concurrency and context; the best length differs.
    • Prune the draft LM head's vocabulary.
    • Keep sampling on the GPU.
    • Make sure verify runs with decode-style attention.
  4. Concurrency and KV sizing:
    • Set max running requests and the KV pool to N × context.
    • For hybrid SSM/GDN models, size the state slots. They, not KV, often cap concurrency.
    • Explicitly enable prefix caching and confirm cached_tokens > 0.
  5. Prefill:
    • Tune chunk size; it trades throughput against decode stalls.
    • Use a faster attention library.
    • Use int8/fp8 prefill GEMMs as a labelled quality trade.
    • Avoid padding cliffs in grouped MoE GEMMs.
  6. Kernel thread and tile sizes: defaults are often tuned for a bigger card.
  7. Kernel work: profile, read the ISA, fix the dominant kernel. Do this only after config levers plateau.
When a lever fails
  • If kernel-count reductions (fusion) give nothing under graphs, the time is inside the main kernels. Stop fusing.
  • If higher acceptance doesn't give higher tok/s, the draft costs too much. Measure per-step time.
  • If a weird result appears, suspect the measurement first: tokens counted per streamed chunk, prompt length not measured with the real tokenizer, cache hits, thermal state, or the battery.
Plateau

Declare a plateau when three consecutive levers land inside noise. Write a STATUS section in PLAN.md with:

  • best numbers vs baseline vs ceiling;
  • what limits further gains, with evidence;
  • a ranked list of the remaining levers and their estimated cost.

Then keep the loop alive at a slower cadence if the user said "don't stop", or hand back.

4. Qualify and publish

See references/publishing.md.

  • Pin weights by revision, engine by image digest or source commit, and kernels by commit. Build the published artifact from those pins in a clean checkout, never from a dirty tree.
  • Run the acceptance gates on the clean build: load, chat, reasoning, tools round-trip, a long-context needle, and speed.
  • Then A/B the clean build against your best image to prove they match.
  • Report which numbers come from which harness. Gate numbers (e.g. thinking on, first 30 s) and hill-climb numbers (thinking off, medians) are not comparable.

5. Reporting to the user

  • Start with a table: metric, baseline, now, ceiling, how it was measured (median of N, cold or warm, which harness).
  • Then: what got the gains, why it stops here, the known limitations, and the system state you changed (power profile, running servers, uncommitted code, branches that must not be deleted).
  • Say plainly what was not verified.
  • When asked to free the GPU: drain, stop the server, confirm no processes and an idle clock, and end any loop.
1---
2name: inference-optimisation
3description: >
4 Hill-climb the serving speed of a local LLM on any hardware (NVIDIA, AMD, Intel,
5 Apple, unified-memory laptops, multi-GPU) and any quantization (GGUF, EXL2/EXL3,
6 AWQ/GPTQ, FP8, MLX, BF16): single-user and concurrent decode tok/s, speculative
7 decoding, prefill tok/s, KV cache size, startup time and TTFT, under a memory cap
8 and with full-context stability. Computes the theoretical ceiling first, runs
9 noise-controlled A/B experiments, keeps a ledger, and qualifies the winner before
10 publishing it. Use when asked to optimise, tune, speed up, benchmark or hill-climb
11 a model server (vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, TensorRT-LLM...),
12 to find the maximum possible tok/s on a machine, or to publish a tuned recipe.
13 Triggers: hill climb, tok/s, decode speed, prefill speed, MTP, EAGLE, draft model,
14 speculative decoding, KV cache, TTFT, startup time, theoretical maximum.
15---
16 
17# Inference optimisation (hill-climbing a model server)
18 
19You improve how fast a model server runs on one machine, with evidence.
20The loop is:
21 
221. Measure the baseline.
232. Calculate the ceiling.
243. Change one thing.
254. Measure it against controls.
265. Keep it or reject it, and write down why.
276. Repeat until the remaining gap needs work out of scope, then qualify and publish.
28 
29Detailed references, read when needed:
30 
31- `references/ceilings.md`: how to compute theoretical maxima. It includes a worked example.
32- `references/methodology.md`: measurement pitfalls and statistics.
33- `references/levers.md`: levers by area, ordered by typical payoff, with measured evidence.
34- `references/gotchas.md`: engine- and hardware-specific traps.
35- `references/publishing.md`: qualification gates, pinning, poster and report.
36- `scripts/`: an engine-agnostic harness. It has a bench, ceiling calculator, memory watchdog, serialized queue and ledger.
37 
38## 0. Before touching anything
39 
40- **Write down the targets and constraints** the user gave. Ask only if one is missing and it changes what you do.
41 - Metrics that matter: c1 decode, cN decode, prefill at several lengths, TTFT, startup.
42 - Memory cap.
43 - Context length that must stay stable.
44 - Which quality changes are allowed: lossless only, or labelled lossy trades such as int8 prefill.
45- **Inventory the machine:**
46 - accelerator, memory type and size, and whether memory is unified;
47 - driver, engine version, model, quantization, current launch command.
48- **Measure, don't trust spec sheets:**
49 - real memory bandwidth (device copy);
50 - real matmul throughput (large GEMM);
51 - clocks under load.
52- **Read any existing experiment log for this machine and model.** Never rerun a rejected experiment without a new reason.
53- **Create the working directory:**
54 - `PLAN.md`: a timestamped log of every experiment, with numbers, verdict and reason.
55 - `THEORY.md`: the ceilings.
56 - `ledger.jsonl`: one line per run, from the harness.
57 - `configs/`: one JSON per run.
58 - `best-config.json`.
59 
60## 1. Ceilings first
61 
62Compute the theoretical maximum for each metric before optimising (`scripts/ceiling.py`, `references/ceilings.md`):
63 
64- **Decode is memory-bandwidth bound.** Count the bytes read per emitted token:
65 - dense weights;
66 - active experts: top-k ÷ experts × expert bytes;
67 - LM head;
68 - KV read at the working context.
69 - Divide by measured bandwidth.
70 - Speculative decoding divides the target's bytes by the accepted length. It also adds the draft's bytes, and the draft's LM head is often the biggest part.
71- **Prefill is compute bound.** FLOPs per token ≈ 2 × active params, plus attention at about 4 × layers × hidden × position. Divide by ~50% of measured peak for a practical ceiling.
72- **TTFT floor:** one forward pass plus tokenize and schedule.
73- **Startup floor:** weight read from disk or page cache plus allocation, assuming JIT and graph caches persist.
74- **KV:** bytes per token per layer × the layers that actually have KV (hybrid models have few). That gives how many full-length sequences fit under the memory cap.
75 
76Report progress as a percentage of the practical ceiling. If the user's target is above the ceiling, say so early, with the arithmetic.
77 
78## 2. Ground rules (non-negotiable)
79 
801. **One change per experiment.** Give each its own config file. Derive new configs from the current best.
812. **Control for noise.** Run-to-run noise is typically ±5–10%.
82 - Take ≥6 decode samples per run and use the median.
83 - Run controls on both sides of each candidate: control, candidate, control.
84 - Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%.
85 - Repeat any single surprising win.
863. **Keep the environment fixed.** Stay on AC power, on a fixed performance profile, with an idle host and no other GPU jobs.
87 - Make the harness refuse to run otherwise.
88 - Record the state at the end of each run, and void the run if it changed.
89 - Restore any profile you changed when you finish, and tell the user.
904. **Run one GPU job at a time.** Serialize them with a lock (`scripts/queue.sh`).
91 - Never use `pkill -f`/`pgrep -f` patterns that can match your own shell.
92 - Detach long queues (`setsid nohup … &`) so they survive agent restarts.
935. **Enforce the memory cap with a watchdog** that kills the run.
94 - On unified memory, GPU allocations do not show in cgroup counters or MemAvailable. Measure driver residency (e.g. `drm-resident-*` in `/proc/<pid>/fdinfo`, `nvidia-smi`, or `ioreg`/`vm_stat` on Apple) plus process anonymous memory.
956. **Stability gates.** A config is only "best" after it passes:
96 - a long-context retrieval run, with needles in a ~50% then ~100% context document, found exactly;
97 - a soak with concurrent full-length requests, if the KV pool is sized for N users;
98 - a greedy output equivalence check, if the change claims to be lossless.
997. **Never force-kill a server mid-batch.** Abort requests, wait for the drain, then stop with a timeout. A forced stop mid-prefill has caused GPU driver faults.
1008. **Verify outputs, not just speed.** Flag degenerate or looping output, failed requests and an idle GPU. A suspiciously fast result is usually broken output.
1019. **Log everything in `PLAN.md`,** including failures and why they failed. Negative results are the most reusable output.
10210. **Secrets:** never ask twice for a credential the user already saved. Never print it, and never copy it or its path into anything published.
103 
104## 3. The loop
105 
106```
107baseline → profile → pick the lever with the largest (ceiling gap × likelihood) → config → A/B with controls
108 ↑ ↓
109 └──────── update best-config + PLAN.md + ledger ←──── accept / reject with reason ←───────┘
110```
111 
112### Lever order
113 
114Cheapest first; see `references/levers.md` for evidence.
115 
1161. **Persistent JIT, compile and graph caches** (startup −50–70%). Also use eager safetensors loading instead of mmap where it is slow.
1172. **Decode graphs** (CUDA/XPU graphs): often 2× c1 on their own.
1183. **Speculative decoding:**
119 - Choose the method (built-in MTP, EAGLE, a draft model, n-gram).
120 - Sweep draft length per concurrency and context; the best length differs.
121 - Prune the draft LM head's vocabulary.
122 - Keep sampling on the GPU.
123 - Make sure verify runs with decode-style attention.
1244. **Concurrency and KV sizing:**
125 - Set max running requests and the KV pool to N × context.
126 - For hybrid SSM/GDN models, size the state slots. They, not KV, often cap concurrency.
127 - Explicitly enable prefix caching and confirm `cached_tokens` > 0.
1285. **Prefill:**
129 - Tune chunk size; it trades throughput against decode stalls.
130 - Use a faster attention library.
131 - Use int8/fp8 prefill GEMMs as a labelled quality trade.
132 - Avoid padding cliffs in grouped MoE GEMMs.
1336. **Kernel thread and tile sizes:** defaults are often tuned for a bigger card.
1347. **Kernel work:** profile, read the ISA, fix the dominant kernel. Do this only after config levers plateau.
135 
136### When a lever fails
137 
138- **If kernel-count reductions (fusion) give nothing** under graphs, the time is inside the main kernels. Stop fusing.
139- **If higher acceptance doesn't give higher tok/s,** the draft costs too much. Measure per-step time.
140- **If a weird result appears,** suspect the measurement first: tokens counted per streamed chunk, prompt length not measured with the real tokenizer, cache hits, thermal state, or the battery.
141 
142### Plateau
143 
144Declare a plateau when three consecutive levers land inside noise. Write a STATUS section in `PLAN.md` with:
145 
146- best numbers vs baseline vs ceiling;
147- what limits further gains, with evidence;
148- a ranked list of the remaining levers and their estimated cost.
149 
150Then keep the loop alive at a slower cadence if the user said "don't stop", or hand back.
151 
152## 4. Qualify and publish
153 
154See `references/publishing.md`.
155 
156- Pin weights by revision, engine by image digest or source commit, and kernels by commit. Build the published artifact from those pins in a clean checkout, never from a dirty tree.
157- Run the acceptance gates on the clean build: load, chat, reasoning, tools round-trip, a long-context needle, and speed.
158- Then A/B the clean build against your best image to prove they match.
159- Report which numbers come from which harness. Gate numbers (e.g. thinking on, first 30 s) and hill-climb numbers (thinking off, medians) are not comparable.
160 
161## 5. Reporting to the user
162 
163- **Start with a table:** metric, baseline, now, ceiling, how it was measured (median of N, cold or warm, which harness).
164- **Then:** what got the gains, why it stops here, the known limitations, and the system state you changed (power profile, running servers, uncommitted code, branches that must not be deleted).
165- **Say plainly** what was not verified.
166- **When asked to free the GPU:** drain, stop the server, confirm no processes and an idle clock, and end any loop.
167 
View all files on GitHub →

Details

License
MIT
Stars on the repo
★ 974
Checked
Source
sybil-solutions/inference-optimisation-skill ↗
Creator
sybil-solutions · 1 listing
View creator

Similar skills

See all →
Genome Query: 1000 GenomesK-Dense-AI/scientific-agent-skills · by K-Dense-AIQuery the 1000 Genomes Project dataset (3,202 whole-genome-sequenced individuals, GRCh38) at the level of individual participants. Use when a question is about individuals or variants in the 1000 Genomes Project cohort: which individuals carry variants matching specific criteria in a gene or region, which individuals are homozygous-reference at a position, which variants exist in the dataset or carried by specified individuals in a gene or region, the relatedness between two specified individuals. Variants are returned with 1000 Genomes allele frequencies (AF), gnomAD v4.1 exome and genome AF, AlphaMissense score, and HGVSp annotations.★ 46k GitHub starsBayesian Modeling FrameworkK-Dense-AI/scientific-agent-skills · by K-Dense-AIBayesian modeling with PyMC. Build hierarchical models, MCMC (NUTS), variational inference, LOO/WAIC comparison, posterior checks, for probabilistic programming and inference.★ 46k GitHub starsStatistical Modeling LibraryK-Dense-AI/scientific-agent-skills · by K-Dense-AIStatistical models library for Python. Use when you need specific model classes (OLS, GLM, mixed models, ARIMA) with detailed diagnostics, residuals, and inference. Best for econometrics, time series, rigorous inference with coefficient tables. For guided statistical test selection with APA reporting use statistical-analysis.★ 46k GitHub starsHugging Face TransformersK-Dense-AI/scientific-agent-skills · by K-Dense-AIHugging Face Transformers for loading Hub models, running pipeline inference, text generation, and Trainer fine-tuning on NLP, vision, audio, and multimodal tasks. Use when working with AutoModel, pipelines, tokenizers, or TrainingArguments—not for general ML outside the Transformers library.★ 46k GitHub starsWaypoint Microbiome ModelsK-Dense-AI/scientific-agent-skills · by K-Dense-AIUse when working with Outpost Bio's open microbiome foundation models - the Waypoint checkpoints (Waypoint-6m, Waypoint-45m, Waypoint-170m), the Atlas pretraining corpus, the Compass eight-task benchmark, or the `waypoint` CLI from the `waypoint-bio` package. Covers embedding microbiome samples, fine-tuning on taxonomic abundance data, benchmarking a checkpoint on Compass, pretraining a GPT-2 model on taxonomic abundance profiles, and converting MetaPhlAn, Kraken2, QIIME 2, or MGnify abundance tables into waypoint format.★ 46k GitHub starsTrading Backtest Frameworkwshobson/agents · by wshobsonBuild robust backtesting systems for trading strategies with proper handling of look-ahead bias, survivorship bias, and transaction costs. Use when developing trading algorithms, validating strategies, or building backtesting infrastructure.★ 39.9k GitHub stars