A. The engine, tests and commands

Everything in this appendix is in the code download (the code/ folder of the book’s repository). Chapter 0 sets it up.

Layout

engine/      YOUR engine: every module, class and signature, with the chapter's functions left as TODO
izh/         the complete reference engine (the answer key): same modules, same names
tests/       one milestone test file per chapter, run against engine/ by default
run.py       one demo command per chapter; --impl engine runs yours
serve.py     OpenAI-compatible server launcher; see Chapter 36
izh/serve/   unified scheduler, block manager, runner, API and adapter slots
izh/bench.py, izh/evaluation.py
             endpoint load generator and correctness/quality harnesses (Chapter 44)
data/        The Verdict, BALLM's 1,100 instructions (with fixed splits), steering prompts
rust/        the Rust track: CPU engine for Qwen3, tests against the Python reference
cpp/         the C++ track: header-only companions and standalone CUDA kernels
rust-cuda/   optional NVIDIA Rust GPU examples
finetune.py, quantize_checkpoint.py, edit_checkpoint.py, model_workflows.py
             Hugging Face-based workflows for real checkpoints (Chapters 20-23)

engine/ is generated from izh/ by tools/make_engine.py: every function whose docstring says “(Your engine: Chapter N)” keeps its signature and docstring and has its body replaced with raise NotImplementedError("TODO(Chapter N): ..."). Triton kernels marked that way become pass, and the CUDA kernels in cuda_ops.cu lose their bodies. Everything not marked is provided, so you write the ideas and skip the boilerplate.

Warning

Regenerating overwrites engine/. The book’s repository does this to keep the skeleton in sync with the reference; you never need to. If you do, commit your work first.

Where to make exercise changes

All exercise paths are relative to the extracted code download’s root: the directory containing run.py, engine/ and tests/. In this repository that directory is docs/inference-zero-to-hero/src/code/.

• Engine milestones: fill in the named TODOs in engine/. izh/ is the complete reference used for comparison. A code tab that includes code/izh/... shows that reference; the matching engine/... file is your implementation target.
• Stretch exercises: each Where note names the edit target. These extensions can add functions, classes or arguments beyond the milestone’s existing TODOs. A file marked new or create it is a suggested file to create, not a supplied starter.
• Experiments: create experiments/chNN.py for a chapter’s measurements, plots or standalone comparisons. Import your completed engine modules there. From the code root, run, for example, python -m experiments.ch02; this keeps the code root on Python’s import path. Use a notebook in the same code root if you prefer. Save handwritten predictions and observations alongside it.
• Tests: run the chapter’s milestone suite after engine changes. Add extension checks in a new tests/test_chNN_stretch.py, importing the relevant engine modules. Milestone tests cover the required implementation; they do not automatically validate an optional extension.
• Native tracks: edit the explicitly named cpp/ or rust/src/ file. For a new CUDA operation in engine/kernels/cuda_ops.cu, add its host launcher and PYBIND11_MODULE entry in that same file; engine/kernels/cuda.py loads the extension. Existing cpp/kernels.cu is the standalone companion.

The Check your understanding questions are written answers; they require no engine edits. Appendix C contains answers and stretch-exercise hints.

The chapter loop

pytest tests/test_ch16_kv_cache.py                 # red: NotImplementedError("TODO(Chapter 16): ...")
$EDITOR engine/kv_cache.py                          # implement the TODOs
pytest tests/test_ch16_kv_cache.py                 # green
python run.py cache --impl engine                   # watch your code run
IZH_IMPL=izh pytest tests/test_ch16_kv_cache.py    # the reference passes the same tests

Later chapters build on earlier ones: Chapter 17’s tests use your Chapter 5 attention and Chapter 16 cache. If a late test fails in an early function, fix the early one; its own tests may not have covered the case.

Environment variables:

Test markers: gpu (skipped without CUDA), reference (needs transformers; compares with the official implementation), slow. Run pytest -m "not slow" for a quick pass.

Milestones by chapter

rg "TODO\(Chapter" engine/ lists what’s left. Chapters 0, 1 and 45 have no implementation milestone.

run.py commands

Every command runs on a CPU unless noted; --impl engine runs your code; python run.py <command> --help lists the options.

Server and measurement commands

Install serve-requirements.txt for HTTP, tokenizer and benchmark labs; Pillow is optional for image data URIs.

python serve.py --model-dir models/Qwen3-0.6B --device cpu --dtype fp32
python serve.py --help
python -m izh.bench --help
python -m izh.evaluation --help

Chapter 36 documents generation/API settings, Chapter 43 the preloaded --lora NAME=DIR adapters and embedding/reranking hooks, and Chapter 44 the complete workload/launch recipes and service limits. engine has the same module CLIs after its TODOs are implemented.

Workflows for real checkpoints

They use Hugging Face Transformers for the model so that the chapter can focus on data, measurement and evaluation; your own engine implements the same ideas from scratch elsewhere in the chapter.