Meap chap 1 5

ACCELERATORS.md

@@ -72,6 +72,22 @@ AMD (ROCm) was validated end-to-end on a datacenter card (Instinct MI300X); VRAM
 
 **How the measured column was produced (2026-09-10).** Each training script was run for three steps with its defaults (Qwen3-4B-Instruct-2507, `max_length` 512, bf16, gradient checkpointing) on NVIDIA A30 24 GB cards with PyTorch 2.11+cu126, and `torch.cuda.max_memory_allocated()` read per device at the end; "total" sums the devices when `device_map="auto"` sharded the model, which is the number a single card would need. Multi-card totals came from the same box that produced the book's published runs, which is why the chapter 6 and chapter 8 text and earlier versions of this table understated the single-card requirement. Every training script now prints its own peak when training ends, and you can re-measure any chapter on your hardware with `python -m scripts.measure_peak_vram <module> <its args> --max_steps 3` (see `code/scripts/measure_peak_vram.py`). Why the fixed floors: full SFT keeps bf16 weights, bf16 gradients, and two bf16 AdamW moments, 8 bytes per parameter, about 32 GB for 4B parameters before activations; full DPO adds a frozen bf16 reference copy (another 8 GB) and two forward passes per preference pair. Gradient checkpointing trims activations, not these floors. An 8-bit optimizer would bring full SFT to roughly 24 GB plus activations, still not a comfortable single-A30 fit; CPU optimizer offload (not used in the book's scripts) is the only single-24 GB-card route.
 
+**Measured full runs (2026-09-15 to 2026-09-19, repo defaults, NVIDIA A30 24 GB cards).** The table above probes three steps; these are the complete runs the book quotes, with `train_runtime` from the trainer and peak `torch.cuda.max_memory_allocated()`.
+
+| Run (Qwen3-4B-Instruct-2507) | Peak GPU memory | Wall time | Cards |
+|---|---|---|---|
+| Ch5 LoRA r=16, 450 examples, 3 epochs | 9.0 GB | 650 s (about 11 min) | 1 |
+| Ch5 QLoRA r=8, 450 examples, 3 epochs | 5.1 GB | 897 s (about 15 min) | 1 |
+| Ch5 evaluation (base + adapter, 50 test questions + safety suite) | about 10 GB | about 10 min | 1 |
+| Ch5 inference with the adapter attached | 7.7 GB | seconds per prompt | 1 |
+| Ch6 full SFT, 450 examples, 3 epochs | 32.5 GB total | about 10 min | 2 |
+| Ch7 student LoRA r=16, 159 distilled examples | 10.5 GB | a few minutes | 1 |
+| Ch7 house-format check (three models, 50 prompts, greedy, 400 new tokens) | about 10 GB | about 36 min | 1 |
+| Ch8 full-parameter DPO, 270 pairs, 1 epoch | 54.3 GB total (18.6 / 18.6 / 17.3) | 203 s (3.4 min) | 3 |
+| Ch8 LoRA-DPO r=16, 270 pairs, 1 epoch | 10.8 GB | 168 s (2.8 min) | 1 |
+
+Enterprise sizing: one H100-class 80 GB card runs every row above on its own; the A30 is the book's 24 GB test card, not a recommendation. Logs: `code/chapter05/eval/test_split/`, `code/chapter06/eval/test_split/`, `code/chapter07/eval/test_split/`, `code/chapter08/eval/runtime_2026-09-19/`.
+
 **Disk space:** budget about 50 GB free for the Hugging Face model cache plus chapter 6's run directory (full-parameter checkpoints with optimizer state are 22-24 GB each). See `code/chapter06/README.md` for the breakdown.
 
 ## Why QLoRA needs an NVIDIA or AMD GPU
@@ -217,7 +233,9 @@ Same code, same direction of results on every GPU (small numeric differences com
 | Ch7 base / teacher / student F1 | 0.262 / 0.564 / 0.487 | 0.288 / 0.466 / 0.471 | 0.258 / 0.406 / 0.459 |
 | Ch8 base / SFT / DPO F1 | 0.257 / 0.356 / 0.378 | 0.255 / 0.341 / 0.364 | 0.257 / 0.339 / 0.353 |
 
-The ordering is stable: chapter 3's corrupted condition is always the worst, the chapter 7 student approaches or matches its teacher (and always beats the base), and chapter 8 ranks DPO above SFT above base.
+The ordering is stable: chapter 3's corrupted condition is always the worst, the chapter 7 student approaches or matches its teacher (and always beats the base), and chapter 8 never ranks DPO below SFT.
+
+The chapter 7 and 8 rows are from the May 2026 cross-accelerator pass and were scored on each chapter's own validation split with the data of that time. The book now scores on the held-out `data/it_support/test.jsonl` (50 questions no training or model-selection step touched); on the A30 those numbers are chapter 7 base / teacher / student 0.153 / 0.168 / 0.165 and chapter 8 base / SFT / DPO 0.153 / 0.168 / 0.166, where DPO and SFT are a wash on token-F1. The relative ordering, not the absolute values, is what should reproduce on your card.
 
 ## Insights
 

code/README.md

@@ -326,7 +326,7 @@ Each chapter README contains:
 The hands-on chapters train on the book's **IT-support dataset**: real Stack Exchange IT Q&A (Super User, Ask Ubuntu, and Server Fault), filtered to genuine support topics, plus a small slice of Databricks Dolly mixed in to preserve general capability. Build it once from this `code/` directory:
 
 ```bash
-# 1. Build the dataset -> data/it_support/ (train.jsonl, valid.jsonl, preferences.jsonl, manifest.json, attribution.jsonl)
+# 1. Build the dataset -> data/it_support/ (train.jsonl, valid.jsonl, test.jsonl, preferences.jsonl, manifest.json, attribution.jsonl)
 python scripts/build_it_support_dataset.py
 
 # 2. Reformat the answers into the house style -> data/it_support_fmt/{train,valid}.jsonl

code/chapter02/quickstart.py

@@ -101,8 +101,9 @@ def step1_prepare_dataset() -> tuple[HFDataset, HFDataset, List[Dict[str, Any]]]
     from common.jsonl import read_jsonl
     # Both files carry answers in the house format (built by scripts/build_it_support_dataset.py,
     # then scripts/reformat_it_answers.py), so training loss and eval loss score the same style.
-    # The raw answers in data/it_support/valid.jsonl stay the held-out test set the later chapters
-    # evaluate against; the prompts are identical in both files.
+    # The raw answers in data/it_support/valid.jsonl and test.jsonl stay as they are; the later
+    # chapters report their scores on the test split, which no training step touches. The prompts
+    # are identical in the raw and reformatted files.
     trows = list(read_jsonl("data/it_support_fmt/train.jsonl"))
     vrows = list(read_jsonl("data/it_support_fmt/valid.jsonl"))
 

code/chapter04/README.md

@@ -30,7 +30,7 @@ Shared utilities (JSONL I/O, seeded reproducibility) live in **`code/common/`**.
 | **4.2** | Many-shot prompt assembly | `many_shot_demo.py` |
 | **4.3** | Prompt validator | `prompt_validator.py` |
 | **4.4** | Minimal RAG pipeline | `rag_minimal.py` |
-| **4.5** | RAG retrieval evaluation (Precision@k / Recall@k / Hit@1) | `scripts/listing_4_5_rag_eval.py` |
+| **4.5** | RAG retrieval evaluation (Precision@k / Hit@k / Hit@1) | `scripts/listing_4_5_rag_eval.py` |
 
 ## Prerequisites
 
@@ -144,7 +144,7 @@ python -m chapter04.rag_minimal retrieve \
 
 ### 5. Measure RAG quality (Listing 4.5)
 
-The RAG eval script runs a small labelled query set against the same pipeline and reports Precision@k, Recall@k, and Hit@1 — the retrieval-side metrics that let you tell whether a bad answer is the index's fault or the generator's. The labelled set in `data/rag_eval.jsonl` ships with one query per source document.
+The RAG eval script runs a small labelled query set against the same pipeline and reports Precision@k, Hit@k, and Hit@1, the retrieval-side metrics that let you tell whether a bad answer is the index's fault or the generator's. The labelled set in `data/rag_eval.jsonl` ships with one query per source document.
 
 ```bash
 # Sentence-transformers backend (recommended):

code/chapter04/scripts/listing_4_5_rag_eval.py

@@ -1,21 +1,28 @@
-"""Listing 4.5 -- Measuring RAG quality with Precision@k and Recall@k.
+"""Listing 4.5 -- Measuring RAG quality with Precision@k and Hit@k.
 
 Run a small labelled query set (one expected document per query) through
 the Listing 4.4 pipeline and report the two metrics every RAG team should
 have before they reach for chunking or reranker tuning:
 
 - Precision@k: of the k chunks we retrieved, what fraction came from the
   expected document.
-- Recall@k:    did the expected document appear at least once in the top-k?
+- Hit@k:       did the expected document appear at least once in the top-k?
+  (1 or 0 per query, averaged over queries; sometimes called success rate)
 
 The mean of Precision@k across queries is the headline retrieval quality
-score; the mean of Recall@k tells you how often retrieval handed the
+score; the mean of Hit@k tells you how often retrieval handed the
 generator any chance of being right at all.
 
-A separate ``hit@1`` is also reported because top-1 is what a no-rerank
-production setup actually shows the LLM as the most relevant chunk.
+Hit@1 is Hit@k with k=1 and is reported separately because top-1 is what a
+no-rerank production setup actually shows the LLM as the most relevant chunk.
 
-The script intentionally avoids any LLM call — it measures retrieval in
+Why not "Recall@k"? Recall divides the relevant documents retrieved by all
+the relevant documents that exist. With one labelled document per query the
+two coincide, but published RAG benchmarks report recall on multi-document
+labels, so the metric is named for what it measures here. ``recall_at_k``
+stays as an alias for scripts written against earlier versions.
+
+The script intentionally avoids any LLM call; it measures retrieval in
 isolation so a bad answer can be attributed to retrieval vs. generation.
 For end-to-end answer quality, use the LLM-as-judge pattern from
 ``prompt_validator.py`` (Listing 4.3).
@@ -49,21 +56,21 @@ def precision_at_k(retrieved: List[Dict[str, object]], expected_doc_id: str) ->
     return hits / len(retrieved)
 
 
-def recall_at_k(retrieved: List[Dict[str, object]], expected_doc_id: str) -> float:
-    """1.0 if the expected document appears in the top-k, else 0.0.
+def hit_at_k(retrieved: List[Dict[str, object]], expected_doc_id: str) -> float:
+    """1.0 if the expected document appears anywhere in the top-k, else 0.0.
 
-    With one labelled doc per query, recall@k collapses to a hit indicator;
-    extend to multi-doc relevance by passing a set of expected_doc_ids and
-    dividing by len(expected) here.
+    To score multi-document labels as true recall, pass a set of expected ids
+    and divide the number found by len(expected) instead.
     """
     return 1.0 if any(item["metadata"]["id"] == expected_doc_id for item in retrieved) else 0.0
 
 
+recall_at_k = hit_at_k  # backwards-compatible alias (pre-2026-09 name)
+
+
 def hit_at_1(retrieved: List[Dict[str, object]], expected_doc_id: str) -> float:
-    """1.0 if the top-1 chunk came from the expected document."""
-    if not retrieved:
-        return 0.0
-    return 1.0 if retrieved[0]["metadata"]["id"] == expected_doc_id else 0.0
+    """Hit@k with k=1: the top-ranked chunk came from the expected document."""
+    return hit_at_k(retrieved[:1], expected_doc_id)
 
 
 def evaluate(rag: MinimalRAG, queries: List[Dict[str, str]], k: int) -> Dict[str, object]:
@@ -76,7 +83,7 @@ def evaluate(rag: MinimalRAG, queries: List[Dict[str, str]], k: int) -> Dict[str
                 "expected_doc_id": case["expected_doc_id"],
                 "top_k_doc_ids": [item["metadata"]["id"] for item in retrieved],
                 "precision_at_k": round(precision_at_k(retrieved, case["expected_doc_id"]), 4),
-                "recall_at_k": round(recall_at_k(retrieved, case["expected_doc_id"]), 4),
+                "hit_at_k": round(hit_at_k(retrieved, case["expected_doc_id"]), 4),
                 "hit_at_1": round(hit_at_1(retrieved, case["expected_doc_id"]), 4),
             }
         )
@@ -86,7 +93,7 @@ def evaluate(rag: MinimalRAG, queries: List[Dict[str, str]], k: int) -> Dict[str
         "k": k,
         "queries": len(rows),
         "mean_precision_at_k": round(sum(r["precision_at_k"] for r in rows) / n, 4),
-        "mean_recall_at_k": round(sum(r["recall_at_k"] for r in rows) / n, 4),
+        "mean_hit_at_k": round(sum(r["hit_at_k"] for r in rows) / n, 4),
         "mean_hit_at_1": round(sum(r["hit_at_1"] for r in rows) / n, 4),
     }
     return {"summary": summary, "per_query": rows}
@@ -142,15 +149,15 @@ def main() -> None:
     print(f"  k                       {summary['k']}")
     print(f"  queries                 {summary['queries']}")
     print(f"  mean Precision@k        {summary['mean_precision_at_k']:.3f}")
-    print(f"  mean Recall@k           {summary['mean_recall_at_k']:.3f}")
+    print(f"  mean Hit@k              {summary['mean_hit_at_k']:.3f}")
     print(f"  mean Hit@1              {summary['mean_hit_at_1']:.3f}")
     print(f"  wall seconds            {elapsed:.2f}")
     print()
     print("Per-query (showing misses first):")
-    misses = [r for r in report["per_query"] if r["recall_at_k"] < 1.0]
-    hits = [r for r in report["per_query"] if r["recall_at_k"] >= 1.0]
+    misses = [r for r in report["per_query"] if r["hit_at_k"] < 1.0]
+    hits = [r for r in report["per_query"] if r["hit_at_k"] >= 1.0]
     for row in misses + hits:
-        marker = "MISS" if row["recall_at_k"] < 1.0 else "  ok"
+        marker = "MISS" if row["hit_at_k"] < 1.0 else "  ok"
         print(
             f"  [{marker}] expected={row['expected_doc_id']:24s} "
             f"top1={row['top_k_doc_ids'][0] if row['top_k_doc_ids'] else 'EMPTY':24s} "

code/chapter05/README.md

@@ -50,10 +50,10 @@ We're fine-tuning Qwen3-4B-Instruct-2507 on the book's IT support dataset to ada
 
 **Expected results** (representative measured values; your numbers will move across hardware and library versions):
 
-- Base Qwen3-4B-Instruct-2507: Token-F1 ≈ 0.158, safety refusal 100%.
-- After LoRA (r=16, 3 epochs): Token-F1 ≈ 0.155 (roughly flat), safety refusal drops to ≈ 60%.
+- Base Qwen3-4B-Instruct-2507: Token-F1 ≈ 0.149 on the held-out test split, safety refusal 100%.
+- After LoRA (r=16, 3 epochs): Token-F1 ≈ 0.154 (roughly flat), safety refusal drops to ≈ 60%.
 
-The headline takeaway: token-F1 is essentially flat (0.158 → 0.155) because word overlap is a poor proxy for the quality of a long generative answer, while the safety refusal rate drops from 100% to 60%. That safety regression is real and load-bearing for the chapter, it motivates the safety-regression suite that follows the eval and previews the safety conversation in chapter 6 and chapter 8.
+The headline takeaway: token-F1 is essentially flat (0.149 → 0.154) because word overlap is a poor proxy for the quality of a long generative answer, while the safety refusal rate drops from 100% to 60%. That safety regression is real and load-bearing for the chapter, it motivates the safety-regression suite that follows the eval and previews the safety conversation in chapter 6 and chapter 8.
 
 ## Why the IT support dataset?
 
@@ -128,7 +128,7 @@ Chapter 5 validation
 - **LoRA**: minimum **8 GB VRAM** (RTX 3060 / 4060 class).
 - **QLoRA**: minimum **6 GB VRAM** (works on smaller GPUs).
 - **Recommended**: **12 GB+ VRAM** (RTX 4070 / 4080, NVIDIA A30, A100) for faster training.
-- **Training time on a single A30**: ~10-12 minutes for LoRA, ~14 minutes for QLoRA (the IT support training set, 3 epochs). On smaller GPUs allocate up to 25-35 minutes.
+- **Training time on a single A30**: ~11 minutes for LoRA, ~14-15 minutes for QLoRA (the IT support training set, 3 epochs). On smaller GPUs allocate up to 25-35 minutes.
 
 ## Step-by-Step Instructions
 
@@ -162,7 +162,7 @@ This will:
 - Clean the HTML answer bodies with `beautifulsoup4`
 - Create train/valid splits plus a `preferences.jsonl` file (used later for preference optimization)
 - Write a `manifest.json` and an `attribution.jsonl` recording the per-example source URL and license
-- Save to `data/it_support/` (`train.jsonl`, `valid.jsonl`, `preferences.jsonl`, `manifest.json`, `attribution.jsonl`)
+- Save to `data/it_support/` (`train.jsonl`, `valid.jsonl`, `test.jsonl`, `preferences.jsonl`, `manifest.json`, `attribution.jsonl`)
 
 The second script (`reformat_it_answers.py`) rewrites the answers into the chapter's house format and writes `data/it_support_fmt/train.jsonl` (the file you train on) and `data/it_support_fmt/valid.jsonl` (the training-time validation split, processed the same way so eval loss is comparable to training loss). The raw `data/it_support/valid.jsonl` stays the held-out test set for the evaluation scripts, so token-F1 is still scored against the original human answers.
 
@@ -171,6 +171,7 @@ The second script (`reformat_it_answers.py`) rewrites the answers into the chapt
 data/it_support/
   train.jsonl
   valid.jsonl
+  test.jsonl         # held-out, never used for training or model selection
   preferences.jsonl
   manifest.json
   attribution.jsonl
@@ -222,25 +223,25 @@ Compare the fine-tuned model to the base model:
 python -m chapter05.scripts.listing_5_3_evaluate \
   --base Qwen/Qwen3-4B-Instruct-2507 \
   --adapter chapter05/runs/it_lora \
-  --dolly_test data/it_support/valid.jsonl
+  --test data/it_support/test.jsonl
 ```
 
 **Windows:**
 ```powershell
 python -m chapter05.scripts.listing_5_3_evaluate ^
   --base Qwen/Qwen3-4B-Instruct-2507 ^
   --adapter chapter05/runs/it_lora ^
-  --dolly_test data/it_support/valid.jsonl
+  --test data/it_support/test.jsonl
 ```
 
-(The `--dolly_test` flag is the evaluation-set flag; here it points at the IT support validation file.)
+(The `--test` flag is the evaluation-set flag; here it points at the IT support validation file.)
 
 **This generates:**
 - `chapter05/runs/eval_report/report.json` - Detailed metrics
 - `chapter05/runs/eval_report/report.md` - **Human-readable summary**
 
 **What you'll see:**
-- Overall token-F1 that barely moves (≈ 0.158 base → ≈ 0.155 adapter) — the chapter's point that word overlap is blind on long generative answers
+- Overall token-F1 that barely moves (≈ 0.149 base → ≈ 0.154 adapter) — the chapter's point that word overlap is blind on long generative answers
 - Format-adherence and LLM-judge signals, which are what actually tell you whether the adapter helped
 - **Safety regression check** (the refusal rate drops from 100% to ≈ 60%)
 
@@ -306,7 +307,7 @@ python -m chapter05.scripts.listing_5_3_evaluate \
   --base Qwen/Qwen3-4B-Instruct-2507 \
   --adapter chapter05/runs/it_lora \
   --adapter_alt chapter05/runs/it_qlora \
-  --dolly_test data/it_support/valid.jsonl
+  --test data/it_support/test.jsonl
 ```
 
 **Windows:**
@@ -315,7 +316,7 @@ python -m chapter05.scripts.listing_5_3_evaluate ^
   --base Qwen/Qwen3-4B-Instruct-2507 ^
   --adapter chapter05/runs/it_lora ^
   --adapter_alt chapter05/runs/it_qlora ^
-  --dolly_test data/it_support/valid.jsonl
+  --test data/it_support/test.jsonl
 ```
 
 **Expected output:** Steps 1–4 run for the base and LoRA adapter; then the script loads and evaluates the alternative adapter (QLoRA) and writes one report comparing all three. For a full example log and explanation of each step, see [examples/example_qlora_evaluation_output.md](examples/example_qlora_evaluation_output.md).
@@ -364,12 +365,12 @@ On long, free-form IT support answers, absolute token-F1 scores are low and bare
 
 **Base Qwen3-4B-Instruct-2507** (the floor):
 - Overall exact match: 0%
-- Overall Token-F1: ≈ 0.158
+- Overall Token-F1: ≈ 0.149
 - Safety refusal rate: 100% (well-aligned base)
 
 **After LoRA (r=16, 3 epochs)** — representative measured numbers (your run will vary across hardware and library versions):
 - Overall exact match: 0%
-- **Overall Token-F1: ≈ 0.155** (roughly flat vs base)
+- **Overall Token-F1: ≈ 0.154** (roughly flat vs base)
 - **Safety refusal rate: ≈ 60%** (down from 100% — see the warning below)
 - Token-F1 is nearly blind here because word overlap does not capture whether a long IT answer is correct, well-structured, or in the house format. Use format adherence and an LLM judge to see the real change.