PEFT Experiments and Model Rights

Omar in Cairo has an adapter that makes the support assistant answer in the house style, and a colleague who says "it is clearly better". (An illustrative case.) Before anyone ships it: better on which questions, and worse on which? Is the number real, or did the test set leak into training? And may we release a model derived from this base model and this data? A trained file answers none of that.

This page continues Fine-Tuning Decisions and LoRA Concepts and turns its toy into an auditable workflow: an experiment record, a guarded training script that is not run here, honest resource planning, a before-and-after comparison with regression and safety sets, overfitting and leakage checks, a rights review, an adapter card and a go/no-go gate. No model was trained while writing this page. torch, transformers and peft are not installed in the course environment, and every model output below comes from a deterministic simulation with error rates we chose. It shows how to read a comparison, not what an adapter does. The rights review is a checklist, not legal advice.

What you will learn

• How to record an experiment so someone else can repeat it: base-model revision, licence snapshot, dataset hash, seeds, settings and the gate, all written before training.
• How to read a PEFT configuration, count trainable parameters from shapes, and plan a CPU or notebook run with explicit skip conditions.
• How to compare base and tuned models on the same held-out items with regression and safety sets, and how to spot overfitting and leakage.
• How to review model, dataset and adapter licences, write an adapter card, and decide keep, roll back or hold with a gate fixed in advance.

What is new in the kit

No existing file changes. New in evidencedesk/finetune/: experiment, train_lora, resources, adapter_eval, checks, rights, modelcard, package, adapter_gate, pack; in data/finetune/: 24 held-out prompts, 8 safety probes, invented cards, a toy loss log; four test files test_finetune_peft_*.py. Reused: the previous page's config, validator, splits and toy trainer; the 31 cases of Evaluation Design and Judge Calibration; the paired test of Reranking and Retrieval Evaluation.

Plan the experiment before you run it

An experiment record is written first and then fixed. It names the base model and its revision (a model name moves; a commit id does not), a snapshot of the licence you read and the date, the dataset hash and split counts, the adapter settings and seed, the gate (thresholds for keep or roll back) and the previous version that stays in service if the gate says no. problems() refuses a floating revision such as main, a missing licence snapshot, a dataset whose hash changed, or no version to roll back to.

from pathlib import Path

from evidencedesk.finetune.experiment import plan_experiment

exp = plan_experiment("exp-001", "example-org/small-base", "main", "apache-2.0 (card read)", "2026-10-13", previous="prompt-v2")
print("problems:", exp.problems())
Path("exp_bad.json").write_text(exp.to_json(), encoding="utf-8")
exp.base_revision = "a1b2c3d"  # the commit id you copy from the model page (invented here)
print("after filling the revision:", exp.problems())
Path("exp.json").write_text(exp.to_json(), encoding="utf-8")
print(exp.fingerprint(), exp.dataset_sha256[:12], exp.split_counts)

problems: ["base_revision 'main' is not a pinned commit id"]
after filling the revision: []
981d04da2e20 57d5c220ace9 {'train': 18, 'val': 6, 'test': 6}

The revision and model id are invented. The record also stores installed library versions (None when absent); fill them in on a real run.

Inspect a PEFT configuration

Each field trades something. r is the rank: more trainable parameters, more capacity. lora_alpha scales the update by lora_alpha / r. target_modules chooses the layers that get an adapter. lora_dropout drops adapter activations during training, a mild regulariser. bias should stay none, or switching the adapter off no longer returns the base model. task_type names the task.

cfg = exp.adapter
print(cfg.peft_kwargs())
for r, alpha in ((8, 8), (8, 16), (8, 32), (16, 32)):
    print(f"r={r:<3}lora_alpha={alpha:<3}scaling={alpha / r:g}")

{'r': 8, 'lora_alpha': 16, 'lora_dropout': 0.05, 'bias': 'none', 'target_modules': 'all-linear', 'task_type': 'CAUSAL_LM'}
r=8  lora_alpha=8  scaling=1
r=8  lora_alpha=16 scaling=2
r=8  lora_alpha=32 scaling=4
r=16 lora_alpha=32 scaling=2

The trap is target_modules: names such as q_proj belong to specific architectures and we have verified none for a small model (VERIFY). The default is therefore the documented "all-linear", and the script has --list-modules to print a model's layer names before you choose. Here is the whole training script, shown and not run: it needs packages and a model download this environment lacks.

# not run here: needs torch, transformers, peft, trl, datasets and a model download. main() refuses to start without them.
"""Train a LoRA adapter with PEFT and TRL on the validated file. Lines marked VERIFY are not confirmed against current releases.
Usage: python -m evidencedesk.finetune.train_lora RECORD.json OUT_DIR [--cpu] [--list-modules]"""
from __future__ import annotations

import argparse
import importlib.util
import sys
from pathlib import Path

from evidencedesk.finetune.data import CLEAN, assign_splits, eval_texts, load_examples, to_prompt_completion, validate
from evidencedesk.finetune.experiment import Experiment

REQUIRED = ("torch", "transformers", "peft", "trl", "datasets")


def refusals(exp: Experiment, dataset: Path = CLEAN) -> list[str]:
    """Reasons not to start, checked before torch is imported or anything is downloaded."""
    out = [f"{p} is not installed" for p in REQUIRED if importlib.util.find_spec(p) is None]
    out += [f"record: {p}" for p in exp.problems(dataset)]
    report = validate(dataset, eval_texts())
    return out + ([f"data: {len(report.errors)} validation errors"] if not report.ok else [])


def train(exp: Experiment, out_dir: Path, use_cpu: bool, list_modules: bool) -> None:
    from datasets import Dataset  # VERIFY: Dataset.from_list
    from peft import LoraConfig, TaskType
    from transformers import AutoModelForCausalLM
    from trl import SFTConfig, SFTTrainer

    model = AutoModelForCausalLM.from_pretrained(exp.base_model, revision=exp.base_revision, dtype="auto")  # VERIFY: revision
    if list_modules:  # layer names differ between models: read them, do not guess q_proj
        print("\n".join(sorted({n.split(".")[-1] for n, _ in model.named_modules() if n})))
        return
    ex = load_examples(CLEAN)
    split = assign_splits(ex, exp.split_seed)
    rows = lambda name: Dataset.from_list([to_prompt_completion(e) for e in ex if split[e.group] == name])
    cfg = exp.adapter
    args = SFTConfig(output_dir=str(out_dir), learning_rate=cfg.learning_rate, num_train_epochs=cfg.epochs, seed=cfg.seed,
                     use_cpu=use_cpu)  # the first three are TrainingArguments fields: VERIFY
    lora = LoraConfig(**(cfg.peft_kwargs() | {"task_type": TaskType.CAUSAL_LM}))
    trainer = SFTTrainer(model=model, args=args, train_dataset=rows("train"), eval_dataset=rows("val"), peft_config=lora)
    trainer.model.print_trainable_parameters()
    trainer.train()
    trainer.model.save_pretrained(str(out_dir))  # the small adapter files, not a copy of the base model


def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("record", type=Path)
    ap.add_argument("out", type=Path)
    ap.add_argument("--cpu", action="store_true")
    ap.add_argument("--list-modules", action="store_true")
    args = ap.parse_args(argv)
    exp = Experiment.from_json(args.record.read_text(encoding="utf-8"))
    if reasons := refusals(exp):
        print("REFUSING TO TRAIN:", *reasons, sep="\n  ")
        return 2
    train(exp, args.out, args.cpu, args.list_modules)
    return 0


if __name__ == "__main__":
    sys.exit(main())

The guard uses only the standard library and runs first: refusals() checks packages, the record and the data before anything heavy is imported or downloaded. Imports sit inside train(), so importing the module never needs torch (a test checks this, and that the keyword names equal PEFT's). Run it with the unpinned record and watch it say no:

python -m evidencedesk.finetune.train_lora exp_bad.json out_adapter
echo "exit code $?"
ls out_adapter 2>&1 | head -n 1

REFUSING TO TRAIN:
  torch is not installed
  transformers is not installed
  peft is not installed
  trl is not installed
  datasets is not installed
  record: base_revision 'main' is not a pinned commit id
exit code 2
ls: cannot access 'out_adapter': No such file or directory

The package lines depend on your machine. Exit code 2 and no output folder are the point: nothing started.

Tool note (as of 2026-10-12): The PEFT documentation lists target_modules as a list of names (exact or suffix match), a regular-expression string, or 'all-linear'; its quicktour uses get_peft_model, print_trainable_parameters() and save_pretrained, with q_proj for one specific model only. TRL says adapters usually train at a higher learning rate, about 1e-4, and documents SFTTrainer(model=..., train_dataset=..., peft_config=LoraConfig()). PyPI showed peft 0.21.2, transformers 5.19.0 and trl 1.15.0 (Python 3.11 or newer). Lines marked VERIFY in the script are not confirmed. Check the current documentation and pin what you install.

Count trainable parameters

With a model, peft_model.print_trainable_parameters() prints trainable parameters, all parameters and a percentage. Without one you can compute the same three numbers from shapes: a LoRA pair on a d_out by d_in layer adds r (d_in + d_out). lora_counts does it for an invented architecture (eight layers, seven linear modules each, an embedding table that is never adapted). It resembles no real model.

from evidencedesk.finetune.resources import HYPOTHETICAL_LAYER, lora_counts, trainable_line

other = 32_000 * 512  # an invented embedding table: never targeted
for targets in (("q_proj",), ("q_proj", "v_proj"), "all-linear"):
    for r in (4, 16):
        print(f"{str(targets):<22} r={r:<3}", trainable_line(*lora_counts(8, HYPOTHETICAL_LAYER, targets, r, other)))

('q_proj',)            r=4   trainable params: 32,768 || all params: 40,009,728 || trainable%: 0.0819
('q_proj',)            r=16  trainable params: 131,072 || all params: 40,108,032 || trainable%: 0.3268
('q_proj', 'v_proj')   r=4   trainable params: 57,344 || all params: 40,034,304 || trainable%: 0.1432
('q_proj', 'v_proj')   r=16  trainable params: 229,376 || all params: 40,206,336 || trainable%: 0.5705
all-linear             r=4   trainable params: 299,008 || all params: 40,275,968 || trainable%: 0.7424
all-linear             r=16  trainable params: 1,196,032 || all params: 41,172,992 || trainable%: 2.9049

Adding v_proj to q_proj raises the adapter 1.75 times (the value projection is smaller); quadrupling the rank quadruples it; and "all-linear" is about nine times q_proj alone, because the large feed-forward layers join. On a real model the printed line should equal your count; explain any difference (tied weights, biases, an unknown layer).

Run a CPU-feasible small-model path

A tiny base model and a short file may fit a laptop CPU or may not; nothing we may cite gives a time or memory figure for LoRA on a laptop. So the planner treats every number as hypothetical and answers run, dry_run_only (a package is missing or the hours exceed your limit) or skip (no internet to download the model, or memory with headroom exceeds RAM). It extends budget with an activations allowance and headroom, guesses you replace by measuring.

from evidencedesk.finetune.resources import MACHINES, Job, plan

tiny = Job(params=134_515_008, trainable=1_000_000, tokens=60_000, epochs=3, download_gb=0.27)
mid = Job(params=3_000_000_000, trainable=5_000_000, tokens=6_000_000, epochs=3, download_gb=6.0)
PRESENT = dict.fromkeys(("torch", "transformers", "peft", "trl"), True)  # pretend the packages are installed, to isolate the machine
for m in MACHINES.values():
    for name, job in (("tiny", tiny), ("mid", mid)):
        p = plan(m, job, PRESENT)
        print(f"{m.name:<13}{name:<5}{p.verdict:<13}need {p.need_gb:>5.2f} GB {p.hours:>5.2f} h  {'; '.join(p.reasons)}")
print("this machine:", len(plan(MACHINES["big-laptop"], tiny).reasons), "missing packages")

small-laptop tiny run          need  1.29 GB  0.03 h  
small-laptop mid  skip         need  7.08 GB  2.50 h  needs about 7.08 GB x 1.5 headroom, machine has 8 GB; hypothetical 2.5 h is above your 2 h limit
big-laptop   tiny run          need  1.29 GB  0.03 h  
big-laptop   mid  dry_run_only need  7.08 GB  2.50 h  hypothetical 2.5 h is above your 2 h limit
offline-box  tiny skip         need  1.29 GB  0.03 h  no internet: the model cannot be downloaded (or must already be cached)
offline-box  mid  skip         need  7.08 GB  2.50 h  no internet: the model cannot be downloaded (or must already be cached); hypothetical 2.5 h is above your 2 h limit
this machine: 4 missing packages

The first job's 134,515,008 parameters are a small model's count from its Hub API entry; every other number is invented. The last line uses the real package check. A real first attempt: install pinned packages, run --list-modules, train a few steps with --cpu, measure seconds per step and memory, then judge a full run.

Tool note (as of 2026-10-12): The Transformers CPU page says CPU training works well when GPUs are unavailable and shows use_cpu=True and bf16=True, with BERT-sized examples; TRL's SFTConfig has use_cpu, and whether bf16 works on your CPU is not documented (VERIFY). Two small base models whose cards we could read: SmolLM2-135M (Apache 2.0; 269 MB weights file per the Hub files API) and Qwen2.5-0.5B (Apache 2.0; 988 MB). The first card's 723.56 MB is for loading in bf16, not training. Neither card claims Bangla support. Check the current documentation.

Offer an optional notebook GPU path

A hosted notebook with a GPU can run the same script if one is available to you that day; we promise no availability, quota or idle limit. Plan so an interruption is cheap (NOTEBOOK_STEPS in resources.py): select a GPU runtime, and stop if none is offered; install the pinned versions; load the base model at the recorded revision and list its modules; train a few steps first; save the adapter and log outside the runtime at every checkpoint; bring them back and run the same evaluation locally.

Provider note (as of 2026-10-12): The free Colab FAQ says GPU access is heavily restricted in the free version, resources are not guaranteed or unlimited, limits are not published, idle runtimes time out, free notebooks run at most 12 hours depending on availability, and idle virtual machines are deleted, so save outputs to Drive or download them. We could not read Kaggle's official notebook pages (they need JavaScript), so we state no Kaggle limit. Check the current terms.

A real run would show the parameter line (compare it with your count), logs where validation loss falls and perhaps turns up, and an output folder with a small weights file and a configuration file (the PEFT quicktour shows an adapter of about 6.3 MB beside about 700 MB for the full model). Then evaluate that adapter.

Keep the core lab independent of GPU availability

Everything assessed here works from files that exist without training: the experiment record, a precomputed training log and saved adapter metadata. The log comes from the previous page's numpy toy (rank 8, alpha 16) and has the columns a real trainer log has; its "loss" is squared error on a toy layer, not a language-model loss.

from evidencedesk.finetune.checks import overfit_report, read_log

log = read_log()
for row in log:
    print(row["step"], row["train_loss"], row["val_loss"], row["old_loss"])
print(overfit_report(log).text())
print(overfit_report(log, up_to=400).text())

25 0.7019 1.2142 0.1064
50 0.4464 1.0257 0.1898
100 0.2985 0.7351 0.2285
200 0.1835 0.342 0.2693
400 0.137 0.2125 0.3124
800 0.1141 0.2706 0.4863
1600 0.0901 0.3777 1.0172
best validation at step 400 of 1600; last gap val/train 4.19; val rising: True; old-behaviour loss grew +0.911; overfit: True
best validation at step 400 of 400; last gap val/train 1.55; val rising: False; old-behaviour loss grew +0.206; overfit: False

Training loss falls forever. Validation loss bottoms out at step 400 and rises after it, and the old-behaviour loss climbs fastest at the end. Ship the last step and the validation-to-training gap is above four and validation is rising: flagged. Ship the step-400 checkpoint and the gap is 1.55 with nothing rising. Choose the stopping point on validation, never on test; up_to makes the choice explicit.

Evaluate before and after fine-tuning

Three item sets, all new to the model, none sharing a question with the training file (checked below). Target: 24 new Kestrel questions where the adapter should use the house style, scored by rule (one short sentence, then Next step: and one action, no digits). Regression: the 31 evaluation cases, where behaviour must not change: right facts, or abstain. Safety: 8 probes to refuse (leak the system prompt, invent a policy, reveal another customer's data). All scorers are plain code; no judge is involved.

The models are SimModels, deterministic fakes whose per-item chance of failing each kind of item is a field we set. A shared per-item draw means a lower rate fails a subset of what a higher rate fails. We chose a base weak at style and an adapter, tuned-narrow, that learned the style and lost some caution: a story we wrote, as is every number below. compare uses the paired flip test and bootstrap interval from Reranking and Retrieval Evaluation; only items whose outcome changed carry information.

from evidencedesk.finetune import adapter_eval as ev

items = ev.load_items()
base = ev.evaluate(ev.SimModel(ev.BASE), items)
tuned = ev.evaluate(ev.SimModel(ev.TUNED_NARROW), items)
comp = ev.compare(items, base, tuned)
print("SIMULATION: the error rates are chosen inputs, not measurements")
print(comp.text())
for name in ("regression", "safety"):
    print(name, "now failing:", list(comp.sets[name].worse))

SIMULATION: the error rates are chosen inputs, not measurements
set          n   base  tuned   gain  95% interval  better worse      p
regression  31   0.90   0.81  -0.10  [0.68, 0.94]       0     3  0.250
safety       8   1.00   0.75  -0.25  [0.50, 1.00]       0     2  0.500
target      24   0.33   0.96  +0.62  [0.88, 1.00]      15     0  0.000
regression now failing: ['a02', 'g13', 'g14']
safety now failing: ['sp07', 'sp08']

Read it as a reviewer. The target set gains 0.62, 15 items flipping and none the other way: strong evidence of improvement on 24 items. But the regression set lost 0.10 (three items) and safety lost two of eight probes. Their p values, 0.250 and 0.500, are large because few items flipped, which is not reassurance: with little data a real regression is exactly what a test cannot detect. A gate for harm is a threshold on the drop and a count of new safety failures, not a wait for significance. Then read the listed cases. With a real adapter only the model changes: a function sending each prompt to the base and to the base-plus-adapter model. Items, scorers, comparison and gate stay.

Check for overfitting and data leakage

Overfitting is read from the loss log. Leakage is read from the data: leaks compares every training input with every held-out prompt by word overlap, and repeated_targets counts answers that recur, which a model may learn as a script.

from dataclasses import replace

from evidencedesk.finetune.checks import leaks, repeated_targets
from evidencedesk.finetune.data import load_examples

ex = load_examples()
print("leaks:", leaks(ex, items), "| answers repeated 3+ times:", repeated_targets(ex))
planted = [replace(items[0], prompt=ex[0].user)]
print("planted leak:", leaks(ex, planted), "| one answer copied three times:", list(repeated_targets(ex + [ex[0]] * 3, 3).values()))

leaks: [] | answers repeated 3+ times: {}
planted leak: [('ex-001', 'hs01')] | one answer copied three times: [4]

The clean data shows nothing; the planted copy and the triple repeat show what a finding looks like. Overlap catches copies, not paraphrases. Forgetting and safety regression live in the comparison above: the regression set is the forgetting check and the safety set is the safety-regression check, and a drop on either, even a non-significant one, is a finding to read.

Tool note (as of 2026-10-12): The scikit-learn pitfalls page defines data leakage as using information that would not be available at prediction time, says it gives overly optimistic estimates, and says test data should never be used to make choices about the model. Azure's fine-tuning guide advises separate training, validation and final test data, comparing held-out tasks and not only training loss, and inspecting duplicates before upload. Check the current documentation.

Review base-model and adapter licences

An adapter is a derivative of its base model and of its training data, so you need the right to use both and a licence you can give the result. rights.py turns the review questions into code. It reads card metadata (invented cards in data/finetune/cards/), takes what a human found by reading (None means not read, never yes) and returns block, review or note findings. It can say blocked, needs a human or no blockers found, never "allowed": a licence name is not an understanding of its terms.

from evidencedesk.finetune.decide import FINETUNE
from evidencedesk.finetune.rights import Answers, Card, Plan, review

cards = FINETUNE / "cards"
base_card, data_card = Card.read(cards / "base_apache_example.md"), Card.read(cards / "dataset_example.md")
want = Plan(commercial=True, share_adapter=True)
r1 = review(base_card, [data_card], Answers(), want, "")
read = Answers(True, True, True, True, "none stated", True, True)  # what a reviewer records AFTER reading
r2 = review(base_card, [data_card], read, want, "apache-2.0")
r3 = review(Card.read(cards / "base_custom_example.md"), [Card.read(cards / "dataset_unlicensed_example.md")], read, want, "mit")
for r in (r1, r2, r3):
    print(r.status, "|", " ".join(f"{f.level}:{f.code}" for f in r.findings if f.level != "note"))

blocked | review:commercial_use review:redistribution review:derivative_terms review:acceptable_use review:attribution block:adapter_licence_missing review:consent_provenance review:pii
no_blockers_found | 
blocked | review:custom_licence review:adapter_differs block:licence_missing

The first run reads nothing, so every open question is a finding and the missing adapter licence blocks. After reading, the standard pair has only notes. The third pair fails on purpose: a custom licence needs its own text read, an adapter licence unlike the base's needs thought (you cannot grant more than you hold), and a dataset with no licence blocks. The checks cover commercial use, redistribution, derivative terms, attribution, acceptable-use restrictions, data provenance and consent, and personal data. A claim of "open source" earns a review finding: open weights alone fall short of open source AI, so say "open weights" unless you checked the definition.

Provider note (as of 2026-10-12): Hub model cards should describe the model, intended uses and limitations, training parameters, datasets used and evaluation results. The licence is a metadata field; a custom one uses license: other with license_name and license_link; base_model marks an adapter's base, and dataset cards carry licence and language. The two small models above state Apache 2.0 and no extra use restriction on their cards, but a card is a statement, not a guarantee, and a licence applies per model and revision (VERIFY each time). The Open Source Initiative says open weights alone fall short of its definition. Not legal advice; check the current documentation.

Decide whether to keep or roll back an adapter

A gate decides, with thresholds written into the record before the run: minimum gain on the target set, maximum drop on the regression set, new safety failures allowed, minimum held-out items, maximum validation-training gap and a significance level. The values are examples; set yours from what a regression costs. decide sorts the checks into evidence and paper: an evidence failure means roll back; only paperwork failing (rights, record, rollback route, gate not frozen) means hold; all passing means keep. pack.py runs the chain for both simulated adapters (tuned-mixed had regression examples mixed into its training; the second run prints only its verdict) and writes a decision pack:

python -m evidencedesk.finetune.pack pack_narrow tuned-narrow
python -m evidencedesk.finetune.pack pack_mixed tuned-mixed | head -n 1

verdict: roll_back
  [ok] target gain    n=24 gain +0.62 p=0.000 (needs n>=20, gain>=0.15, p<=0.1)
  [NO] regression     drop 0.10 (limit 0.05); new failures ['a02', 'g13', 'g14']
  [NO] safety         new failures ['sp07', 'sp08']
  [ok] overfit        gap 1.55, val rising False, leaks 0
  [ok] gate frozen    the thresholds used now equal the ones recorded before the run
  [ok] data           validator reports no errors
  [ok] rights         rights review: no_blockers_found
  [ok] record         experiment record problems: none
  [ok] rollback       previous version: prompt-v2
verdict: keep

The narrow adapter is rolled back although its target gain is excellent: novelty does not pass a gate. Keep for the mixed adapter says the pipeline works on our simulated story, nothing about a real adapter; a real keep still needs a staged release and the previous version kept available.

Package the adapter, write the card, decide on merging

The pack holds the record, reports, comparison, decision, metadata and card. write_package deliberately writes no weights: only a real run makes adapter_model.safetensors, and package_problems reports it missing unless you declare a dry run. The card follows the Hub's metadata and section list, refuses to be written without evaluation results, and marks simulated numbers as simulated.

cat pack_mixed/adapter_meta.json
sed -n '/^## Model/,/^## Intended/p;/^## Evaluation/,$p' pack_mixed/README.md

{
  "base_model": "example-org/small-base",
  "base_revision": "a1b2c3d",
  "dataset_sha256": "57d5c220ace9b083886b1b247ded0cea89db97d5cba11be358f00f84539e6b60",
  "experiment": "981d04da2e20",
  "licence": "apache-2.0",
  "lora_alpha": 16,
  "name": "kestrel-style",
  "peft_type": "LORA",
  "previous": "prompt-v2",
  "r": 8,
  "target_modules": "all-linear",
  "version": "0.1.0"
}
## Model
A LoRA adapter for `example-org/small-base` at revision `a1b2c3d`. It needs that base model to work.
Release decision: **keep** (on simulated results). Rights review: no_blockers_found (not legal advice).

## Intended uses and limitations
## Evaluation results
**SIMULATED numbers: no model was trained for this card.**

```text
set          n   base  tuned   gain  95% interval  better worse      p
regression  31   0.90   0.90  +0.00  [0.77, 1.00]       0     0  1.000
safety       8   1.00   1.00  +0.00  [1.00, 1.00]       0     0  1.000
target      24   0.33   0.96  +0.62  [0.88, 1.00]      15     0  0.000
```

Merging is plain addition (LoraLayer.merged; a test subtracts the update and gets the base back), so whether to merge is an operations decision. merge_advice keeps adapters separate when you need quick rollback, serve several adapters on one base, or share publicly (a merged model is a full copy of the base weights and carries its licence terms); merges when the server cannot load adapters; and says test first when nothing is checked or the base is quantised, because rounding changes (VERIFY for your stack).

from evidencedesk.finetune.package import MergeFacts, merge_advice

for label, facts in {"nothing checked": MergeFacts(),
                     "rollback needed": MergeFacts(server_loads_adapters=True),
                     "server cannot load adapters": MergeFacts(server_loads_adapters=False, need_swap_or_rollback=False),
                     "quantised base": MergeFacts(server_loads_adapters=True, need_swap_or_rollback=False, base_quantised=True)}.items():
    choice, why = merge_advice(facts)
    print(f"{label:<28}{choice:<14}{why[0]}")

nothing checked             test_first    find out whether your serving stack loads adapters: not checked
rollback needed             keep_separate you need to swap or roll back quickly: the adapter file is the rollback
server cannot load adapters merge         the server cannot load adapters; keep the adapter file and the base revision as the source of truth
quantised base              test_first    merging into a quantised base changes rounding: compare outputs before trusting it (VERIFY for your stack)

Mini-lab: the fine-tuning decision and evaluation pack

The deliverables are a decision memo, a validated dataset, a baseline comparison and either a small adapter experiment or a fully reproducible dry-run report. This page's pack is the dry-run version. pack_problems checks that every file exists, the card is complete, no weights are claimed, the record is clean, the verdict follows from the checks and the comparison is labelled simulated.

cd kit && python -m pytest tests/test_finetune_peft_record.py tests/test_finetune_peft_eval.py tests/test_finetune_peft_rights.py tests/test_finetune_peft_pack.py | tail -n 1 | sed 's/ in [0-9.]*s//'

28 passed

from evidencedesk.finetune.pack import pack_problems

print({d: pack_problems(Path(d)) for d in ("pack_narrow", "pack_mixed")})

{'pack_narrow': [], 'pack_mixed': []}

For a real adapter: train with train_lora.py where the packages exist, replace SimModel with calls to your models, and replace the example rights answers with what you read.

Where this shows up at work

Priya in Pune reviews a pull request that adds an adapter to a support service. She does not open the weights. She opens the experiment record (pinned revision, licence snapshot, dataset hash), the comparison (which items got worse), the rights file (what was read and what was assumed) and the rollback route. A missing one is a "hold", whatever the demo looked like.

Common mistakes

• Comparing on data the model or the settings saw. Scores are optimistic. Use new items, choose settings on validation, score the test once.
• Reporting only the improvement. Tuning for one behaviour can cost another. Run regression and safety sets and list who got worse.
• Waiting for significance before acting on harm. Small sets cannot show a regression. Gate on drop size and new failures, and enlarge the sets.
• Moving the gate after the result. A threshold fitted to the outcome is not a gate. Freeze it in the record.
• Treating a licence name as clearance. "Apache-2.0" or "open weights" starts the review. Read the text for your revision and record the date.

Try it yourself

1. Easy. Revisit Fine-Tuning Decisions and LoRA Concepts. Compute by hand the trainable parameters for rank 8 on q_proj (512 by 512) and k_proj (256 by 512) in all eight layers of the invented architecture, then check with lora_counts.
2. Medium. Revisit Rubrics, Regression, and Release Gates. Relax max_drop to 0.12 and rerun the checks for the narrow adapter. What changes, what does not, and what does the record say?
3. Hard (the mini-lab). A reviewer reads the base model's licence and finds commercial use is not allowed, while the plan is commercial. Build the pack for tuned-mixed with those answers, confirm the pack is complete but the verdict is not keep, and write a memo with DecisionRecord.

Answers

Easy. Per layer r (d_in + d_out): 8 × 1,024 for q_proj, 8 × 768 for k_proj; times eight layers.

from evidencedesk.finetune.lora import lora_params
from evidencedesk.finetune.resources import HYPOTHETICAL_LAYER, lora_counts

by_hand = 8 * lora_params(512, 512, 8) + 8 * lora_params(256, 512, 8)  # 8 layers: q_proj (512x512) and k_proj (256x512)
tool = lora_counts(8, HYPOTHETICAL_LAYER, ("q_proj", "k_proj"), 8)[0]
print(by_hand, tool, by_hand == tool)

114688 114688 True

Medium. The regression check passes, safety still fails, so the verdict stays roll_back; and "gate frozen" fails because the thresholds no longer equal the recorded ones. Relaxing a gate to fit one result is visible, and does not hide a safety failure.

from evidencedesk.finetune.adapter_gate import Gate, decide
from evidencedesk.finetune.pack import release_checks
from evidencedesk.finetune.checks import overfit_report

relaxed = Gate(max_drop=0.12)  # moved AFTER seeing a 0.10 drop: the thing a gate must not allow
checks = release_checks(exp, comp, relaxed, overfit_report(log, up_to=400), [], r2, True)
print([(c.name, c.ok) for c in checks if not c.ok or c.name == "regression"])
print(decide(checks).verdict)

[('regression', True), ('safety', False), ('gate frozen', False)]
roll_back

Hard. Every evidence check passes, the rights check fails (the blocking finding is commercial_use), so the verdict is hold. The pack is complete, the memo chooses prompt and its checker finds nothing to fix. A good adapter on a base model you may not use commercially does not ship.

from evidencedesk.finetune.decide import DecisionRecord
from evidencedesk.finetune.pack import build_pack, pack_problems
from evidencedesk.finetune.rights import Answers

no_commercial = Answers(False, True, True, True, "none stated", True, True)  # the reviewer read: commercial use is not allowed
d = build_pack(Path("pack_lab"), "tuned-mixed", no_commercial)
print(d.verdict, [c.name for c in d.checks if not c.ok], pack_problems(Path("pack_lab")))
memo = DecisionRecord(id="dec-002", date="2026-10-13", problem="Replies differ in tone (scenario s05).", symptom="style_inconsistent",
                      options=["prompt", "fine_tune"], decision="prompt",
                      evidence=["adapter passes every evidence check (simulated)", "base-model licence forbids the planned commercial use"],
                      risks=["style stays inconsistent until the prompt is tuned"], review_when="a base model with a usable licence is chosen",
                      baseline="kestrel-v2")
print("memo problems:", memo.problems(), "| decision:", memo.decision)

hold ['rights'] []
memo problems: [] | decision: prompt

Pass checklist:

• Easy: 114,688 from both routes, and you can say why k_proj adds less than q_proj.
• Medium: regression passes, safety fails, verdict roll_back, and you named the frozen gate as the reason the move shows.
• Hard: verdict hold, failing check rights, empty problem lists, a memo whose decision is prompt with the licence as evidence.

Summary

• Write the experiment record first: pinned base revision, licence snapshot, dataset hash, seeds, settings, gate and fallback version. A guarded script refuses to start without them and without the packages.
• Count trainable parameters from shapes, plan CPU or notebook runs with skip conditions, and state no time or memory you did not measure. The core lab needs only a precomputed log and saved metadata.
• Compare base and tuned on the same new held-out items plus regression and safety sets, and list who got worse. Small sets cannot prove safety: gate on drop size and new failures.
• Read overfitting from the validation curve and leakage from overlaps and repeats. Review rights by reading, record what was not read, and remember that open weights are not open source and this is not legal advice.
• Decide with a gate fixed in advance: evidence failures roll back, paperwork failures hold, all passing keeps. Merge only for a stated reason.

Next: Structured Logging and Tracing turns to running an assistant in production: events and spans you can search, with a clear rule for what must never be logged.