wshobson / lora-qlora-recipes

LoRA & QLoRA Recipes

This skill assumes the routing decision already
happened — finetuning-method-selection should
have already pointed here because the data shape
is demonstrations (SFT), not preference pairs or
a verifiable reward signal. What follows is the
current best-practice recipe for configuring the
adapter itself: which modules to target, how to
size rank and alpha, what learning rate to use,
and when QLoRA buys real headroom versus when it
just adds risk. Dataset preparation and quality
checks are a separate concern — see
dataset-curation.

Input: a routing decision (SFT via LoRA/
QLoRA) plus a target size class.
Output format: a validated adapter config —
the kwarg values below, not free-form advice —
that llm-finetuning-training-engineer consumes
directly when it generates a runnable script.

The Reference Recipe

The reference recipe is "LoRA Without Regret"
(Thinking Machines/Schulman, 2025-09), now the
settled convention for LoRA/QLoRA SFT.

Target Modules

Target all-linear modules, not just
attention:

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",   # attention
    "gate_proj", "up_proj", "down_proj",      # MLP — matters most
]

The MLP layers (gate_proj, up_proj,
down_proj) matter most — attention-only
targeting was the older, weaker convention.
Dropping modules to save memory is a Failure
Mode below, not a valid optimization.

Alpha and Learning Rate

• lora_alpha = 2 * r is the settled
  convention (NeurIPS 2025 "intruder dimensions"
  result). Don't hand-tune alpha independently of
  rank — derive it from rank every time.

• **LoRA learning rate ≈ 10x the equivalent
  full-fine-tune LR.** For QLoRA specifically,
  2e-4 is the standard starting point. Full
  hyperparameter tables and worked examples:
  references/hyperparameters.md.

Rank by Task

Rank is task-shaped, not a single global default:

Higher rank isn't automatically better — it
raises capacity to memorize as fast as it raises
capacity to generalize. Start at the row matching
the task, and only move up a row if the lower
rank measurably underfits on held-out eval, not
as a default hedge.

Effective Batch Size

Keep effective batch size under 32. This
recipe was validated at that scale — pushing
effective batch higher is an untested
extrapolation, not a free throughput win.

Unsloth Defaults

Unsloth is the reference implementation this
plugin assumes as the default fast path — except
for messages-shaped conversational SFT with
assistant_only_loss=True, where Unsloth
2026.7.x's compiled trainer has no messages-shaped
path at all and the plain-TRL escape hatch
(references/unsloth-trl-mapping.md) is the
default for that combination, not a rare-regression
fallback. Its out-of-the-box defaults, and why
each one is set that way:

• lora_dropout=0 — the optimized kernel
  path assumes zero dropout; setting a nonzero
  value forfeits the fused-kernel speedup.

• bias="none" — bias terms add adapter
  parameters for negligible quality gain at this
  rank range.

• use_gradient_checkpointing="unsloth" —
  Unsloth's checkpointing variant, not vanilla HF
  checkpointing; saves roughly 30% VRAM over
  no checkpointing.

• optim="adamw_8bit" — 8-bit AdamW cuts
  optimizer-state memory with negligible quality
  impact at LoRA/QLoRA adapter scale.

• random_state fixed — pins LoRA
  initialization for reproducibility across runs;
  treat it like any other seed, not a tunable.

These show up together on the get_peft_model
call:

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=target_modules,
    lora_alpha=64,               # 2 * r
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

Exact kwarg names and their plain-TRL/PEFT
equivalents, plus a full worked config including
SFTConfig: references/unsloth-trl-mapping.md
and references/hyperparameters.md.

LoRA vs QLoRA vs Full FT

• QLoRA = NF4-quantized frozen base weights +
  BF16 adapters. This is what makes a 65B-class
  model trainable on 48GB — the quantized base
  is the memory win, not the adapter itself.

• Full fine-tuning is not a default. Reserve
  it for dense knowledge injection where the goal
  is changing what the model knows at the weight
  level, not adapting a behavior. For everything
  else in this skill's scope, LoRA or QLoRA is
  the starting assumption.

• **On DGX Spark, QLoRA can OOM before an
  equivalent bf16 LoRA run would**, even though
  QLoRA's steady-state footprint is smaller —
  bitsandbytes dequantization buffers are
  transient CUDA-side allocations that spike
  during load. A QLoRA OOM is not proof the model
  doesn't fit; the dgx-spark-ops plugin's
  spark-memory-thermal-ops skill covers the
  full OOM remediation ladder (bf16 LoRA is the
  next thing to try, not a further QLoRA
  shrink).

Failure Modes

• fp16 divergence on non-BF16 GPUs. Training
  in fp16 on hardware that doesn't have solid
  BF16 support is a known source of loss spikes
  and silent divergence. Force bf16=True
  wherever the hardware supports it; don't fall
  back to fp16 as if it were equivalent. Check
  hardware support before picking a dtype:

  python -c "import torch; print(torch.cuda.is_bf16_supported())"

• Rank too high on a small dataset overfits.
  A rank picked for "SFT at scale" (up to ~256)
  on a dataset that doesn't have scale behind it
  memorizes rather than generalizes. Match rank
  to the Rank by Task table above, not to the
  largest number available.

• **Removing target modules to save memory costs
  quality for negligible savings.** The adapter
  parameters on gate_proj/up_proj/down_proj
  are a small fraction of total model size — cutting
  them barely moves memory but measurably hurts
  quality. If memory is tight, move to QLoRA or
  reduce rank/batch/pack length before trimming
  target modules.

All three failure modes share a pattern: they
look like a training-loop bug (loss spikes,
plateaus, memorization) but are actually a
config choice that contradicts the reference
recipe above. Check configuration against this
skill before debugging the training loop itself.

References

• references/hyperparameters.md — full rank/
  alpha/LR tables by task type, rsLoRA notes,
  batch/packing interactions, and a complete
  worked Unsloth config block.

• references/unsloth-trl-mapping.md — every
  Unsloth kwarg mapped to its TRL/PEFT
  equivalent, current TRL API notes, and the
  escape-hatch rule for when to drop back to
  plain TRL.

Related skills: finetuning-method-selection
routes here; dataset-curation covers the data
side this skill doesn't; llm-finetuning-training-engineer
is the downstream consumer of the config this
skill produces.