Parameter-Efficient Fine-Tuning

What actually changes, and what does not.

Full fine-tuning updates every weight in the model. Parameter-efficient fine-tuning (PEFT) freezes all of them and trains a small set of new parameters alongside. LoRA — low-rank adaptation, and the only variant most teams need — is the dominant form: for a weight matrix W, it learns two skinny matrices A and B and computes y = Wx + (α/r)·BAx. The base is untouched; the adapter is the difference.

• The parameter count collapses. For a 4096×4096 projection, full fine-tuning trains 16.8 million weights. At rank 16, LoRA trains 16 × (4096 + 4096) = 131,072 — about 0.8% of them. Across a whole model this typically lands between 0.1% and 2% of parameters trained.
• The memory saving is mostly from the optimiser, not the weights. Adam keeps two moment estimates per trainable parameter. Training 1% of the weights removes 99% of the optimiser state and 99% of the gradient buffers, which is the bulk of what makes full fine-tuning need a cluster. QLoRA goes further by holding the frozen base in 4-bit while keeping the adapter in bf16 — the combination is what put fine-tuning a 70B model on a single card.
• The artefact is a file, not a model. A trained adapter is tens to a few hundred megabytes. You can store thousands of them, version them in ordinary object storage, and ship one without shipping weights you may not be licensed to redistribute.
• Inference cost is unchanged. The adapter adds a small extra matrix multiply per layer — single-digit percent latency, and zero if you merge it back into the base. PEFT is a training-time and storage-time saving that does not tax you at serving time.

Rank is a capacity dial, and it decides what you are able to teach.

The rank r sets how much information the adapter can hold. This is the single setting that most often explains a disappointing result, because the failure looks like "fine-tuning didn't work" rather than "the container was too small".

• Rank 8–16 teaches behaviour. Output format, tone, refusal boundaries, a schema the model keeps drifting from, a house style, a tool-calling convention. This is what the overwhelming majority of production LoRAs do, and it is where the technique is at its most reliable — the model already knows how to do the task and you are moving where it lands.
• Rank 64–256 is where you start teaching content. A new domain vocabulary, an unusual reasoning pattern, a language the base handles poorly. It costs more to train and more to store, and it starts to need real data volume rather than a few hundred examples.
• No rank teaches facts economically. If the model needs to know your product catalogue, your incident history, or anything that changes, the answer is retrieval, not weights — the whole argument in fine-tuning, RAG, or prompting. Teams reach for rank 128 to force facts in, get a model that is confidently wrong in a new way, and conclude the method is broken.
• Alpha is a scale, not a second capacity knob. The α/r factor scales the adapter's contribution. The common convention is α = 2r; changing rank without adjusting alpha silently changes the effective learning rate, which is a frequent cause of "it worked at rank 16 and broke at rank 64".
• Which layers you attach to matters more than most people expect. The original recipe adapted only the attention query and value projections. Current practice targets every linear layer including the feed-forward blocks, which costs more parameters and consistently closes more of the gap to full fine-tuning.

The property that changes system design: one base, many adapters.

Here is the part that matters for agents rather than for training budgets. Because the base weights are shared and untouched, a serving stack can hold one copy of the model in GPU memory and apply a different adapter per request. Multi-adapter serving is a standard feature of modern inference engines, built on batching kernels that let requests using different adapters share the same forward pass.

• Specialisation stops being a deployment. Ten fine-tuned variants used to mean ten model deployments and ten times the memory. Now it means ten files and one deployment, and the choice of which to apply is made at request time — the same shape as model routing, but one layer down.
• Per-tenant and per-task adapters become affordable. An adapter per customer, per document type, or per stage of an agent loop is a reasonable design when the marginal cost is a file and a few hundred megabytes of pooled memory. It is an absurd one when it is a GPU each.
• Merge or don't merge is a real trade. Merging the adapter into the base removes the extra multiply and gives you a plain model — best when you have exactly one specialisation and want maximum throughput. Keeping it separate preserves hot-swapping and lets one fleet serve everybody. You cannot have both on the same fleet.
• The ranks have to be compatible. A serving cluster is configured with a maximum adapter rank, and adapters above it will not load. This is the most common day-one failure when a training team and a serving team pick their defaults independently.
• The base version is now a shared dependency. Every adapter is trained against exact base weights and is not portable to a different checkpoint. When the base is deprecated, every adapter is invalidated at once — the retraining event described in model deprecation and migration, with a multiplier on it.

When it is the right tool, and what it costs after you ship.

PEFT is cheap to start and not cheap to own. The training run is an afternoon; the obligations last as long as the adapter is in production.

• Reach for it when the base model can do the task but will not do it consistently. Format adherence that survives long contexts, a domain register, a tool-calling style, reliable structured output on a schema the model keeps bending. These are behaviour problems and behaviour is what adapters move.
• Do not reach for it before prompting has been tried properly. A prompt change ships in minutes and is reversible; an adapter is a dataset, a training run, an eval, and a permanent artefact. The ordering argued in prompt, fine-tune, or RL holds — and the honest version is that most teams that skip to fine-tuning had a prompt they never seriously iterated on.
• Every adapter needs its own eval, forever. This is the cost people miss. Ten adapters means ten eval suites, ten regression surfaces, and ten things to re-verify on every base upgrade. The evaluation burden scales with adapter count, and it is what makes per-tenant adapters expensive despite the serving economics being excellent.
• Watch for narrowing. Training hard on one behaviour degrades others, and the losses land outside your training distribution where your eval is not looking. Keep a held-out general-capability check alongside your task eval; a LoRA that improved your metric by four points and lost eight on instruction-following is a common and invisible outcome.
• Assume it will need retraining on someone else's schedule. Adapters bind to a base checkpoint, and base checkpoints get deprecated. Budget the retraining as recurring cost, and keep the training data and the recipe in version control — the adapter file is worthless without them.

Related: distillation and quantization for the other two ways to change what a model costs, post-training for the layer an adapter is editing, RL fine-tuning on open weights for when supervised adaptation is not enough, and self-hosted inference for agents for the serving stack that makes multi-adapter deployment worth having.