Hugging Face released PEFT v0.22.0 on 19 November 2026, and its headline addition is a new fine-tuning method called PaCA (Partial Connection Adaptation). Instead of adding LoRA’s extra adapter matrices, PaCA fine-tunes a small set of randomly selected input columns of the existing weights. The result is an adapter that trains with fewer sequential operations and stores less activation memory. The release also adds a model health check, two new LoRA initialisations, and better tensor-parallel and FSDP2 support.
This tutorial shows how to fine-tune with PaCA, when to choose it over LoRA, how to use the new health check, and which changes can affect existing code. Sources: the PEFT v0.22.0 release notes and the PaCA reference documentation.
Install
PEFT 0.22 requires Python 3.11 or newer (3.10 support was dropped because it reached end of life).
pip install --upgrade "peft==0.22.0" transformers accelerate
How PaCA differs from LoRA
LoRA freezes a weight matrix W and learns a low-rank update B @ A that runs alongside it. During training, the adapter layers run after the base layer, and LoRA still needs to keep the full input activation for the backward pass.
PaCA takes a different route, according to the PEFT docs:
- One matmul per layer. The adapted columns are written into the pretrained weight just before the matmul and restored right after, so forward and backward each run a single matrix multiplication, just like the frozen base layer.
- Partial activations. The gradient of the adapted columns depends only on the matching
rinput channels, so only those are saved for backward. - Same parameter count as LoRA’s B matrix. For a weight of shape
(out_features, in_features), PaCA trainsr * out_featuresparameters.
The release notes quote the paper’s results: 22% less training time, 16% less memory and 23% longer sequences than LoRA, with comparable accuracy. Treat those as the paper’s numbers; your results depend on model, data and hardware.
Mathematically, a PaCA adapter is equivalent to a LoRA adapter of rank r whose A matrix is a fixed column selector. That means it can be converted to a LoRA adapter without loss using convert_to_lora(), which is handy if your serving stack only supports LoRA.
Fine-tune a model with PaCA
The API mirrors LoRA. Create a PacaConfig and wrap the model with get_peft_model:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PacaConfig, get_peft_model
model_id = "Qwen/Qwen2.5-0.5B" # any causal LM with nn.Linear layers
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
config = PacaConfig(
r=16, # columns fine-tuned per targeted layer
paca_alpha=16, # update is scaled by paca_alpha / r
target_modules="all-linear",
random_seed=0, # reproducible column selection
task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_healthcheck() # new in 0.22, see below
From here you train exactly as you would with LoRA, for example with the Transformers Trainer or TRL’s SFTTrainer:
from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling
from datasets import load_dataset
ds = load_dataset("imdb", split="train[:2000]")
ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, max_length=512),
batched=True, remove_columns=ds.column_names)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
trainer = Trainer(
model=model,
args=TrainingArguments(
output_dir="paca-out",
per_device_train_batch_size=4,
num_train_epochs=1,
learning_rate=2e-4,
bf16=True,
logging_steps=20,
),
train_dataset=ds,
data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)
trainer.train()
model.save_pretrained("paca-adapter")
Key parameters
r– number of input columns fine-tuned in each targeted layer. Must not exceedin_features.paca_alpha– scaling factor. Likelora_alpha, the docs note it acts as a multiplier on the effective learning rate. Start withpaca_alpha = rand tune the learning rate.target_modules– module names, a regex, or"all-linear"(all linear modules except the output layer).random_seed– makes column selection reproducible. The selected indices are always saved in the checkpoint, so you don’t need the seed to load an adapter.paca_indices_pattern– choose specific columns per layer instead of random ones:
config = PacaConfig(
r=16,
paca_alpha=16,
target_modules=["q_proj", "v_proj"],
paca_indices_pattern={
"layers.0.self_attn.q_proj": [0, 2, 4, 6],
"layers.1.self_attn.v_proj": list(range(32)),
},
)
The docs say random selection performed on par with selecting columns by weight norm or accumulated gradients in the paper, so random is a sensible default.
Inference
Merge the adapter before serving. The unmerged forward still swaps columns in and out on every call:
merged = model.merge_and_unload()
merged.save_pretrained("paca-merged")
Because the update is a plain column update, it merges exactly into the base weights.
When to pick PaCA (and when not to)
Choose PaCA when:
- Training step time or activation memory is your bottleneck, for example when you want longer sequences or bigger batches on the same GPU.
- You fine-tune quantized models (the release mentions QPaCA).
- You want an adapter that merges exactly.
Stick with LoRA (or test carefully) when:
- You only adapt light modules such as
q_proj/v_proj, or train with few tokens per step. The release notes say PaCA is less effective there. - You use non-
nn.Linearlayers. PaCA supports only linear layers (including quantized linear layers). - You rely on forward hooks on base layers. PaCA runs the matmul itself, so those hooks aren’t called (a warning is emitted). Register hooks on the PaCA layer instead.
- You need mixed adapter batches (
adapter_names), which aren’t supported. - You train with sharded weights (FSDP, DeepSpeed ZeRO-3), which the docs say hasn’t been tested yet.
The new model health check
model.print_healthcheck() prints a summary of adapter state, dtypes, devices, trainable vs total parameters and layer types, and flags issues PEFT can detect. model.healthcheck() returns the same as a JSON-serialisable report, which is useful in CI or experiment tracking:
import json
report = model.healthcheck()
with open("peft-healthcheck.json", "w") as f:
json.dump(report, f, indent=2, default=str)
The release notes recommend using it instead of print_trainable_parameters(), since it reports the same counts and more.
Two new LoRA initialisations
If you’re staying on LoRA, 0.22 adds two options:
- NoRA (
init_lora_weights="nora"): like standard LoRA,Bstarts at zero, but each column ofAis normalised along the rank axis. Setlora_alphaequal torwhen using it. - Astra (
init_lora_weights="astra"): uses a small, task-specific calibration dataset to initialise LoRA in a task-aware subspace. You must runpreprocess_astrabefore injecting the adapter.
from peft import LoraConfig
config = LoraConfig(r=16, lora_alpha=16, init_lora_weights="nora",
target_modules="all-linear", task_type="CAUSAL_LM")
Changes that can affect existing code
- Python 3.11+ required.
- Adapter loading warnings.
PeftModel.from_pretrained()andload_adapter()now warn about missing and unexpected adapter keys by default. Usestrict_adapter_check="error"to fail or"ignore"to silence. modules_to_savematching is stricter. It used to match by suffix, so["classifier"]also matchedmodel.other_classifier. Now the whole sub-module name must match. List every module explicitly if you relied on the old behaviour.- OFT
Conv2dfix. A forward-pass bug was fixed, so existing OFT checkpoints targetingConv2dlayers may produce different outputs when used without merging. - k-bit training checkpointing.
prepare_model_for_kbit_training()now uses the Transformers default (non-reentrant on current versions) when no kwargs are passed.
The PEFT team also says the next release will be PEFT 1.0, with some API clean-up and a release candidate for testing.
Key takeaways
- PaCA fine-tunes selected weight columns instead of adding adapter matrices, aiming for LoRA-level accuracy with faster steps and less activation memory.
- Use
PacaConfig(r=..., paca_alpha=..., target_modules=...)with the usualget_peft_modelworkflow, thenmerge_and_unload()for inference. - Replace
print_trainable_parameters()withprint_healthcheck(). - Check
modules_to_save, adapter-key warnings and your Python version before upgrading.
