Hugging Face released PEFT v0.22.0 on 19 November 2026, and its headline addition is a new fine-tuning method called PaCA (Partial Connection Adaptation). Instead of adding LoRA’s extra adapter matrices, PaCA fine-tunes a small set of randomly selected input columns of the existing weights. The result is an adapter that trains with fewer sequential operations and stores less activation memory. The release also adds a model health check, two new LoRA initialisations, and better tensor-parallel and FSDP2 support.

This tutorial shows how to fine-tune with PaCA, when to choose it over LoRA, how to use the new health check, and which changes can affect existing code. Sources: the PEFT v0.22.0 release notes and the PaCA reference documentation.

Install

PEFT 0.22 requires Python 3.11 or newer (3.10 support was dropped because it reached end of life).

pip install --upgrade "peft==0.22.0" transformers accelerate

How PaCA differs from LoRA

LoRA freezes a weight matrix W and learns a low-rank update B @ A that runs alongside it. During training, the adapter layers run after the base layer, and LoRA still needs to keep the full input activation for the backward pass.

PaCA takes a different route, according to the PEFT docs:

  • One matmul per layer. The adapted columns are written into the pretrained weight just before the matmul and restored right after, so forward and backward each run a single matrix multiplication, just like the frozen base layer.
  • Partial activations. The gradient of the adapted columns depends only on the matching r input channels, so only those are saved for backward.
  • Same parameter count as LoRA’s B matrix. For a weight of shape (out_features, in_features), PaCA trains r * out_features parameters.

The release notes quote the paper’s results: 22% less training time, 16% less memory and 23% longer sequences than LoRA, with comparable accuracy. Treat those as the paper’s numbers; your results depend on model, data and hardware.

Mathematically, a PaCA adapter is equivalent to a LoRA adapter of rank r whose A matrix is a fixed column selector. That means it can be converted to a LoRA adapter without loss using convert_to_lora(), which is handy if your serving stack only supports LoRA.

Fine-tune a model with PaCA

The API mirrors LoRA. Create a PacaConfig and wrap the model with get_peft_model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PacaConfig, get_peft_model

model_id = "Qwen/Qwen2.5-0.5B"  # any causal LM with nn.Linear layers
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

config = PacaConfig(
    r=16,                    # columns fine-tuned per targeted layer
    paca_alpha=16,           # update is scaled by paca_alpha / r
    target_modules="all-linear",
    random_seed=0,           # reproducible column selection
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
model.print_healthcheck()    # new in 0.22, see below

From here you train exactly as you would with LoRA, for example with the Transformers Trainer or TRL’s SFTTrainer:

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling
from datasets import load_dataset

ds = load_dataset("imdb", split="train[:2000]")
ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, max_length=512),
            batched=True, remove_columns=ds.column_names)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="paca-out",
        per_device_train_batch_size=4,
        num_train_epochs=1,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=20,
    ),
    train_dataset=ds,
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)
trainer.train()
model.save_pretrained("paca-adapter")

Key parameters

  • r – number of input columns fine-tuned in each targeted layer. Must not exceed in_features.
  • paca_alpha – scaling factor. Like lora_alpha, the docs note it acts as a multiplier on the effective learning rate. Start with paca_alpha = r and tune the learning rate.
  • target_modules – module names, a regex, or "all-linear" (all linear modules except the output layer).
  • random_seed – makes column selection reproducible. The selected indices are always saved in the checkpoint, so you don’t need the seed to load an adapter.
  • paca_indices_pattern – choose specific columns per layer instead of random ones:
config = PacaConfig(
    r=16,
    paca_alpha=16,
    target_modules=["q_proj", "v_proj"],
    paca_indices_pattern={
        "layers.0.self_attn.q_proj": [0, 2, 4, 6],
        "layers.1.self_attn.v_proj": list(range(32)),
    },
)

The docs say random selection performed on par with selecting columns by weight norm or accumulated gradients in the paper, so random is a sensible default.

Inference

Merge the adapter before serving. The unmerged forward still swaps columns in and out on every call:

merged = model.merge_and_unload()
merged.save_pretrained("paca-merged")

Because the update is a plain column update, it merges exactly into the base weights.

When to pick PaCA (and when not to)

Choose PaCA when:

  • Training step time or activation memory is your bottleneck, for example when you want longer sequences or bigger batches on the same GPU.
  • You fine-tune quantized models (the release mentions QPaCA).
  • You want an adapter that merges exactly.

Stick with LoRA (or test carefully) when:

  • You only adapt light modules such as q_proj/v_proj, or train with few tokens per step. The release notes say PaCA is less effective there.
  • You use non-nn.Linear layers. PaCA supports only linear layers (including quantized linear layers).
  • You rely on forward hooks on base layers. PaCA runs the matmul itself, so those hooks aren’t called (a warning is emitted). Register hooks on the PaCA layer instead.
  • You need mixed adapter batches (adapter_names), which aren’t supported.
  • You train with sharded weights (FSDP, DeepSpeed ZeRO-3), which the docs say hasn’t been tested yet.

The new model health check

model.print_healthcheck() prints a summary of adapter state, dtypes, devices, trainable vs total parameters and layer types, and flags issues PEFT can detect. model.healthcheck() returns the same as a JSON-serialisable report, which is useful in CI or experiment tracking:

import json

report = model.healthcheck()
with open("peft-healthcheck.json", "w") as f:
    json.dump(report, f, indent=2, default=str)

The release notes recommend using it instead of print_trainable_parameters(), since it reports the same counts and more.

Two new LoRA initialisations

If you’re staying on LoRA, 0.22 adds two options:

  • NoRA (init_lora_weights="nora"): like standard LoRA, B starts at zero, but each column of A is normalised along the rank axis. Set lora_alpha equal to r when using it.
  • Astra (init_lora_weights="astra"): uses a small, task-specific calibration dataset to initialise LoRA in a task-aware subspace. You must run preprocess_astra before injecting the adapter.
from peft import LoraConfig

config = LoraConfig(r=16, lora_alpha=16, init_lora_weights="nora",
                    target_modules="all-linear", task_type="CAUSAL_LM")

Changes that can affect existing code

  • Python 3.11+ required.
  • Adapter loading warnings. PeftModel.from_pretrained() and load_adapter() now warn about missing and unexpected adapter keys by default. Use strict_adapter_check="error" to fail or "ignore" to silence.
  • modules_to_save matching is stricter. It used to match by suffix, so ["classifier"] also matched model.other_classifier. Now the whole sub-module name must match. List every module explicitly if you relied on the old behaviour.
  • OFT Conv2d fix. A forward-pass bug was fixed, so existing OFT checkpoints targeting Conv2d layers may produce different outputs when used without merging.
  • k-bit training checkpointing. prepare_model_for_kbit_training() now uses the Transformers default (non-reentrant on current versions) when no kwargs are passed.

The PEFT team also says the next release will be PEFT 1.0, with some API clean-up and a release candidate for testing.

Key takeaways

  • PaCA fine-tunes selected weight columns instead of adding adapter matrices, aiming for LoRA-level accuracy with faster steps and less activation memory.
  • Use PacaConfig(r=..., paca_alpha=..., target_modules=...) with the usual get_peft_model workflow, then merge_and_unload() for inference.
  • Replace print_trainable_parameters() with print_healthcheck().
  • Check modules_to_save, adapter-key warnings and your Python version before upgrading.