Hugging Face’s Transformers library has moved quickly in the last few weeks. v5.22.0 (29 October 2026) added Mistral Large 4 and made it possible to train adapters on FP8 weights. v5.22.2 (6 November 2026) followed as a security and distributed-training patch. The 5.21 release just before them also added an FP8 backend for the quantized KV cache. This tutorial shows how to use the FP8 features, what Mistral Large 4 support means, and how to upgrade without surprises.

What Shipped

From the official release notes:

  • v5.22.2 (6 Nov): hardened Jinja sandboxing, which matters because chat templates are Jinja, and an NCCL-only device_id fix for distributed runs.
  • v5.22.1 (30 Oct): a documentation fix.
  • v5.22.0 (29 Oct): Mistral Large 4, adapter training on FP8 weights, memory savings in the FP8 MoE backward pass, and a breaking change that moves expert-parallel (EP) logic into its own file.
  • v5.21.0 (26 Oct): an "fp8" backend for QuantizedCache, new tensor-parallel embedding sharding, plus new models including MiniCPM-OCR, Flint, UltraVBert and Pocket TTS.

Step 1: Upgrade

pip install --upgrade "transformers>=5.22.2" accelerate
python -c "import transformers; print(transformers.__version__)"

If you render chat templates from models you didn’t train yourself, treat 5.22.2 as the minimum version because of the Jinja sandbox hardening.

Step 2: Use the FP8 Quantized KV Cache

During generation, the key/value (KV) cache can use more memory than the model weights once prompts get long. Transformers already had quantized caches backed by quanto and hqq. v5.21 added an "fp8" backend that stores KV states as torch.float8_e4m3fn with a single scale per tensor. You turn it on with two arguments to generate():

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-0.5B-Instruct"  # any causal LM you have access to

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

messages = [{"role": "user", "content": "Explain KV caching in three sentences."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=200,
    cache_implementation="quantized",
    cache_config={"backend": "fp8"},
)

print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Measure the trade-off yourself

FP8 is lower precision, so compare quality and memory on your prompts before switching production traffic. Here’s a simple A/B test on a GPU:

import torch

def run(cache_kwargs: dict) -> tuple[str, float]:
    torch.cuda.reset_peak_memory_stats()
    out = model.generate(**inputs, max_new_tokens=200, do_sample=False, **cache_kwargs)
    text = tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
    peak_mb = torch.cuda.max_memory_allocated() / 2**20
    return text, peak_mb

baseline, base_mb = run({})
fp8, fp8_mb = run({"cache_implementation": "quantized", "cache_config": {"backend": "fp8"}})

print(f"baseline peak: {base_mb:.0f} MB | fp8 cache peak: {fp8_mb:.0f} MB")
print("identical output:", baseline == fp8)

With short prompts you may see almost no difference, because the cache is small. The savings show up with long contexts and large batches.

Step 3: Train LoRA Adapters on FP8 Weights

v5.22.0 adds backward passes to the dense and grouped FP8 matmuls, and marks FP8 quantized models as trainable (is_trainable is now True). In practice, you can attach PEFT adapters such as LoRA to an FP8 checkpoint and fine-tune it without first converting the weights to BF16. The release notes also say the MoE layer backward now dequantizes experts in chunks, cutting peak memory in that layer from about 28.9 GiB to 2.8 GiB at 8k tokens.

The PEFT side looks the same as usual. The difference is that the base model you load is an FP8-quantized checkpoint:

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(
    "your-org/your-fp8-checkpoint",   # an FP8-quantized model you have access to
    device_map="auto",
)

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)

model = get_peft_model(base, lora)
model.print_trainable_parameters()
# ...then train with Trainer / TRL's SFTTrainer as you normally would

Check the target_modules names against your model’s architecture. Layer names differ between model families.

Mistral Large 4 Support

v5.22.0 adds support for Mistral Large 4. According to the release notes, it’s a natively multimodal Mixture-of-Experts model with about 1 trillion total parameters and about 52 billion active per token (49B excluding embeddings and output layers). It uses 128 routed experts with 4 active per token plus 1 shared expert, supports a 1M-token context and accepts text and image input. Recommended settings from the notes:

  • Reasoning effort: low, medium or high (the default is high). Use high for complex and agentic work and low for quick chat replies.
  • Sampling: temperature=1.0 and top_p=0.95 at every effort level.
  • License: the Mistral Ecosystem Open-Weight License, a modified MIT license. Running a Model-as-a-Service business needs a separate commercial licence from Mistral.

A model this size needs multi-GPU infrastructure, so most developers will use it through a hosted endpoint or a serving stack like vLLM. Transformers support still matters for fine-tuning, evaluation and tooling.

Breaking Changes to Check

  • v5.22.0: expert-parallel logic moved into its own file. If you import EP internals directly, update the import paths.
  • v5.21.0: tensor parallelism now has a dedicated embedding style and shards input embeddings by default. ColwiseParallel and RowwiseParallel now apply only to linear layers, so custom TP plans that used them on embeddings have to switch to the new style.
  • v5.21.0: every MoE TP/EP plan now shards its experts. Custom MoE parallel plans need to add expert sharding.

Upgrade Checklist

  • Pin transformers>=5.22.2 in requirements.txt or pyproject.toml.
  • Run your evaluation set with and without the FP8 KV cache before turning it on.
  • Re-test any custom TP or EP plans on a small multi-GPU job.
  • Only load chat templates and remote code from sources you trust.

Takeaway

Transformers 5.21 and 5.22 make FP8 practical from inference to fine-tuning. Add cache_implementation="quantized" with cache_config={"backend": "fp8"} to cut KV-cache memory, attach LoRA adapters to FP8 checkpoints, and upgrade to 5.22.2 for the security fix. Measure quality on your own data before you change production settings. Release notes: huggingface/transformers releases.