Sentence Transformers 6.3.0, released on 11 November 2026, focuses on making multi-vector retrieval models cheaper to store and faster to score. Its headline feature is token-pooling-aware training: you train the encoder so its document embeddings survive being compressed to a fraction of their size. It also adds Matryoshka training for multi-vector models, optional GPU kernels, and native AWS Neuron inference. This tutorial explains what multi-vector token pooling is, then walks through compressing documents, calibrating a budget, and training with the new losses.

Background: why multi-vector models are expensive

A normal embedding model turns a document into one vector. A multi-vector (late-interaction, ColBERT-style) model keeps one vector per token and scores a query against a document with MaxSim: every query token finds its best-matching document token. That is usually more accurate, but storage grows with document length. The release notes give an example where the average document representation was about 193 embeddings.

Token pooling merges similar document token embeddings (using hierarchical Ward clustering) so you store and score fewer vectors. The catch: a model trained without pooling loses some quality when you compress it. Version 6.3 lets you train for the compression you plan to deploy.

Install

pip install "sentence-transformers==6.3.0"

# with training support and optional kernels
pip install "sentence-transformers[train,kernels]==6.3.0"

Step 1: compress documents at encode time

You can try pooling on an existing multi-vector model without retraining. Pass a pooler to encode_document. By default, queries are not pooled. This snippet is from the official v6.3.0 release notes:

from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling

model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling")

queries = model.encode_query(["Which planet is known as the Red Planet?"])
documents = model.encode_document(
    ["Mars has a reddish surface.", "Jupiter is the largest planet."],
    token_pooling=HierarchicalTokenPooling(pool_factor=4),
)
scores = model.similarity(queries, documents)
print(scores)

pool_factor=4 keeps roughly a quarter of each document’s vectors. You can also set a runtime default with MultiVectorEncoder(..., token_pooling=...).

Step 2: choose a token budget

HierarchicalTokenPooling takes one of three controls:

  • pool_factor=4: keep about one quarter of each document’s vectors.
  • max_tokens=16: keep at most 16 vectors per document, including protected tokens.
  • distance_threshold=...: allocate vectors by embedding geometry with a shared Ward distance threshold, so complex documents keep more vectors and simple ones fewer.

The threshold option is the most interesting. Calibrate it on representative documents so the average lands near your target:

calibration_documents = [
    "Mars is the fourth planet from the Sun and has a reddish surface.",
    "Jupiter is a gas giant with many moons and a prominent Great Red Spot.",
]
embeddings = model.encode_document(calibration_documents)

threshold = HierarchicalTokenPooling.calibrate_distance_threshold(
    embeddings, target_tokens=8
)
pooling = (
    HierarchicalTokenPooling(distance_threshold=threshold)
    if threshold is not None else None
)
compressed = model.encode_document(calibration_documents, token_pooling=pooling)

If the sample already fits the budget, calibration returns None and no pooling is needed. Use a few hundred real documents from your corpus, not two toy sentences, when you do this for real.

Step 3: train for pooling

To keep more quality after compression, wrap your retrieval loss in MultiVectorTokenPoolingLoss with the pooling settings you intend to deploy. This trains on the average of the unpooled and pooled losses:

from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.losses import (
    CachedMultiVectorMultipleNegativesRankingLoss,
    MultiVectorTokenPoolingLoss,
)
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling

model = MultiVectorEncoder("answerdotai/ModernBERT-base")
pooling = HierarchicalTokenPooling(pool_factor=4)

base_loss = CachedMultiVectorMultipleNegativesRankingLoss(model, mini_batch_size=64)
loss = MultiVectorTokenPoolingLoss(
    model,
    base_loss,
    token_poolings=[None, pooling],
    weights=[0.5, 0.5],
)
# Pass `loss` to MultiVectorEncoderTrainer with your (query, positive, negative) dataset.

Both variants reuse the same encoder embeddings, so training adds clustering and scoring work but not another encoder pass.

How much does it help? In the maintainers’ matched experiment (ModernBERT-base, 120,000 MS MARCO triplets), the ordinary model kept 94.6% of its unpooled nDCG@10 at factor-4 pooling, while the pooling-aware model kept 97.8%, with factor-4 pooling cutting the average document from 193.22 to 48.68 embeddings. That is one experiment on one dataset, so measure on your own data.

Step 4: Matryoshka for shorter vectors

Pooling reduces the number of vectors; Matryoshka training reduces their width. MultiVectorMatryoshkaLoss trains a model to keep quality when token embeddings are truncated:

from sentence_transformers.multi_vector_encoder.losses import MultiVectorMatryoshkaLoss

loss = MultiVectorMatryoshkaLoss(
    model,
    base_loss,
    matryoshka_dims=[128, 64, 32],
    matryoshka_weights=[1 / 3] * 3,
)

After training, pass truncate_dim=64 to both encode_query and encode_document. To get fewer and shorter vectors together, wrap MultiVectorMatryoshkaLoss inside MultiVectorTokenPoolingLoss, so pooling happens before truncation.

Optional: kernels and AWS Neuron

With use_kernels=True, models can use compatible Hub kernels for their transformer backbones, and multi-vector models also get kernels for MaxSim scoring and Ward clustering:

model = MultiVectorEncoder("lightonai/mLateOn", use_kernels=True)

These target CUDA (MaxSim needs Triton) and fall back to reference implementations otherwise. Kernels are a runtime setting, so enable them again after reloading a saved model. On AWS, with TorchNeuron installed, device="neuron" runs dense, cross-encoder and multi-vector inference natively.

Compatibility notes before you upgrade

  • Loss scoring changed: the four built-in multi-vector contrastive and distillation losses now follow the model’s similarity function, so MeanMaxSim models use mean scores instead of MaxSim sums. Check your scale and teacher-score conventions.
  • Evaluation scores can shift: reranking and paraphrase-mining evaluators now use the model’s configured similarity, and some metrics normalize against all known positives.
  • Pooling is not saved: truncate_dim set in the constructor is saved with the model, but token_pooling is a runtime default; supply the pooler again after reload.

Takeaway

Sentence Transformers 6.3 makes multi-vector retrieval much more practical: compress documents with HierarchicalTokenPooling, calibrate a budget for your corpus, and train with MultiVectorTokenPoolingLoss and MultiVectorMatryoshkaLoss so the compressed index keeps its quality. Start by trying pooling on an existing model and evaluating nDCG on your own queries before deciding whether retraining is worth it.