Most semantic search tutorials start with an embedding model of 20 to 100+ million parameters. On 19 November 2026, NeuML (the team behind txtai) published Atto, a family of compact embedding models that asks a different question: how small can a useful retrieval model get? The family runs from an 840-parameter ColBERT configuration up to 28-million-parameter dense models, all available on the Hugging Face Hub under the Apache 2.0 licence.

This tutorial explains what Atto is, how its published benchmark numbers compare with a familiar baseline, and how to use an Atto model for semantic search in Python with Sentence Transformers and txtai. Sources: the official Atto announcement on the Hugging Face blog and the atto-14m-384d-embeddings model card.

What is Atto?

Atto is a family of compact contextual models for retrieval in three styles:

  • Dense: one vector per text, for normal vector search.
  • Sparse: sparse vectors over a learned latent space (not a vocabulary-sized output).
  • ColBERT / late interaction: one small vector per token, for finer query-document matching.

Instead of stacking standard Transformer blocks, Atto uses two building blocks described in the post: a GatedConvAttentionMixer (depthwise convolution combined with global multi-head attention) and a PhaseGatedMoE (local and dilated convolutional experts). It also experiments with very small vocabularies (from 250 to 30K tokens) and hash embeddings to keep parameter counts down.

The published numbers

NeuML evaluated the models on three BEIR datasets (ArguAna, NFCorpus, SciFact) using nDCG@10. A few rows from the dense table in the announcement:

Model Parameters Dimensions Average nDCG@10
atto-90k-2048d 90K 2048 0.4257
atto-2m-8192d 2M 8192 0.4707
atto-14m-384d 14M 384 0.4892
all-MiniLM-L6-v2 (baseline) 23M 384 0.4802
atto-28m-8192d 28M 8192 0.5138

Two takeaways from NeuML’s own figures:

  1. atto-14m-384d scores slightly above all-MiniLM-L6-v2 on this subset while using fewer parameters and the same 384 dimensions, which makes it a natural drop-in candidate.
  2. Tiny models such as atto-90k-2048d are far from useless: the announcement notes it reaches 0.4257 with roughly 256 times fewer parameters than MiniLM, though it uses wider 2048-dimensional vectors.

These are three datasets, measured by the authors. Treat them as a starting point and benchmark on your own data before switching production search.

Picking a checkpoint

The Hub collection includes dense models such as NeuML/atto-36k-384d-embeddings, NeuML/atto-1m-2048d-embeddings, NeuML/atto-14m-384d-embeddings and NeuML/atto-28m-8192d-embeddings, sparse variants such as NeuML/atto-17m-8192d-sparse-embeddings, and ColBERT models such as NeuML/colbert-atto-950k-128d.

A practical way to choose:

  • Replacing MiniLM in an existing 384-dim index? Start with atto-14m-384d-embeddings or atto-25m-384d-embeddings, so your vector size stays the same.
  • Edge devices or in-browser search? Try the sub-million models and measure recall.
  • Best quality in the family? The 17M and 28M models with 8192 dimensions score highest, but large vectors cost more storage. The announcement’s comments suggest vector compression for very wide outputs.

Setup

python -m venv .venv && source .venv/bin/activate
pip install -U sentence-transformers txtai

Atto uses a custom architecture, so models load with trust_remote_code=True. That runs Python code from the model repository on your machine. Review the repo first, and in production pin a specific revision.

Semantic search with Sentence Transformers

The model card’s config defines two prompts: query: for queries and document: for documents, matching how the model was trained. Use them via prompt_name:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "NeuML/atto-14m-384d-embeddings",
    trust_remote_code=True,
)

docs = [
    "Reset your password from the account settings page.",
    "Invoices are emailed on the first business day of each month.",
    "Two-factor authentication can be enabled with any TOTP app.",
    "Refunds are processed within five working days.",
]

doc_emb = model.encode(docs, prompt_name="document", normalize_embeddings=True)
query_emb = model.encode(
    ["how do I turn on 2FA"], prompt_name="query", normalize_embeddings=True
)

scores = model.similarity(query_emb, doc_emb)[0]
for score, doc in sorted(zip(scores.tolist(), docs), reverse=True):
    print(f"{score:.3f}  {doc}")

The model card lists cosine as the similarity function, so normalising embeddings and using dot product or cosine gives the same ranking.

A searchable index with txtai

txtai, also from NeuML, wraps model loading, indexing and storage. The model card shows this pattern:

import txtai

embeddings = txtai.Embeddings(
    path="neuml/atto-14m-384d-embeddings",
    content=True,
    vectors={"trust_remote_code": True},
)

embeddings.index([
    "Reset your password from the account settings page.",
    "Invoices are emailed on the first business day of each month.",
    "Two-factor authentication can be enabled with any TOTP app.",
])

print(embeddings.search("enable 2FA", 2))
embeddings.save("support-index")   # reload later with embeddings.load(...)

content=True stores the original text alongside vectors, so search results include the matching text, not just IDs.

Benchmark it on your own data

Small models shine or fail depending on the domain. A quick recall check needs only a few labelled pairs:

from sentence_transformers import SentenceTransformer

pairs = [  # (query, index of the correct document in docs)
    ("how do I turn on 2FA", 2),
    ("when do I get my invoice", 1),
    ("money back timeline", 3),
]

def recall_at_1(model_id, docs, pairs, **kw):
    m = SentenceTransformer(model_id, **kw)
    d = m.encode(docs, normalize_embeddings=True, **({"prompt_name": "document"} if m.prompts else {}))
    hits = 0
    for q, gold in pairs:
        qe = m.encode([q], normalize_embeddings=True, **({"prompt_name": "query"} if m.prompts else {}))
        hits += int(m.similarity(qe, d)[0].argmax().item() == gold)
    return hits / len(pairs)

print("atto-14m", recall_at_1("NeuML/atto-14m-384d-embeddings", docs, pairs, trust_remote_code=True))
print("MiniLM  ", recall_at_1("sentence-transformers/all-MiniLM-L6-v2", docs, pairs))

Use a few hundred real queries for a meaningful result, and also time encode() on your target hardware: speed and memory are the main reasons to go small.

Where compact embeddings make sense

  • First-stage filtering on edge devices or laptops, with a bigger model or reranker for the top results.
  • Domain-specific models: the authors’ broader goal is making it cheap to train specialised embedding models for a single dataset or vocabulary.
  • High-volume pipelines where embedding cost dominates.

Takeaway

Atto shows that useful retrieval is possible with far fewer parameters than most of us assume. If you use all-MiniLM-L6-v2 today, NeuML/atto-14m-384d-embeddings is an easy experiment because it keeps the same 384 dimensions. Load it with trust_remote_code=True, use the query/document prompts, and benchmark on your own data. Read the Atto announcement for the full tables and architecture details.