Hugging Face released Datasets 5.2.0 on 9 November 2026. It is a short but useful release for anyone training models on images, audio, video or 3D data: streaming decode for folder-based media datasets is now multithreaded by default, .gltf files are accepted as 3D meshes, a pickle-based remote code execution issue (CVE-2026-57845) is closed, and several data-loss bugs in distributed and Hub workflows are fixed. This tutorial shows what changed and how to use it in a real data pipeline.

Upgrade

pip install -U "datasets>=5.2.0"
python -c "import datasets; print(datasets.__version__)"

The release also bumps fsspec and removes the direct aiohttp dependency, so check your lockfile if you pin either of those.

1. Faster streaming for image, audio and video folders

Folder-based builders such as imagefolder and audiofolder let you load a directory of files without writing a loading script. In streaming mode, examples are read and decoded on the fly instead of being converted to Arrow on disk first, which is ideal for large datasets or quick experiments.

In 5.2.0, the maintainers enabled multithreaded decoding by default for these media folder datasets (PR #8791). Mesh datasets use less multithreading, because they load whole files into memory and could otherwise run out of RAM. You do not need to change your code to benefit:

from datasets import load_dataset

# A local folder laid out as data/train/<label>/*.jpg
ds = load_dataset(
    "imagefolder",
    data_dir="data",
    split="train",
    streaming=True,
)

for i, example in enumerate(ds):
    image, label = example["image"], example["label"]
    print(image.size, label)
    if i == 4:
        break

The same applies to datasets on the Hub that are stored as plain media files:

ds = load_dataset("your-username/your-image-dataset", split="train", streaming=True)

Measure it on your own data

Speedups depend heavily on file sizes, codecs, disk and CPU, so time it yourself before and after the upgrade:

import time
from datasets import load_dataset

ds = load_dataset("imagefolder", data_dir="data", split="train", streaming=True)

start = time.perf_counter()
n = 0
for example in ds:
    _ = example["image"].size   # force decoding
    n += 1
    if n == 2000:
        break
elapsed = time.perf_counter() - start
print(f"{n} images in {elapsed:.1f}s ({n / elapsed:.0f} img/s)")

Run it in two virtual environments (the old version and 5.2.0) to see the real difference for your workload.

Feed a PyTorch training loop

A streaming dataset is an IterableDataset, so you can apply transforms lazily and hand it to a PyTorch DataLoader:

import torch
from torch.utils.data import DataLoader
from torchvision import transforms
from datasets import load_dataset

ds = load_dataset("imagefolder", data_dir="data", split="train", streaming=True)

to_tensor = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

def preprocess(batch):
    batch["pixel_values"] = [to_tensor(img.convert("RGB")) for img in batch["image"]]
    return batch

ds = ds.map(preprocess, batched=True, remove_columns=["image"])
ds = ds.with_format("torch")

loader = DataLoader(ds, batch_size=32)
for batch in loader:
    print(batch["pixel_values"].shape, batch["label"][:5])
    break

Because decoding was a common bottleneck in pipelines like this, faster decode means the GPU spends less time waiting for data.

2. glTF models in the Mesh feature

The Mesh feature and the meshfolder builder previously accepted only .glb, .ply and .stl. In 5.2.0 they also accept .gltf (PR #8958). Before this change, a folder of .gltf files produced no Mesh column at all, and a folder of glTF models with a few texture images was mistaken for an image dataset.

from datasets import load_dataset

meshes = load_dataset("meshfolder", data_dir="models", split="train")
print(meshes.features)

One limitation remains: format inference still picks the most common extension. If your glTF models are outnumbered by their own texture images, the folder may still be detected as imagefolder, so pass "meshfolder" explicitly as shown above.

3. Security: pickled pandas files are no longer loaded

5.2.0 fixes CVE-2026-57845, an arbitrary code execution issue through pickle deserialization in the pandas dataset builder (PR #8879). Unpickling data runs code, so loading a malicious pickled DataFrame from an untrusted source could compromise your machine.

The fix is blunt and safe: the pandas builder now raises a migration error before opening or unpickling the file. If you relied on pickled DataFrames, convert them to a safe format once, in a trusted environment:

import pandas as pd
from datasets import Dataset

# Only do this for files you created yourself and trust
df = pd.read_pickle("trusted_data.pkl")
df.to_parquet("trusted_data.parquet")

ds = Dataset.from_parquet("trusted_data.parquet")

Parquet is columnar, compressed and does not execute code when read, which makes it the better default for sharing datasets anyway.

4. Bug fixes that protect your data

Several fixes in this release prevent silent data problems, which are the worst kind in ML:

  • push_to_hub dropping rows when num_proc was larger than the number of shards is fixed. If you uploaded datasets this way, re-check the row counts on the Hub.
  • concatenate_datasets after split_dataset_by_node no longer gives every rank all of the data. Distributed training jobs using this pattern were seeing duplicated samples.
  • WebDataset now rejects duplicate fields instead of silently overwriting data.
  • save_to_disk no longer fails when a batch holds more than 2 GB of image or audio files.
  • Dataset.take no longer raises IndexError when you ask for more rows than exist.

A quick sanity check after any upload is cheap insurance:

from datasets import load_dataset

local = load_dataset("imagefolder", data_dir="data", split="train")
local.push_to_hub("your-username/your-image-dataset", num_proc=8)

remote = load_dataset("your-username/your-image-dataset", split="train")
assert len(remote) == len(local), (len(remote), len(local))

Key takeaway

Datasets 5.2.0 is a safe, worthwhile upgrade: media folder streaming decodes with multiple threads by default, glTF meshes are supported, a pickle RCE path is closed, and distributed and Hub workflows lose fewer rows. Upgrade, re-time your input pipeline, convert any pickled DataFrames to Parquet, and verify row counts on datasets you pushed with multiple processes. Full notes are on the huggingface/datasets releases page.