Databricks UCDataset streams Unity Catalog tables straight to your GPUs — 10x faster training, no Parquet exports
Databricks released UCDataset on Monday, November 9 — a native PyTorch data loader that streams Unity Catalog tables directly to the GPUs in a training job, no Parquet export in between. On a simulated production workload, a 300 GB feature table streamed across eight NVIDIA H100 GPUs, it cut training time 10x versus reading an exported Parquet copy. The catch, for now: it is a beta, and it cannot read tables that carry row filters or column masks.
What we checked: Databricks’ November 9 blog post (published 19:00 UTC), cross-checked against Databricks’ November 2026 release notes and Releasebot’s Databricks release-notes tracker, both carrying the same November 9 date.
- What: UCDataset (Unity Catalog Dataset), a native PyTorch data loader from Databricks.
- When: released in beta on November 9, 2026, on Databricks AI Runtime; the announcement was published at 19:00 UTC.
- Performance: 10x faster training on a simulated production workload — a 300 GB feature table streamed across 8 NVIDIA H100 GPUs to train a recommendation model — versus loading data from an exported Parquet copy.
- How: reads the table’s files directly from cloud storage and splits matching rows evenly across every GPU and data loader worker; no exports, no second copy, no custom sharding code.
- Governance: training jobs read as the identity that runs them and Unity Catalog permissions apply; filters and deletion vectors are applied as the data is read, and new rows are picked up automatically.
- Limits: the beta reads Delta tables in Unity Catalog and cannot read tables with row filters or column masks. Install with
%pip install databricks-uc-dataset.
Eight NVIDIA H100s, rented by the hour, sit mostly idle. The model is ready, the training script is ready — but the data is still trickling in from an exported Parquet copy made hours earlier, and every second those chips wait is money burned. That is the scene Databricks opens with in its announcement of UCDataset: "GPUs are usually the most expensive part of any training run, and most teams can't keep them busy. The bottleneck usually isn't the model. It's getting data from where it lives to the GPU fast enough."
On Monday, November 9, the company shipped its answer. UCDataset — short for Unity Catalog Dataset — is a native PyTorch data loader, released in beta on Databricks AI Runtime, that streams a Unity Catalog table directly to every GPU in a training job. No export step, no second copy of the data, no hand-written sharding code.
What Databricks announced#
UCDataset turns a governed table into a first-class PyTorch data source. You describe what to read — a table, the columns you need, a row filter — and it delivers training-ready batches to each GPU. It reads the table's files directly from cloud storage, so nothing sits between your data and the accelerators, and it splits the matching rows evenly across every GPU and data loader worker. Wrap it in a standard PyTorch DataLoader with batch_size=None and train with an ordinary distributed-data-parallel loop; there is no sampler or sharding code to write.
The example Databricks walks through is a click-through-rate model. A team keeps user and item features with a click label in a main.recsys.ctr_features table, with new impressions landing every day. They train on 16 GPUs across two nodes, straight from the table, describing the job in a workload.yaml that starts train.py with torchrun on every node and launching it with the AI Runtime CLI. To train on more GPUs, they change one number — num_accelerators in the YAML — and the training code stays the same. A complete train.py and workload.yaml, with a sample table to try them on, are in the AI Runtime cookbook.
Installing it is one line in a notebook: %pip install databricks-uc-dataset. The beta reads Delta tables registered in Unity Catalog, including tables with deletion vectors.
The three workarounds it replaces#
Databricks frames UCDataset against the three ways teams currently feed governed tables into training — and each one, in its telling, costs time or money:
| Today's workaround | What it costs | What UCDataset does instead |
|---|---|---|
| Export to Parquet before every run | A second copy to store, stale the moment new data lands, outside Unity Catalog lineage | Reads the live table directly from cloud storage |
Pull the table into memory (toPandas()) | Works for small tables, fails on large ones, lands everything on one node | Streams batches to every GPU in the job |
| Stream through a warehouse | Warehouses weren't built to feed GPUs and throttle when many GPUs read at once | Reads table files directly, split evenly across GPUs and workers |
There is a subtler cost the announcement flags, too: every GPU and every worker needs its own balanced slice of the data, and teams build that sharding by hand. A subtle bug can mean the model trains on duplicated or skipped rows without anyone noticing. UCDataset takes that slicing over, which is the part ML engineers will feel first — less data plumbing to maintain, fewer silent errors.
The 10x number, and what it actually measured#
The headline figure is a 10x cut in training time — but the fine print matters. Databricks benchmarked UCDataset on a simulated production workload: a 300 GB feature table streamed across eight NVIDIA H100 GPUs to train a recommendation model. Against loading data from an exported Parquet copy, training time fell 10x on average.
Two caveats belong next to that number. It is a simulated workload, not a production training run, and every figure is vendor-reported — no independent benchmark exists yet. What the number does establish is the shape of the win: when the data path is the bottleneck, removing the export step is worth an order of magnitude, not a few percent. That tracks with the broader story in data infrastructure this year — Databricks took AI Runtime to GA in October with serverless GPUs for lakehouse-native training, and Google open-sourced TorchTPU the same month to make TPUs a native PyTorch device. The vendors are converging on the same idea: the training loop should read governed data where it lives.
Governance is the other half of the pitch, and the one that decides whether enterprises can actually use it. Because the job reads the governed table directly, there is no exported copy to secure separately: training jobs read data as the identity that runs them, and Unity Catalog permissions apply exactly as they do everywhere else on Databricks. Filters and deletion vectors apply as the data is read, so the next run picks up newly appended rows automatically.
What's next#
Beta means limits. UCDataset cannot yet read tables that have row filters or column masks — the governed tables that need those controls most are, for now, out of reach. Databricks also points to the edges of the same idea: for unstructured data in Unity Catalog volumes there is UCVolumeDataset, which streams files with local caching, and the company says it is expanding UCDataset toward multimodal training via the FILE type, so teams could train on images, audio and documents directly from a table.
For teams already on Databricks, the practical test is small: install the package, point it at a table, and watch whether the GPUs stay busy. If the 10x holds on real workloads — and that "if" is doing real work until independent numbers exist — the Parquet export step that every ML team maintains today starts to look like a habit, not a necessity. And if it doesn't hold, the governance story alone may still be enough: one governed copy of the data, read directly, is a simpler system to defend than a lakehouse plus a trail of stale exports.