Every AI project eventually hits the same wall. The training loop wants 200 random rows, and Parquet makes you decompress the whole file to find them. The retrieval layer wants nearest-neighbour search over embeddings, and Parquet has no index for that — so you bolt on a vector database and keep two copies of the data in sync. The dataset changes, and Parquet gives you no way back to the version that produced last week's model.

Lance answers all three from inside the file format. It is a columnar format in the Apache Arrow family, so it reads and writes through the same PyArrow tables as Parquet, but it stores a row-location index that makes random access cheap, ships vector and full-text indices as part of the dataset, and versions every write with ACID transactions, so you can check out an earlier version of a table the way you check out an earlier commit. It runs on local disk or object storage (S3, Azure, GCS, Hugging Face Hub), and it plugs into Pandas, Polars, DuckDB, Spark, and Ray.

The numbers explain the timing: the project has 7,336 stars and 944 forks, 699 releases, and pylance 14.0.3 landed on PyPI on November 15, 2026 — the release that put it on GitHub's daily trending list the next morning. This tutorial installs that exact version and puts each of its three headline features through a real run: conversion, random access, vector search, and versioning, with the timings we measured rather than the ones on the tin.

What you'll need#

  • Python 3.10 or newer. We used Python 3.12.3. A virtual environment is recommended.
  • pip and about 500 MB of disk for the libraries and the tutorial datasets. The 200,000-row benchmark file is about 105 MB; the main dataset is 11 MB.
  • No account, no API key, no GPU, no server. Everything below runs locally; Lance has no hosted component to sign into.

1. Install pylance#

Create a virtual environment and install the pinned versions we verified. pylance is the Python package; the other four are the libraries this tutorial uses alongside it.

python3 -m venv lance-env
source lance-env/bin/activate
pip install pylance==14.0.3 pyarrow==26.0.1 pandas==3.1.1 duckdb==2.0.0 numpy==2.5.5

Confirm the install answers with the version you asked for:

python -c "import lance; print(lance.__version__)"
14.0.3

That is the whole install. There is no server to start, no daemon, and no configuration file. If the import fails, the usual cause is a Python older than 3.10 or a stale pip — upgrade pip and retry before anything else.

2. Build a test dataset#

Real AI datasets pair rows of metadata with embedding vectors. This script builds a stand-in with the same shape: 20,000 rows, each with an id, a text field, a label, and a 128-dimensional float32 embedding. The vectors are random and seeded, so your run reproduces ours exactly; in your own work you would swap in embeddings from your encoder and keep the rest of the script unchanged.

import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

N, DIM = 20_000, 128
rng = np.random.default_rng(42)
labels = rng.choice(["cat", "dog", "bird", "fish"], size=N)
texts = [f"sample {i} a {labels[i]} in the wild" for i in range(N)]
vectors = rng.standard_normal((N, DIM)).astype(np.float32)
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)

table = pa.table({
    "id": pa.array(np.arange(N, dtype=np.int64)),
    "text": pa.array(texts),
    "label": pa.array(labels),
    "vector": pa.FixedSizeListArray.from_arrays(vectors.reshape(-1), DIM),
})
pq.write_table(table, "data.parquet")

Two details matter for what follows. First, the vector column is a fixed_size_list of 128 floats — Lance's vector index expects exactly this shape, so build it that way from the start rather than storing vectors as a variable-length list. Second, the vectors are L2-normalized, which makes Euclidean distance equivalent to cosine similarity for the nearest-neighbour queries in step 5.

3. Convert Parquet to Lance#

The conversion is the two lines the README promises: open the Parquet file as a dataset and write it out as Lance.

import lance

parquet = pa.dataset.dataset("data.parquet", format="parquet")
ds = lance.write_dataset(parquet, "data.lance")
print(ds.count_rows())   # 20000
print(ds.schema)
20000
id: int64
text: string
label: string
vector: fixed_size_list<item: float>[128]

The schema survives the round trip unchanged, and the dataset object you get back is itself a PyArrow dataset, so every Arrow tool keeps working. On our run the Parquet file was 11,163,841 bytes and the Lance file 10,509,460 bytes — essentially the same size, which is the point: Lance does not buy its features with a storage penalty. Read it back the same way you would read anything else:

ds = lance.dataset("data.lance")
df = ds.to_table().to_pandas()
df.head(3)

If you already have a Parquet lake, this is the migration story in full: convert in place, keep the same tooling, and gain the three features below. Nothing about the conversion is destructive — the Parquet file stays where it was.

4. Random access: fetch rows by id#

This is the feature that justifies the format for training workloads. A training loop, an evaluation harness, or an annotation tool fetches scattered rows by id; Parquet has no row index, so the honest way to do it is to read the whole file and filter. Lance keeps a map from row number to file location, so take() reads only the pages that hold your rows.

import time
import pyarrow.compute as pc

ids = sorted(rng.choice(20_000, size=200, replace=False).tolist())
t0 = time.perf_counter()
rows = ds.take(ids)
print(f"lance take(200): {(time.perf_counter() - t0) * 1000:.1f} ms, got {rows.num_rows} rows")

full = pq.read_table("data.parquet")
t0 = time.perf_counter()
sel = full.filter(pc.is_in(full["id"], value_set=pa.array(ids, type=pa.int64())))
print(f"parquet read-all + filter: {(time.perf_counter() - t0) * 1000:.1f} ms, got {sel.num_rows} rows")

Our run answered 13.0 ms for Lance against 77.1 ms for the Parquet read-and-filter, on the 20,000-row file. At 200,000 rows (a 105 MB file) the gap widened: Lance took 183–586 ms across three trials while Parquet took 378–2,528 ms. The ranges are wide because this ran on a shared 4-core VM, so treat the numbers as direction, not gospel — but the mechanism is not noisy: Lance reads only the pages it needs, Parquet decompresses the whole file, and the larger the file, the worse that ratio gets. Lance's own published benchmark, on the Oxford Pets image dataset, puts batched random access at 100x over Parquet; our small-scale run points the same way without reaching that multiple.

The practical consequence is mundane and valuable: random sampling for training batches, spot-checking rows during annotation, and fetching examples by id all stop being full-file scans.

Illustration of a magnifying lens picking out a few glowing rows from a vast grid of data columns
Random access reads only the pages that hold your rows. Illustration generated for AI Frontier Post.

5. Vector search: build an IVF_PQ index#

Storing embeddings next to their metadata is only half the job; the other half is searching them. Lance builds the index inside the dataset, so there is no second system to keep in sync. IVF_PQ is the standard choice at this scale: inverted-file partitioning (IVF) narrows the search to a few partitions, product quantization (PQ) compresses the vectors so the index stays small.

ds.create_index("vector",
                index_type="IVF_PQ",
                num_partitions=16,   # IVF
                num_sub_vectors=8)   # PQ
print(ds.list_indices())
[{'name': 'vector_idx', 'type': 'IVF_PQ', 'fields': ['vector'], 'version': 1, ...}]

The build took 4.3–6.0 seconds on 20,000 rows. Query it with the nearest argument to to_table — column, k, and the query vector:

q = vectors[0]
res = ds.to_table(nearest={"column": "vector", "k": 10, "q": q})
print(res["id"].to_pylist())
print([round(d, 4) for d in res["_distance"].to_pylist()])
[0, 19108, 9097, 6273, 2913, 5557, 3123, 19671, 10221, 6910]
[0.4814, 1.0149, 1.0154, 1.0353, 1.0429, 1.0432, 1.0614, 1.0615, 1.067, 1.0685]

The first query took 129.4 ms cold and 3.8 ms warm. Two things in that output deserve a second look. First, row 0 ranks first, as it must — the query vector is row 0's own embedding. Second, its distance is 0.4814, not 0. That is not a bug: product quantization compresses the stored vectors, so the distance you get back is measured against the quantized copy. Run the same query with the index switched off and the exact answer appears:

exact = ds.to_table(nearest={"column": "vector", "k": 10, "q": q, "use_index": False})
print(exact["id"].to_pylist()[:3])          # [0, 19043, 14806]
print(exact["_distance"][0].as_py())        # 0.0

Exact search took 24.5 ms on 20,000 rows and returned distance 0.0 for row 0, as expected. That comparison is the honest way to think about approximate indices: IVF_PQ trades a little accuracy for speed and size, and the trade is worth it once the row count makes brute force slow. On 20,000 rows brute force is still faster than the cold index query; at a million rows it is not. Notice also that the exact top 10 and the indexed top 10 disagree after rank 1 — the quantization reorders near-ties. If your application needs exact recall, run with use_index=False or add a re-ranking pass over the indexed candidates; if it needs speed at scale, IVF_PQ is the default for a reason.

6. Versioning and time travel#

Every mutation of a Lance dataset — a write, an append, an index build — creates a new version, and old versions stay readable. Append 5,000 fresh rows and watch the version counter move:

extra = pa.table({
    "id": pa.array(np.arange(20_000, 25_000, dtype=np.int64)),
    "text": pa.array([f"sample {i} extra" for i in range(20_000, 25_000)]),
    "label": pa.array(rng.choice(["cat", "dog", "bird", "fish"], size=5000)),
    "vector": pa.FixedSizeListArray.from_arrays(
        (lambda v: v / np.linalg.norm(v, axis=1, keepdims=True))(
            rng.standard_normal((5000, 128)).astype(np.float32)).reshape(-1), 128),
})
ds2 = lance.write_dataset(extra, "data.lance", mode="append")
print(ds2.count_rows())                    # 25000
print([v["version"] for v in ds2.versions()])  # [1, 2, 3]

Three versions: v1 was the original write, v2 the index build in step 5, v3 the append. Check out the first version and it is exactly as it was:

v1 = lance.dataset("data.lance", version=1)
print(v1.count_rows())   # 20000
latest = lance.dataset("data.lance")
print(latest.count_rows())  # 25000

Nothing is copied when you time-travel — versions share the underlying files and only the manifest changes, which is why the feature is described as zero-copy. For ML work this is the missing piece Parquet never had: the dataset version that produced a model is recoverable by number, so an experiment is reproducible from the data side, not just the code side. Tags and branches exist for naming versions (a prod tag, a dev branch); the version numbers are enough for most teams.

Illustration of dataset versioning as a branching timeline of stacked translucent data layers
Every write is a version; old versions stay readable without copying data. Illustration generated for AI Frontier Post.

7. Query it with DuckDB#

Lance datasets are Arrow datasets, so DuckDB can query them without any export step. Register the table and run SQL over it:

import duckdb

con = duckdb.connect()
con.register("lance_tbl", latest.to_table())
print(con.execute(
    "SELECT label, count(*) AS n FROM lance_tbl GROUP BY label ORDER BY label"
).fetchall())
[('bird', 6246), ('cat', 6357), ('dog', 6271), ('fish', 6126)]

The counts sum to 25,000 — the appended dataset, as expected. For columns you filter on often, add a scalar index the same way you added the vector index:

latest.create_scalar_index("label", index_type="BTREE")
print([v["version"] for v in latest.versions()])  # [1, 2, 3, 4]

The scalar index is itself a version bump — v4 — which is a useful reminder that in Lance, indices are part of the dataset's history, not sidecar files you can lose. The same dataset also answers to Pandas, Polars, Spark, and Ray without conversion; the DuckDB path above is the shortest route to SQL analytics over data that also serves vector search.

When to use this vs alternatives#

Use Lance when the workload is AI-shaped. Training loops that sample random rows, retrieval corpora where each row carries an embedding, datasets that mix images or audio with their metadata, and any pipeline where you need to roll a dataset back to the version that trained a model. That is the workload the format was designed for, and the three features above are the reason to switch.

Keep Parquet when the workload is pure SQL analytics. Parquet has the widest tool support of any columnar format, every warehouse and query engine reads it natively, and for scan-heavy analytics it is as fast as anything. If nothing in your pipeline fetches random rows or searches vectors, Lance adds nothing you need.

Use LanceDB when you want a database API, not a file format. LanceDB is a separate project built on the Lance format; it adds collections, a server, and a query API. Choose it when you want to call a vector database; choose Lance when you want files you can copy, version, and open in any Arrow tool.

Use DuckDB or pgvector when the data already lives in a database. If your embeddings are already in Postgres with pgvector, or your analytics already run in DuckDB, moving the data into Lance files is a migration, not a free win. Lance's advantage is for data that lives as files — in a lake, on object storage, or on a training machine.

Do not use Lance as a transactional database. It versions tables and supports ACID writes, but it is not a replacement for Postgres or SQLite for row-level updates and concurrent writers. Its transaction model is built for dataset-scale writes: appends, overwrites, and index builds.

The takeaway#

The interesting thing about Lance is not any single feature — random access, vector search, and versioning each exist elsewhere — but that all three live in the file format, with no server and no second system. A dataset that used to need Parquet plus a vector database plus a versioning scheme becomes one directory you can copy, query with DuckDB, and check out by version number.

The workflow to keep: convert a Parquet dataset with two lines, build an IVF_PQ index on the embedding column, and version every append so the data behind each model run is recoverable. Start with the 20,000-row script above, swap in your own embeddings, and measure the random-access gap on your own file sizes before deciding how much of your lake to move. The conversion is cheap and reversible — the Parquet file stays put — which makes the experiment nearly free. Our run converted, indexed, queried, and time-travelled on a 4-core VM in under a minute of compute; the format's case gets stronger as the files get bigger.

Sources: the lance-format/lance repository and lance.org. Star count, fork count, and trending placement as observed November 16, 2026; pylance 14.0.3 (PyPI, November 15, 2026) and all command outputs verified by running them in our sandbox. The 100x random-access figure is Lance's own published benchmark on the Oxford Pets dataset, cited as their claim.