Embedded ParqDB

The embedded runtime runs in the Python process and is ParqDB’s primary supported path. It uses DataFusion for relational execution, native Rust code for index construction and distance kernels, SQLite for catalog state, and Parquet for the index tables.

Use it for local development, single-node analytical workloads, batch jobs, or applications that already expose Parquet data to an embedded query engine.

Install

python -m pip install parqdb

See getting started for the supported Python and platform matrix and a complete first query.

Open a Session

The shortest configuration keeps the catalog and index data together:

import parqdb

session = parqdb.connect("./parqdb-data")

The directory contains the SQLite catalog, immutable metadata documents, and Parquet index snapshots. Reopening the same path restores persistent source registrations and index mappings.

Store index tables on shared storage while keeping the local session state under one directory:

session = parqdb.connect(
    "/var/lib/parqdb",
    warehouse="s3://lakehouse-indexes/parqdb",
    storage_options={"aws_region": "us-east-1"},
)

See configuration for S3-compatible storage and HDFS settings.

Register a Source

Register one absolute file, directory, or wildcard pattern:

session.register_parquet(
    "documents",
    "file:///data/documents/*/part-*.parquet",
)
documents = session.table("documents")

The registration is durable. A new session opened on the same catalog can call session.table("documents") without registering it again. ParqDB stores the location pattern rather than a one-time list of matching files.

Use session.deregister_table("documents") before rebinding the same logical name to a different location.

Build and Monitor an Index

from datetime import timedelta

documents.create_index(
    "documents_embedding",
    column="embedding",
    key=["document_id"],
    config=parqdb.IVF(nlist=4096),
)
documents.wait_for_index(
    "documents_embedding",
    timeout=timedelta(minutes=30),
)

Construction runs asynchronously. Inspect progress without blocking:

status = documents.index_status("documents_embedding")
print(status.state, status.phase, status.completed, status.total)

Tune physical Parquet output independently:

documents.create_index(
    "documents_embedding",
    column="embedding",
    key=["document_id"],
    config=parqdb.IVF(nlist=4096, encoding="lvq8"),
    writer_options=parqdb.WriteOptions(
        compression="zstd(3)",
        target_file_size=512 * 1024 * 1024,
    ),
)

Start with defaults. nlist and query-time nprobes have the largest effect on IVF recall and candidate work. Writer settings primarily affect persistence and scan layout.

The default source encoding keeps vectors in the source table. lvq4 and lvq8 store compact codes in postings. Set metric="cosine" to normalize source and query vectors and report cosine distance; the default is l2_squared.

Query and Continue with DataFusion

Collect a portable Arrow result:

query = (
    documents.search(query_vector, column="embedding")
    .where("tenant_id = 42")
    .nprobes(64)
    .limit(1_000)
    .select(["document_id", "category"])
)
hits = session.collect(query)

Compile the vector query as a SQL subquery when relational work follows:

search_sql = session.to_sql(query)
result = session.sql(f"""
    SELECT category, COUNT(*) AS matches, AVG(_distance) AS avg_distance
    FROM ({search_sql}) AS hits
    GROUP BY category
    ORDER BY category
""")

The vector search and aggregation remain in one DataFusion plan. The generated SQL is executable in the originating session because it refers to registered source and index tables.

Page Cache

Repeated queries use the session’s bounded decompressed Parquet Page cache:

stats = session.parquet_page_cache_stats()
print(stats.resident_bytes, stats.hits, stats.misses)
session.clear_parquet_page_cache()

The cache is capacity-bounded and shared by Parquet index and source scans in the session. Configure it with parqdb.parquet.page_cache.capacity.

Refresh and Remove

After the source changes, rebuild and atomically publish a new snapshot:

documents.refresh_index("documents_embedding")
documents.wait_for_index("documents_embedding")

Dropping an index removes catalog visibility but deliberately leaves immutable objects for retention-safe cleanup:

documents.drop_index("documents_embedding")

Retention and orphan cleanup behavior is summarized in current limitations.

Diagnose a Query

print(session.explain(query))
print(session.analyze(query))

analyze reports candidate counts, distance and Top-K time, bytes scanned, and other physical metrics. See troubleshooting for common schema, catalog, storage, and performance failures.