⚡ RaBitQ Gets Faster: Higher Recall, Lower Latency, Query-Time Control

High-recall vector search usually requires a refine step that re-ranks candidates against raw full-precision vectors — expensive in memory and tail latency. Multi-bit IVF_RQ in LanceDB now reaches 96.2% recall@10 with 5-bit codes at 2.6x lower p99 latency than unrefined IVF_PQ, no raw vectors needed.

New SIMD kernels and a fast rotation pass push throughput to 3.3x higher QPS per core. The approx_mode parameter lets you tune recall vs. latency at query time — fast, normal, or accurate — on the same index.

Read more →

💡 Case Study

How ByteDance’s Volcano Engine Rebuilt Its AI Stack on Lance, From Data Lake to Agent Memory at 100K+ QPS

Volcano Engine replaced single-node IVFPQ training (500GB+ memory, multi-day runs) with distributed index training across Lance fragments, cutting a 7-day model training pipeline to 1 day at 95% GPU utilization. Binary copy compaction skips decode/re-encode entirely, dropping compaction time from 418s to 15s on 5M-row tables.

ArkClaw, their managed OpenClaw deployment, runs memory-lancedb-ultra at 100K+ QPS with hybrid FTS+vector search and GitforMemory—agent memory branching built on Lance’s native branching API.

Volcano Engine’s Lance-Based AI Stack →

Rebuilding the Data Foundation for Embodied AI with Lance: From Long Videos to Random-Access-Friendly Multimodal Samples

China Merchants Lion Rock AI Lab rebuilt their robotics data pipeline on Lance to handle 100 Hz robot-arm states, multi-camera video streams, and per-frame annotations in a single table row. Long videos are sliced into GOP-sized blobs, preserving video-level compression while enabling frame-level random access without full-video decoding.

The result: 1.7–6.0× faster random reads than LeRobot, ~42% storage savings, and a single S3 copy that collection, processing, and training all read/write directly—eliminating the package→upload→download→reupload cycle.

Lance for Embodied AI Data →

📚 Also Published

📅 Upcoming Events

Actuate — Aug 18-19, 2026 · San Francisco, CA

LanceDB is sponsoring Actuate by Foxglove. Stop by Booth 15 to discuss data infrastructure for robotics—specifically how to make video, LiDAR, sensor data, and embeddings searchable for curation, training, and evaluation workflows.

We also have a lightning talk at 1pm on Aug 18 at the breakout stage!

Register →

Ray Summit — Aug 24-26, 2026 · San Francisco

LanceDB is sponsoring Ray Summit by Anyscale, stop by our booth!

🎤 How Exa Indexes the Entire Web: Ray Data and Lance at Billion-Document Scale

Lei Xu (LanceDB) · Hubert Yuan (Exa)

Catch our joint session with Exa covering how they index hundreds of billions of web pages using Lance and Ray Data—including dependency-graph-driven execution, fragment-level patching to avoid write amplification, and stateful model reuse via Ray Actors.

Register →

Composable Data Management Systems (CDMS) — September 4, 2026 · Boston, MA

🎤 The AI Frontier for Composable Data Systems

Weston Pace (LanceDB)

The talk explores how composable data systems with well-defined API boundaries and plugin architectures are positioned to support agentic workloads, covering architectural challenges like stability boundaries, business considerations around balancing enterprise and open source value, and the shift in development focus from implementation to review and design as AI-assisted contributions accelerate.

Register →

🏗️ LanceDB Enterprise Updates

Performance

  • Lock-free per-table freshness check — A lock-free atomic check replaces the per-table lock on read freshness, removing a scheduler handoff that capped throughput on busy tables; on a 126M-row table, ID-based row lookups at concurrency 64 rose from about 1,570 to 5,954 requests per second.
  • Full-text search index metadata caching — A new cache for full-text-search index snapshots avoids a storage read on every request; at 768 concurrent requests, a diagnostic benchmark measured 93.7% higher throughput and 56% lower median latency.
  • Faster result transfer for wide-column queries — Higher default connection-pool and buffer-pool sizes speed up result transfer between nodes; on wide-column take queries (150KB rows, 100 results, ~16MB per response), throughput rose 13.5% and median wire-transfer latency dropped from 226ms to 71ms.
  • Lower-contention index cache backend — A lower-contention backend for the index cache's memory tier removes per-read bookkeeping that scaled with core count; on a 320-core host under cache-heavy full-text-search load, throughput rose from 18.9 to 191.3 queries per second.
  • Faster distributed full-text-search planning — Distributed full-text-search query plans now decode index segment metadata synchronously instead of routing it through blocking-pool scheduling; in a controlled benchmark, throughput rose 27.9–31.1% and p99 latency fell 22.3–24.0% across query types.

Features

Feature Description
Job management Job views now support instant search across all fields, job cancellation, per-job failure reasons with retryability, archived job history, and a SQL SHOW JOBS statement, giving unified visibility into background jobs like index builds and cache prewarming alongside Feature Engineering jobs.
Table history Tables now have a history view showing every version with its schema changes and row count, with a diff against main for any version and pagination to scroll back through history.
Bring-your-own-bucket Enterprise clusters can bind to a customer-owned storage bucket or container instead of provisioned storage, covering the full path from cluster creation through the data plane, with single-writer ownership enforcement and an installer access preflight check.
Distributed ngram, bloom filter, r-tree, zone map & HNSW indexes The distributed indexer now supports building these five additional index types across multiple workers, broadening the set of index types that can be built at scale on large tables.

🌟 Open Source Releases

Project Description
Lance v8.0.0 – v9.0.0
Release notes		 • FTS v2 is now the default index format, with bulk MAXSCORE search for top-k disjunctions (#7512, #7603)
• Data overlay files enable in-place column updates without full fragment rewrites; compaction now triggers on overlay count (#7535, #7536, #7772)
• Streaming IVF k-means training for large datasets (#6913)
• Cold reads up to 8× faster via lazy column metadata loading (#7375)
• Object store metrics now published via the metrics crate and exposed through OpenTelemetry in Python (#7533, #7537)
• MemWAL adds row-level deletes via tombstones, prefiltered vector/FTS search, and snapshot-consistent fresh-tier membership (#7417, #7138, #7215)
LanceDB v0.34.0 – v0.36.0
Release notes		 • Branch diff and merge: promote columns added on a branch onto main via new client APIs (#3686)
• Elastic dataloader for PyTorch: iterable dataset that dynamically adjusts batch sizes based on memory pressure (#3509); remote tables now work with PyTorch dataloaders (#3432)
• OpenTelemetry metrics: Lance internal metrics now exposed via OTel in Python and Node bindings (#3609)
• OAuth authentication: new OAuth header provider for connecting to LanceDB Cloud with OAuth tokens (#3579, #3586)
lance-namespace-impls v0.4.1
Release notes		 • Lance tables in Hive Metastore are now stored as external tables, preventing accidental data deletion when dropping tables via HMS (#146)
lance-namespace v0.9.0
Release notes		 • Breaking: REST spec now includes response context, requiring updates to clients consuming the API (#358)
• Added tag field to DescribeTableRequest for table version/tag-based lookups (#345)
lance-context v0.5.1 – v0.6.5
Release notes		 • New RolloutDB for RL rollout storage: native RolloutStore with MemWAL ingest, server-id sharding, count/time-triggered WAL merges, and filtered/ordered trajectory reads (#124, #126, #142, #158)
• User-defined schemas via SchemaSpec and GenericStore, exposed across Rust core, HTTP server, client, and Python bindings (#218, #220)
• Control-plane master service with React UI for experiment browsing, centralized compaction/WAL-merge scheduler (etcd-backed HA), Prometheus /metrics endpoints, and stats-table observability (#137, #139, #140, #154)
lance-ray v0.5.0
Release notes		 • Distributed index building now supports IVF_RQ vector indexes (#5228), bitmap indexes (#5169), and ZoneMap indexes (#5214), with configurable num_segments for controlling parallelism (#5229)
• New add_columns_from enables distributed column addition across Ray workers (#4923); nested field paths now supported in Ray workflows (#5173)
• Vector search performance improved by sharing IVF centroids and PQ codebooks across Ray tasks (#4744) and reusing a global Ray pool with pickled dataset references (#5149, #5157)
lance-spark v0.6.0 – v0.7.1
Release notes		 • Spark 4.2 support added (#696)
• Full-text search SQL extension for querying Lance tables directly via Spark SQL (#501)
• DDL support for branch and tag operations, enabling version control workflows from Spark (#576, #654)
• Deferred index creation with WITH (train=false) for creating indexes without immediate training (#558)

🫶 Community Contributions

Thank you to contributors from Bytedance, Uber, Pinterest, Tencent, Baidu, Adobe, NVIDIA for improvements across storage, indexing, query execution, distributed processing, and ecosystem integrations in LanceDB, Lance, and the broader ecosystem.

Notable contributions this month:

  • @XuQianJin-Stars — Added GooseFS object store support and implemented namespace directory table operations (add/alter/drop columns, update, delete)
  • @zhangyue19921010 — Introduced RowAddrRemap structure to prevent OOM during compaction and migrated distributed BTree builds to segmented index framework
  • @ddupg — Added TOS (Volcengine) object store support and enabled distributed vector/FTS segment builds with direct commit paths
  • @tobocop2 — Enabled runtime SIMD dispatch for pre-Haswell x86_64 builds, expanding hardware compatibility for from-source installations
  • @valkum — Added Dict-to-value-type casting in alter_columns schema evolution and fixed datafusion filter coercion for dictionary-encoded columns
  • @wombatu-kun — Accelerated regex and infix LIKE queries using ngram indexes and exposed per-query I/O metrics on ANN operators
  • @Ali2Arslan — Enabled reading column min/max directly from ZoneMap without full scans and implemented single-flight scalar index opening
  • @beinan — Implemented FM-Index scalar index for exact substring search and added Java schema override for fragment writes
  • @gstamatakis95 — Added shared RaBitQ rotation for distributed IVF_RQ builds and eliminated HEAD calls when opening vector indexes
  • @LuciferYang — Added Spark 4.2 support and contributed zero-copy BFloat16Array construction for improved memory efficiency

A heartfelt thank you to our community contributors of Lance and LanceDB this past month:

@2dmurali • @a-agmon • @adibaadi • @aimanmalib • @alowator • @ar-maan05 • @bugwz • @chakshu-dhannawat • @charleshuang119 • @chuenchen309 • @ckarnell • @claydugo • @clearlove10-c • @coyasong • @cswpy • @danielmao1 • @dcfocus • @dentiny • @devteamaegis • @ebyhr • @ecthlion • @erandagan • @everysympathy • @expyron • @fangbo • @fanng1 • @farmerchillax • @fl0-m • @geserdugarov • @ghx5t-sol • @glitch-ux • @goutamadwant • @haochengliu • @haroldbenoit • @hashwnath • @hellower • @hfutatzhanghb • @huahuay • @hushengquan • @ivscheianu • @j7nhai • @jay-ju • @jiaoew1991 • @jiaqizho • @jo-migo • @joaquinhuigomez • @jsap0914 • @jtuglu1 • @julianyg • @kaan-simbe • @kobihikri • @lakshjain7 • @leepokai • @leohoare • @leoreeyang • @lixmgl • @malinjawi • @mansiverma897993 • @markmcd • @mateuszossgit • @mediamana • @missing-identity • @mmatczuk • @mocobeta • @moongtnt • @morales-t-netflix • @neo-x7 • @niraj-mx07 • @nuthalapativarun • @nyl3532016 • @omkar-334 • @plotor • @pranavachar01 • @prrao87 • @puchengy • @rtmalikian • @ryantqiu • @saitejabandaru-in • @sanskar-singh-2403 • @sapnilb15 • @say-5 • @sbrunk • @sezruby • @shizoqua • @skycutter • @skyshineb • @sohumt123 • @solaris-star • @spectual • @ssrheart425 • @stumpylog • @summaryzb • @sushanth012 • @timsaucer • @touch-of-grey • @u70b3 • @vibhujawa • @vitaliy-pikalo • @vortex-captain • @wayneadams • @wending-y • @whitewooood • @wirybeaver • @wulansari999 • @xingsuo-zbz • @xixigoodluck • @xloya • @xtangxtang • @xuxiaoqiang666 • @xuzha • @yanghua • @yangjunz • @yangshangqing95 • @yaodong-shen • @yesunbmh • @yeung108 • @yohahaha • @ytyky • @yuju-huang • @yuvalif • @ywu342 • @yyzhao2025 • @zhangyang0418 • @zlepper • @ztorchan

🤝 Lance Community Sync Recap

Community sync sessions this month covered several significant proposals and releases. On the proposal side, discussions included new low-level APIs, a blob session API, multi-table commit and branching scalability improvements, and a design for scaling to millions of fragments. The team announced the Lance 9.0.0 SDK release with a 10.0.0 beta incoming, highlighted a 7x FTS performance improvement, and introduced Lance Gatekeeper, an automated PR review bot. Additional topics included the lance.org website redesign with plans for community content, and a proposal to streamline the format-change voting process by moving votes into PRs with a shortened 72-hour window.

The next Lance Community Sync will take place on Thursday, August 13 @ 9am PT.