Indexing speed, quantization, semantic chunking, and search-quality improvements #1

Merged

Sadykhzadeh merged 28 commits from indexing-improvements into master

2026-06-28 13:16:29 +00:00

Sadykhzadeh commented

2026-06-17 15:04:56 +00:00

(Migrated from github.com)

Indexing — faster and smaller

Quantize index vectors to int8 with a per-vector scale (~4x smaller per
vector, query stays f32 for accuracy). Index gains a format_version so a
stale index reports a clear "rebuild" message instead of decoding garbage.
Batch chunk embedding across files instead of one model call per file.
Reindex changed files against a precomputed path map (was O(files x records)),
and parallelise file read + change detection.
Classify text vs binary from a 1 KB prefix so each file is fully read once.
Write the index atomically (temp file + rename) to avoid torn reads.
Exclude .git, apply the configured exclude globs (previously a dead
field), and guard against reusing vectors after a model/dimension change.

Search — better results

treesitter feature: chunk on function/type boundaries across 17 languages,
with a tuned size cap for focused snippets; line-window fallback otherwise.
hnsw feature: optional approximate index that auto-activates above ~20k
chunks; the exact flat scan stays the default for smaller repos.
Scope MCP search to a subdirectory against the warm index.
Hot-reload the MCP index when it changes on disk (shared between watch and
the server).
Bounded lexical boost so exact-identifier queries rank well without disturbing
semantic ranking.

Product maturity

New status command (index size, chunk/file counts, model, staleness).
install cursor alongside install claude-code; install no longer clobbers
existing MCP servers.
Graceful handling of empty repositories and missing indexes.
Apache-2.0 LICENSE file, crate repository metadata, cleaned-up path output.

Tests, CI, and verification

Unit + integration tests across chunking, config, index, MCP, walking,
quantization, and the full index/search/incremental flow (lexical backend, no
model download).
A labelled retrieval benchmark and a real-model semantic test, both gated and
#[ignore]d so CI needs no download. On a multi-domain corpus they report
recall@1 = 1.00 and MRR = 1.000.
GitHub Actions running fmt, clippy (--all-features, denying warnings), the
default test suite, the pure-Rust feature tests, and an all-features build.

Verified end-to-end with the real model: semantic (by-meaning) retrieval, the
full MCP handshake, content-hash incremental indexing, and ~4 ms warm searches.

Works through the project roadmap (now / next / later) while adding a test suite, CI, and end-to-end verification with the real embedding model. The default build stays lean (lexical backend, line chunking); the heavier capabilities are opt-in cargo features. ## Indexing — faster and smaller - Quantize index vectors to int8 with a per-vector scale (~4x smaller per vector, query stays f32 for accuracy). Index gains a `format_version` so a stale index reports a clear "rebuild" message instead of decoding garbage. - Batch chunk embedding across files instead of one model call per file. - Reindex changed files against a precomputed path map (was O(files x records)), and parallelise file read + change detection. - Classify text vs binary from a 1 KB prefix so each file is fully read once. - Write the index atomically (temp file + rename) to avoid torn reads. - Exclude `.git`, apply the configured `exclude` globs (previously a dead field), and guard against reusing vectors after a model/dimension change. ## Search — better results - `treesitter` feature: chunk on function/type boundaries across 17 languages, with a tuned size cap for focused snippets; line-window fallback otherwise. - `hnsw` feature: optional approximate index that auto-activates above ~20k chunks; the exact flat scan stays the default for smaller repos. - Scope MCP `search` to a subdirectory against the warm index. - Hot-reload the MCP index when it changes on disk (shared between `watch` and the server). - Bounded lexical boost so exact-identifier queries rank well without disturbing semantic ranking. ## Product maturity - New `status` command (index size, chunk/file counts, model, staleness). - `install cursor` alongside `install claude-code`; install no longer clobbers existing MCP servers. - Graceful handling of empty repositories and missing indexes. - Apache-2.0 LICENSE file, crate `repository` metadata, cleaned-up path output. ## Tests, CI, and verification - Unit + integration tests across chunking, config, index, MCP, walking, quantization, and the full index/search/incremental flow (lexical backend, no model download). - A labelled retrieval benchmark and a real-model semantic test, both gated and `#[ignore]`d so CI needs no download. On a multi-domain corpus they report recall@1 = 1.00 and MRR = 1.000. - GitHub Actions running fmt, clippy (`--all-features`, denying warnings), the default test suite, the pure-Rust feature tests, and an all-features build. Verified end-to-end with the real model: semantic (by-meaning) retrieval, the full MCP handshake, content-hash incremental indexing, and ~4 ms warm searches.

Commenting is not possible because the repository is archived.

No reviewers