One search that answers any shape of question about the code #2

Merged

Sadykhzadeh merged 29 commits from retrieval/experiment-run into master

2026-07-29 12:16:20 +00:00

Sadykhzadeh commented

2026-07-26 20:50:46 +00:00

(Migrated from github.com)

Turns search into a single retrieval primitive that answers "where is the code
that does X" whether X is an identifier, an error string, a half-remembered name,
or a plain-language description. Previously only one of those shapes worked well
depending on which retriever you happened to hit.

What changed

A lexical arm, fused with the vector arm. src/lexical.rs adds code-aware
BM25 over per-chunk term vectors, and search merges the two retrievers by rank
fusion rather than exposing them as separate tools. An adjacent "second opinion"
tool is one agents skip, so there is one ranked call and no exhaustive regex mode.

A symbol layer. src/symbol.rs names the declaration each chunk belongs to
and carries that name as a searchable term, so a result can say which function it
is rather than just which byte range.

Documentation indexed as records of its own. Doc comments and docstrings
become separate short records pointing at the same declaration. This is the
single largest gain in the run (0.679 → 0.733, every corpus improved), and a 2×2
showed why: doc records alone +0.009, BM25 length normalisation alone +0.009,
together +0.054. A doc record is a short document naming a declaration, which is
exactly what the length term rewards.

Declarations ranked above the prose describing them. Chunks belonging to a
declaration now outrank those that do not (0.735 → 0.785, every corpus and every
query shape improved). This came from dumping the results for every unanswered
question rather than theorising: 18 of 50 top-5 slots were documentation or
manifests and 9 were tests, against 23 of code. Both retrievers prefer prose
honestly — a changelog repeats the question's words better than the function does.

Deduplication and tie-breaking. One result per place instead of per record,
and ranking ties broken by confidence.

Indexing fixes. Machine-written files (lockfiles, generated output) are
excluded by default, and internal-path filters are judged relative to the index
root instead of the process CWD.

Measurement

tests/retrieval_quality.rs is the substance of this branch: 73 labelled queries
in 4 shapes across 4 pinned corpora (ripsem's own src, requests v2.34.2,
gorilla/mux v1.8.1, got v14.4.5), 2304 chunks.

arm	MRR
merged	0.785
lexical only	0.631
vector only	0.627

Reading cost to reach the answer: 439 bytes, against 14410 for the whole file it
lives in.

Which arm is stronger reverses between repositories — BM25 wins on Go and
Rust, the embedding model wins decisively on Python (requests: 0.747 vs 0.531),
TypeScript is level. An earlier README claim that BM25 beats the model on real
code was an artefact of only ever measuring ripsem's own Rust source. Merged is
not best on every corpus (it costs 0.034 on requests); it wins because nothing
tells you in advance which arm a given repository needs.

Reviewer notes

The corpora are pinned and must stay pinned. Before pinning, the corpus was
ripsem's live source and moved with every commit, so two runs a few commits
apart were not comparable — that confounding sent two decisions the wrong way.
Never compare numbers across a corpus change.
The benchmarks must run one at a time. They rebuild the cached corpora's
index in place and cargo parallelises tests, so a whole-file run had two
measurements overwriting each other and reporting plausible wrong numbers. The
harness mutex that serialises them is load-bearing; don't replace it with
--test-threads=1.
Known bias, recorded in the README: every label names a source file, so a
rule preferring source is partly rewarded by construction.
The misses diagnostic test is deliberately kept. It distinguishes "absent
from every arm at any depth" (an indexing problem) from "sitting at rank 40" (a
scoring problem) — a distinction the aggregate MRR cannot make.
Two commits are docs: only, recording measured dead ends so they are not
retried: a larger embedding model (bge-base-en-v1.5 is the best arm alone at
0.564 vs 0.547 and the worst merged at 0.628 vs 0.679), a context header on the
embedded text, and arm-competence weighting.

Open problems

Descriptions in the reader's own words remain the weakest shape at 0.595, and
eight labelled questions still miss the top five — only one of those is a true
retrieval failure, the rest are indexed but ranked below five. Messages are still
better served by lexical alone (1.000 vs 0.944) and misremembered names by vector
alone (0.750 vs 0.702): rank fusion counts absence from an arm as evidence
against a hit, so a silent arm votes down what the other found. Nothing folds
plurals, so models does not reach ModelInfo.

Turns `search` into a single retrieval primitive that answers "where is the code that does X" whether X is an identifier, an error string, a half-remembered name, or a plain-language description. Previously only one of those shapes worked well depending on which retriever you happened to hit. ## What changed **A lexical arm, fused with the vector arm.** `src/lexical.rs` adds code-aware BM25 over per-chunk term vectors, and `search` merges the two retrievers by rank fusion rather than exposing them as separate tools. An adjacent "second opinion" tool is one agents skip, so there is one ranked call and no exhaustive regex mode. **A symbol layer.** `src/symbol.rs` names the declaration each chunk belongs to and carries that name as a searchable term, so a result can say which function it is rather than just which byte range. **Documentation indexed as records of its own.** Doc comments and docstrings become separate short records pointing at the same declaration. This is the single largest gain in the run (0.679 → 0.733, every corpus improved), and a 2×2 showed why: doc records alone +0.009, BM25 length normalisation alone +0.009, together +0.054. A doc record is a short document naming a declaration, which is exactly what the length term rewards. **Declarations ranked above the prose describing them.** Chunks belonging to a declaration now outrank those that do not (0.735 → 0.785, every corpus and every query shape improved). This came from dumping the results for every unanswered question rather than theorising: 18 of 50 top-5 slots were documentation or manifests and 9 were tests, against 23 of code. Both retrievers prefer prose honestly — a changelog repeats the question's words better than the function does. **Deduplication and tie-breaking.** One result per place instead of per record, and ranking ties broken by confidence. **Indexing fixes.** Machine-written files (lockfiles, generated output) are excluded by default, and internal-path filters are judged relative to the index root instead of the process CWD. ## Measurement `tests/retrieval_quality.rs` is the substance of this branch: 73 labelled queries in 4 shapes across 4 pinned corpora (ripsem's own `src`, requests v2.34.2, gorilla/mux v1.8.1, got v14.4.5), 2304 chunks. | arm | MRR | |---|---| | merged | **0.785** | | lexical only | 0.631 | | vector only | 0.627 | Reading cost to reach the answer: 439 bytes, against 14410 for the whole file it lives in. Which arm is stronger **reverses between repositories** — BM25 wins on Go and Rust, the embedding model wins decisively on Python (requests: 0.747 vs 0.531), TypeScript is level. An earlier README claim that BM25 beats the model on real code was an artefact of only ever measuring ripsem's own Rust source. Merged is not best on every corpus (it costs 0.034 on requests); it wins because nothing tells you in advance which arm a given repository needs. ## Reviewer notes - **The corpora are pinned and must stay pinned.** Before pinning, the corpus was ripsem's live source and moved with every commit, so two runs a few commits apart were not comparable — that confounding sent two decisions the wrong way. Never compare numbers across a corpus change. - **The benchmarks must run one at a time.** They rebuild the cached corpora's index in place and cargo parallelises tests, so a whole-file run had two measurements overwriting each other and reporting plausible wrong numbers. The harness mutex that serialises them is load-bearing; don't replace it with `--test-threads=1`. - **Known bias, recorded in the README:** every label names a source file, so a rule preferring source is partly rewarded by construction. - The `misses` diagnostic test is deliberately kept. It distinguishes "absent from every arm at any depth" (an indexing problem) from "sitting at rank 40" (a scoring problem) — a distinction the aggregate MRR cannot make. - Two commits are `docs:` only, recording measured dead ends so they are not retried: a larger embedding model (bge-base-en-v1.5 is the best arm alone at 0.564 vs 0.547 and the worst merged at 0.628 vs 0.679), a context header on the embedded text, and arm-competence weighting. ## Open problems Descriptions in the reader's own words remain the weakest shape at 0.595, and eight labelled questions still miss the top five — only one of those is a true retrieval failure, the rest are indexed but ranked below five. Messages are still better served by lexical alone (1.000 vs 0.944) and misremembered names by vector alone (0.750 vs 0.702): rank fusion counts absence from an arm as evidence against a hit, so a silent arm votes down what the other found. Nothing folds plurals, so `models` does not reach `ModelInfo`.

Commenting is not possible because the repository is archived.

No reviewers