Benchmarks

All benchmarks run on PostgreSQL 18, Apple M-series (12 CPU, 64 GB RAM) with shared_buffers=4GB, work_mem=256MB, maintenance_work_mem=2GB. Warm cache, average of 5 runs.

EXPLAIN ANALYZE – query latency and buffer reads

1M rows (71 MB)

Query sorted_heap heap + btree heap seqscan
Point (1 row) 0.035 ms / 1 buf 0.046 ms / 7 bufs 15.2 ms / 6,370 bufs
Narrow (100 rows) 0.043 ms / 2 bufs 0.067 ms / 8 bufs 16.2 ms / 6,370 bufs
Medium (5K rows) 0.434 ms / 33 bufs 0.492 ms / 52 bufs 16.1 ms / 6,370 bufs
Wide (100K rows) 7.5 ms / 638 bufs 8.9 ms / 917 bufs 17.4 ms / 6,370 bufs

10M rows (714 MB)

Query sorted_heap heap + btree heap seqscan
Point (1 row) 0.034 ms / 1 buf 0.047 ms / 7 bufs 117.9 ms / 63,695 bufs
Narrow (100 rows) 0.037 ms / 1 buf 0.062 ms / 7 bufs 130.9 ms / 63,695 bufs
Medium (5K rows) 0.435 ms / 32 bufs 0.549 ms / 51 bufs 131.0 ms / 63,695 bufs
Wide (100K rows) 7.6 ms / 638 bufs 8.8 ms / 917 bufs 131.4 ms / 63,695 bufs

100M rows (7.8 GB)

Query sorted_heap heap + btree heap seqscan
Point (1 row) 0.045 ms / 1 buf 0.506 ms / 8 bufs 1,190 ms / 519,906 bufs
Narrow (100 rows) 0.166 ms / 2 bufs 0.144 ms / 9 bufs 1,325 ms / 520,782 bufs
Medium (5K rows) 0.479 ms / 38 bufs 0.812 ms / 58 bufs 1,326 ms / 519,857 bufs
Wide (100K rows) 7.9 ms / 737 bufs 10.1 ms / 1,017 bufs 1,405 ms / 518,896 bufs

At 100M rows, a point query reads 1 buffer (vs 8 for btree, 519,906 for sequential scan).

pgbench throughput (TPS)

Prepared mode (-M prepared)

Query planned once, re-executed with parameters. 10 s, 1 client.

Query 1M (sh / btree) 10M (sh / btree) 100M (sh / btree)
Point 46.9K / 59.4K 46.5K / 58.0K 32.6K / 43.6K
Narrow 22.3K / 29.1K 22.5K / 28.8K 17.9K / 18.1K
Medium 3.4K / 5.1K 3.4K / 4.8K 2.4K / 2.4K
Wide 295 / 289 293 / 286 168 / 157

Simple mode (-M simple)

Each query parsed, planned, and executed separately. 10 s, 1 client.

Query 1M (sh / btree) 10M (sh / btree) 100M (sh / btree)
Point 28.4K / 38.0K 29.1K / 41.4K 18.7K / 4.6K
Narrow 19.6K / 24.4K 21.8K / 27.6K 7.1K / 5.5K
Medium 3.1K / 3.7K 3.4K / 4.8K 2.1K / 1.6K
Wide 198 / 290 200 / 286 163 / 144

At 100M rows in simple mode, sorted_heap wins all query types. Point queries reach 18.7K TPS vs 4.6K for btree (4x).

INSERT and compaction throughput

Scale sorted_heap heap + btree heap (no index) compact time
1M 923K rows/s 961K 1.91M 0.3 s
10M 908K rows/s 901K 1.65M 3.1 s
100M 840K rows/s 1.22M 2.22M 41.3 s

Storage

Scale sorted_heap heap + btree heap (no index)
1M 71 MB 71 MB (50 + 21) 50 MB
10M 714 MB 712 MB (498 + 214) 498 MB
100M 7.8 GB 7.8 GB (5.7 + 2.1) 5.7 GB

sorted_heap stores data + zone map in the same space as a heap table. The zone map replaces the btree index, so total storage is comparable to heap-without-index – roughly 30% less than heap + btree at scale.

Current local synthetic benchmark (sorted_hnsw Index AM)

Repo-owned harnesses:

  • python3 scripts/bench_gutenberg_local_dump.py --dump /tmp/cogniformerus_backup/cogniformerus_backup.dump --port 65473
  • REMOTE_PYTHON=/path/to/python SH_EF=32 EXTRA_ARGS='--sh-ef-construction 200' ./scripts/bench_gutenberg_aws.sh <aws-host> /path/to/repo /path/to/dump 65485
  • scripts/bench_sorted_hnsw_vs_pgvector.sh /tmp 65485 10000 20 384 10 vector 64 96
  • python3 scripts/bench_ann_real_dataset.py --dataset nytimes-256 --sample-size 10000 --queries 20 --k 10 --pgv-ef 64 --sh-ef 96 --zvec-ef 64 --qdrant-ef 64
  • python3 scripts/bench_qdrant_synthetic.py --rows 10000 --queries 20 --dim 384 --k 10 --ef 64
  • python3 scripts/bench_zvec_synthetic.py --rows 10000 --queries 20 --dim 384 --k 10 --ef 64

Current AWS restored-corpus benchmark (~104K x 2880D, Gutenberg dump)

AWS ARM64 host (4 vCPU, 8 GiB RAM), top-10, restored PostgreSQL custom dump. Ground truth is recomputed by exact heap search on the restored svec table. In the current rerun the stored bench_hnsw_gt table matched the exact heap GT on 100% of the 50 benchmark queries, so the fresh exact heap GT and the historical GT table agree. This rerun uses sorted_hnsw ef_construction=200 and ef_search=32, and the harness reconnects after build before timing ordered scans.

Method p50 latency Recall@10 Notes
Exact heap (svec) 458.762 ms 100.0% brute-force GT on restored corpus
sorted_hnsw (svec) 1.287 ms 100.0% ef_construction=200, ef_search=32, index 404 MB, total 1902 MB
sorted_hnsw (hsvec) 1.404 ms 100.0% ef_construction=200, ef_search=32, index 404 MB, total 1032 MB
pgvector HNSW (halfvec) 2.031 ms 99.8% ef_search=64, index 804 MB, total 1615 MB
zvec HNSW 50.499 ms 100.0% in-process collection, ef=64, ~1.12 GiB on disk
Qdrant HNSW 6.028 ms 99.2% local Docker on same AWS host, hnsw_ef=64, 103,260 points

The precision-matched PostgreSQL comparison on Gutenberg is now sorted_hnsw (hsvec) vs pgvector halfvec: 1.404 ms @ 100.0% versus 2.031 ms @ 99.8%, with total footprint 1032 MB versus 1615 MB. The raw fastest PostgreSQL row on this corpus is still sorted_hnsw (svec) at 1.287 ms, but that uses float32 source storage. sorted_hnsw keeps the same 404 MB index in both cases because the AM stores SQ8 graph state; the storage gain from hsvec appears in the base table and TOAST footprint instead.

Synthetic 10K x 384D cosine corpus, top-10, warm query loop. PostgreSQL methods were rerun across 3 fresh builds and the table below reports median p50 / median recall. Qdrant uses 3 warm measurement passes on one local Docker collection.

Method p50 latency Recall@10 Notes
Exact heap (svec) 2.03 ms 100% Brute-force ground truth
sorted_hnsw 0.158 ms 100% shared_cache=on, ef_search=96, index ~5.4 MB
pgvector HNSW (vector) 0.446 ms 90% median (90-95 range) ef_search=64, same M=16, ef_construction=64, index ~2.0 MB
zvec HNSW 0.611 ms 100% local in-process collection, ef=64
Qdrant HNSW 1.94 ms 100% local Docker, hnsw_ef=64

Current local real-dataset sample (nytimes-256-angular)

Repo-owned harness:

  • python3 scripts/bench_ann_real_dataset.py --dataset nytimes-256 --sample-size 10000 --queries 20 --k 10 --pgv-ef 64 --sh-ef 96 --zvec-ef 64 --qdrant-ef 64

ANN-Benchmarks nytimes-256-angular, sampled to 10K base vectors and 20 queries, top-10. The table below reports medians across 3 full harness runs. Ground truth comes from exact PostgreSQL heap search on the sampled svec corpus.

Method p50 latency Recall@10 Notes
Exact heap (svec) 1.557 ms 100% brute-force ground truth
sorted_hnsw 0.327 ms 85.0% median (83.5-85.5 range) shared_cache=on, ef_search=96, index ~4.1 MB
pgvector HNSW (vector) 0.751 ms 79.0% median (78.5-79.0 range) ef_search=64, same M=16, ef_construction=64, index ~13 MB
zvec HNSW 0.403 ms 99.5% local in-process collection, ef=64, ~14.1 MB on disk
Qdrant HNSW 1.704 ms 99.5% local Docker, hnsw_ef=64

This corpus is materially harder than the deterministic synthetic one. It is a better signal for default-parameter recall, while the synthetic table remains useful for controlled same-host engine comparisons and regression tracking.

Current local GraphRAG benchmark (person -> parent -> city, stable fact contract)

Repo-owned harness:

  • python3 scripts/bench_graph_rag_multihop.py --num-pairs 5000 --query-count 64 --runs 3 --dim 384 --ann-k 64 --top-k 10 --ef-search 128 --ef-construction 200 --m 24 --pgv-ef-search 64 --zvec-ef 64 --qdrant-ef 64 --shared-buffers-mb 64 --backend-mode fresh

Deterministic fact graph, 5K chains / 10K rows total, 384D, top-10. This is the current balanced stable GraphRAG point for the narrow fact-shaped fact-retrieval workload. The stable-facing SQL entry point is sorted_heap_graph_rag(...); the table below also shows the underlying helper and wrapper paths because the harness measures the dispatched execution path directly.

Method p50 latency hit@1 hit@k Notes
Heap two-hop SQL 0.762 ms 75.0% 96.9% exact rerank over expanded heap set
sorted_heap_graph_rag_twohop_scan() 0.720 ms 75.0% 96.9% old city-only rerank contract
sorted_heap_expand_twohop_rerank() 0.712 ms 75.0% 96.9% same city-only seed point
sorted_heap SQL pathsum baseline 0.847 ms 98.4% 98.4% hop1_distance + hop2_distance in SQL
sorted_heap_graph_rag_twohop_path_scan() 0.739 ms 98.4% 98.4% fused path-aware wrapper
sorted_heap_expand_twohop_path_rerank() 0.726 ms 98.4% 98.4% same knobs, fused path-aware helper
pgvector HNSW + heap expansion 2.588 ms 90.6% 90.6% path-aware rerank, ef_search=64
zvec HNSW + heap expansion 2.507 ms 100.0% 100.0% path-aware rerank, ef=64
Qdrant HNSW + heap expansion 4.947 ms 100.0% 100.0% path-aware rerank, hnsw_ef=64

The path-aware helper changes the local conclusion materially: the dominant quality issue on this fact-shaped workload was the old hop-2-only rerank contract, not seed ANN quality. The fused path-aware helper now gives the best local latency/quality point at the same m=24, ef_construction=200, ann_k=64, ef_search=128 operating point.

Under the same path-aware scorer contract, the current local conclusion gets sharper: sorted_heap keeps the latency lead, while zvec and Qdrant reach the strongest observed answer quality on this deterministic fact graph.

A repeated-build local protocol then quantified how much of this is just single-run luck. Using three independent fresh builds of the same 5K/384D balanced point:

  • sorted_heap_expand_twohop_path_rerank() median 0.798 ms, range 0.771-0.819 ms, hit@1 = 98.4%, hit@k = 98.4% on every build
  • sorted_heap_graph_rag_twohop_path_scan() median 0.796 ms, range 0.778-0.804 ms, hit@1 = 98.4%, hit@k = 98.4% on every build
  • pgvector path-aware parity row median 1.405 ms, hit@1/hit@k 85.9-89.1%
  • zvec path-aware parity row median 1.076 ms, 100.0% / 100.0%
  • Qdrant path-aware parity row median 2.799 ms, 100.0% / 100.0%

So on the local balanced point, the current sorted_heap path-aware rows are not a fragile one-off. The latency band is tight, and the answer quality did not drift across the three rebuilds.

shared_cache=on vs off for multihop (post-fix, 0490cc4)

A multi-index shared-cache corruption bug was fixed in commit 0490cc4. Previously, when two or more sorted_hnsw indexes existed in the same database and shared_cache=on, a publish for the second index could overwrite shared memory that an attached cache for the first index was still reading through bare pointers. This caused silent data corruption and 0% retrieval quality on the affected index.

The fix deep-copies all data (L0 neighbors, SQ8 vectors, upper-level neighbor slabs) from shared memory into local palloc’d buffers on attach. The exact failure mode is regression-guarded by the multi-index overwrite phase (B5) in scripts/test_hnsw_chunked_cache.sh.

Post-fix verification on the 5K x 384D multihop benchmark (fresh backends, 3 runs, ann_k=64, ef_search=128, m=24):

python3 scripts/bench_graph_rag_multihop.py \
  --num-pairs 5000 --dim 384 --query-count 64 --runs 3 \
  --ann-k 64 --top-k 10 --ef-search 128 --ef-construction 200 --m 24 \
  --shared-cache {on,off} --backend-mode fresh \
  --skip-zvec --skip-qdrant --skip-pgvector
Method shared_cache p50 hit@1 hit@k
sorted_heap_expand_twohop_path_rerank() off 0.780 ms 98.4% 98.4%
sorted_heap_expand_twohop_path_rerank() on 0.766 ms 98.4% 98.4%
sorted_heap_graph_rag_twohop_path_scan() off 0.804 ms 98.4% 98.4%
sorted_heap_graph_rag_twohop_path_scan() on 0.786 ms 98.4% 98.4%

Also verified at 10K x 384D:

python3 scripts/bench_graph_rag_multihop.py \
  --num-pairs 10000 --dim 384 --query-count 64 --runs 3 \
  --ann-k 64 --top-k 10 --ef-search 128 --ef-construction 200 --m 24 \
  --shared-cache {on,off} --backend-mode fresh \
  --skip-zvec --skip-qdrant --skip-pgvector
Method shared_cache p50 hit@1 hit@k
sorted_heap_graph_rag_twohop_path_scan() off 0.904 ms 95.3% 96.9%
sorted_heap_graph_rag_twohop_path_scan() on 0.972 ms 95.3% 96.9%

Quality is identical between on and off at both measured scales. Latency at these measured scales is mixed rather than a clear universal win. shared_cache=off is no longer needed as a correctness workaround.

Larger-scale cold-start measurement (100K x 32D, 200K rows, 47 MB index):

The deep-copy fix means attach now copies all L0 neighbors (~38 MB), SQ8 data (~6 MB), and node metadata (~6 MB) from shared memory into local buffers on each fresh backend’s first query. At 200K nodes, this upfront cost exceeds the lazy page-decode path used by shared_cache=off:

# 100K pairs, 32D, m=24, ef_construction=200, ef_search=128, 20 fresh backends per mode
# PG buffer cache warm (shared_buffers=256MB):
shared_cache=off: cold KNN p50=6.31ms
shared_cache=on:  cold KNN p50=8.68ms

# PG buffer cache cold (PG restarted between modes, shared_buffers=64MB):
shared_cache=off: cold KNN first=5.3ms  p50=5.7ms
shared_cache=on:  cold KNN first=8.6ms  p50=9.5ms

The off path loads L0 pages lazily (only pages visited by beam search). The on path deep-copies all bulk data upfront under LWLock. At 200K nodes, the upfront copy dominates. Quality remains identical.

Current conclusion: shared_cache=on is a correctness-safe default but not a performance feature at the measured scales. The deep-copy overhead from the multi-index fix neutralized the original latency benefit.

Synthetic multi-hop depth scaling (relation_path depth 1..5)

Repo-owned harness:

  • python3 scripts/bench_graph_rag_multidepth.py --num-pairs 5000 --max-depth 5 --query-count 32 --runs 3 --dim 384 --ann-k 64 --top-k 10 --ef-search 128 --ef-construction 200 --m 24 --shared-buffers-mb 64

Deterministic chain-shaped fact graph, 25K rows total, 384D, top-10, fresh backend, path-aware scorer. This is a narrow scaling check for the generic unified syntax:

  • relation_path := ARRAY[1]
  • relation_path := ARRAY[1,2]
  • relation_path := ARRAY[1,2,3]
  • relation_path := ARRAY[1,2,3,4]
  • relation_path := ARRAY[1,2,3,4,5]
Method depth 1 depth 2 depth 3 depth 4 depth 5 Quality
Heap SQL path baseline 0.573 ms 0.589 ms 0.613 ms 0.622 ms 0.622 ms 100.0% / 100.0%
sorted_heap SQL path baseline 0.774 ms 0.732 ms 0.776 ms 0.712 ms 0.772 ms 100.0% / 100.0%
sorted_heap_graph_rag(…, score_mode := ‘path’) 0.674 ms 0.651 ms 0.643 ms 0.633 ms 0.649 ms 100.0% / 100.0%

On this synthetic chain benchmark, the unified path-aware function does not show a latency cliff through depth 5; it stays in the same ~0.63-0.67 ms band while preserving 100.0% / 100.0% quality. This is a controlled scaling signal, not a claim about arbitrary deep graph workloads.

Larger synthetic GraphRAG scale envelope (1M and 10M)

Repo-owned harnesses:

  • local: python3 scripts/bench_graph_rag_multidepth.py --num-pairs 200000 --max-depth 5 --query-count 8 --runs 1 --dim 384 --ann-k 64 --top-k 10 --ef-search 128 --ef-construction 64 --m 24 --shared-buffers-mb 64 --max-wal-size-gb 32 --maintenance-work-mem-mb 4096 --table-scope sorted_heap_only
  • AWS: NUM_PAIRS=2000000 MAX_DEPTH=5 QUERY_COUNT=1 RUNS=1 DIM=32 ANN_K=16 TOP_K=10 EF_SEARCH=32 EF_CONSTRUCTION=8 M=8 SHARED_BUFFERS_MB=64 MAX_WAL_SIZE_GB=16 MAINTENANCE_WORK_MEM_MB=2048 TABLE_SCOPE=sorted_heap_only ./scripts/bench_graph_rag_multidepth_aws.sh <aws-host> /path/to/repo 65494

Current verified scale picture:

  • local 1M rows (200K pairs x 5 hops, 384D, cheap-build point):
    • generate_csv: 162.667 s
    • load_data: 45.694 s
    • build_indexes: 377.968 s
    • unified path-aware sorted_heap_graph_rag(...) p50:
      • depth 1: 1.250 ms
      • depth 2: 1.429 ms
      • depth 3: 1.777 ms
      • depth 4: 2.181 ms
      • depth 5: 2.604 ms
    • quality stayed 100.0% / 100.0% at every depth
  • local 10M x 384D no longer dies in Python generation after the streaming CSV change; the process stayed near 31 MiB RSS on the early run where the old generator previously blew up.
  • local 10M x 64D and AWS 10M x 32D both survive generation and load, but the first practical frontier is still the single sorted_hnsw CREATE INDEX build, not query execution.

The strongest bounded AWS signal so far is:

  • 10M x 32D (2M pairs x 5 hops, ultralight build point)
    • generate_csv: 223.114 s
    • CSV size: 3.63 GiB
    • load_data: 485.177 s
    • on the old build path, it then entered a long CREATE INDEX that was still active after ~11-12 minutes in the index-build phase
    • temp-cluster footprint reached about 17-18 GiB with 11-13 GiB disk still free on the AWS host

After removing the per-search visited[] allocation/zeroing from the hot HNSW build loop, the same local diagnostic point that previously established the bottleneck changed materially:

  • local 500K x 32D (100K pairs x 5 hops, m=8, ef_construction=8)
    • old total build: 18.1-18.7 s
    • old graph-construction phase: about 18.27 s
    • new total build: 2.777-2.996 s
    • new graph-construction phase: about 2.42-2.59 s
    • page-writing tail stayed small (~0.20-0.24 s)

That moved the AWS 10M x 32D scale branch from “still stuck in CREATE INDEX” to the first real 10M query pass on the same cheap-build point:

  • generate_csv: 222.814 s
  • load_data: 468.923 s
  • build_indexes: 212.553 s
  • analyze: 14.562 s
  • unified path-aware sorted_heap_graph_rag(...) p50:
    • depth 1: 2.124 ms
    • depth 2: 3.433 ms
    • depth 3: 4.611 ms
    • depth 4: 5.846 ms
    • depth 5: 5.139 ms

This is not a release-quality GraphRAG point. The build/query knobs here were intentionally ultra-light (ann_k=16, ef_search=32, ef_construction=8, m=8) to get the first 10M latency envelope. At those settings, answer quality on the single-query AWS probe was poor (0.0% / 0.0%) and should be treated as a scale smoke, not a publishable quality frontier.

I then kept that AWS temp cluster alive and ran query-only sweeps on the same cheap-built 10M x 32D graph to test whether query-time tuning alone could recover depth-5 quality without paying for another CREATE INDEX. It did not:

  • ef_search=128, ann_k=64 -> depth-5 2.626 ms, 0.0% / 0.0%
  • ef_search=128, ann_k=128 -> depth-5 3.040 ms, 0.0% / 0.0%
  • ef_search=256, ann_k=128 -> depth-5 3.284 ms, 0.0% / 0.0%
  • ef_search=256, ann_k=256 -> depth-5 4.076 ms, returned 10 rows, but still 0.0% / 0.0%
  • one pathological point, ef_search=32, ann_k=64, spiked to 1833.809 ms and 259709.5 shared reads while still returning 0.0% / 0.0%

I then ran the same question on a smaller but shape-matched local 1M x 32D graph (200K pairs x 5 hops) to separate build quality from query contract:

  • cheap local build points (m=8, ef_construction=8/32, m=16, ef_construction=32/64) stayed weak at the old narrow query contract (ann_k=64, top_k=10), topping out around 25.0% hit@k
  • but with a wider query contract on the same 1M x 32D graph (ann_k=256, top_k=32, ef_search=128), both
    • the strong build (m=24, ef_construction=200), and
    • the much cheaper build (m=16, ef_construction=64) reached the same 96.9% hit@k over 32 queries
  • exact heap seeds on that 1M x 32D graph matched the ANN result exactly at the widened point, so once the query budget is large enough, heavier builds were no longer the decisive lever there

That suggested a second AWS 10M x 32D probe on the same cheap build but with the widened local winner (ann_k=256, top_k=32, ef_search=128). The result still failed:

  • ANN path-aware sorted_heap_graph_rag(...):
    • top_k=10 -> 1832.345 ms, 0.0% / 0.0%
    • top_k=32 -> 1832.755 ms, 0.0% / 0.0%
  • exact heap seeds + the same path-aware expansion/rerank contract:
    • ann_k=256, top_k=32 -> 0.0% / 0.0%

So the cheap-build 10M x 32D graph can produce latency numbers, but its quality is not recoverable by either:

  • larger query-time ANN budgets, or
  • exact heap seeds on the same low-dimensional corpus

The 10M x 32D frontier is therefore no longer “weak HNSW build quality.” It is the low-dimensional scale contract itself. The next meaningful scale branch is a higher-dimensional 10M point or a different retrieval contract, not another m/ef_construction tweak on the same 32D setup.

The first higher-dimensional calibration point was 64D, and it changed the local picture immediately. On a local 1M x 64D graph (200K pairs x 5 hops), a relatively cheap build (m=16, ef_construction=64) plus the wider query contract (ann_k=256, top_k=32, ef_search=128) gave:

  • ANN path-aware sorted_heap_graph_rag(...): 65.6% hit@1, 96.9% hit@k
  • exact heap seeds + the same path-aware expansion/rerank contract: 65.6% hit@1, 96.9% hit@k

At top_k=10, the same 1M x 64D point already held 96.9% hit@k, so unlike 32D, the result-budget cliff largely disappeared there.

The first file-backed 10M x 64D AWS attempt on the current ubuntu@dev.rigelstar.com host (4 vCPU, 8 GiB RAM) failed for operational reasons before query timing:

  • generate_csv: 414.397 s
  • CSV size: 6.67 GiB
  • temp dir grew to about 28 GiB
  • /tmp fell to 1.3 GiB free (99% used) while the run was still active

That led to a footprint reduction pass in the harness:

  • stream fact rows directly into COPY instead of materializing a giant CSV
  • drop facts_heap after loading facts_sh in sorted_heap_only mode
  • allow query-only reuse from a kept temp cluster
  • when facts_heap is not needed, copy rows straight into facts_sh before sorted_heap_compact(...) instead of staging through facts_heap and INSERT .. ORDER BY

That direct facts_sh load path held up on bounded local checks:

  • 200K rows (40K pairs x 5 hops, 64D, hop_weight=0.05)
    • old staged path: 6321.453 ms load, depth-5 unified GraphRAG 24.504 ms, 100.0% / 100.0%
    • direct COPY facts_sh: 5638.134 ms load, depth-5 unified GraphRAG 24.312 ms, 100.0% / 100.0%
  • 1M rows (200K pairs x 5 hops, 64D, load only)
    • old staged path: 31.392 s
    • direct COPY facts_sh: 28.231 s

So the current conclusion is narrow but useful: for synthetic sorted_heap_only multidepth runs, direct COPY facts_sh is a real ~10% ingest win without giving up the compacted query-time locality that the earlier “skip compaction” falsifier showed we still need.

I also bounded the obvious follow-up on the same direct ordered load: parameterizing the post-load maintenance step as none, merge, or compact.

  • 200K rows (40K pairs x 5 hops, 64D, hop_weight=0.05)
    • none: 5159.888 ms load, depth-5 unified GraphRAG 62.148 ms
    • merge: 5749.190 ms load, depth-5 unified GraphRAG 23.277 ms
    • compact: 5626.887 ms load, depth-5 unified GraphRAG 24.591 ms
  • 1M rows (200K pairs x 5 hops, 64D, load only)
    • none: 25.820 s
    • merge: 28.142 s
    • compact: 28.108 s

So merge is now exposed as an experiment knob in the multidepth harness, but it is not a proven new default. The current evidence says:

  • none is too expensive at query time
  • merge is viable on ordered synthetic loads
  • merge does not materially beat compact on the larger 1M load point

That leaves compact as the stable default for large-scale multidepth runs.

With that lower-footprint path, the same 10M x 64D AWS point advanced materially further on the same host:

  • streamed generate_csv: 0.000 s (csv_bytes=0, no materialized CSV)
  • load_data: 916.030 s
  • temp dir plateaued around 19 GiB
  • filesystem still had about 11 GiB free at the start of CREATE INDEX

The next frontier was no longer disk headroom. On the same 10M x 64D cheap-build point (m=16, ef_construction=64), the old local scan-cache seeding path then failed inside PostgreSQL:

  • CREATE INDEX facts_sh_ann_idx ON facts_sh USING sorted_hnsw (embedding) WITH (m = 16, ef_construction = 64)
  • ERROR: invalid memory alloc request size 1280000000

That request size matched the old contiguous local L0 neighbor cache layout (n_nodes * 2 * M * sizeof(int32) at 10M x M=16), so the next fix was to replace local l0_neighbors + sq8_data slabs with page-backed storage. Shared immutable scan caches remain contiguous; local build seeding and local shnsw_load_cache() no longer require a single giant L0 allocation.

With the chunked local-cache fix in place, the next retained branch was the constrained-memory rerun via:

  • sorted_hnsw.build_sq8 = on
  • lower-hop synthetic contract (hop_weight = 0.05)

On the same AWS ARM64 host (4 vCPU, 8 GiB RAM, 4 GiB swap), the monolithic 10M x 64D point then completed cleanly:

  • load_data: 787.809 s
  • build_indexes: 846.795 s
  • kept temp cluster: /home/ubuntu/graphrag_tmp/graph_rag_y8cuntf9

That is a real constrained-memory result: the same host that previously hit early large-scale frontiers now built the full monolithic sorted_hnsw index on 10,000,000 rows without being OOM-killed.

The first query-only reuse pass on that exact built graph (query_count=4, runs=1, ann_k=256, top_k=32, ef_search=128, sorted_heap_only) gave:

  • depth 1
    • SQL path: 1072.525 ms, 100.0% / 100.0%
    • unified GraphRAG path: 840.607 ms, 100.0% / 100.0%
  • depth 2
    • SQL path: 1070.437 ms, 75.0% / 100.0%
    • unified GraphRAG path: 2055.479 ms, 75.0% / 100.0%
  • depth 3
    • SQL path: 1050.793 ms, 50.0% / 100.0%
    • unified GraphRAG path: 2070.894 ms, 50.0% / 100.0%
  • depth 4
    • SQL path: 1080.123 ms, 50.0% / 100.0%
    • unified GraphRAG path: 2066.717 ms, 50.0% / 100.0%
  • depth 5
    • SQL path: 1064.405 ms, 75.0% / 100.0%
    • unified GraphRAG path: 2084.155 ms, 75.0% / 100.0%

So the 10M x 64D monolithic branch is now narrowed sharply:

  • constrained-memory monolithic build is viable
  • quality stayed aligned between the SQL baseline and the unified GraphRAG path
  • the remaining issue is not build survival and not quality drift
  • the remaining issue is latency: at depth 2+, the current generic unified GraphRAG path is still about 2x slower than the SQL path baseline on this host and graph

That is exactly why the next scale story should move toward segmentation + pruning rather than trying to turn one monolithic 10M graph into the final operating model for small hosts.

I then added an opt-in stage breakdown path to the multidepth harness via --report-stage-stats, using sorted_heap_graph_rag_stats() after each unified path-aware call. On the local 1M x 64D lower-hop point (hop_weight=0.05, ann_k=256, top_k=32, ef_search=128), that narrowed the runtime picture sharply:

  • depth 2:
    • end-to-end unified GraphRAG: 109.222 ms
    • internal stage stats:
      • ann_ms = 107.590
      • expand_ms = 0.369
      • rerank_ms = 0.004
  • depth 5:
    • end-to-end unified GraphRAG: 110.507 ms
    • internal stage stats:
      • ann_ms = 109.178
      • expand_ms = 0.691
      • rerank_ms = 0.011

So the current 1M x 64D depth cost is not expansion-bound. On the widened contract, almost all measured time is already in the ANN seed stage; the multi-hop expansion and rerank work remain sub-millisecond.

I then added an opt-in low-memory sorted_hnsw build path via SET sorted_hnsw.build_sq8 = on. This keeps the final on-disk/index-query contract the same, but the graph is built from SQ8-compressed build vectors instead of a full float32 build slab. The tradeoff is one extra heap scan during CREATE INDEX.

Bounded local A/B on the same 1M x 64D lower-hop point (m=16, ef_construction=64, ann_k=256, top_k=32, ef_search=128, sorted_heap_only, streamed load):

  • build_sq8=off
    • load_data: 28.044 s
    • build_indexes: 48.606 s
    • unified depth-5 GraphRAG: 111.578 ms, 87.5% / 100.0%
  • build_sq8=on
    • load_data: 27.935 s
    • build_indexes: 46.541 s
    • unified depth-5 GraphRAG: 110.541 ms, 87.5% / 100.0%

So the first retained result is narrow but real:

  • the low-memory build path did not regress the measured 1M x 64D GraphRAG quality point
  • it slightly improved build time on that point instead of paying a visible penalty for the extra heap scan
  • and by construction it cuts the build-vector slab from 4 * N * D bytes to 1 * N * D
    • 10M x 64D: about 2.56 GiB -> 0.64 GiB
    • 10M x 384D: about 15.36 GiB -> 3.84 GiB

So the narrow conclusion is:

  • the multi-hop GraphRAG path itself survives to at least 1M rows and gives measured query latencies there
  • the old 10M frontier was build-bound, not Python-memory-bound, not CSV-generation-bound, and not an early PostgreSQL failure
  • the current build optimization moved that frontier enough to obtain the first real 10M query numbers on a cheap-build point
  • the remaining 10M x 32D problem is no longer “can it finish?” and no longer just “is the HNSW build too weak?”
  • the stronger falsifier is that even exact seeds fail at that scale and dimensionality, so the next branch is a different scale contract, not a narrower HNSW tuning loop
  • 64D is the first scale contract that looks healthy locally at 1M
  • on the current AWS host, the 10M x 64D branch now clears ingest, build, and the first query pass; the remaining cheap-build frontier is deep-path quality, not allocator failure

I then added a segmented multidepth harness in scripts/bench_graph_rag_multidepth_segmented.py. This is the first partitioning benchmark, but it is intentionally harness-side, not an extension API: current GraphRAG helpers require a concrete sorted_heap table, so the benchmark fans out across multiple sorted_heap shards and merges the shard-local top-k rows in Python.

The local 1M x 64D lower-hop point (200K pairs, 5 hops, 32 queries, ann_k=256, top_k=32, ef_search=128, m=16, ef_construction=64, build_sq8=on) gives a clean first answer:

  • monolithic unified GraphRAG:
    • depth 1: 50.104 ms, 100.0% / 100.0%
    • depth 5: 121.524 ms, 81.2% / 100.0%
  • segmented into 8 shards, all-shard fanout:
    • build_indexes: 57.885 s versus monolithic 45.714 s
    • depth 1: 87.677 ms, 100.0% / 100.0%
    • depth 5: 142.472 ms, 81.2% / 100.0%
  • segmented into 8 shards, exact routing to the owning shard:
    • depth 1: 10.574 ms, 100.0% / 100.0%
    • depth 5: 16.822 ms, 100.0% / 100.0%

That makes the partitioning lesson explicit:

  • segmentation by itself is not a free latency win
  • all-shard fanout preserves quality on this benchmark, but pays a clear fanout tax
  • the real win appears only when routing/pruning can avoid most shards
  • so the next scale story should be “partitioning + pruning contract”, not just “more shards”

This is the correct shape for large knowledge bases:

  • on constrained hosts, sharding bounds per-index build memory
  • on real workloads, the query path must also prune by tenant / knowledge-base / relation family / time window (or a future segment router), otherwise the system only trades one monolith for a broad fanout query

That local result transferred cleanly to the constrained AWS host on the first full 10M x 64D streamed segmented run (2M pairs, 5 hops, 8 shards, hop_weight=0.05, ann_k=256, top_k=32, ef_search=128, m=16, ef_construction=64, build_sq8=on):

  • streamed segmented build-only:
    • generate_csv: 0.000 s
    • load_data: 500.474 s
    • build_indexes: 784.778 s
  • segmented, route=exact query-only reuse on the same built graph:
    • depth 1: 126.057 ms, 100.0% / 100.0%
    • depth 2: 261.986 ms, 75.0% / 100.0%
    • depth 3: 259.794 ms, 75.0% / 100.0%
    • depth 4: 258.879 ms, 75.0% / 100.0%
    • depth 5: 258.766 ms, 100.0% / 100.0%
  • segmented, route=all query-only reuse on the same built graph:
    • depth 1: 898.440 ms, 100.0% / 100.0%
    • depth 2: 2090.866 ms, 75.0% / 100.0%
    • depth 3: 2089.650 ms, 50.0% / 100.0%
    • depth 4: 2088.114 ms, 50.0% / 100.0%
    • depth 5: 2093.652 ms, 75.0% / 100.0%

That gives the first full large-scale constrained-memory comparison on the same AWS box:

  • monolithic 10M x 64D, low-memory build:
    • depth 1 unified GraphRAG: 840.607 ms, 100.0% / 100.0%
    • depth 5 unified GraphRAG: 2084.155 ms, 75.0% / 100.0%
  • segmented 10M x 64D, route=all:
    • effectively the same latency/quality envelope as the monolith
  • segmented 10M x 64D, route=exact:
    • about 6.7x faster than the monolith at depth 1
    • about 8.1x faster than the monolith at depth 5
    • and still stable at 100.0% / 100.0% for depth 5 on this benchmark

So the large-scale result now matches the local 1M lesson:

  • segmentation without pruning is not a performance story
  • streamed segmented ingest is operationally better than front-loaded shard CSV materialization
  • segmented routing is the first constrained-memory scale path that improves both build viability and query latency on the same host

The next local step was to move routing out of benchmark-only Python and into a real SQL reference path. On a kept local 5K-row segmented smoke cluster (4 shards, 64D, ann_k=32, top_k=8, ef_search=64), the new range-routed SQL path matched the segmented SQL merge baseline on quality and row counts:

  • segmented SQL merge, route=exact, depth 5:
    • 0.215 ms, 100.0% / 100.0%
  • metadata-routed SQL wrapper, same exact-route point:
    • 0.245 ms, 100.0% / 100.0%

So the current reference stack is now:

  • sorted_heap_graph_rag_segmented(...) for explicit candidate shard arrays
  • sorted_heap_graph_rag_routed(...) for simple metadata-driven range routing

The next local routing pass added an exact-key companion for tenant / KB style selection. On another kept local 5K-row segmented smoke cluster (4 shards, 64D, ann_k=32, top_k=8, ef_search=64), the exact-key routed path stayed aligned with the exact-route segmented SQL merge path:

  • exact-route segmented SQL merge, depth 5:
    • 0.183 ms, 100.0% / 100.0%
  • exact-key routed SQL wrapper, same point:
    • 0.202 ms, 100.0% / 100.0%

That is still not the final routing story for large knowledge bases, but it is the first usable SQL-level bridge from harness-side segmentation to productized segmented GraphRAG, with both:

  • range-based routing
  • exact-key routing

Monolithic vs segmented exact-routed: constrained-memory 10M x 64D comparison

This table consolidates the AWS 10M x 64D results already reported above (monolithic in “Larger synthetic GraphRAG scale envelope”, segmented in “Segmented GraphRAG at scale”) into a direct side-by-side.

Both runs used the same AWS ARM64 host (4 vCPU, 8 GiB RAM, 4 GiB swap), the same workload (2M pairs x 5 hops, hop_weight=0.05), and the same build/query knobs (m=16, ef_construction=64, build_sq8=on, ann_k=256, top_k=32, ef_search=128). Segmented mode used 8 shards.

  Monolithic Segmented exact Segmented all
load_data 787.8 s 500.5 s —
build_indexes 846.8 s 784.8 s —
depth 1 p50 840.6 ms 126.1 ms 898.4 ms
depth 5 p50 2084.2 ms 258.8 ms 2093.7 ms
depth 1 hit@1/hit@k 100%/100% 100%/100% 100%/100%
depth 5 hit@1/hit@k 75%/100% 100%/100% 75%/100%
speedup at depth 5 1x 8.1x ~1x

Exact commands (monolithic build + query-only reuse):

NUM_PAIRS=2000000 MAX_DEPTH=5 QUERY_COUNT=4 RUNS=1 DIM=64 \
  ANN_K=256 TOP_K=32 EF_SEARCH=128 EF_CONSTRUCTION=64 M=16 \
  SHARED_BUFFERS_MB=64 MAX_WAL_SIZE_GB=16 HOP_WEIGHT=0.05 \
  BUILD_SQ8=on TABLE_SCOPE=sorted_heap_only \
  ./scripts/bench_graph_rag_multidepth_aws.sh <host> <dir> 65493

Exact commands (segmented build + query-only reuse):

NUM_PAIRS=2000000 MAX_DEPTH=5 QUERY_COUNT=4 RUNS=1 DIM=64 \
  ANN_K=256 TOP_K=32 EF_SEARCH=128 EF_CONSTRUCTION=64 M=16 \
  SHARED_BUFFERS_MB=64 MAX_WAL_SIZE_GB=16 HOP_WEIGHT=0.05 \
  BUILD_SQ8=on SHARDS=8 ROUTE=both \
  ./scripts/bench_graph_rag_multidepth_segmented_aws.sh <host> <dir> 65494

Conclusion: on the measured 10M x 64D constrained-memory point, segmented exact routing is 8.1x faster at depth 5 with better quality (100%/100% vs 75%/100%), while build time is comparable. All-shard fanout offers no latency benefit — the win comes entirely from shard pruning.

This is not a universal claim. Exact routing requires knowing which shard owns the query entity, which maps to tenant-id / knowledge-base-id style workloads. Queries that cannot be pruned to a single shard will see all-shard-fanout latency (~1x monolithic).

Bounded fanout: routing robustness on 1M x 64D

This measures how the segmented win degrades when routing is imperfect. Instead of routing to exactly one shard (exact) or all shards (all), bounded fanout routes to K adjacent shards (always including the correct one). This simulates imperfect routing where the router knows roughly which shard, but hedges by including neighbors.

Local, 200K pairs x 5 hops, 64D, 8 shards, 32 queries, 3 runs, ann_k=256, top_k=32, ef_search=128, m=16, ef_construction=64, build_sq8=on, hop_weight=0.05:

python3 scripts/bench_graph_rag_multidepth_segmented.py \
  --num-pairs 200000 --max-depth 5 --query-count 32 --runs 3 \
  --dim 64 --ann-k 256 --top-k 32 --ef-search 128 \
  --ef-construction 64 --m 16 --build-sq8 on \
  --shards 8 --route {exact,bounded,all} --fanout {2,4} --hop-weight 0.05

Monolithic baseline (sorted_heap_graph_rag(...) on same workload):

python3 scripts/bench_graph_rag_multidepth.py \
  --num-pairs 200000 --max-depth 5 --query-count 32 --runs 3 \
  --dim 64 --ann-k 256 --top-k 32 --ef-search 128 \
  --ef-construction 64 --m 16 --build-sq8 on \
  --hop-weight 0.05 --table-scope sorted_heap_only

Depth-5 comparison (the hardest hop):

Route mode Shards hit p50 (d5) hit@1 (d5) hit@k (d5) Speedup vs monolith
monolithic 1 table 120.7 ms 81.2% 100.0% 1x
segmented all (8/8) 8 149.5 ms 81.2% 100.0% 0.8x
segmented bounded (4/8) 4 74.1 ms 93.8% 100.0% 1.6x
segmented bounded (2/8) 2 35.9 ms 96.9% 100.0% 3.4x
segmented exact (1/8) 1 17.2 ms 100.0% 100.0% 7.0x

Depth-1 comparison:

Route mode Shards hit p50 (d1) hit@1 (d1) hit@k (d1)
monolithic 1 table 49.4 ms 100.0% 100.0%
segmented all (8/8) 8 94.3 ms 100.0% 100.0%
segmented bounded (4/8) 4 46.3 ms 100.0% 100.0%
segmented bounded (2/8) 2 22.8 ms 100.0% 100.0%
segmented exact (1/8) 1 11.3 ms 100.0% 100.0%

Observations:

  • Latency scales roughly linearly with the number of shards hit.
  • Quality degrades gracefully: bounded(2/8) still reaches 96.9% hit@1 at depth 5 vs 100.0% for exact, compared to 81.2% for monolithic/all.
  • Even bounded(4/8) at half the shards is 1.6x faster than monolithic and has better quality (93.8% vs 81.2% hit@1).
  • All-shard fanout is worse than monolithic (fanout overhead dominates).
  • The win is not exact-or-nothing — bounded fanout preserves most of the benefit even with imperfect routing.

Bounded fanout at AWS 10M x 64D scale

Same AWS ARM64 host (4 vCPU, 8 GiB RAM, 4 GiB swap), same workload as the 10M x 64D comparison above (2M pairs x 5 hops, hop_weight=0.05, 8 shards, ann_k=256, top_k=32, ef_search=128, m=16, ef_construction=64, build_sq8=on). Build once, query-only reuse for each route mode. 4 queries, 1 run.

Build: load_data=500.7s, build_indexes=764.2s.

Query commands:

# Build once with --keep-temp --stop-after build_indexes
NUM_PAIRS=2000000 MAX_DEPTH=5 QUERY_COUNT=4 RUNS=1 DIM=64 \
  ANN_K=256 TOP_K=32 EF_SEARCH=128 EF_CONSTRUCTION=64 M=16 \
  SHARED_BUFFERS_MB=64 MAX_WAL_SIZE_GB=16 HOP_WEIGHT=0.05 \
  BUILD_SQ8=on SHARDS=8 ROUTE=exact \
  EXTRA_ARGS="--stream-copy --keep-temp --stop-after build_indexes" \
  bash scripts/bench_graph_rag_multidepth_segmented_aws.sh \
  <host> <dir> 65494

# Query-only reuse for each route mode:
ssh <host> "cd <dir> && python3 scripts/bench_graph_rag_multidepth_segmented.py \
  --port 65494 --num-pairs 2000000 --max-depth 5 --query-count 4 --runs 1 \
  --dim 64 --hop-weight 0.05 --ann-k 256 --top-k 32 --ef-search 128 \
  --ef-construction 64 --m 16 --build-sq8 on --shared-buffers-mb 64 \
  --shards 8 --route {exact,bounded,all} [--fanout {2,4}] \
  --backend-mode fresh --reuse-temp <kept_temp>"

Depth-5 comparison (monolithic baseline from prior section):

Route mode Shards hit p50 (d5) hit@1 (d5) hit@k (d5) vs monolith
monolithic 1 table 2084 ms 75% 100% 1x
segmented all (8/8) 8 2107 ms 75% 100% ~1x
segmented bounded (4/8) 4 1042 ms 75% 100% 2.0x
segmented bounded (2/8) 2 520 ms 100% 100% 4.0x
segmented exact (1/8) 1 259 ms 100% 100% 8.0x

Depth-1 comparison:

Route mode Shards hit p50 (d1) hit@1 (d1) hit@k (d1) vs monolith
monolithic 1 table 841 ms 100% 100% 1x
segmented all (8/8) 8 920 ms 100% 100% ~1x
segmented bounded (4/8) 4 473 ms 100% 100% 1.8x
segmented bounded (2/8) 2 233 ms 100% 100% 3.6x
segmented exact (1/8) 1 137 ms 100% 100% 6.1x

The bounded-fanout gradient from the local 1M point transfers cleanly to 10M:

  • Latency scales linearly with the number of shards hit at both scales.
  • bounded(2/8) at 10M is 4.0x faster than monolithic at depth 5 — close to the local 3.4x ratio.
  • Even bounded(4/8) at half the shards gives a 2.0x win.
  • Quality: hit@k stays at 100% across all modes. hit@1 varies with fanout width but stays ≥75%.
  • All-shard fanout remains ~1x monolithic (fanout overhead cancels the per-shard size reduction).

Conclusion: bounded fanout remains materially useful on this measured 10M synthetic point. A router that can narrow to 2 out of 8 candidate shards captures ~50% of the exact-routing speedup. The performance gradient is smooth on this benchmark, not a cliff.

Routing-miss tolerance on 1M x 64D

This measures what happens when the router sometimes picks the wrong shards — the correct shard is not always included in the candidate set. Uses --route bounded_recall --fanout 2 --recall-pct N where N% of queries get the correct shard and the rest get 2 adjacent wrong shards. Miss schedule is deterministic per-query via SHA-256 hash (seed=42).

Local, 200K pairs x 5 hops, 64D, 8 shards, 32 queries, 3 runs, ann_k=256, top_k=32, ef_search=128, m=16, ef_construction=64, build_sq8=on, hop_weight=0.05:

python3 scripts/bench_graph_rag_multidepth_segmented.py \
  --num-pairs 200000 --max-depth 5 --query-count 32 --runs 3 \
  --dim 64 --ann-k 256 --top-k 32 --ef-search 128 \
  --ef-construction 64 --m 16 --build-sq8 on \
  --shards 8 --route bounded_recall --fanout 2 \
  --recall-pct {100,90,75,50,25,0} --hop-weight 0.05

Depth-5 routing-miss tolerance (fanout=2, 8 shards):

Router recall Queries hitting correct shard p50 (d5) hit@1 (d5) hit@k (d5)
100% 32/32 54.5 ms 96.9% 100.0%
90% 29/32 64.7 ms 87.5% 90.6%
75% 23/32 57.0 ms 68.8% 71.9%
50% 18/32 45.4 ms 53.1% 56.2%
25% 10/32 79.1 ms 31.2% 31.2%
0% 0/32 44.8 ms 0.0% 0.0%
(ref: monolithic) — 120.7 ms 81.2% 100.0%

Observations:

  • Quality tracks router recall nearly linearly. There is no sharp cliff.
  • A router with 90% recall keeps 87.5% hit@1 — close to the monolithic baseline’s 81.2%, while remaining 2-3x faster.
  • A router with 75% recall gives 68.8% hit@1 — below monolithic quality, so the latency win comes at a quality cost.
  • Latency stays roughly constant across recall levels because the fanout (2 shards) is fixed. The router-miss doesn’t cost extra I/O — it just returns wrong answers.
  • This means: routing quality determines answer quality, not latency. The segmented latency win is structural (fewer rows per shard), but the quality win depends entirely on the router including the correct shard.

Routing-miss tolerance at AWS 10M x 64D

Same AWS ARM64 host and workload as the bounded-fanout section above. Build once (load=500.5s, build=803.5s), query-only reuse. 4 queries, 1 run, fanout=2, 8 shards.

# Build: same as bounded-fanout section
# Query-only reuse with bounded_recall:
ssh <host> "cd <dir> && python3 scripts/bench_graph_rag_multidepth_segmented.py \
  --port 65494 --num-pairs 2000000 --max-depth 5 --query-count 4 --runs 1 \
  --dim 64 --hop-weight 0.05 --ann-k 256 --top-k 32 --ef-search 128 \
  --ef-construction 64 --m 16 --build-sq8 on --shared-buffers-mb 64 \
  --shards 8 --route bounded_recall --fanout 2 --recall-pct {90,75} \
  --backend-mode fresh --reuse-temp <kept_temp>"

Depth-5 routing-miss tolerance at 10M:

Route mode Router recall p50 (d5) hit@1 (d5) hit@k (d5)
bounded optimistic 100% (4/4 hit) 519 ms 100% 100%
bounded_recall 90% (3/4 hit) 521 ms 75% 75%
bounded_recall 75% (3/4 hit) 522 ms 75% 75%
(ref: monolithic) — 2084 ms 75% 100%
(ref: exact) 100% 259 ms 100% 100%

The 90% and 75% recall points produce the same result because with only 4 queries, SHA-256 hash gives 3/4 correct-shard inclusion at both settings. The 1 missed query returns wrong results, giving 75% hit@1.

Despite limited query-count resolution, the directional signal is clear:

  • Bounded(2/8) latency stays at ~520 ms regardless of recall (~4x faster than monolithic)
  • One missed query out of four drops hit@1 from 100% to 75% — matching the monolithic baseline’s own 75% hit@1
  • hit@k drops from 100% to 75% (monolithic gets 100% hit@k because its broader candidate set is more likely to include the target)

This confirms the local finding: routing quality determines answer quality, not latency. The latency win is structural and survives routing errors. At 90% router recall on this 10M point, bounded routing matches the monolithic hit@1 (75%) while staying 4x faster.

A higher query_count run would give finer crossover resolution. On the measured 4-query point, the evidence is narrower: at 90% recall, bounded routing matches the monolithic hit@1 while staying 4x faster, and the true crossover is not shown to be above that point.

Current AWS GraphRAG benchmark (person -> parent -> city, stable fact contract)

Repo-owned harness:

  • REMOTE_PYTHON=/path/to/python QUERY_COUNT=64 RUNS=3 NUM_PAIRS=5000 DIM=384 ANN_K=64 TOP_K=10 EF_SEARCH=128 EF_CONSTRUCTION=200 M=24 PGV_EF_SEARCH=64 ZVEC_EF=64 QDRANT_EF=64 SHARED_BUFFERS_MB=64 BACKEND_MODE=fresh ./scripts/bench_graph_rag_multihop_aws.sh <aws-host> /path/to/repo 65492

AWS ARM64 host (4 vCPU, 8 GiB RAM), deterministic fact graph, 5K chains / 10K rows total, 384D, top-10, 64 queries, 3 runs. This is the current portable stable multihop GraphRAG point for the narrow fact-shaped contract.

Method p50 latency hit@1 hit@k Notes
Heap two-hop SQL 1.088 ms 75.0% 96.9% exact rerank over expanded heap set
sorted_heap_expand_twohop_rerank() 0.952 ms 75.0% 96.9% older city-only helper
sorted_heap_graph_rag_twohop_scan() 1.012 ms 75.0% 96.9% older city-only wrapper
sorted_heap SQL pathsum baseline 1.204 ms 98.4% 98.4% same ANN seeds, hop1_distance + hop2_distance
sorted_heap_expand_twohop_path_rerank() 0.955 ms 98.4% 98.4% fused path-aware helper
sorted_heap_graph_rag_twohop_path_scan() 1.018 ms 98.4% 98.4% fused path-aware wrapper
pgvector HNSW + heap expansion 1.422 ms 85.9% 85.9% path-aware rerank, ef_search=64
zvec HNSW + heap expansion 1.720 ms 100.0% 100.0% path-aware rerank, ef=64
Qdrant HNSW + heap expansion 3.435 ms 100.0% 100.0% path-aware rerank, hnsw_ef=64

The new AWS result matches the local diagnostic cleanly: the dominant quality loss on this workload was the old hop-2-only rerank contract, not the seed ANN frontier. The path-aware helper preserves the quality gain on ARM64 with only trivial latency cost versus the older helper.

On the apples-to-apples path-aware contract, the portable frontier is now:

  • sorted_heap fastest
  • zvec and Qdrant strongest on answer quality
  • pgvector still behind on both latency and quality at this operating point

One intermediate AWS all-engines rerun temporarily dropped the sorted_heap path-aware rows to 96.9% / 96.9%. An immediate sorted_heap-only control and a second full all-engines rerun both returned the stable 98.4% / 98.4% point above, so the published table uses the confirmed rerun rather than the single outlier.

An AWS repeated-build protocol then tightened that confidence band on the same balanced 5K point. Using three independent fresh builds:

  • sorted_heap_expand_twohop_path_rerank() median 0.962 ms, range 0.956-0.965 ms, hit@1/hit@k = 98.4/98.4 on all three builds
  • sorted_heap_graph_rag_twohop_path_scan() median 1.025 ms, range 1.018-1.043 ms, hit@1/hit@k = 98.4/98.4 on all three builds
  • pgvector path-aware parity row median 1.434 ms, hit@1/hit@k 84.4-89.1
  • zvec path-aware parity row median 1.711 ms, 100.0/100.0
  • Qdrant path-aware parity row median 3.355 ms, 100.0/100.0

So on the current portable 5K point, the earlier AWS outlier now looks like an anomaly rather than a broad instability. The balanced sorted_heap path-aware rows stayed fixed across all three rebuilds.

The larger 10K-chain AWS rerun now tells a different story than the older city-only benchmark. At the same portable point:

  • heap two-hop SQL: 1.319 ms, hit@1 71.9%, hit@k 92.2%
  • city-only sorted_heap_graph_rag_twohop_scan(): 1.197 ms, hit@1 73.4%, hit@k 93.8%
  • SQL pathsum baseline: 1.436 ms, hit@1 96.9%, hit@k 98.4%
  • sorted_heap_expand_twohop_path_rerank(): 1.185 ms, hit@1 96.9%, hit@k 98.4%
  • sorted_heap_graph_rag_twohop_path_scan(): 1.212 ms, hit@1 96.9%, hit@k 98.4%

So the old larger-scale caveat now narrows materially: the main 10K loss was also the city-only rerank contract, not a fundamental collapse of the seed frontier at that scale.

An AWS repeated-build protocol then checked whether the remaining 10K difference was really a build-variance problem. Using three independent fresh builds on the same 10K path-aware point:

  • sorted_heap_expand_twohop_path_rerank() median 1.177 ms, range 1.148-1.191 ms, hit@1/hit@k = 95.3/96.9 on all three builds
  • sorted_heap_graph_rag_twohop_path_scan() median 1.236 ms, range 1.211-1.240 ms, hit@1/hit@k = 95.3/96.9 on all three builds
  • pgvector path-aware parity row median 1.667 ms, hit@1/hit@k 76.6-82.8
  • zvec path-aware parity row median 2.788 ms, 98.4/100.0
  • Qdrant path-aware parity row median 3.818 ms, 98.4/100.0

So the larger 10K AWS point is now repeated-build stable too. The remaining issue is scale frontier, not build instability: the 10K quality band is lower than 5K, but it stayed fixed across fresh builds.

An exact-seed diagnostic on the local 5K and 10K points did not improve hit@1 or hit@k versus the ANN-seeded sorted_heap helper. So on this benchmark shape, the remaining gap is not explained by ANN approximation alone. The stronger result is that seed coverage itself was identical for ANN and exact seeds: 98.4% at 5K and 96.9% at 10K. So the remaining loss is downstream of seeding. The new rerank-rank diagnostic narrows that further: at 5K, the correct city is still within the top 6 for 95% of reachable queries, and at 10K it is still within the top 3 for 95% of reachable queries. The quality drop is therefore driven by a few severe outliers (max rank 17 at 5K, 20 at 10K), not by a broad collapse.

A path-aware SQL rerank baseline then changed the picture materially. Keeping the same ANN seeds and the same two-hop expansion, but scoring candidates as hop1_distance + hop2_distance, moved the local balanced points to:

  • 5K: 0.957 ms, hit@1 98.4%, hit@k 98.4%
  • 10K: 1.179 ms, hit@1 95.3%, hit@k 96.9%

That branch is now implemented in the extension and verified on both local and AWS ARM64 runs. The fused path-aware helper measured:

  • local 5K: 0.726 ms, hit@1 98.4%, hit@k 98.4%
  • local 10K: 0.823 ms, hit@1 95.3%, hit@k 96.9%
  • AWS 5K: 0.955 ms, hit@1 98.4%, hit@k 98.4%
  • AWS 10K: 1.185 ms, hit@1 96.9%, hit@k 98.4%

So the current strongest portable GraphRAG result is no longer the SQL baseline or the old city-only helper. It is the fused path-aware helper.

Current real code-corpus GraphRAG reference benchmark (cogniformerus CrossFile)

Repo-owned harnesses:

  • python3 scripts/bench_graph_rag_code_corpus.py --runs 3 --backend-mode fresh --ann-k 16 --top-k 4
  • python3 scripts/repeat_graph_rag_code_corpus_builds.py --repeats 3 --runs 3 --backend-mode fresh
  • REMOTE_PYTHON=/path/to/python REPEATS=3 RUNS=3 BACKEND_MODE=fresh bash scripts/repeat_graph_rag_code_corpus_builds_aws.sh <aws-host> /path/to/repo 65320

This benchmark uses the actual cogniformerus source tree (40 files, 840 rows after chunk + summary expansion) and the real CrossFile prompts from butler_code_test.cr. The current real-corpus conclusion is not a single universal winner. The frontier splits by embedding mode:

Mode Best case Local repeated-build p50 AWS repeated-build p50 Keyword coverage Full hits Notes
generic prompt_summary_snippet_py 0.613 ms 0.955 ms 100.0% 100.0% symbol-aware variant is strictly slower with no quality gain
code-aware prompt_symbol_summary_snippet_py 0.963 ms 1.541 ms 100.0% 100.0% exact prompt-symbol rescue is required in summary seeding

The most important diagnostic result was the old code-aware miss:

  • prompt_summary_snippet_py on facts_sh
    • local repeated-build: 97.6% keyword coverage, 83.3% full hits
    • AWS repeated-build: 97.6%, 83.3%
  • prompt_symbol_summary_snippet_py on facts_sh
    • local repeated-build: 100.0%, 100.0%
    • AWS repeated-build: 100.0%, 100.0%

So the code-aware quality win is now both repeated-build stable and cross-environment stable. The change in winner is not a local-only artifact.

Larger in-repo cogniformerus transfer gate

The smaller 40-file code-corpus slice above is a useful stable benchmark, but it is not the only in-repo transfer check anymore. The same repeated-build protocol was rerun on the full cogniformerus repository (183 Crystal files), still using the real CrossFile prompts from butler_code_test.cr.

Control point at the old tiny-budget contract (top_k=4, 1 fresh build):

Mode Best case Local p50 Keyword coverage Full hits Avg returned rows Notes
generic prompt_summary_snippet_py 0.770 ms 87.1% 66.7% 3.67 larger corpus exposes a result-budget cliff
code-aware prompt_symbol_summary_snippet_py 1.824 ms 87.6% 66.7% 4.00 same cliff under code-aware embeddings

Bounded recovery point (top_k=8, 3 fresh builds):

Mode Best case Local repeated-build p50 Keyword coverage Full hits Avg returned rows Notes
generic prompt_summary_snippet_py 0.819 ms 100.0% 100.0% 6.33 larger in-repo Crystal transfer now verified
code-aware prompt_symbol_summary_snippet_py 1.814 ms 100.0% 100.0% 7.50 same winner, but needs the larger final budget

Interpretation:

  • the current real code-corpus contracts do transfer beyond the tiny 40-file slice
  • the dominant larger-corpus issue on the in-repo Crystal side is result budget, not a new retrieval failure
  • this larger-corpus Crystal gate is now covered and serves as a transfer check for the benchmark-side code-retrieval logic, not as the stable release contract for GraphRAG

Mixed-language external code-corpus GraphRAG reference benchmark (pycdc)

The code-corpus harness now also supports:

  • JSON question fixtures
  • configurable source extensions
  • quoted local #include "..." dependency edges for C/C++ corpora

The first mixed-language adversary corpus was pycdc, using a repo-owned fixture in scripts/fixtures/graph_rag_pycdc_questions.json. This run used the real pycdc source tree (138 files, 1281 rows after chunk + summary expansion, 72 local dependency edges) and 3 fresh builds at top_k=8.

Mode Best case Local repeated-build p50 Keyword coverage Full hits Avg returned rows Notes
generic prompt_symbol_summary_snippet_py 0.850 ms 90.0% 60.0% 6.40 fastest mixed-language point, but it does not close the corpus
code-aware prompt_compactseed_require_summary_snippet_fn 8.006 ms 100.0% 100.0% 5.80 helper-backed compact lexical seed + one-hop include rescue closes the corpus

Interpretation:

  • the mixed-language gate is now covered for a real ~/Projects/C corpus
  • the result split is sharper than on the Crystal corpora:
    • the fast generic path plateaus below perfect quality
    • the code-aware include-rescue path closes the corpus, but at a much higher latency
  • so ~/Projects/C is now covered as part of the broader code-corpus reference matrix, even though the fastest generic point remains partial

Archive-side code-corpus GraphRAG reference benchmark (ninja/src)

The same widened harness was then pointed at an archive-side corpus under ~/SrcArchives: apple/ninja/src. This run used a second repo-owned fixture in scripts/fixtures/graph_rag_ninja_questions.json and the local include graph inside ninja/src (103 files, 1757 rows after chunk + summary expansion, 282 dependency edges).

Initial smoke at top_k=8:

Mode Best fast case Local p50 Keyword coverage Full hits Notes
generic prompt_summary_snippet_py 0.898 ms 95.0% 80.0% already close without rescue
code-aware prompt_summary_snippet_py 0.928 ms 85.0% 80.0% code-aware mode is weaker on this corpus

Bounded budget probe (top_k=12, 3 fresh builds):

Mode Best case Local repeated-build p50 Keyword coverage Full hits Avg returned rows Notes
generic prompt_summary_snippet_py 0.914 ms 100.0% 100.0% 7.80 archive-side gate closes with a small result-budget bump
code-aware prompt_summary_snippet_py 0.871 ms 85.0% 80.0% 7.60 still not the winner on this corpus

Interpretation:

  • the ~/SrcArchives side is now covered by a real repeated-build gate
  • unlike pycdc, this archive corpus does not need a dependency-rescue contract to close
  • the winner is the simple generic summary-snippet path at a slightly larger final budget (top_k=12)
  • the larger real-corpus verification matrix for the narrow 0.13 fact-graph release now spans:
    • ~/Projects/Crystal
    • ~/Projects/C
    • ~/SrcArchives

External folding stress corpus for GraphRAG reference logic (folding/src)

The same harness was then pointed at a second real code corpus outside this repository: folding/src with prompts from butler_folding_test.cr. This is not the primary publishable frontier for the repository, but it is a strong adversary corpus because it falsifies overfit retrieval contracts quickly.

Current repeated-build result:

Mode Case Local repeated-build p50 AWS repeated-build p50 Keyword coverage Full hits Notes
generic prompt_summary_snippet_py 1.048 ms 1.540 ms 90.5% 83.3% fast baseline drifts below perfect quality on this corpus
generic prompt_compactseed_require_summary_snippet_fn 5.940 ms 8.839 ms 100.0% 100.0% compact lexical seed table + helper-backed one-hop REQUIRES_FILE rescue
generic prompt_lexseed_require_summary_snippet_fn 28.266 ms 41.960 ms 100.0% 100.0% historical full-summary lexical rescue, now dominated
code-aware prompt_summary_snippet_py 1.080 ms 1.775 ms 79.8% 66.7% worse baseline than the primary cogniformerus corpus
code-aware prompt_compactseed_require_summary_snippet_fn 5.804 ms 8.392 ms 100.0% 100.0% compact lexical seed table + helper-backed one-hop REQUIRES_FILE rescue
code-aware prompt_lexseed_require_summary_snippet_fn 36.676 ms 60.457 ms 100.0% 100.0% historical full-summary lexical rescue, now dominated

Interpretation:

  • the external folding miss was a real seed-selection problem, not a snippet extraction bug
  • the rescue is now verified on both local Apple Silicon and AWS ARM64
  • the current documented external rescue is no longer the old full-summary lexical path; it is the compact lexical-seed table variant
  • compact lexical seeding keeps 100.0% / 100.0% while cutting the old rescue by about 4.8x locally and 4.7-7.2x on AWS, depending on mode
  • an isolated local timing split shows the helper-backed rescue is still dominated by lexical-seed + REQUIRES_FILE fetch work (~10.7-11.0 ms/query) with snippet postprocess as a secondary cold-start cost (~7.7-8.0 ms/query)
  • the old full-summary lexical rescue remains useful as a diagnostic, but it is no longer the external default frontier
  • even the compact rescue is still slower than the primary in-repo winners, so it does not replace them as the default GraphRAG contract

Legacy/manual IVF-PQ benchmark

The sections below are still useful for the explicit IVF-PQ API (svec_ann_scan), but they are no longer the default ANN baseline for the repository. Those measurements target the legacy/manual vector path, not the planner-integrated sorted_hnsw Index AM.

All IVF-PQ benchmarks below use svec_ann_scan (C-level) with residual PQ. 1 Gi k8s pod, PostgreSQL 18.

103K vectors, 2880-dim (Gutenberg corpus)

Residual PQ (M=720, dsub=4), 256 IVF partitions. 100 cross-queries (self-match excluded):

Config R@1 Recall@10 Avg latency
nprobe=1, PQ-only 54% 48% 5.5 ms
nprobe=3, PQ-only 79% 71% 8 ms
nprobe=3, rerank=96 82% 74% 10 ms
nprobe=5, rerank=96 89% 86% 12 ms
nprobe=10, rerank=200 97% 94% 22 ms

Self-query (vector in dataset): R@1 = 100% at nprobe=3 / 8 ms.

10K vectors, 2880-dim (float32 precision test)

Same corpus, pure svec (float32), nlist=64, M=720 residual PQ. 100 cross-queries:

Config R@1 Recall@10
nprobe=1, PQ-only 56% 56%
nprobe=3, PQ-only 72% 82%
nprobe=5, rerank=96 93% 93%
nprobe=10, rerank=200 99% 99%

float32 vs float16 precision impact

Tested the same 10K Gutenberg vectors in two configurations:

  • float32 (svec): native 32-bit storage, independently trained codebooks
  • float16-degraded (hsvec): svec → hsvec → svec roundtrip, independently trained

Result: no measurable recall difference. Float16 precision loss (~1e-7) is 1000× smaller than typical distance gaps between consecutive neighbors (~1e-4). The recall bottleneck is PQ quantization and IVF routing, not input precision. This confirms hsvec is a safe storage choice for ANN workloads.

CRUD performance (500K rows, svec(128), prepared mode)

Operation eager / heap lazy / heap Notes
SELECT PK 85% 85% Index Scan via btree
SELECT range 1K 97% – Custom Scan pruning (eager only)
Bulk INSERT 100% 100% Always eager
DELETE + INSERT 63% 63% INSERT always eager
UPDATE non-vec 46% 100% Lazy skips zone map flush
UPDATE vec col 102% 100% Parity both modes
Mixed OLTP 83% 97% Near-parity with lazy

Eager mode (default) maintains zone maps on every UPDATE for scan pruning. Lazy mode (sorted_heap.lazy_update = on) trades scan pruning for UPDATE parity with heap. Compact/merge restores pruning.

Self-query vs cross-query

Self-query: query vector is in the dataset (typical RAG case — you embedded documents, now you search them). The vector is always found as its own closest neighbor, so R@1 = 100%.

Cross-query: query vector is NOT in the dataset (e.g., user question embedded at search time). R@1 depends on nprobe and PQ fidelity.

When comparing benchmarks, verify whether self-match is included or excluded. The tables above use cross-query (self-match excluded) for honest comparison.

Methodology notes

  • EXPLAIN ANALYZE: warm cache (pg_prewarm), average of 5 runs, actual execution time + buffer reads reported
  • pgbench: 10 s runtime, 1 client, includes pgbench overhead (connection management, query dispatch); useful for relative throughput comparison
  • INSERT: COPY path via INSERT ... SELECT generate_series()
  • Compact time: wall-clock time for sorted_heap_compact() on warm data
  • Vector search: 100 random queries from the dataset, self-match excluded by requesting lim := 11 and taking positions 2–11. Ground truth via exact brute-force cosine (<=> operator). Latency measured via clock_timestamp() per-query in PL/pgSQL loop (20 queries, warm cache)
  • Current local sorted_hnsw comparison: deterministic synthetic 10K x 384D corpus via scripts/bench_sorted_hnsw_vs_pgvector.sh, 3 fresh builds for PostgreSQL methods, median p50 / median recall reported; Qdrant via scripts/bench_qdrant_synthetic.py, 3 warm measurement passes on one local Docker collection; zvec via scripts/bench_zvec_synthetic.py, 3 warm measurement passes on one local in-process collection
  • Current local real-dataset sample: scripts/bench_ann_real_dataset.py on ANN-Benchmarks nytimes-256-angular, sampled to 10K base vectors and 20 queries. Ground truth is exact PostgreSQL svec heap search on the sampled corpus. Numbers above are medians across 3 full harness runs.