ContextNest vs Amazon Bedrock Knowledge Bases (vector + GraphRAG), v3.1

Misha Sulpovar, PromptOwl · benchmark run 2026-10-13 to 2026-10-14 · paper v3.1, 2026-10-28 (first draft 2026-10-14) · amends and extends the report of 2026-10-02 (reproduced unchanged in the Annex)

1. What changed since v2 and why
2. Executive results
3. Method
4. Results: full tables
5. GraphRAG analysis
6. Governance results
7. Cost
8. What we fixed in ContextNest because of this
9. Limitations
10. Reproduce it
11. Annex: the v2 report (2026-10-02), unchanged

1. What changed since v2 and why

The v2 report tested one question on 30 questions over 11 short documents. The reviewer's verdict was that it was "not a defensible conclusion". This table maps each request to what v3 does about it.

2. Executive results

Retrieval: headline comparison per dataset (NDCG@10)

"Bedrock best" is the strongest fair Bedrock arm on that dataset: vector + Cohere Rerank 3.5 (B2) on the BEIR and MultiHop-RAG sets, and vector with a status=current metadata filter (B1-filter) on VersionedDocs, since B2 there returns superseded versions in 90% of top-10 lists. "ctx" is ctx hybrid + passage rerank (C3p).

GraphRAG was not run on the BEIR sets (A9). p = paired randomization test, as reported by make_results.py (0.0000 means no permutation reached the observed difference). Source: results/final.md.

MultiHop-RAG, plainly: the lower CI bound is −0.025 and the rule needs it above −0.020, so parity is not established there. Neither is a Bedrock advantage (p = 0.3482). The answer-level result on the same dataset is a tie.

Answers (Claude Sonnet 4.6 answers; Claude Opus 4.6 judge)

Hedged answers make up the remainder of each row; full rows, including the superseded whole-document ctx arm, are in section 4. Judge agreement (Cohen's κ, primary vs check): 0.7462 on MultiHop-RAG, 0.8781 on VersionedDocs.

Stale versions (VersionedDocs, old and new copies both present)

Validated items only (A21). On the 278 questions whose answer really changed, the pattern is the same and the stale-answer gap is wider:

Answer rates are on the 132 questions of the 250-question answer sample that pass the audit; Stale@10 on all 278.

Stale@10 = share of questions whose top 10 contains the superseded version of the gold page. The flat ctx arm leaks like Bedrock does: the advantage comes from versioning, not from the search engine.

3. Method

Datasets

VersionedDocs. Built from public git history of kubernetes/website content/en/docs: current 51675a5 (2026-10-13) against previous fb49b3e (2024-10-12). 180 pages exist in both with a changed fact-bearing sentence (A2). A model generated 1,250 questions in two passes; a validator kept 541 (56.7% dropped), requiring that the gold fact appears only in the current page and the stale fact only in the previous one, and rejecting questions that copy the page's wording. 155 strict paraphrases (no 4+-letter word from the page except names) have a query-term overlap of 0.353 against 0.796 for their originals (A14). The model's questions are a known risk; see Limitations.

Question validity (A21). A 50-item spot-check found that many generated items do not test a real change: the "change" is rewording, a typo fix or a rename of the same thing, or the gold or stale fact does not answer the question. All 541 items were then audited by Claude Sonnet 4.6 against four criteria (natural single-answer question answered by the current page; gold fact answers it; stale fact is what the previous page said; the answer genuinely changed). 278 of 541 pass (91 of the 155 paraphrases). The audit agrees with the spot-check on 39 of 50 items overall and 44 of 50 on "changed"; the spot-check was also model-made, not a human hand-check. NDCG is defined by the page a question is about, so it stays meaningful on every item; the answer-level stale label is only meaningful on valid items. Every VersionedDocs table is reported on both sets.

Arms

All arms return a ranked list of document ids; chunk hits collapse to their parent document at their best rank. Every arm received identical text (frontmatter stripped; title + body). The ctx arms were measured on the release candidate (engine commit 9e1ab02) and re-run on the published @promptowl/contextnest-cli@3.4.0 (A20; status in section 8).

Models

The answer model is the same for every arm, so a same-family bias in the judge would apply to every arm alike and cancel in arm-to-arm differences.

Metrics and statistics

• Primary: NDCG@10. Secondary: MRR@10, MAP@10, Recall@20 (cross-system; Bedrock's retrieve caps at 100 chunks, so Recall@100 is local-only, A5).
• Headline difference per dataset: 95% bootstrap CI of the per-query NDCG@10 difference and a paired randomization test.
• Parity, defined before results: the CI's lower bound lies above −0.02.
• VersionedDocs: Stale@10. MultiHop-RAG: Evidence-Complete@k (A11).
• Answers: share of answers per label, judge agreement as Cohen's κ.

Protocol amendments

The full text is in PROTOCOL.md. Every amendment that changes a reported number is marked.

4. Results: full tables

Every number in this section is copied from results/final.md, which make_results.py generates from the raw runs in runs/.

SciFact (300 queries, 5,183 documents), development set

C3p − B2 NDCG@10 = +0.035, 95% CI [+0.015, +0.058], p = 0.0008. Parity rule met.

NFCorpus (323 queries, 3,633 documents), development set

C3p − B2 NDCG@10 = +0.020, 95% CI [+0.007, +0.033], p = 0.0013. Parity rule met.

MultiHop-RAG (529 queries, 609 documents), development set for ctx lexical; C3p post-hoc

C3p − B2 NDCG@10 = −0.008, 95% CI [−0.025, +0.009], p = 0.3482. Parity rule not met. On this set plain BM25 (0.735) also beats Bedrock vector without rerank (0.685).

VersionedDocs (541 queries, 180 documents + 180 previous versions), held-out

C3p − B1-filter NDCG@10 = +0.075, 95% CI [+0.047, +0.102], p = 0.0000. Parity rule met; ctx significantly better. Each question has one gold page, so MRR@10 and MAP@10 coincide.

VersionedDocs, paraphrased questions (155 queries)

C3p − B1-filter NDCG@10 = +0.048, 95% CI [−0.001, +0.096], p = 0.0620. Parity rule met; not significant. Lexical-only ctx (C1) falls to 0.456 when the question shares little wording with the page: this is where embeddings earn their place. Stale@10 on paraphrases (now measured, A20): Bedrock B1 0.742, B2 0.781; B1-filter, F1 and every nest arm 0.000.

VersionedDocs, validated items only (278 queries, A21)

C3p − B1-filter NDCG@10 = +0.077, 95% CI [+0.038, +0.117], p = 0.0000. Parity rule met; ctx better.

VersionedDocs, paraphrased, validated items only (91 queries, A21)

C3p − B1-filter NDCG@10 = +0.048, 95% CI [−0.022, +0.116], p = 0.1924. Parity rule not met on this subset: 91 questions are too few to bound the difference at −0.02.

Answers, full rows

Claude Sonnet 4.6 answers at temperature 0 from a 6,000-character context; Claude Opus 4.6 labels; gpt-oss-120b check labels. κ is per arm; "mean ctx chars" is the context actually supplied.

All-answer agreement: MultiHop-RAG κ 0.7462, raw agreement 0.8522, 133 disagreements; VersionedDocs κ 0.8781, raw agreement 0.9312, 86 disagreements. The MultiHop-RAG sample is stratified by question type from the frozen 600 (seed 13) and includes no-evidence questions (35 of 300, the null_query items in bedrock/out/multihop.sample300.qids). The VersionedDocs sample is stratified by change type (seed 13).

The A19 correction, stated plainly. The first ctx answer run fed the model the opening 6,000 characters of whole documents, while Bedrock arms fed retrieved ~300-token chunks. VersionedDocs pages average 16.6k characters, so the model mostly saw the top of the top page: 46.8% correct and 48.8% missing, despite the best retrieval. The engine returns each hit's best-matching passage in --json, which is what an agent receives. C3pass uses those passages under the same 6,000-character budget. We publish both rows; the amendment was written before the corrected result.

5. GraphRAG analysis

Result-count sensitivity

The protocol pre-registered GraphRAG at 100 results, Bedrock's maximum. At that setting it scored 0.169 NDCG@10 on MultiHop-RAG. As the requested count falls, it improves, peaking at 10 results on MultiHop-RAG and at 5 on VersionedDocs. At high counts, graph-expanded chunks displace the best direct hits (A18). We report every count and use the best one as the headline, which favours Bedrock.

Evidence-Complete@k on MultiHop-RAG (A11)

A graph is meant to help assemble evidence spread across several articles. Evidence-Complete@k is the share of the 529 questions whose top k contains every gold evidence article. The table also shows how many distinct documents each arm returned per question, since GraphRAG's chunks collapse to few documents.

Not in final.md: produced by the harness scorer, bedrock/bx score.py multihop B1 B2 B3 C1 C2 C3p bm25 /lab/runs/multihop_k{5,10,20,50}.B3.json --ks 2 4 5 10 (output in bedrock/out/scores/multihop.json). Its NDCG@10 column reproduces final.md for every arm, which checks the run files. Not significance-tested.

Reading it

• On its intended case, GraphRAG with default configuration did not reach parity with Bedrock's own vector arm, or with ctx. Its best NDCG@10 (0.529) is below plain BM25 (0.735).
• Its answers follow its retrieval: 39.0% correct and 54.7% missing on MultiHop-RAG, against 66.0% and 65.7% correct for B2 and ctx.
• At its best count it returns 4.4 distinct documents per question, which caps how much multi-article evidence reaches the model. Evidence-Complete@10 is 0.3025 against 0.6994 (B2) and 0.7032 (ctx).
• Configuration: AWS defaults for chunking and retrieval, CHUNK_ENTITY_EXTRACTION with Claude Haiku 4.5, Titan v2, 16 m-NCU. We did not tune GraphRAG beyond the result count. A tuned graph may do better; the harness takes any configuration.
• Indexing time: about 2 hours per corpus (MultiHop-RAG, 609 documents: 7,535 s and 7,432 s for the two builds; VersionedDocs, 360 documents: 7,011 s), against seconds to minutes for ctx embed (section 7).

6. Governance results

VersionedDocs asks: when a page changes and the old version is still stored, does the old answer reach the model? Each question's gold fact exists only in the current page, and its stale fact only in the previous one.

• Bedrock leaks without a filter. With old and new copies side by side, Bedrock vector put a superseded version in the top 10 for 86% of questions (Stale@10 0.863). Its reranker made it worse (0.900), because the old page is also on topic. GraphRAG at its best count: 0.708. Answers from B2 committed to the stale fact 13.6% of the time; from GraphRAG, 12.0%.
• A maintained filter fixes it. A status=current metadata attribute on every object, with a filter on every query, takes Bedrock to 0.000. Your ingestion pipeline must set that attribute correctly on every file, every time.
• The nest does it by default. Every versioned ctx arm scored 0.000 Stale@10. Superseded versions live in history, reachable through ctx history and ctx reconstruct, not in search. ctx answers were stale 2.0% of the time (C3pass).
• The win is versioning, not search. Store old copies as plain files in ctx and it leaks like Bedrock: Stale@10 0.861 (keyword) and 0.843 (hybrid).
• The two compose. F1 feeds Bedrock only the nest's current published snapshot. It beats Bedrock's own filter on retrieval (0.838 vs 0.827 NDCG@10, not significance-tested) and gave the best answers in the study: 74.4% correct, 1.2% stale.

7. Cost

Measured: what this benchmark cost

The harness ledger (bedrock/out/spend.jsonl) records every billable call at AWS list price (us-east-1, fetched 2026-10-13). ./bx plan.py spend reports TOTAL committed $56.59 for the 10-13/14 run (A20 added the first MultiHop graph the harness had missed), of which:

Correction (A20). The harness first counted only two graph lifetimes. Measured lifetimes from progress.log: MultiHop-RAG graph 03:11 to 05:29 UTC (about 2.3 h), VersionedDocs graph 06:21 to 08:33 (about 2.2 h), MultiHop-RAG rebuild 10:53 to 13:18 (about 2.4 h), at $0.48/h. The first graph is now in the ledger, which brings the total to $56.59. The 2026-10-28 additions were the validity audit ($4.72) and the re-run on the released build (Titan v2 embeddings and Cohere rerank, a few dollars).

Indexing (measured)

GraphRAG extraction per document = $9.0908 ÷ (2 × 609) = $0.0075 on MultiHop-RAG (~2.6k tokens per document). cost_report.py now divides by both builds (A20). Its ctx rows for MultiHop-RAG and VersionedDocs read "0 s", which reflects an incremental re-run over an existing index, not a cold build, so they are not cited. Bedrock vector ingestion times include Knowledge Base sync polling.

Projected monthly cost

Same formulas and assumptions as cost_report.py, with the extraction rate corrected to $0.0075 per document: 51,000 reads a month (the nest TCO model's mid-market case: 200 people × 5 questions a day × 21 days, plus 20 agents × 10 runs × 5 reads × 30 days), 10% of documents changing per month, 2,600 tokens per document. Reranking ($102/month at 51,000 reads) is included on both sides. Answer-generation tokens are excluded because they are the same whichever retriever feeds them.

Fixed floors: Neptune Analytics minimum 16 m-NCU = $0.48/h ≈ $350/month while the graph exists; OpenSearch Serverless 2-OCU floor ≈ $350/month (a NextGen collection can scale to zero, per the TCO node). Community Nest hosting: Lightsail 4 GB ($24) + managed PostgreSQL ($15). Excluded: the Community Nest self-hosted license ($10 per person per month, 30-person minimum, per the TCO node), which applies only to a self-hosted team server; the ctx CLI and engine are free.

Script for the corrected figures

# same constants and formulas as cost_report.py; two corrections
ext = 9.0908 / (2 * 609)                      # ledger "ingest graph:multihop" covers two builds of 609 docs
hours = {"MultiHop graph 1": 2.3, "VersionedDocs graph": 2.2, "MultiHop rebuild": 2.4}
nep = sum(hours.values()) * 0.48              # 6.9 h, $3.31
missing = 2.3 * 0.48                          # first MultiHop graph, not in plan.py TOTAL: $1.10
total = 55.48 + missing                       # $56.58
READS, CHURN, TOK = 51_000, 0.10, 2_600
TITAN, RR, OSS, NEPM, HOST = 0.02, 2.00, 350.0, 16*0.03*730, 24+15
rerank = READS/1000*RR
for n in (1_000, 10_000, 100_000):
    ch = n*CHURN; emb = ch*TOK/1e6*TITAN; qe = READS*30/1e6*TITAN
    s3 = emb+qe+rerank; oss = s3+OSS; graph = s3+NEPM+ch*ext; ctx_hosted = HOST+s3
# -> 1k: 102 / 452 / 453 / 141   10k: 102 / 452 / 460 / 141   100k: 103 / 453 / 528 / 142

8. What we fixed in ContextNest because of this

The released ctx 3.0.0 (@promptowl/contextnest-engine@3.0.0, pinned in the lab image) scored 0.604 NDCG@10 on SciFact (results/gate.md), against 0.690 for plain BM25. That was a real gap in our product, and the benchmark found it. These changes shipped in ctx 3.4.0 (PromptOwl/ContextNest#190). They were developed on branch misha/retrieval-modes of PromptOwl/ContextNest, commits through 9e1ab02, the release candidate measured here.

Shipped-only rule. The protocol counts only shipped ctx code (section 2 of PROTOCOL.md). The first measurements used the release candidate (commit 9e1ab02); every arm is being re-run on the published @promptowl/contextnest-cli@3.4.0. Every arm finished so far is identical to the release candidate; the hybrid + rerank arms for NFCorpus, MultiHop-RAG and VersionedDocs are still running (A20). The search code between the two differs only in hardening (configuration checks, index file format, retry budget) and whitespace folding in exact-title matching.

9. Limitations

• Only one held-out set against Bedrock. SciFact and NFCorpus chose ctx defaults (A7). MultiHop-RAG was used to fix ctx lexical scoring (A8), and the passage rerank was prompted by a MultiHop-RAG result (A17). VersionedDocs and its paraphrases are the only held-out comparison.
• FiQA not run. FiQA-2018 (57,638 documents), pre-registered as held-out and lexical's worst case, was stopped for memory. Its absence leaves the paraphrase-heavy case resting on 155 VersionedDocs paraphrases.
• One run per arm. No re-indexing variance. Bedrock ingestion, especially LLM extraction for GraphRAG, may vary between builds.
• Titan v2, not Cohere Embed v4. Bedrock Knowledge Bases rejected Cohere Embed v4 (A16), so both sides use Titan v2, a weaker embedder. Preliminary local numbers with Cohere v4 exist (A13: e.g. SciFact ctx+Cohere 0.761 vs Cohere dense alone 0.777, whole documents, no chunking) and are not part of the comparison.
• Model-generated questions on VersionedDocs. A model wrote the questions and a validator dropped 56.7%, but a later audit (A21) found only 278 of the remaining 541 describe a real change of fact. Results are reported on both sets; the validated subset is itself chosen by a model (agreement with a 50-item spot-check: 39 of 50), and no human hand-check was done. A stricter generator with a validity gate before any run is the fix for v4.
• GraphRAG configuration. AWS defaults except the result count. Haiku 4.5 extraction (AWS-documented). Other graph configurations were not tested.
• Answer samples. 300 MultiHop-RAG and 250 VersionedDocs questions per arm, only on the arms the claims rest on (A15). Answer rates have no confidence intervals in this paper.
• The A19 correction. The ctx answer figure (C3pass) comes from a corrected harness. The amendment predates the corrected result, and both results are published, but a reader should know the first run looked very different.
• Not reported from the protocol: Holm correction across arms, paired t-tests, p50/p95 latency, the open-source dense anchor (bge-base) in the final tables, and third-pass adjudication of judge disagreements. The headline uses bootstrap CIs and randomization tests only.
• Cost projections use our assumptions for read volume and churn, and the TCO model's engineering estimates, which are assumptions, not measurements.

10. Reproduce it

• Harness: retrieval-v3-lab/. PROTOCOL.md (frozen protocol and amendments), runs/ (raw per-query run files for every arm, Bedrock chunk sidecars), results/make_results.py → results/final.md and final.json, bedrock/score.py (Evidence-Complete, Stale@10), cost_report.py.
• Pinned images: everything runs in Docker. retrieval-v3-bedrock:0.1 (the lab image plus boto3/botocore 1.43.100) for the Bedrock arms; the lab image pins @promptowl/contextnest-engine@3.0.0 and stemmer@2.0.1; the released ctx 3.4.0 used for the re-runs is installed in engine-release-3.4.0/ (run_release.sh, cmp_release.py); the release candidate is snapshotted in engine-snapshot-9e1/.
• Bedrock configuration: full Knowledge Base and data-source JSON for every KB in bedrock/out/manifests/.
• Your models: every model is an environment variable (EMBED_MODEL_ID, EXTRACTION_MODEL_ID, RERANK_MODEL_ID, GEN_MODEL_ID, JUDGE_1, JUDGE_2, …); see "Rerun with your own models" in bedrock/README.md. The default embedder is Titan v2, as used here.
• Cost safety: ./dryrun.sh runs the whole pipeline against stubs with no AWS calls; ./bx estimate.py prices the plan first; the spend ledger enforces a hard cap; teardown.sh deletes only the resources the harness created.

Annex: the v2 report (2026-10-02), unchanged

Reproduced as delivered. Its conclusions are superseded where v3 measures the same thing, and section 1 maps each objection to v3. Two statements in it are out of date: "semantic seed selection designed, not shipped" (hybrid semantic search shipped in ctx 3.4.0; see section 8) and "Paraphrase recall on large corpora: weaker, lexical" (see the paraphrase table in section 4).