Skip to the yard
Waxsealai

OpenWiki and Evidence Graphs Improve Agents’ Cross-Document Answers

Shipped
Shipper
James Johnson
Readers
734

In tests on synthetic insurance documents, an OpenWiki memory layer improved cross-source accuracy; adding evidence links raised accuracy and citation consistency.

OpenWiki and Evidence Graphs Improve Agents’ Cross-Document Answers

Agents that answer questions across a company’s documents need more than a way to find passages. They must connect related material, track which rules have changed, and support answers with evidence. Tests of OpenWiki, a compiled wiki memory layer, suggest that it can help with this work: on questions requiring connections across documents, the agent was about five times as accurate as one using keyword search alone. Adding an evidence graph improved accuracy further and made citations more consistent.

Those gains came from a synthetic insurance library designed to stress cross-document reasoning. They don’t establish that a compiled wiki will pay off for every workload. Simple lookup questions showed little accuracy benefit, and the reported answer costs leave out the work of compiling and updating the wiki. The practical case depends on how often agents need to reason across the same sources.

Why compile documents into a wiki?

Search systems typically return relevant passages, but an agent may still need to work out how those passages fit together. A provision might be amended elsewhere, or a newer document might replace an older one. When each question requires the agent to reconstruct those relationships, retrieval can become expensive and answers can vary between runs.

OpenWiki is an open-source command line tool built on Deep Agents. It compiles source material into linked wiki pages that summarize concepts and relationships, with citations back to source lines. The resulting context can be reused across questions. The tool was first built to help coding agents retain information about software architecture and code behavior instead of rediscovering it for every task. Its documentation is available at OpenWiki.

Each statement in OpenWiki is represented as a claim linked to supporting source lines and a hash of those lines. If the source changes, the hash changes too. OpenWiki can then flag affected claims without calling a model, and send only those claims for confirmation, revision, or removal. This gives the memory a way to respond to source changes while keeping the initial staleness check deterministic. More detail is available in Building self-correcting memory in OpenWiki.

A test corpus built around document dependencies

The evaluation used 111 long-form synthetic insurance documents. Insurance provided a useful test case because policies and endorsements can refer to, amend, or override one another. The corpus included outdated versions, contradictions, indirect reference chains, different wording for similar concepts, and very long documents.

To score answers, the researchers created a ledger of 972 facts and controlled where each fact appeared, how documents referred to each other, and what changed between versions. An LLM generated the surrounding prose, and automated checks confirmed that the required facts were present.

The same retrieval agent was tested in two configurations: keyword search over the original documents, and access to an OpenWiki compiled from those documents. The model, instructions, source material, and questions stayed the same; access to compiled memory was the variable. The evaluation included 80 simple questions answerable from a single passage and 50 connected questions that required following relationships across documents. Each question was run three times.

Accuracy measured whether answers included required facts and evidence while avoiding contradictions, using a checklist drawn from the ledger. Citation consistency measured how much the cited evidence overlapped across the three runs, using average pairwise Jaccard similarity. That score indicates whether an agent returns to the same sources; it does not show whether those sources or the answer are correct. The evaluation also tracked model cost in US dollars and wall-clock time per question.

Memory helped most on connected questions

OpenWiki reduced the 111-document library to a wiki about one-tenth the size of the original corpus. Compilation cost $4.15, and adding 17 documents later cost about $0.09 per document. The reported retrieval and answering costs do not include either compilation or update costs.

For simple lookups, search and wiki performance was nearly identical. Accuracy was 0.84 with search and 0.83 with the wiki. Citation overlap was 0.72 and 0.74, respectively, and cost per correct answer was $0.020 with search versus $0.021 with the wiki. Answers took 9 seconds with search and 12 seconds with the wiki. In this part of the test, the extra context did not improve accuracy and added latency.

The result changed on connected questions. Search alone reached 0.10 accuracy, compared with 0.52 for the wiki, a roughly fivefold improvement. The compiled context gave the agent a clearer way to apply domain relationships across sources, while reducing the need for repeated tool calls. Cost per correct answer fell from about $0.55 to $0.19.

Yet accuracy gains did not ensure that the agent found the same evidence each time. Citation overlap for the wiki agent was 0.25 on connected questions, compared with 0.74 on simple lookups. That matters in settings where sources conflict: a different set of supporting passages can lead to a different answer and make it harder to audit how the agent reached it.

Using a graph to make evidence easier to follow

The next step was a lightweight evidence graph that turns citations and document references into typed links. The wiki remains responsible for synthesizing information; the graph gives the agent explicit routes from claims and pages to supporting paragraphs and related documents.

Graph construction starts by splitting each document into paragraphs and assigning every paragraph a unique identifier. That makes a paragraph, rather than an entire document, the smallest unit of evidence. The system then links claims, paragraphs, wiki pages, and other documents with relationships such as cite, supersede, and refers_to. Other edge types include amends and attaches_to, along with their inverse relationships.

Some links come directly from document references and can be extracted deterministically. Existing citations in wiki claims can also be turned into graph edges without an additional model call. The graph took under a second to build and resolved every explicit cross-reference in the test library. Its scope is evidence navigation: it does not attempt to represent everything an organization knows. Links derived from the wiki still rely on the accuracy of the model-generated citations.

Two tools give the agent access to the graph. locate finds a starting point by matching question terms against document titles, section headings, wiki pages, and claim statements. expand follows selected edges to related nodes and supporting passages. When there is a single path, it can continue automatically; when the path branches, the agent decides which connection to follow. A similar pattern of locating relevant nodes and collecting connected context appears in Microsoft’s GraphRAG.

What the graph changed—and what it costs

Compared with the wiki alone, adding the evidence graph raised accuracy by 27%, doubled citation consistency, reduced median answer time by 40%, and cut cost per correct answer by 58%. The graph directs retrieval along relationships already represented in citations and document references. It improved the results without making the agent’s answers fully deterministic.

These figures point to a specific use case: recurring questions that require reasoning across shared, interconnected documents. For a one-off lookup, compiling and maintaining a wiki may add time and expense without improving accuracy. The experiment’s answering costs exclude compilation and maintenance, so teams assessing the economics need to account for those costs and how often the memory will be reused.

When multiple agents repeatedly work across the same disconnected sources, a shared wiki can preserve useful relationships, while graph edges help them return to evidence. The remaining work is to test that balance on real document collections and workloads. Planned directions include compiling only the context needed for cross-source questions and exploring claims and typed links as building blocks for factual, episodic, and procedural agent memory.

Source: James Johnson · www.langchain.com

Same vesselRelated cargo