Document Audit Agent

The problem

A legal team reviewing contracts at volume needs a playbook: the canonical version of each clause, and a rule for what counts as an acceptable deviation. Building one by hand means a lawyer reading a few hundred existing contracts and reconciling every variant of the same clause into one. It is weeks of work, it is never finished, and the moment it is finished it is out of date.

Then there is the other half. Once the playbook exists, every incoming contract has to be checked against it, including the counterparty’s tracked changes — which is the part where a human reviewer’s attention runs out first, and where the expensive mistakes live.

The approach

Two pipelines over the same corpus. The first builds the playbook; the second audits against it.

Neither is an agent. There is no planner deciding what to do next — the orchestration is ordinary Python, and the model is invoked at specific stages to do specific bounded jobs. That distinction is the single most consequential design decision in the service, and the reasoning is below.

Playbook construction, five stages

1. Structural clause extraction with no model involved at all — regex and raw OOXML parsing, parallel across documents. Document structure is a solved problem; paying a model to rediscover it would be slower, more expensive and less reliable.
2. Clause naming by a light model call at minimal thinking budget, parallel.
3. Embeddings in batches, computed concurrently.
4. Greedy Star clustering by cosine similarity above a tuned threshold, filtered by a minimum document frequency, with the invariant that a single document contributes at most one clause to any cluster. Without that invariant, one verbose contract can manufacture a cluster on its own and a house style becomes a “standard”.
5. Consolidation — one model call merges the variants into a single standard clause, with anonymisation required in the prompt so identifiers, values and dates come back as placeholders.

Audit, hybrid retrieval

Matching an incoming clause to its playbook entry by embedding alone is not good enough: two clauses about payment terms are semantically close whether or not they are the same clause. So retrieval combines the embedding with title similarity and normalised token overlap, and a high-confidence title match overrides the vector score outright.

Two model calls then run in parallel: a full compliance check that returns a verdict plus proposed wording, and a separate judgement on the counterparty’s redlines that sorts each one into acceptable, unacceptable, out of scope, or needs a human.

Decisions worth defending

The model is a step, not a planner. An agent with tools could have done this. It would also have made every run a different shape, and a legal audit whose method varies per execution cannot be reviewed, reproduced, or explained to the person who has to sign the contract. Fixing the pipeline in code costs flexibility the domain does not want, and buys back the ability to say exactly which stage produced which conclusion. The trade-off is real: adding a new analysis means writing a stage rather than editing a prompt. That was the right price.

Textual anchors are validated against the document, not trusted. When the model proposes a literal snippet to anchor an inserted clause, the code rejects that anchor unless it survives a set of checks — no line breaks, a length inside a narrow band, not beginning as a sub-item, and crucially occurring exactly once in the document. A plausible-looking anchor that appears twice would insert the clause in the wrong place, and nothing downstream would notice. Failed anchors fall through a defined cascade rather than guessing.

An empty response is a failure, not an empty result. Reasoning tokens draw from the same budget as the answer, so a model that thinks too hard about a hard clause returns nothing at all. A tolerant parser that treated that as “no findings” would silently pass non-compliant contracts. Instead the parser fails loudly on an empty body, which is what triggers the retry — and max_output_tokens is calibrated per stage because of it.

Prompts are calibrated against false positives explicitly. A reviewer who gets ten spurious flags stops reading the eleventh, so the instruction is to prefer “compliant” when genuinely uncertain. An audit tool’s real failure mode is not missing a problem; it is being ignored.

Closing the loop

A suggestion that a reviewer has to retype by hand is a suggestion that does not get used. A companion plugin and relay apply accepted changes directly inside the collaborative document editor with track changes enabled, so an AI-authored edit arrives as a revision someone accepts or rejects — never as a silent write.

That path chains three transports: the command’s metadata arrives over server-sent events, and the plugin fetches the full payload in a separate single-use request. Sending the payload down the event stream would have been simpler and would have put large documents through a relay that has no business holding them. Inside the editor, a permanent bidirectional message channel replaced raw cross-frame posting, because the plugin and its host sit on different origins and a persistent pipe beats round-tripping through the backend for every acknowledgement.

Six document operations are supported: replace with tracking, insert before a textual anchor, delete a counterparty insertion, restore a removal, and accept or reject a native revision.

Scale

• ~4,300 lines of service code, ~1,050 lines of tests, 47 modules
• 2 pipelines, 7 distinct LLM call types, each with token and cost accounting
• 768-dimension embeddings in pgvector; up to 25 concurrent workers with context propagation
• ~945 lines in the editor bridge: 6 operations, 3 chained transport layers