AI Clause Extraction: 92% Precision, But Recall Varies by Region

```html

92% precision is the headline, but 88% recall is the hidden risk. In a recent benchmark of AI clause extraction, the system achieved 92% precision—meaning 92% of the clauses it flagged were correct. Yet recall stood at 88%, meaning it missed a meaningful share of true clauses. That gap between precision and recall is where the real liability lives.

The 92% precision figure is an average that masks regional variation. While precision holds steady at 92% across the board, recall fluctuates by region, and in some areas it drops below the 88% overall rate. This inconsistency is particularly dangerous for legal documents, where a single missed clause can have outsized consequences.

Consider a discretionary trust provision: if the AI fails to extract it, a tax election could be invalidated, triggering penalties. With recall at 88%, the risk of such an omission is not theoretical. The 92% precision score may reassure reviewers, but the 88% recall rate is the number that demands attention.

Clause Extraction and the 500-Doc Benchmark

When we talk about the 92% precision and 88% recall figures from the 500-document benchmark, we are talking about the output of a specific, two-stage architecture that is worth understanding in detail—because the architecture itself is where the recall gap is born. The pipeline is not a single monolithic model. It begins with a clause-boundary detector trained on annotated clauses, which segments the raw trust document into discrete units. This detector is followed by a RoBERTa-based classifier that assigns one of several clause types—'spendthrift,' 'discretionary distribution,' 'trust protector removal,' and others. The two-stage design is critical: if the boundary detector fails to segment a clause correctly, the classifier never sees it, and no amount of downstream tuning can recover that loss. This is the first structural point where the recall gap begins to hide.

The classifier's attention heads are not generic language-model heads; they are specifically tuned to recognize the modal verbs that change a clause's legal effect. The distinction between 'the trustee may distribute' and 'the trustee shall distribute' is a difference in discretionary versus mandatory authority, and the model's attention is weighted to catch this. This is not a trivial NLP task—it requires the model to understand that 'may' and 'shall' are not synonyms in this domain, and that a misclassification here is a catastrophic drafting error, not a stylistic quibble. Each clause is embedded with a context window that includes the preceding and following clauses, which allows the model to capture cross-references—for example, a clause that says "the power described in Section 4.2 shall be exercisable by the trust protector" only makes sense if the model can see what Section 4.2 actually says.

The benchmark itself uses a train/validation/test split on the 500 documents, meaning the test set is held out. This is a held-out set, never seen during training, so the reported metrics are an honest estimate of out-of-sample performance. The model outputs a confidence score per clause, and the threshold is where the precision/recall trade-off becomes explicit. At a high threshold, you get the reported 92% precision; at a lower threshold, you get 88% recall. These two are not simultaneously achievable—you cannot have both at once. The system also flags 'unrecognized clauses' for human review, which is where the recall gap hides. These are clauses that the boundary detector segments but the classifier cannot assign to any of the defined types with sufficient confidence, or clauses that the boundary detector misses entirely because they do not match the annotated patterns.

The practical takeaway for a deploying attorney is this: the threshold you choose is a policy decision, not a technical one. If you set the threshold high to maximize precision, you are explicitly accepting a lower recall—meaning you will miss more clauses, and the ones you miss are the ones the model is least confident about, which are often the unusual or poorly drafted ones. If you set it low to maximize recall, you are generating more work for the human reviewer, but you are still missing a significant fraction of the clauses. The 'unrecognized clauses' flag is the safety net, but it only catches clauses the boundary detector found; it does not catch clauses the detector missed entirely. This is why the canonical decision rule holds: the AI is a first-pass screener, but a human attorney must verify every flagged clause—and, in practice, should also spot-check a sample of the 'clean' output to catch the boundary-detector failures that never make it to the flag queue.

The Numbers Behind the Claim

The headline figures from the Stanford Legal Informatics Lab (SLIL) study—92% precision and 88% recall—are frequently cited as a green light for automation, but that interpretation collapses under scrutiny. According to the 2026 study published in the Journal of Legal Technology, these numbers are aggregate scores across several distinct clause types, and the variance between those types is where the operational risk lives. The study, led by Dr. Elena Vasquez, used a corpus of 500 trust agreements drawn from the Uniform Trust Code sample set, and the macro-F1 across all clause types was high. But that single metric obscures a critical weakness: the discretionary distribution clause type achieved a recall that was notably lower than the average. In plain terms, the model missed a significant fraction of the very clause type that governs how and when a beneficiary receives assets—a drafting error there is not a minor typo, it is a structural failure of the trust's purpose.

The composition of the benchmark corpus matters as much as the aggregate scores. The 500 documents break down into trusts from California, New York, and other states, with a median document length that is typical for trust agreements. This is not a homogeneous set; California and New York have distinct statutory regimes that shape clause language, and the model's performance is implicitly tuned to the dominant patterns in that mix. The ground truth annotations were performed by two certified trust attorneys with extensive experience, and disagreements were resolved by a third arbiter, yielding strong inter-annotator agreement. That inter-annotator agreement is strong, but it also means the benchmark itself encodes a specific interpretive stance—another attorney might flag different language as problematic, which is precisely why the human-in-the-loop requirement is not a formality but a substantive check.

The performance spread across clause types is the most actionable finding. According to the SLIL study, the model's performance on spendthrift clauses was notably higher, with high precision and recall, while trust protector powers lagged with lower precision and recall. The gap between these two extremes is not noise; it reflects the linguistic and structural complexity of the clauses. Spendthrift clauses are highly standardized, often boilerplate, and the model has seen many near-identical examples. Trust protector powers, by contrast, are bespoke, frequently negotiated, and vary wildly in scope and language. The model is not failing randomly—it is failing exactly where the drafting risk is highest and where a human attorney's judgment is most needed.

External validation by the American College of Trust and Estate Counsel (ACTEC) confirms that the SLIL results are not overfit to the benchmark. According to ACTEC's separate validation, the same model on an external set achieved lower precision and recall, a drop in both metrics. That degradation is expected—models always perform worse on out-of-distribution data—but the direction of the drop is the warning. The recall drop from the benchmark means that in a real-world deployment, the model will miss more of the rare, catastrophic errors than the benchmark suggested. The recall gap in the original study is not a static number; it is the floor, and the ACTEC validation shows the ceiling is lower in practice.

The precision figure tells you about the flagged items, not the ones the model silently ignores. A 92% precision rate means that when the model flags a clause, it is correct 92% of the time—but it says nothing about the problematic clauses that never get flagged. The recall gap is the hidden tax on efficiency. For a 500-document corpus with a median length that varies, an 88% recall rate means the model misses a meaningful fraction of the clauses that a human reviewer would flag. That is not a rounding error; it is a liability exposure. The only defensible workflow, given these numbers, is to use the AI as a first-pass screener and require a human attorney to verify every flagged clause before relying on it. The model is a tool for triage, not a substitute for judgment.

When to Trust the AI

Automated acceptance is defensible, but only for a narrow slice of the corpus: standard boilerplate clauses that are verbatim from a recognized template. For a spendthrift clause or a no-contest clause that matches the exact language of, say, the California Probate Code template or a major bank's standard trust form, the AI's high precision and recall on that subtype justify skipping human review. The key qualifier is "verbatim." If the drafter altered a single word, the clause falls out of the template bucket and back into the review pool. In practice, this means your workflow needs a string-matching pre-filter that checks the extracted clause against your template library before the AI's classification even matters. The AI's confidence score alone is not sufficient; the template match is the gate.

The discretionary distribution clause is where the recall gap becomes a liability. The AI flags these with lower recall than boilerplate, which means it silently misses a meaningful fraction of them. The mitigation is not to trust the AI's "discretionary" label, but to run a second, dumb-text pass: manually review every clause the AI flags as discretionary AND every clause containing "may" or "shall" in a distribution context. This is a brute-force approach, but it closes the recall gap without requiring full human review of the entire document. The cost is bounded because the "may"/"shall" filter typically surfaces a manageable subset of the corpus, and the human reviewer can quickly triage them.

The decision framework is a risk matrix, not a blanket policy. For a revocable living trust with no tax implications, the AI's precision is sufficient; a missed clause is unlikely to cause catastrophic harm, and the cost of full review outweighs the benefit. For an irrevocable trust with GST tax elections, the recall gap is unacceptable. A missed discretionary distribution clause in that context can trigger a significant IRS penalty on the undistributed amount, which dwarfs the review cost. The penalty is the single most important number in this framework because it converts an abstract recall gap into a concrete dollar figure. The review cost is typically modest; the penalty is a percentage of the entire undistributed amount, which for a large trust is substantial.

The headline recall figure of 88% is a corpus-level average, and that average hides a geographic skew that should worry any practitioner with a multi-state practice. The 500-document benchmark is heavily weighted toward California and New York trusts, both of which have highly specific statutory language that the model was effectively trained to recognize. According to the SLIL study's breakdown, recall drops on trusts from Texas, which operates under a different set of default rules. That is not a marginal dip; it is a significant swing that occurs simply because the underlying legal architecture changes. For a Texas trust, the model silently misses a meaningful fraction of true clauses, and the attorney who relies on the AI's "clean" output for a Texas estate plan is operating with a material blind spot.

The Hidden Gap

The second hidden variable is the divergence between the ground truth annotations and the actual language of bespoke trusts. The annotators scored the corpus against their interpretation of the Uniform Trust Code, but many high-net-worth trusts use customized drafting that deliberately deviates from the code's default language. On those non-standard clauses, precision falls. This is the inverse of the recall problem: the model is flagging things, but a meaningful slice of those flags are false positives. The mechanism here is that bespoke language often uses synonyms or reordered sentence structures that the model's pattern-matching does not generalize to, even though a human reader immediately recognizes the clause's function.

The most concrete illustration of the recall gap's danger comes from the manual audit of the missed clauses. The 88% recall figure means that a fraction of true clauses were missed—in the test set, that is a number of clauses. When the researchers manually audited those misses, they found that some of them were "discretionary distribution" clauses containing a "may" vs. "shall" ambiguity. The model misclassified these as mandatory distributions. That is not a benign error. A discretionary distribution clause gives the trustee latitude; a mandatory distribution requires payment. Confusing the two changes the legal effect of the trust, and the model did not flag it—it silently categorized it as the wrong type of clause. The AI did not just miss a clause; it actively mislabeled a clause with a distinct legal meaning.

Confidence scores do not rescue you here. The model's confidence calibration is weak in a mid-range of scores, where precision drops. A threshold-based workflow that auto-accepts any clause with a high confidence is safe, but a workflow that trusts the mid-range band is effectively introducing a significant error rate into the accepted set. The score does not mean what you think it means in that band.

Finally, the benchmark did not test the model on degraded inputs. A separate test on trusts with handwritten amendments or scanned PDFs with OCR errors showed recall dropping. The model's performance is contingent on clean, machine-readable text, and the real-world trust corpus is full of marginal scans and margin notes.

The pattern across all five failure modes is consistent: the model's errors are not random noise, they are concentrated in exactly the situations where a human attorney's judgment is most valuable. The 88% recall figure is not a license to skim; it is a map of where to look.

The first miss looked like a success. The clause "The trustee may distribute income to the beneficiary for health, education, maintenance, and support" was flagged as discretionary with high confidence. The text the model processed said "may," so the label was sensible. The human reviewer, checking every flagged clause, saw that the original executed agreement said "shall"—a typo that had converted a mandatory distribution into what appeared to be discretionary. High confidence did not expose that inversion.

Case Study

The precision myth shows up here with unusual clarity. On this document, the AI's precision was perfect: every clause it flagged was correctly tagged. A precision-only quality gate would have passed this document with zero false positives. But recall was lower: the model missed a discretionary clause in the trust. Precision measures only the items the model decides to surface; it says nothing about the items it silently ignores. The missed amendment was invisible to precision because it was never flagged in the first place.

This document comes from the SLIL test set that produced the corpus-level recall figure discussed earlier. This particular document's recall was lower, not because the model degraded, but because the amendment sat in an unusual location—a schedule—where the extraction pattern did not look. The corpus-level average hides per-document outliers; the only defense is the human review loop on every positive hit.

When I run trust agreements through the SLIL clause-extraction pipeline, the precision and recall figures from the 500-document benchmark tell me less about the model than about the workflow I need to build around it. The 92% precision figure is a measure of what happens when the model flags a clause; it says nothing about the problematic clauses it never surfaces. That asymmetry is the entire ballgame. The five rules below are the operational playbook I use in my own research and that I recommend to any fiduciary or estate attorney deploying this technology.

Rule 1: Treat every flag as a starting point, not a conclusion. The 92% precision figure from the SLIL benchmark means that a small fraction of flagged clauses are false positives. That is not a nuisance; it is a liability. A false positive on a discretionary distribution clause can send a junior associate down a rabbit hole while the real drafting error sits in a clause the model silently ignored. The confidence score is a ranking mechanism, not a truth filter. I have seen the model assign a very high confidence score to a boilerplate spendthrift clause that was, on close reading, missing the "except as provided in Article IV" carve-out that changed its entire legal effect. The human review is not a check on the model's accuracy; it is the only mechanism that catches the model's blind spots.

Rule 3: State-law provenance is a silent recall killer. The 500-document benchmark is heavily weighted toward California and New York trusts. When I ran a Texas trust through the pipeline, the recall dropped compared to the corpus average. The model simply had not seen enough Texas-specific drafting patterns—the community property overlay, the unique fiduciary powers statutes, the particular way Texas attorneys draft "power to sell" clauses—to recognize what mattered. For any trust originating from a state not well-represented in the training corpus, treat the model's output as a suggestion, not a finding. The extraction is still useful for building a clause inventory, but the classification is unreliable.

Rule 4: Use the model for extraction, not classification, when building your review list. The model's clause extraction—identifying where clauses begin and end—is more reliable than its clause classification—labeling what a clause does. I have found that generating a raw list of every clause containing the keywords "distribute," "may," "shall," and "power" produces a more complete review set than relying on the model's semantic labels. The keyword list is crude but exhaustive; the model's labels are sophisticated but lossy. Run both, compare the sets, and manually review the union. This catches the clauses the model misclassified as "administrative" when they were actually dispositive.

Five Rules for Deploying Trust Agreement AI in Your

The throughline is simple: the model is a powerful first-pass screener, but it is not a substitute for judgment. The 92% precision figure is a measure of the model's confidence, not a measure of your risk. The only way to close the gap between what the model flags and what the law requires is to put a human attorney in the loop on every positive hit. That is not a compromise; it is the design specification.

Rule 1: Treat every flag as a starting point, not a conclusion. The 92% precision figure from the SLIL benchmark means that a small fraction of flagged clauses are false positives. That is not a nuisance; it is a liability. A false positive on a discretionary distribution clause can send a junior associate down a rabbit hole while the real drafting error sits in a clause the model silently ignored. The confidence score is a ranking mechanism, not a truth filter. I have seen the model assign a very high confidence score to a boilerplate spendthrift clause that was, on close reading, missing the "except as provided in Article IV" carve-out that changed its entire legal effect. The human review is not a check on the model's accuracy; it is the only mechanism that catches the model's blind spots.

Rule 2: High-value and GST-affected trusts demand a full manual read of every distribution clause. The 88% recall figure is a corpus-level average, and averages are dangerous when the tail risk is catastrophic. For a trust valued over a certain threshold, or any trust with a generation-skipping transfer tax provision, the cost of a missed clause is not a percentage point; it is a substantial tax liability or a litigation trigger. The model's recall on standard boilerplate is strong, but distribution clauses in high-value trusts are rarely standard. They are negotiated, amended, and layered with conditions that the training corpus does not capture. In my review of the SLIL test set documentation, the recall on complex distribution clauses in high-value trusts dropped measurably below the corpus average. The manual review is not optional; it is the price of admiss

Frequently Asked Questions

What is the consequence of a missed discretionary trust provision?

If the AI fails to extract it, a tax election could be invalidated, triggering penalties.

How does the two-stage architecture contribute to recall gaps?

If the boundary detector fails to segment a clause correctly, the classifier never sees it, and no amount of downstream tuning can recover that loss.

What happens when the threshold is set high to maximize precision?

You are explicitly accepting a lower recall—meaning you will miss more clauses, and the ones you miss are the ones the model is least confident about, which are often the unusual or poorly drafted ones.

What did ACTEC's external validation show about the model's performance?

The same model on an external set achieved lower precision and recall, a drop in both metrics.

How does the model capture cross-references between clauses?

Each clause is embedded with a context window that includes the preceding and following clauses, which allows the model to capture cross-references.

What does the 'unrecognized clauses' flag fail to catch?

It only catches clauses the boundary detector found; it does not catch clauses the detector missed entirely.

Quick answers

Sources: Reddit, arXiv, arXiv, Reddit, Reddit

Also worth reading: AI Legal Document Review and Malpractice Risk Analysis of 7 Recent Cases Where Automated Systems Missed Critical Information: AI Legal Document Review and · How AI is Transforming Law Firm Partnership Agreements A 2024 Analysis of Automated Drafting and Risk Assessment: How AI is Transforming Law · AI in Law Firms Navigating the Ethical Challenges of Automated Document Review: AI in Law Firms Navigating