Six NLP Systems, One F1 Winner: Probate Disclaimer Clauses

One in six. Across its evaluation corpus of de-identified probate packets, the best general-purpose LLM failed to surface one in six buried disclaimer clauses - the exact passages that demand a written, irrevocable notice to the executor within nine months of death under IRS Code Section 2518. Every miss burns directly against the qualified-disclaimer window that decides whether a beneficiary's renunciation keeps its tax treatment intact.

The 2026 winner was not the biggest model. An older-generation architecture, fine-tuned on annotated probate pages, out-extracted GPT-4o, Claude, and Gemini by 5-9 F1 points on precisely the clauses carrying the statutory deadline. Context-window size, parameter counts, and benchmark prestige predicted nothing; labeled pages did. In a domain where a single overlooked sentence can redirect an inheritance, depth on narrow clause types beat raw scale.

The economics sharpen the point. Per-task pricing on the Artificial Analysis Intelligence Index runs from $0.21 for GPT-5.6 Luna - roughly 80% below GPT-5.6 Sol's $1.04 - yet dollars bought general intelligence, not disclaimer recall. For executors and beneficiaries racing a nine-month clock, the lesson is unglamorous: a small supervised model that finds the clause beats a frontier system that merely prices it.

Anatomy of the Winner

The winning architecture is deliberately unglamorous: commodity OCR, a Longformer-base span extractor initialized from LegalBERT, and a frontier LLM that never sees an entire packet. Nothing in that stack is novel in 2026 — the margin comes from sequencing. Each stage exists to shrink the next stage's input, and each fails in a characteristic, predictable way.

Stage one is Tesseract or AWS Textract converting scanned probate pages to text. It breaks on skewed scans: county-recorder digitizations routinely arrive off-axis, and one garbled character inside "disclaims" renders the span unfindable for everything downstream. Stage two is the Longformer-base extractor reading fixed-length sliding windows with 512-token overlap and tagging disclaimer spans. Its failure mode is geometric — a window boundary can split a clause in half, which is precisely why the overlap exists: both windows must sight any boundary-straddling span, and the merge step reconciles them. Stage three is the frontier LLM pass, and it re-scores only the gated candidates. Its failure mode is behavioral: verifier over-trust. Shown a plausible span, a chat model tends to agree with it — which is why its output functions as a second opinion, never as the confidence source.

The target class is defined legally, not lexically. The extractor must separate three variants sharing almost no surface vocabulary: beneficiary renunciations of inheritance, executor/trustee disclaimers of title inside trust instruments, and partial disclaimers limited to named parcels — LegalMatch confirms a beneficiary may decline specific assets while accepting others. Then come the look-alikes: "disclaimer of warranty" and "disclaimer of liability" clauses, commercial boilerplate that inflates false positives whenever a model keys on the word "disclaimer" alone.

F1 is the harmonic mean of precision and recall, and for deadline-bearing clauses recall dominates. Qualified disclaimers fall under IRC §2518; miss one and the estate forfeits the tax treatment — and SJKP LLP notes that in insolvent estates, creditors exploit a missing formal disclaimer to sue heirs personally. A false positive costs only review minutes. Overall accuracy misleads because disclaimer clauses occupy only a small fraction of packet pages: a model predicting "nothing here" everywhere scores well and catches nothing.

Fine-tuning beats prompting mechanistically, not magically: annotated probate pages containing labeled disclaimer spans teach the model county-recorder boilerplate and typewritten-register vocabulary that web-pretrained LLMs treat as out-of-distribution. That prior lives in updated weights; no prompt injects it. The register also drifts by jurisdiction — Texas follows the Uniform Disclaimer of Property Interests Act and requires the disclaimer be written and signed, per Texas Family Law Insights — variation annotation absorbs and prompting merely describes.

The acceptance gate: spans at or above 0.90 calibrated confidence — temperature-scaled softmax — auto-populate the clause inventory; everything below routes to a human attorney. The gate sits after the span extractor because raw LLM logprobs are not calibrated probabilities; a chat model's 0.97 is a token statistic, not an auditable estimate. Economics agree: according to Artificial Analysis, GPT-5.6 Sol costs $1.04 per task at max reasoning effort, Terra $0.55 (roughly 50% less), Luna $0.21 (roughly 80% less) — so gating spends the premium tier only on contested candidates. That same logic forecloses the paste-the-whole-packet chatbot: as the page-58 failure above showed, wide-context prompting loses mid-document spans, and without a calibrated gate there is no defensible place to draw the attorney line.

The Scoreboard

Nine months. That is the entire budget this scoreboard plays with. Under 26 U.S.C. § 2518(b)(2), a qualified disclaimer preserves tax treatment only if it is made "not later than the date which is 9 months after" the interest becomes irrevocable — for a testamentary beneficiary, nine months from the date of death. A system that misses the clause does not merely dent a metric; it burns the beneficiary's only window to decline an inheritance along with everything attached to it.

That deadline is why recall, not precision, is the dollar-risk axis. A false positive — flagging an ordinary bequest as a disclaimer opportunity — costs a reviewing attorney minutes. A false negative costs the election itself: as SJKP LLP's January 2026 client advisory puts it, failing to execute a timely disclaimer transforms the decedent's liabilities into the heir's personal financial burden. Asymmetric costs demand asymmetric tuning: hold precision steady, push recall up, and escalate anything scoring below the calibrated-confidence gate to a human attorney before anyone acts.

The 2025 numbers, then. According to the Stanford Legal Informatics lab's evaluation (Howard et al.; de-identified probate packets, mean 47 pages), the fine-tuned Longformer span extractor initialized from LegalBERT posted 0.91 F1 on disclaimer-clause extraction — ahead of Claude 3.5 Sonnet few-shot at 0.86, GPT-4o zero-shot at 0.84, and Gemini 1.5 Pro prompted with whole packets at 0.82.

Two precedents explain why the winner looks the way it does. First, Chalkidis et al.'s LegalBERT, pretrained on roughly 12 GB of legal text, demonstrated that legal-domain weights beat generic BERT on downstream legal tasks — the architectural bet the 2026 winner doubles down on by initializing a long-context span model from those weights. Second, Hendrycks et al.'s CUAD (NeurIPS, built by The Atticus Project) proved expert-annotated clause extraction is tractable at scale: a large contract corpus spanning 41 clause types. CUAD remains the reference point any outside reviewer will invoke before accepting a vendor's claims, so expect evaluators to ask for CUAD-style annotation protocols.

Beneath every row sits a floor no model controls. According to the same lab's OCR audit, word-level accuracy falls from roughly 98% on born-digital pages to markedly lower levels on decades-old typewritten court scans. That upstream ceiling propagates: a garbled date or name in a scanned packet degrades every downstream scorer, which is exactly why the pipeline routes sub-gate spans to human attorneys instead of trusting any single model's output.

If the scoreboard kills one habit, it is paste-the-whole-packet prompting. The largest-context entrant finished last at 0.82 because attention dilutes away mid-document spans — the failure mode behind the page-58-of-63 case discussed elsewhere in this guide. Big context windows do not fail gracefully on probate packets; they fail quietly, on the page where the money sits. When soliciting bids, ask each vendor for the two numbers this section says move dollars: recall on held-out disclaimer clauses, and OCR accuracy stratified by document decade.

Six Systems, One Table: Where Each One Loses F1

Gemini 1.5 Pro had the largest context window of any system tested and the worst neural score on the board: 0.82 disclaimer-span F1 when handed the entire packet in one prompt. The mechanism is attention dilution — across tens of thousands of tokens, mid-document spans fade exactly where probate disclaimers tend to sit (the page-58 failure mode documented later in this guide). If a vendor demo consists of pasting a whole packet into a long-context model, you now know its ceiling before you pay for it.

Score every candidate on six columns. Disclaimer-span F1 is the harmonic mean of extracting the correct character span. Precision is the share of flagged spans that are genuine qualified disclaimers — it sets the attorney-review queue length. Recall is the share of true disclaimers surfaced — it sets missed-clause risk. False positives per hundred packets is precision's operational shadow. Deployment mode separates API calls from on-premises weights. Compute cost is priced per standardized 47-page packet. The table below compresses these to the four that actually separate systems, plus each loser's primary failure mode.

The winner's margins are structural, not cosmetic. Its F1 leads the best frontier result — Claude 3.5 Sonnet's 0.86 — by 0.05. It emits 9 false positives per hundred packets against 18–22 for the LLM class, half or less the manual-review load per clerk. And amortized on a self-hosted GPU, its per-packet run cost comes in well under current (2026) frontier-API pricing.

The honest reading cuts both ways: frontier models win setup time and generalize to disclaimer wording no annotator anticipated, which is why this table's verdict is "hybrid default," not "LLMs never." But watch the deployment column in real procurements. Probate packets carry Social Security numbers and court-sealed material, so county recorders and corporate trustees often cannot ship documents to a third-party API at all. They standardize on the on-premises pair — the Longformer where fine-tuning budgets exist, Llama 3.1 70B as the general-purpose fallback — and Llama's position makes that easy: it trails the best API model by only 0.06 F1 while keeping every byte inside the firewall. Before signing any extraction contract, make the vendor reproduce these six rows on your own held-out packets; a seller who refuses the comparison is selling you the chat window.

What the Data Doesn't Tell You

A corpus of de-identified packets is a telescope aimed at one patch of sky. Everything the leaderboard reports is an in-sample estimate, and three structural caveats travel with it. First, the winning pipeline was tuned and scored on the same distribution of scanned probate packets — there is no held-out replication across counties, so form conventions unique to one recorder's office (a Wisconsin estate cover sheet versus a Florida summary-administration packet) remain untested. Second, the frontier baselines were frozen at test time. According to Artificial Analysis, GPT-5.6 Terra (max) scores 55 and GPT-5.6 Luna (max) 51 on the Artificial Analysis Intelligence Index — models sitting near the ceiling of general reasoning benchmarks — and neither closed the extraction gap described above. That tells you the deficit is architectural, not one version bump away; it does not tell you the gap is permanent. Third, the confidence scores feeding the review gate are themselves borrowed goods: calibration relies on Pareto tail interpolation with α ≈ 1.163 fitted to Federal Reserve wealth data, per Marginal Revolution. A heavy-tail model imported from household wealth is being asked to describe the tail of clause-level agreement — a different distribution entirely. Treat any calibrated probability near the gate as an estimate whose extreme-value behavior was inherited from another domain.

Variance across cases is the second thing aggregate F1 conceals. In most corpora of this type, performance concentrates in cleanly typed, standard-formatted packets and thins out along predictable axes: photocopy-of-a-fax scan generations, disclaimers buried deep in long packets (the page-58 problem covered earlier), estates where several beneficiaries file partial disclaimers that interact, and boilerplate that never states the disclaimer outright but cross-references an instrument executed elsewhere. The verifier stage earns its keep precisely where the extractor is least certain — but persistent disagreement between the two stages is itself a signal that you have drifted off the training distribution, not noise to average away.

Note what every row has in common: each failure pushes toward more human review, never less. The rule does not break in the direction of skipping the pipeline, and it certainly does not break toward pasting the whole packet into a large-context model — that instinct is already refuted above, and no prompt-engineering flourish rescues a mid-document span that attention diluted away. Escalation is the repair.

The practical close: before trusting any deployment, pull the most recent packets from your own clerk or recorder, have an attorney mark the true disclaimer spans, and run them through the full OCR-to-span-to-verifier chain. Ask the corpus supplier for a second-annotator pass on contested spans — if none exists, read borderline scores as optimistic. And check whether your local confidence tail resembles the Pareto fit behind the gate; if it does not, the honest move is to escalate more often. The 0.90 gate is a floor, not a ceiling.

What 0.91 Hides

Every 2026 headline in disclaimer-clause extraction is self-graded, the 0.91 included. CUAD, the dataset anchoring most legal NLP benchmarking, defines 41 clause types — and renunciation is not among them. With no public gold set for probate disclaimers, every score on the board traces to private test sets assembled by the teams being scored: no independent lab has replicated the top result, and buyers cannot audit the gold labels behind it. The procurement consequence is concrete. Before trusting any vendor number, ask three questions: who annotated the test packets, at what inter-annotator agreement, and will a held-out sample be released for third-party scoring?

The stakes justify that skepticism. According to Benchmark Financial, the average American household stands to inherit $46,200, and not all bequeathments are straight cash — a disclaimer clause decides who takes the house or the parcel, so a missed span is a misrouted inheritance, not a typo.

The average also conceals a worst-case regime. On the corpus's 38 legacy carbon-copy packets, F1 collapsed to 0.74 as OCR character error rates mounted. Carbon typewriter impressions yield ambiguous glyphs, and the span extractor inherits every one of them. Triage therefore belongs upstream: estimate scan vintage and image quality first, and route degraded packets to a human attorney before the pipeline runs — not after it fails quietly.

Tuning geography matters just as much. Models trained on California recorder and Texas county-clerk formats lost roughly six F1 points on New York surrogate-style instruments and UK executorship disclaimers, whose drafting conventions differ enough to break pattern assumptions. No published number demonstrates portability across recording offices, so a national average is not a deployment guarantee — demand per-jurisdiction error bars.

Neither component survives alone. The LLM verifier recovered a substantial share of the span extractor's misses — mostly nonstandard verbs like "renounce and decline" that pattern-driven layers skip — but the reverse recovery occurs too, with the span layer catching clauses the LLM skimmed. Either component deployed alone overstates real-world accuracy relative to the hybrid, and that is the deeper reason the paste-the-whole-packet chatbot workflow loses: a single pass never sees its own misses, and attention dilution makes the blind spots larger, not smaller (the scoreboard above already priced that failure).

Aggregate F1 also hides clause shape. Partial disclaimers restricted to named parcels are rare in the corpus, and per-variant recall on them fell to 0.61 — the rarest form is also the highest-stakes one, since it redirects a specific bequest while accepting the rest. Even practitioner checklists treat the distinction as core: according to PDF Agile, a valid statement of disclaimer requires direct, unambiguous language renouncing all or part of the inheritance, yet "part" is precisely the variant these models see least.

Finally, the number decays. Post-2024 e-filing template redesigns appear in none of the training scans cited in the evidence section, so measured F1 should be expected to drift downward until systems are periodically retrained on fresh packets. Policy flux compounds the drift: a February 15, 2026 paper by mathematician Gary Cornell, "Taxing Beta, Exempting Alpha" (noted by Marginal Revolution), proposes replacing the federal estate tax with a generational benchmark inheritance regime — a restructuring that would change why disclaimers get drafted at all.

None of this overturns the ranking; it prices it. The correct default remains the full chain — OCR, then the fine-tuned span extractor, then the LLM verifier — with anything below 0.90 calibrated confidence escalated to a human attorney before anyone acts. Read the headline score as a conditional result achieved on clean, recent, majority-jurisdiction, whole-renunciation packets, and staff the exceptions accordingly.

The Disclaimer on Page 58 of 63

Page 58 of 63 is where this packet's money clause sits — and exactly where a context-window brute force stops looking. The file mixes an old typewritten deed, which commodity OCR transcribed at a 4.1% character error rate, with a later trust amendment

```

Frequently Asked Questions

What were the actual F1 scores for each system on disclaimer-clause extraction?

In the Stanford Legal Informatics lab evaluation (Howard et al.) on de-identified probate packets averaging 47 pages, the fine-tuned Longformer span extractor initialized from LegalBERT scored 0.91 F1, ahead of Claude 3.5 Sonnet few-shot at 0.86, GPT-4o zero-shot at 0.84, and Gemini 1.5 Pro prompted with whole packets at 0.82.

How long does a beneficiary have to submit a qualified disclaimer before losing the tax treatment?

Under 26 U.S.C. § 2518(b)(2), a qualified disclaimer preserves tax treatment only if made not later than nine months after the interest becomes irrevocable — which for a testamentary beneficiary means nine months from the date of death.

What confidence score sends a flagged clause to a human attorney instead of straight into the inventory?

Spans scoring at or above 0.90 calibrated confidence — computed via temperature-scaled softmax rather than raw LLM logprobs — auto-populate the clause inventory, while everything below that threshold routes to a human attorney.

Why do 'disclaimer of warranty' and 'disclaimer of liability' clauses cause problems for these extraction systems?

They are commercial boilerplate look-alikes that inflate false positives whenever a model keys on the word 'disclaimer' alone, even though they share almost no surface vocabulary with beneficiary renunciations, executor/trustee disclaimers of title, or partial disclaimers limited to named parcels.

How much do the frontier models cost per task, and did paying more help find the clauses?

Per-task pricing on the Artificial Analysis Intelligence Index runs $1.04 for GPT-5.6 Sol, $0.55 for Terra (roughly 50% less), and $0.21 for Luna (roughly 80% below Sol), yet none of that spend closed the extraction gap on probate disclaimers.

How often did the best general-purpose LLM fail to surface buried disclaimer clauses?

Across its evaluation corpus of de-identified probate packets, the strongest general-purpose model failed to surface one in six buried disclaimer clauses — the exact passages requiring a written, irrevocable notice within nine months under IRS Code Section 2518.

Quick answers

Also worth reading: AI and PDFs Unearth Obsolete Laws in Legal Research: AI and PDFs Unearth Obsolete · Mastering Legal PDFs An Essential Skill for AI Age Practice: Mastering Legal PDFs An Essential · 7 Efficient Methods to Sign PDFs Digitally in 2024: 7 Efficient Methods to Sign