```html
| Takeaway | Detail |
|---|---|
| Token price is a cost signal, not a competence signal. | Gemini 3.7 Flash's introductory pricing — $0.75 per 1M input tokens and $3.75 for output through December 31, 2026, then doubling — is itself a buried auto-exec calendar trigger, the exact clause type that leaderboard-topping extractors are never scored on computing. |
| Cheap build tooling has flattened the leaderboard. | AI coding agents have collapsed the technical-expertise barrier — the core claim of the $10/month business analysis — making a clause classifier trivial to assemble, so F1 rank increasingly measures packaging rather than deadline reasoning. |
| Deadline errors land on the clauses that decide retention. | Auto-renewal, pricing, benchmarking, termination, and SLA clauses measurably drive renewal success and retention, so even a 5% error rate on computed notice windows concentrates misses on the highest-value documents — the ones F1 never isolates. |
| Independent harnesses, not vendor boards, should gate procurement. | Terminal-Bench 2.0 — an independent benchmark from the Laude Institute, published at ICLR 2026 with 89 containerized terminal tasks — shows what contract NLP lacks: a third-party test of deadline computation instead of a $0.75-versus-$3.75 price sheet standing in for quality. |
Gemini 3.7 Flash publishes introductory pricing of $0.75 per million input tokens and $3.75 for output — terms that hold only through December 31, 2026, after which they double. Embedded in its own pricing page is the precise artifact that matters here: a hard calendar deadline wired to an automatic trigger, structurally identical to the auto-renewal and termination windows buried in commercial contracts.
Leaderboards reward the wrong skill. Published clause-extraction models are scored on spotting termination, indemnity, and confidentiality language — classification work measured in F1 — yet not one is scored on computing the correct notice deadline when an auto-exec trigger fires. The characteristic miss is quiet and expensive: a term-expiry date off by exactly one day, an error invisible to every metric on the board.
Rank is therefore the wrong buying signal. Independent harnesses already exist in adjacent fields: Terminal-Bench 2.0, built by the Laude Institute and published at ICLR 2026, executes 89 tasks inside containerized terminal environments. Contract NLP has no equivalent test of deadline arithmetic — so buyers keep comparing $0.75-versus-$3.75 token bills, pricing inputs while the calendar risk goes unmeasured.
Two Models, One Contract
A short fixed-size encoder window cannot read a commercial contract many times its length. That arithmetic is why the benchmark's reference design is two models and one deterministic engine rather than any single end-to-end system. Stage 1 is a fine-tuned LegalBERT or RoBERTa span extractor run over short sliding windows, proposing candidate spans in three auto-exec clause families: payment obligations, term/renewal, and termination. Stage 2 is a long-context LLM — GPT-4o-class or Claude-class — that receives each candidate alongside the full document and verifies it against sections the encoder never saw together. Nothing is flagged until Stage 2 confirms. In 2026, with long-context windows a commodity feature, the split persists because Stage 2's job is joint reasoning across the whole document — a verification problem, not a context-length problem.
The construct underneath is the auto-exec clause: a provision whose monetary or exit consequence triggers without further assent. The canonical illustration is a net-60 payment term that begins accruing a 1.5%-per-month late fee on day 61 — no invoice, no election, no counter-signature; the fee simply starts. The lawyer-annotated gold set sorts every trigger into four families:
| Clause family | Trigger condition | Auto-exec consequence |
|---|---|---|
| Payment accrual | Net-60 term lapses at day 61 | 1.5%-per-month late fee starts, no invoice required |
| Evergreen renewal with index escalator | Initial Term ends with no notice sent | Renewal locks in at the index-adjusted rate |
| Insolvency termination | Counterparty insolvency event occurs | Exit right becomes exercisable on the event itself |
| Notice-gated convenience termination | Notice window closes unexercised | Contract rolls into the next term |
Single-pass extraction fails mechanically, not statistically. Commercial contracts in the standard research corpora — CUAD and its peers — average roughly fifteen to twenty times the encoder window. Chunking is the standard workaround, and it is exactly what severs the cross-references: "the Initial Term defined elsewhere in this Agreement" is a pointer from a Notices clause to a definition three sections away, and that pointer is where the renewal trigger lives. A chunk-local extractor sees two unrelated fragments and never reasons over them jointly. That mechanism, not model quality, is what the guide's F1-versus-recall gap is made of.
Stage 3 is deliberately unglamorous. Trigger phrases such as "ninety (90) days prior to the end of the then-current Term" are converted to ISO dates by rule-based tooling — the dateparser library plus a custom grammar for ordinals and rolling periods. The rationale is mechanical: autoregressive generation is unreliable for calendar arithmetic. A verifier can quote the clause verbatim and still compute the wrong deadline, because next-token prediction contains no calendar. Deterministic code does; the same phrase always resolves to the same ISO date.
Scoring follows the architecture. Alongside span-level precision, recall, and F1, the benchmark adds a trigger-correctness metric: did the system output the right calendar deadline, judged against the lawyer-annotated gold set of auto-exec triggers? No existing public leaderboard measures this dimension — they score token spans, never dates. That absence is why a leaderboard crown is evidence about clause typing, not proof that a model can read your contract: a system can post a strong span F1 and still hand you the wrong day 61.
| Stage | Component | Context it handles | Failure mode it eliminates |
|---|---|---|---|
| 1 — Propose | Fine-tuned LegalBERT/RoBERTa span extractor | Short sliding windows | Unsurfaced candidates in payment, term/renewal, termination spans |
| 2 — Verify | GPT-4o-class or Claude-class long-context LLM | Full document per candidate | Cross-section triggers severed by chunking |
| 3 — Compute | dateparser plus custom ordinal/rolling-period grammar | Extracted trigger phrases | LLM calendar-arithmetic drift on deadlines |
The Receipts
Stack CUAD, LEDGAR, ContractNLI, MAUD, and LexGLUE side by side and they stop reading like leaderboards and start reading like a build specification: each dataset licenses exactly one stage of the hybrid pipeline and refuses to support anything more.
Start with CUAD. According to the Atticus Project's 2021 release, it pairs commercially negotiated contracts with lawyer-supervised annotations across 41 clause types. Its per-type results show a wide spread: encoder models score well on boilerplate such as Governing Law but materially lower on renewal- and term-adjacent types — precisely the clauses auto-execution logic depends on. That spread is the empirical kill shot for the belief that topping a clause-extraction leaderboard means a model can read your contracts. The aggregate metric and the auto-exec-relevant slice are simply different measurements, and vendors quote whichever flatters them.
LEDGAR explains the data vacuum on the other side. According to Tuggener et al. at the University of St. Gallen, it supplies a large body of labeled provisions drawn from SEC 10-K filings — strong for provision classification, containing no trigger-date or execution-timing labels at all. That absence is why payment-window extraction has almost no public gold data, and why any vendor quoting a payment-window accuracy figure is scoring against a private set you cannot audit.
ContractNLI supplies the verifier's blueprint. According to Koreeda and Manning at Microsoft Research, evaluating NDAs against 17 entailment hypotheses showed that framing clause questions as verify/refute/unknown checks improves consistency over free-form extraction — the direct design precedent for running a long-context LLM as a yes/no/uncertain checker rather than a prose generator.
MAUD answers the skeptic who calls that framing artificial. According to Northeastern University's collaboration with Wilson Sonsini, its merger agreements covering 92 review points reached expert-parity agreement once the domain was narrow and the question set fixed. Constrained verification beats open-ended reading for high-stakes clauses.
LexGLUE justifies splitting the labor. According to Chalkidis et al., across its 8 legal NLP tasks, fine-tuned encoder models consistently outperform generative zero-shot models on classification metrics, while LLMs win on explanation quality. Extraction goes to the encoder; judgment goes to the LLM.
The economics close the case. A manual first-pass review of a commercial contract runs 45 to 90 minutes of associate time at standard professional hourly rates — versus a trivially small API cost for a 20-page contract at mid-tier 2026 pricing. That gap of about four orders of magnitude is what funds the human-verification layer: automation cheap enough to run on every document, humans affordable enough to confirm every computed deadline.
| Benchmark | Source | Scale | Licenses | Does not provide |
|---|---|---|---|---|
| CUAD | Atticus Project, 2021 | Commercial contracts, 41 clause types | Lawyer-labeled span candidates | Cross-section trigger reasoning |
| LEDGAR | Tuggener et al., Univ. of St. Gallen | Labeled provisions from SEC 10-Ks | Provision classification pretraining | Trigger-date or timing labels |
| ContractNLI | Koreeda & Manning, Microsoft Research | NDAs, 17 hypotheses | Verify/refute/unknown verifier design | Payment or termination coverage |
| MAUD | Northeastern + Wilson Sonsini | Merger agreements, 92 review points | Constrained Q&A at expert parity | Scale beyond narrow M&A scope |
| LexGLUE | Chalkidis et al. | 8 legal NLP tasks | Encoder-vs-LLM division of labor | Auto-exec-specific evaluation |
Before accepting any vendor's auto-exec accuracy claim, map it against this table. If the claim rests on none of these five receipts — or on LEDGAR-style classification data repurposed as timing evidence — treat the number as marketing until a human confirms it against your own contracts.
Pick the Pipeline, Not the Leaderboard
Leaderboards rank models; portfolios rank pipelines. Before shortlisting anything, fix the evaluation frame: five axes — termination-trigger recall, payment-term precision, date-math correctness, cost per contract, and data-privacy posture — scored across four architectures: (a) the fine-tuned span extractor alone, (b) the long-context LLM alone, (c) the hybrid extractor-verifier-date-engine, and (d) the commercial platform of the Kira, Luminance, or Spellbook class. No public benchmark runs this grid, because benchmarks never see axis three or axis five.
Run the grid yourself and one column survives: architecture (c). It is the only design that clears a 0.95 recall floor on termination windows while holding per-document cost negligible. The mechanism matters more than the scoreline — the encoder supplies recall on known clause patterns cheaply, the long-context verifier recovers the cross-reference triggers that chunking severed when a Term definition points into a distant Notices section, and the deterministic date engine converts prose deadlines into calendar dates with zero arithmetic drift. Neither component reaches both numbers alone.
Weight recall on termination triggers at three times precision in the rubric. The asymmetry is structural: a missed non-renewal window is irreversible once it passes, because the renewal binds automatically and no post-hoc tooling unwinds an executed renewal, whereas a false positive costs only minutes of attorney review. Any rubric that weights these equally crowns the wrong architecture — the 3× multiplier is precisely what pushes (c) past (b) despite (b)'s cleaner precision profile.
One axis can overturn everything: privacy. If the corpus contains privileged M&A or deal documents, on-premises or open-weight models — Llama- or Mistral-class — become mandatory regardless of leaderboard rank. A cloud-hosted hybrid that wins on raw scores loses the portfolio the moment privilege attaches, so spec the self-hosted-verifier fallback during procurement, not after outside counsel objects.
Now the disqualification rule: reject any model or vendor whose evaluation reports only aggregate F1. A headline 0.90 with no per-clause-type breakdown hides exactly the termination-recall gap documented above — strong overall scores coexisting with weak evergreen-window recall. Demand the per-clause-type breakdown or walk away. Telecom regulators reached the same conclusion long ago: the mobile-termination benchmarking methodology applied in Namibia benchmarks rates, costs, and best practice separately, precisely because a single blended number conceals the distortion that matters.
Finally, bound the human layer by design. Route only flagged auto-exec candidates to attorneys — typically under 5% of pages — so review burden stays proportional to risk instead of reverting to full manual reading. A workflow that ships entire agreements to lawyers is not automation; it is expensive proofreading with extra steps.
| Architecture | Termination-trigger recall | Precision & date math | Cost per contract | Privacy posture |
|---|---|---|---|---|
| (a) Span extractor alone | Strong on known patterns; misses cross-section triggers | Clean spans; no native date engine | Lowest inference spend of the four | Fully on-premises capable |
| (b) Long-context LLM alone | Recovers severed cross-references; drifts at portfolio scale | Good typing; unreliable calendar arithmetic | Highest token spend, scaling with page count | Cloud API unless built on open weights |
| (c) Hybrid extractor-verifier-date engine | Clears the 0.95 floor on termination windows | Deterministic dates; every deadline reproducible | Negligible per document | Flexible — verifier can run open-weight |
| (d) Commercial platform (Kira / Luminance / Spellbook class) | Vendor-reported; rarely broken out by clause type | Built-in deadline tooling varies by product | Seat-priced; per-document rate usually undisclosed | Enterprise cloud; on-prem uncommon |
The action close: hand every shortlisted candidate a shadow set of roughly fifty contracts, including at least ten evergreen renewals whose notice windows sit in a separate Notices section, score the returns on the five axes with the 3× recall multiplier, and eliminate any submission reporting aggregate F1 alone. Vendors optimize for the metric a buyer can verify in thirty seconds; you should optimize for the one they hope you will not ask about. In most procurement cycles, the survivor is (c) — with an open-weight verifier held in reserve for whatever privileged material lands next quarter.
What the Data Doesn't Tell You
Every number in this guide was measured somewhere, and the somewhere matters. The public corpora that anchor clause-extraction evaluation — CUAD's EDGAR filings, MAUD's merger agreements, ContractNLI's NDAs — overrepresent large-company, US-drafted, English-language boilerplate. Mid-market SaaS MSAs, negotiated vendor agreements, and amended-and-restated credit agreements are thinly represented, so the miss-rate ceiling quoted above is a claim about the benchmark distribution, not about your filing cabinet. Two gaps compound the skew. No widely adopted public benchmark, as of this writing, scores end-to-end date computation: the datasets grade clause typing and span boundaries, and the deadline math is inferred rather than measured. And the labels themselves carry noise, because annotators disagree on whether an evergreen renewal with an escalator counts as auto-executing payment at all — a model cannot out-extract an ambiguous label.
The leaderboard-top myth the F1 section dismantled survives in vendor decks for a structural reason: an aggregate score is the only number a benchmark hands out, so it is the only number a deck can show. What the aggregate conceals is variance. Pipeline performance is not a property of the pipeline; it is a property of the pipeline-contract pair, and misses cluster. The evergreen-renewal split trigger is one topology. Amendment layering is a second: when a restated agreement rewrites the Term without rewriting the notice mechanics, the verifier must reconcile two generations of drafting, and that reconciliation is where errors concentrate. Defined-term drift is a third: a contract that defines the notice period one way in the Term clause and another way in the Notices section hands the date engine two answers, and the engine — being deterministic — picks one, confidently and reproducibly wrong. Determinism is an auditability feature, not a correctness guarantee. A portfolio-level miss average can therefore look acceptable while a single contract family, say equipment leases with fiscal-year-relative triggers, accounts for a disproportionate share of the misses. Slice error analysis by family before trusting any aggregate.
When does the rule itself strain? The volume threshold is a cost default, not a risk measure: a small book of credit agreements carries more consequence per missed flag than a far larger pile of NDAs. The rule's own safeguard — every computed deadline provisional until a human confirms — is what absorbs that mismatch, not the threshold. In practice the rule strains at three edges. The trigger lives in an exhibit incorporated by reference that never entered the corpus: no pipeline flags what it never ingested. Sign-off degrades into a rubber stamp: the miss rate does not move, the human does. Governing law is foreign and notice-validity rules diverge from the US-style logic public corpora teach the verifier: that is a counsel question, not a sign-off question. None of these invert the rule; they mark where the hybrid premium is justified — high consequence per miss, exhibits in play, layered drafting — and where the mandatory human step is the only component that works in every case.
Where the rule strains, and what absorbs the strain:
| Edge case | What fails | What keeps the flag honest |
|---|---|---|
| Trigger defined in an exhibit incorporated by reference | Span extractor never ingests the exhibit, so no candidate is generated | Human confirmation must include pulling and reading the exhibit |
| Notice period defined twice, inconsistently | Date engine resolves deterministically — confidently, reproducibly wrong | Provisional status forces a human to arbitrate the definitions |
| Scanned amendment layer with OCR noise | Extractor recall degrades exactly on the newest drafting | Re-key the amendment text before the run |
| Foreign governing law on notice validity | Verifier applies US-style notice logic learned from public corpora | Route to counsel review, not ordinary sign-off |
| Portfolio just under the volume-threshold line | LLM-only mode skips the deterministic date engine | Keep mandatory sign-off; escalate to the hybrid stack if misses surface |
| Sign-off becomes a rubber stamp | Miss rate returns without any model change | Sample-audit a slice of confirmed flags each quarter |
What F1 Hides
Aggregate F1 is a weighted average, and weighted averages are where extraction models bury their worst failures. A clause-typing model can post roughly 0.85 F1 across a benchmark suite while its recall on evergreen-renewal traps collapses toward 0.6 — not because the model reads poorly, but because the operative trigger is never in one place. In the standard evergreen trap, a Term definition states that the agreement renews automatically absent notice, and the Notices provision, many pages later, specifies how and when such notice counts as delivered. A chunked encoder sees two unrelated passages. Long-context LLMs do better and still fail: attention dilutes across document-length distances, and the model answers the question each section poses separately rather than the question the contract poses jointly.
The second thing F1 hides is where the training text came from. Frontier-model pretraining corpora very plausibly include benchmark-corpus-derived contract text and the SEC filings circulating publicly through EDGAR, and zero-shot scores on those same corpora inherit the overlap. No decontamination audit exists for any current-generation model's legal-data slice, so a surprising zero-shot result may be memorization wearing comprehension's clothes. The practical tell: performance that survives paraphrase and fresh drafting but collapses on lightly perturbed originals indicates recall, not reasoning.
Third, the gold labels themselves wobble. Ask trained lawyers whether a volume-contingent escalator auto-executes once the threshold is met, or whether a most-favored-nation election triggers repricing without further action, and they genuinely disagree. Inter-annotator agreement on ambiguous triggers sits well below the near-perfect agreement that benchmark scoring implicitly assumes — so part of every reported error bar is label noise, not model noise. A model can be penalized for faithfully reflecting real legal ambiguity.
Fourth, the measurement base is narrow. Public corpora skew toward English-language US commercial paper, leaving GDPR-era data processing agreements, FAR/DFARS government contract clauses, and non-English agreements essentially unmeasured. Treat any published score as a claim about EDGAR-style paper until demonstrated otherwise on your portfolio's document class.
Fifth, sophisticated counterparties draft against the parsers. The dual-bound notice window — "not less than sixty (60) nor more than ninety (90) days prior" — defeats regex date grammars, which expect a single anchor, and LLM extraction alike, since the model tends to latch onto one bound and silently drop the other. Nobody has measured how often this phrasing occurs in live portfolios, which is exactly why the deterministic date engine stage matters: it forces the ambiguity into the open instead of resolving it quietly.
Sixth, the taxonomies rot faster than the models ship. Post-2023 supply-chain disruption spawned auto-price-adjustment and shortage-termination language absent from the annotation schemes built for earlier corpora. Annotators cannot label clause types their codebook does not contain, so those clauses enter evaluation as background noise — annotation schemes decay faster than model release cycles, and every score inherits that lag.
| Failure mode | Mechanism | Detection signal |
| Split-trigger | Term definition and Notices provision scored separately | Roughly 0.85 aggregate F1 masking roughly 0.6 recall on evergreen traps |
| Contamination | Benchmark-derived text and EDGAR filings in pretraining | Zero-shot edge vanishes on paraphrased or freshly drafted variants |
| Label instability | Lawyers split on escalators and MFN elections | Annotator agreement far below the near-perfect scoring assumption |
| Distribution shift | Corpora limited to US commercial paper | No measured coverage of DPAs, FAR/DFARS, or non-English deals |
| Range-drafting | Dual-bound windows (sixty-to-ninety days) | Regex and LLM each retain one bound, discard the other |
| Taxonomy decay | Codebooks predate shortage-termination language | New clause types scored as unlabeled noise |
Worked Case
Run this contract through a single-pass LLM and it returns the wrong expiry date with complete confidence. The document: an anonymized 42-page SaaS master subscription agreement carrying a 36-month initial term, auto-renewal in 12-month increments with a CPI-linked escalator, a non-renewal notice due at least 90 days before expiry, and immediate termination upon insolvency. A naive single-pass run o
```
Frequently Asked Questions
Until when do Gemini 3.7 Flash's introductory token prices actually last?
The $0.75-per-million-input-token and $3.75-output rates hold only through December 31, 2026, after which they double.
If my vendor contract has a net-60 payment term, what happens automatically when it lapses?
A 1.5%-per-month late fee starts accruing on day 61 with no invoice, no election, and no counter-signature required.
Can't I just chunk a long contract and run my clause classifier on each piece?
Commercial contracts in CUAD and its peer corpora average roughly fifteen to twenty times the encoder window, and chunking severs cross-references like a Notices clause pointing to a definition three sections away, leaving a chunk-local extractor seeing two unrelated fragments.
Does any existing public leaderboard check whether a model outputs the correct calendar deadline?
No existing public leaderboard measures trigger correctness — they score token spans, never dates — which is why a system can post a strong span F1 and still hand you the wrong day 61.
Is there public gold data I can use to evaluate payment-window extraction?
LEDGAR supplies labeled provisions from SEC 10-K filings but contains no trigger-date or execution-timing labels at all, so any vendor quoting a payment-window accuracy figure is scoring against a private set you cannot audit.
For clause classification, should I deploy a fine-tuned encoder or a generative LLM?
Across LexGLUE's 8 legal NLP tasks, fine-tuned encoder models consistently outperform generative zero-shot models on classification metrics, while LLMs win on explanation quality.
Quick answers
| What is Gemini 3.7 Flash's introductory token pricing and when does it change? | It is $0.75 per million input tokens and $3.75 for output through December 31, 2026, after which the prices double. |
| What is Terminal-Bench 2.0 and who built it? | It is an independent benchmark from the Laude Institute, published at ICLR 2026, that executes 89 containerized terminal tasks. |
| What does Stage 1 of the benchmark's reference design do? | A fine-tuned LegalBERT or RoBERTa span extractor runs over short sliding windows to propose candidate spans in three auto-exec clause families: payment obligations, term/renewal, and termination. |
| Why is Stage 3 rule-based rather than an LLM? | Because autoregressive generation is unreliable for calendar arithmetic, so deterministic tooling like dateparser plus a custom grammar always resolves the same phrase to the same ISO date. |
| What happens in the canonical net-60 auto-exec example? | A net-60 payment term begins accruing a 1.5%-per-month late fee on day 61, with no invoice, election, or counter-signature required. |
Also worth reading: AI and PDFs Unearth Obsolete Laws in Legal Research: AI and PDFs Unearth Obsolete · How to ensure your legal PDF form is valid and enforceable: How to ensure your legal · How to manage and sign legal documents securely online: How to manage and sign