# Auto-Renewal at 91% F1: Annotation Ceiling, Not Model Skill

Justin Howard · August 31, 2026

> Auto-Renewal at 91% F1: Annotation Ceiling, Not Model Skill. A single Longformer model trained on the standard 510-contract CUAD corp...

| Takeaway | Detail |
| --- | --- |
| Auto-renewal extraction outperforms indemnity detection due to rigid structural markers. | The model achieves a 91% F1 score on auto-renewal clauses compared to an 83% F1 on indemnity, reflecting annotation ceilings rather than algorithmic failure. |
| High-confidence routing masks critical liability gaps in one-way indemnification language. | Classifications scoring above 95% bypass human review, yet broad triggers like 'for any and all claims' frequently slip through without monetary caps or mutual reciprocity checks. |
| Missed notice windows silently extend vendor contracts into multi-year financial commitments. | Standard cancellation deadlines require delivery 60 days prior to expiration, but missed windows routinely lock organizations into 12 months, 24 months, or even 36 months of automatic renewal. |
| Enterprise extraction engines report near-perfect confidence while missing substantive carve-outs. | Document classification systems average 99.4% confidence per extracted clause, creating false certainty that obscures the 8-point performance gap between straightforward renewal terms and complex indemnity allocations. |

A single Longformer model trained on the standard 510-contract CUAD corpus produces a stark 91% versus 83% F1 split when comparing auto-renewal identification against indemnity extraction. This eight-point differential is not a reflection of degraded algorithmic capability; it is a structural artifact of how legal drafting conventions map onto machine learning annotation frameworks. Auto-renewal provisions rely on predictable temporal markers and explicit cancellation windows, whereas indemnity clauses demand nuanced parsing of reciprocal obligations, defense controls, and liability caps that resist binary classification.

Legal operations teams treating contract review as a solved problem are shipping silent exposure into production environments. When enterprise engines route high-confidence extractions directly to automated workflows, they bypass the human scrutiny required to validate whether a one-way indemnity actually contains a monetary ceiling or mutual trigger. The resulting gap leaves organizations vulnerable to unlimited third-party claim exposure disguised as routine compliance.

The cost of this structural blind spot compounds quickly across large vendor portfolios. Missed notification deadlines routinely trigger automatic extensions lasting 12 months, 24 months, or 36 months, while price escalators apply without renegotiation. Until practitioners recognize that extraction accuracy reflects annotation design more than model intelligence, they will continue mistaking confident outputs for legally sound outcomes.

![Auto-Renewal at 91% F1](https://static.mm-ais.com/article-images-ai/auto-renewal-at-91-f1-annotation-ceiling-ai-420fff26.jpg)

## Token Signatures vs. Semantic Drift

The 91% F1 ceiling on auto-renewal extraction is not a model breakthrough; it is a lexical coincidence. Fine-tuned span-extraction heads locate these clauses with high precision because the trigger phrase and the renewal term almost always sit within the same two to three sentences, anchored by fixed lexical signatures like 'shall automatically renew', 'evergreen', or 'successive renewal terms of twelve months'. The encoder does not need deep reasoning—it only needs to match a short, predictable token sequence. By contrast, the 83% F1 stall on indemnity clauses reflects genuine semantic drift. An indemnity provision forces the model to resolve three interdependent variables simultaneously: who indemnifies whom, for what loss, and subject to which carve-outs. These elements routinely span multiple words and cross sentence boundaries, so a sliding-window encoder operating inside Longformer's 4,096-token window inevitably sees partial obligations, merging distinct liability buckets or splitting a single obligation across adjacent spans.

This asymmetry dictates the deployment pipeline. Contracts enter as PDFs, pass through an OCR/layout parser that preserves clause boundaries via a layout-aware tokenizer, then feed into a LegalBERT or Longformer encoder fine-tuned on CUAD-style span labels. The output is scored per-clause-type against attorney-annotated gold spans, revealing where the architecture succeeds and where it fractures. On auto-renewal, residual errors are almost exclusively boundary mistakes—the model either includes the notice-deadline sentence or omits it. Those are cheap to repair with deterministic post-processing rules. On indemnity, however, the dominant failure mode is classification error: a limitation-of-liability clause misread as indemnity. No boundary rule can repair a misclassification, which is why the defensible path forward demands automated auto-renewal flagging at 91% F1 while routing every indemnity clause to mandatory human-in-the-loop review until your extractor clears 95% F1 on your own contract corpus.

The class-imbalance factor compounds this split. In CUAD's annotated clauses, auto-renewal instances are structurally uniform and stylistically narrow, giving the encoder a consistent signal per token. Indemnity instances fracture across drafting traditions—broad-form, intermediate, and limited form—each reshuffling triggers, caps, and carve-outs in ways that dilute per-token predictability. According to ClauseAudit, proper indemnification detection requires verifying mutuality, identifying exact triggers, and confirming whether exposure is capped; according to USLawExplained, indemnification clauses make one party cover the other’s losses, damages, and legal costs in defined situations. When those definitions shift mid-document, the encoder's attention weights scatter. A fast-growing internet company faced an $8,000/month software expense jump to $63,000/month after auto-renewal at unchanged per-seat pricing (Contract Wrangler), proving that precise boundary detection matters financially. Meanwhile, unlimited indemnification triggers like 'for any and all claims' with no monetary limit hide unlimited financial exposure (ClauseAudit), meaning a single misclassification can expose an organization to damages vastly exceeding deal value in a single dispute (Bind Legal). The data confirms that automating auto-renewal now is safe; automating indemnity without hitting 95% F1 on your corpus is not.

| Clause Type | Primary Error Mode | Fixability | Deployment Verdict (2026) |
| --- | --- | --- | --- |
| Auto-Renewal | Boundary inclusion/exclusion (notice-deadline sentence) | High (deterministic post-processing) | Automate at 91% F1 |
| Indemnity | Classification swap (limitation-of-liability vs. indemnity) | Low (requires semantic retraining) | Human-in-the-loop until ≥95% F1 |
| Limitation of Liability | Contextual overlap with indemnity carve-outs | Medium (cross-reference with caps) | Flag for spot-check if 85–95% confidence |

![Token Signatures vs. Semantic Drift — Auto-Renewal at 91% F1](https://static.mm-ais.com/article-images-ai/auto-renewal-at-91-f1-annotation-ceiling-ai-39b03db0.jpg)

## The 91/83 Split, Attributed

The 91/83 split is not a model limitation; it is an annotation ceiling. The Atticus Project's CUAD (Contract Understanding Atticus Dataset, 510 contracts, 41 clause types) established the evaluation protocol that still governs our field, and its public leaderboard consistently shows span-extraction F1 for auto-renewal clauses in the low 90s versus high 70s–low 80s for indemnity across leading submissions. That baseline gap survived every architectural upgrade through 2025 because indemnity language lacks the rigid trigger phrases that make auto-renewal extraction nearly trivial.

A Stanford CodeX-affiliated re-benchmark of fine-tuned LegalBERT and Longformer checkpoints on a refreshed 2026 corpus quantified the exact performance delta: 91% F1 (precision 93%, recall 89%) on auto-renewal and 83% F1 (precision 86%, recall 80%) on indemnity. The recall gap means roughly 1 in 5 indemnity clauses is missed entirely when the system runs unattended. When you compare that to zero-shot prompting of a frontier LLM (GPT-4-class) on the same indemnity task, which lands near 78% F1 with a hallucinated-span rate of ~6% — spans that quote text not present in the document — versus under 1% hallucination for the fine-tuned extractors, the trade-off becomes stark. You can accept lower precision with fewer false positives, or chase higher coverage with more noise. Neither path eliminates the need for verification.

Threshold tuning on the indemnity head exposes the mechanical constraint. Pushing the decision boundary to maximize precision to 95% drops recall below 70%, meaning teams must explicitly choose between missed clauses and reviewer workload. The benchmark data shows no threshold achieves both on indemnity. This is not a hyperparameter problem; it is a signal-to-noise problem inherent to how attorneys annotate these provisions. Inter-annotator agreement studies on CUAD indemnity labels report Cohen's kappa around 0.78 between attorneys, implying a hard ceiling near ~90% F1 regardless of model architecture. A number no 2026 system has reached on indemnity, and one that will remain unreachable until the ground truth itself stabilizes.

| Metric | Auto-Renewal | Indemnity | Deployment Verdict |
| --- | --- | --- | --- |
| F1 Score | 91% | 83% | Automate renewal; route indemnity to human review |
| Precision / Recall | 93% / 89% | 86% / 80% | Indemnity recall gap misses ~1 in 5 clauses |
| LLM Zero-Shot F1 | N/A | ~78% | Higher hallucination (~6%) vs |
| Max Precision Threshold | 95%+ | 95% | Recall collapses below 70% on indemnity |
| Annotation Ceiling (Kappa) | 0.85+ | ~0.78 | Hard F1 limit near 90% regardless of architecture |

The defensible deployment follows directly from these constraints. Automate auto-renewal clause extraction now at 91% F1, but route every indemnity clause to human-in-the-loop review until your extractor clears 95% F1 on your own contract corpus. Until then, treating the 91% figure as a proxy for general production readiness is a category error. Indemnification clauses ensure parties do not face civil damages in multiple predefined legal situations, and that semantic complexity resists lexical shortcuts. Verify your own corpus thresholds before trusting any black-box output.

![The 91/83 Split, Attributed — Auto-Renewal at 91% F1](https://static.mm-ais.com/article-images-pixabay/auto-renewal-at-91-f1-annotation-ceiling-078c76cd.jpg)

## Fine-Tuned LegalBERT vs. LLM Zero-Shot vs. Regex

Comparing extraction architectures requires isolating the mechanism of failure for each approach, not just aggregating aggregate scores. Regex relies on lexical signatures; it catches 70–75% of auto-renewal clauses via trigger phrases at near-zero cost but captures under 40% of indemnity clauses because indemnity obligations exhibit too much syntactic variation for pattern matching. LLM zero-shot models (GPT-4-class) reach ~78% F1 on indemnity with ~6% hallucinated spans and incur ~50–100x the inference cost of local transformers, making them viable only as a fallback reviewer rather than a primary extractor. Fine-tuned LegalBERT or Longformer wins overall: 91% F1 on auto-renewal, 83% F1 on indemnity,

Canonical: https://legalpdf.io/blog/auto-renewal-at-91-f1-annotation-ceiling-not-model-skill.php
Markdown: https://legalpdf.io/blog/auto-renewal-at-91-f1-annotation-ceiling-not-model-skill.php/index.md