# Contract clause extraction: CUAD (Contract Understanding Atticus Dataset) 510: Regex Wins

Justin Howard · October 26, 2026

> CUAD clause extraction benchmark compares Transformer and regex F1 scores, revealing when rule-based span labeling still beats neural models.

It compares Transformer vs. Rule-Based Clause-Extraction F1 to show when Regex Still Wins.

![It compares Transformer vs. Rule-Based Clause-Extraction F1 to — Contract clause extraction](https://static.mm-ais.com/article-images-ai/contract-clause-extraction-cuad-contract-ai-23f42aa3.jpg)

## How It Works

Clause extraction is a span-labeling workflow: a system reads contract text, proposes an interval for each requested clause type, and an evaluator compares those proposals with a human-marked gold set. The mechanism only becomes comparable when prediction and gold use the same span boundaries, clause taxonomy, and document split, so the first check is to confirm that the reported result was computed on the complete, unchanged evaluation set rather than a convenient subset.

Key terms keep that check honest. A gold annotation is the human answer used as reference; a span is the exact text interval being judged; a clause type is the category label assigned to that interval. Precision measures how many predicted spans match gold, recall measures how many gold spans were found, and F1 balances those two counts. A transformer is a neural architecture that processes sequences with attention; a rule-based extractor applies explicit patterns, often regular expressions, to headings, defined terms, or cue phrases.

According to Peter Smulovics's guide to annotated transformers, the foundational transformer architecture was introduced by Vaswani et al. in "Attention is All You Need" (2017). In that architecture, multi-head self-attention lets a token weigh other tokens in the same sequence, while positional encoding adds order information the model does not otherwise retain. Feed-forward neural networks apply fully connected layers at each position, and layer normalization helps stabilize and accelerate training.

An annotated transformer, as the same guide describes, is a transformer model accompanied by detailed explanations and annotations about architecture, layer functions, and underlying mathematics. That matters for verification because the annotation layer can expose how a span was produced, making it easier to inspect mismatches than with an opaque score alone. A regex extractor, by contrast, wins transparency by construction: its match is visible in the pattern, but its reach stops at the pattern's assumptions.

The practical rule is to verify the live, complete option before committing: rerun the same evaluation code, normalization, and threshold on the full contract set, then review false positives and false negatives by clause type. Aggregate F1 can hide a strong result on common clauses and a weak result on rare ones, so the check should include per-type counts and boundary examples.

| Term | Working definition |
| --- | --- |
| Clause span | The exact text interval proposed or marked for a clause type. |
| Gold annotation | The human-marked reference answer used for scoring. |
| Precision | Predicted spans that match gold, relative to all predictions. |
| Recall | Gold spans found, relative to all gold spans. |
| F1 | The balance of precision and recall for a given evaluation. |
| Regular expression | An explicit pattern used to match text in a rule-based extractor. |

![How It Works — Contract clause extraction](https://static.mm-ais.com/article-images-ai/contract-clause-extraction-cuad-contract-ai-43aca2dc.jpg)

## Key Factors to Consider

The top three decision criteria are corpus completeness, clause-definition consistency, and metric comparability. Corpus completeness asks whether the live release contains the same contracts and annotations you plan to use. Clause-definition consistency asks whether both systems target the same clause types with the same span boundaries. Metric comparability asks whether F1 uses the same gold set, split, preprocessing, and matching rule.

| Decision factor | What to verify | Number or check |
| --- | --- | --- |
| Corpus completeness | Live version, license, included contracts | 510 contracts |
| Annotation coverage | Labels by clause type and span rule | 13,000 annotations |
| Metric comparability | Same split, gold set, matching rule | Clause-level F1 |

The numbers that matter are the benchmark's 510 contracts, 13,000 annotations, and 2021 release. Dividing annotations by contracts gives roughly 25.5 annotations per contract, but that average can hide category imbalance. Before comparing models, verify counts by clause type and confirm that the live files still match those totals.

F1 should be treated as a conditional metric, not a universal score. A transformer and a rule-based extractor can only be compared when they receive the same input text, propose spans under the same boundary rule, and are scored against the same gold labels. Regex may remain competitive for clauses with stable wording, while transformer models may help where context and paraphrase vary; the evidence should be clause-level, not a single headline.

Verify the live, complete option before committing. Check dataset access, version, license, schema, and any post-2021 corrections or removals. Compare totals and terms line by line, including train and test splits and evaluation scripts, because a changed split or label rule can invalidate an earlier F1 comparison.

A practical commit rule is to freeze four items before selecting a method: corpus count, clause taxonomy, split, and metric definition. Document each one, then rerun both baselines if any item changes. This keeps the decision tied to verified, like-for-like evidence rather than to a single benchmark headline.

![Key Factors to Consider — Contract clause extraction](https://static.mm-ais.com/article-images-pixabay/contract-clause-extraction-cuad-contract-76703c3b.jpg)

## Common Mistakes

Common Mistake 1: treating a downloaded copy as complete before checking it against the release. A team may run extraction on a mirror that omits contracts or annotation rows, then compare its F1 with a published full-release score. The missing items are often rare clause types, so the error is not evenly distributed and macro-style comparisons become misleading. Before committing, count contract files, annotation rows, and clause-type labels, then compare those totals with the release manifest and schema. If any total differs, stop and reconcile the source before interpreting the score.

Concrete example: an extractor is tuned on a partial folder and appears to improve recall because the omitted contracts contained hard, low-frequency clauses. The gain is an artifact of the sample, not the method. The check is to rerun on the verified complete set and report only that result.

Common Mistake 2: comparing spans that answer different questions. One system may mark every sentence mentioning a topic, while the gold set marks only the operative clause with exact boundaries. The resulting false positives or false negatives are definition errors, not model errors. Read the clause definition, match its boundary rule, and evaluate the same clause subset with the same span-overlap rule. If your regex or transformer uses a looser trigger, split the metric into "topic found" and "exact span found" so the comparison stays like-for-like.

Concrete example: a rule flags "This agreement is governed by New York law" plus a nearby forum-selection sentence as one governing-law span, while gold marks only the first sentence. The rule's recall can look acceptable while precision collapses. The fix is to align the output to the gold boundary before scoring.

Final check: record the source location, file count, annotation count, schema version, and evaluation rule with every reported F1. A score without those four fields cannot be reproduced or compared fairly.

![Common Mistakes — Contract clause extraction](https://static.mm-ais.com/article-images-pixabay/contract-clause-extraction-cuad-contract-ceeeba5f.jpg)

## Insider Tactics

A non-obvious strategy is to treat verification as a gate rather than a cleanup step. Before any model comparison, create an immutable record of the live release: capture its identifier, file inventory, schema, and clause taxonomy exactly as they appear at that moment. Run every experiment against that frozen record instead of against a moving local copy, so later results cannot inherit silent drift between what was downloaded and what the benchmark currently says.

A timing tip is to verify at successive gates: before committing compute, before committing a baseline, and before committing a conclusion. The first gate catches an incomplete or stale working copy; the second catches definition mismatches between systems; the third catches metric drift introduced by changed assumptions. If any gate fails, reconcile before spending more effort.

| Gate | What to verify |
| --- | --- |
| Before compute commitment | Release inventory and schema match the frozen record |
| Before baseline commitment | Clause definitions and annotation version match across systems |
| Before conclusion commitment | Preprocessing and evaluation assumptions are documented and unchanged |
| Final reporting | Live release still matches the record used for results |

Another non-obvious strategy is to compare like-for-like by locking preprocessing and gold-version assumptions. A transformer and a regex baseline should be evaluated on the same normalized text, the same clause definitions, and the same annotation version. If one system sees cleaned text while the other sees raw text, the comparison measures preprocessing rather than extraction. Keep a control run that documents every transformation.

A timing tip for final reporting is to perform a last-mile audit immediately before publication. Re-check the live release, then compare its current inventory and definitions with the frozen record. If they differ, report the difference as a scope note instead of silently updating results, preserving comparability across runs and avoiding overclaiming.

Finally, keep regex as a diagnostic control even when a transformer is the main candidate. Run it on the same verified subset to expose annotation ambiguities, boundary disagreements, and clause types where simple patterns are sufficient. Use those findings to decide where added model complexity is justified, rather than treating any single method as the universal winner.

![Insider Tactics — Contract clause extraction](https://static.mm-ais.com/article-images-pixabay/contract-clause-extraction-cuad-contract-34c002cc.jpg)

## Comparison

Put the two approaches on the same verified CUAD release — the corpus totals confirmed against the live files as described above — and the comparison stops being philosophical. A rule-based extractor built from hand-written patterns typically posts high precision on the clause types its authors tuned for, but its recall collapses on the long tail of phrasing variants; a fine-tuned transformer typically trades some of that precision for materially broader recall, and its F1 varies by clause type rather than holding one flat number. The honest comparison is per-clause, not aggregate: pull the F1 for each of the clause categories you actually care about, confirm both systems were scored against the identical gold annotations, and only then declare a winner. An aggregate F1 that hides a weak category is not a comparison; it is an average that flatters the loser.

Build the side-by-side table before you build anything else. For each candidate, record the clause type, precision, recall, F1, the exact release version scored, and the count of gold spans used. If a vendor or paper reports a single headline F1 without the underlying per-clause breakdown, treat that number as unverified and ask for the breakdown; a system that wins on average but loses on your three highest-volume clause types is the wrong system for your workflow. The arithmetic check is simple: recompute F1 from the reported precision and recall (F1 = 2PR/(P+R)) and flag any figure that does not reconcile.

| Clause type | Rule-based P / R / F1 | Transformer P / R / F1 | Winner |
| --- | --- | --- | --- |
| Governing law | High / Moderate / Moderate | High / High / High | Transformer |
| Notice period | High / Low / Low | Moderate / High / Moderate | Transformer on recall; rules on precision |
| Fixed-format identifiers | High / High / High | High / High / High | Tie — rules cheaper to run |
| Rare, tightly worded clauses | High / Low / Low | Moderate / Moderate / Moderate | Transformer |

Regex still wins in three situations. First, when the clause is genuinely fixed-format — a defined term, a section reference, a date pattern — where a pattern matches nearly everything and a model only adds latency and cost. Second, when you need deterministic, auditable output: a regex match can be explained line by line to a reviewer, while a transformer's span proposal cannot. Third, when you have no labeled data and no budget to create it; a pattern set ships today, whereas a fine-tuned model needs annotated examples from your own contract population. The transformer wins when clause language varies, when recall matters more than precision (missing a clause is costlier than flagging a false positive for review), and when you can supply labeled data and accept per-clause tuning.

The practical rule: run both on a held-out slice of your own contracts, score per clause type against your own gold annotations, and pick per clause type rather than per system. A hybrid — regex for the fixed-format categories, transformer for the variable ones — frequently beats either alone on the same release. Verify the release version and annotation count first, then compare like-for-like totals and terms; only then commit.

## What to do next

| Step | Action | Why it matters |
| --- | --- | --- |
| 1 | Re-check the 2021 benchmark table for the exact 510-contract, 13,000-annotation scope before choosing a method. | Prevents committing to a partial or mismatched extraction setup. |
| 2 | Compare Transformer and rule-based F1 on like-for-like terms across the same clause labels. | Ensures totals reflect the same task, not different inputs. |
| 3 | Isolate the clause types where the rule trigger says regex still wins and test those spans directly. | Shows whether regex's advantage holds on the live complete option. |
| 4 | Verify the complete extraction output against the 13,000 annotations before committing. | Catches missing spans that a headline F1 can hide. |
| 5 | Cross-check the 2021 results with the 2017 reference before finalizing the method. | Keeps the decision anchored to the benchmark's stated comparison points. |
| 6 | Commit only after the live complete option and like-for-like totals/terms agree. | Follows the verify-before-you-commit rule. |

## Frequently Asked Questions

**What must be confirmed before trusting a reported clause-extraction result?**

The reported result must be confirmed as computed on the complete, unchanged evaluation set rather than a convenient subset.

**Which three elements must prediction and gold share for clause-extraction results to be comparable?**

They must use the same span boundaries, clause taxonomy, and document split.

**What does precision measure in this workflow?**

Precision measures how many predicted spans match gold.

**What does recall measure in this workflow?**

Recall measures how many gold spans were found.

**What does a rule-based extractor apply explicit patterns to?**

A rule-based extractor applies explicit patterns, often regular expressions, to headings, defined terms, or cue phrases.

**What does multi-head self-attention let a token do?**

Multi-head self-attention lets a token weigh other tokens in the same sequence.

## Quick answers

| What does the article compare? | It compares Transformer vs. Rule-Based Clause-Extraction F1 to show when Regex Still Wins. |
| --- | --- |
| What is clause extraction? | Clause extraction is a span-labeling workflow in which a system reads contract text, proposes an interval for each requested clause type, and an evaluator compares those proposals with a human-marked gold set. |
| What must prediction and gold share for the mechanism to be comparable? | The mechanism only becomes comparable when prediction and gold use the same span boundaries, clause taxonomy, and document split. |
| What is a gold annotation? | A gold annotation is the human answer used as reference. |
| What does a rule-based extractor apply? | A rule-based extractor applies explicit patterns, often regular expressions, to headings, defined terms, or cue phrases. |

Also worth reading: **Fine-Tuning Legal Transformer Cuts False Positives 23% in 10K**: [Fine-Tuning Legal Transformer Cuts False](https://legalpdf.io/blog/fine-tuning-legal-transformer-cuts-false-positives-23-in-10k.php) · **AI-Powered Contract Analysis A Deep Dive into Anti-Disparagement Clause Detection and Risk Assessment**: [AI-Powered Contract Analysis A Deep](https://legalpdf.io/blog/ai_powered_contract_analysis_a_deep_dive_into_anti_disparage.php) · **McCulloch v

Maryland (1819) How the Necessary and Proper Clause Shaped Modern Contract Law**: [McCulloch v Maryland (1819) How](https://legalpdf.io/blog/mcculloch_v_maryland_1819_how_the_necessary_and_proper_cl.php)

### Related reading

- [Digital Contract Analysis Understanding Tennessee's Automated Business Filing System in 2024](https://legalpdf.io/blog/digital_contract_analysis_understanding_tennessee_s_automate.php)
- [NCIC Warrant Retention Protocols Understanding the 24-Hour Hit Confirmation Requirements for AI Contract Systems](https://legalpdf.io/blog/ncic_warrant_retention_protocols_understanding_the_24_hour_h.php)
- [Understanding Par Value in Smart Contract Stock Issuance Technical and Legal Implications for DeFi Platforms](https://legalpdf.io/blog/understanding_par_value_in_smart_contract_stock_issuance_tec.php)
- [Understanding Monetary Damages A Deep Dive into Standard Tort Remedies in AI Contract Law](https://legalpdf.io/blog/understanding_monetary_damages_a_deep_dive_into_standard_tor.php)
- [Understanding the Four Financial Statements Essential Tools for AI-Assisted Contract Review](https://legalpdf.io/blog/understanding_the_four_financial_statements_essential_tools.php)
- [How to Verify NER Clause-Extraction Benchmarks on 2024 SEC 10-K Risk Factors](https://legalpdf.io/blog/how-to-verify-ner-clause-extraction-benchmarks-on-2024-sec-10-k-risk-factors.php)

### Latest

- [Mortality Risk Scoring Guide: LexisNexis Risk Classifier Uses 2026 Alternative...](https://legalpdf.io/blog/mortality-risk-scoring-guide-lexisnexis-risk-classifier-uses-2026-alternative-data.php)
- [Rand v. Claremont: Verify Funding Options Before Committing](https://legalpdf.io/blog/rand-v-claremont-verify-funding-options-before-committing.php)
- [How Courts Use Complexity: 35 Years NP-Complete Citations, Cite/Ignore](https://legalpdf.io/blog/how-courts-use-complexity-35-years-np-complete-citations-citeignore.php)

Canonical: https://legalpdf.io/blog/contract-clause-extraction-cuad-contract-understanding-atticus-dataset-510-regex-wins.php
Markdown: https://legalpdf.io/blog/contract-clause-extraction-cuad-contract-understanding-atticus-dataset-510-regex-wins.php/index.md