Legal PDF clause errors: 500-contract sample, retrain or review?

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

TakeawayDetail
Split the 500-contract sample by clause family before deciding to retrain.The thesis calls for a clause-family split: retrain only families whose hand-labeled recall or precision misses a pre-registered threshold.
Use recall <0.90 or precision <0.85 as the example retrain trigger.The reader rule gives the example method: retrain if recall <0.90 or precision <0.85.
Audit a random 100-PDF slice of the 500 before any retraining.The reader rule requires auditing by clause family on a random 100-PDF slice of the 500 before retraining.
Keep manual review for clause families that clear the threshold.The thesis says manual review remains for families that clear the pre-registered threshold.

As the takeaways table states, the defensible answer for a 500-contract legal-PDF clause-error sample in 2026 is a clause-family split: retrain only families that miss pre-registered recall or precision thresholds, and keep manual review for the rest.

Legal PDF clause errors

How to audit 500 PDFs by clause family

Before you touch a single PDF in the 500-contract sample, name the clause families and freeze the list. Use six: definitions, indemnity, liability cap, termination, governing law, and payment terms. Write them down in the audit protocol before sampling starts. A family added after you see results makes the retrain-versus-review comparison unauditable, because the denominator changes midstream and you can no longer tell whether a miss came from the model or from your own recategorization. Treat the family list as part of the pre-registration, not as a labeling convenience.

The audit unit here is the contract-family pair: one PDF contributes one row per frozen family, so a single agreement can appear in several rows and be judged separately for each. That is the only way a family-level decision stays honest. A contract with a clean indemnity clause and a broken liability cap should not be scored as one pass or one failure; it should produce two rows, one for each family, each carrying its own gold label and its own prediction. Keep the unit at the pair level even when the same text span serves two families, and record the overlap rather than collapsing it.

Parse every PDF twice. Pass one extracts native text with layout and table structure preserved, so a capped-liability sentence sitting inside a schedule or side letter keeps its position. Pass two runs OCR only on pages that fail a text-layer character-count check; do not OCR the whole document by reflex, because OCR noise can manufacture clause errors that the model never made. Tag every page as native or OCR in the audit log. When a clause is missed, the tag tells you whether to blame extraction or the model, and that attribution is what keeps a retrain decision defensible.

Store four cells per contract-family row: gold present, gold span, predicted present, and predicted span. The two presence cells let you count false positives and false negatives at the family level; the two span cells let you check whether a correct presence call actually landed on the right language. A row with gold present and predicted absent is a recall miss. A row with gold absent and predicted present is a precision miss. A row where both say present but the spans diverge is a boundary miss, and it belongs in the same family bucket rather than being quietly marked correct.

Keep the page tag and the four cells together in one store. If a definitions row shows a recall miss on an OCR-tagged page, the extraction layer is the first thing to inspect, not the training set. If the same miss appears on native pages across several contracts in the family, the signal points at the model. This is the check that makes the family split work: you are not asking whether the system got the contract right in the abstract, you are asking whether this family, on this extraction path, cleared the bar you set before you looked.

Definitions deserve particular care in the audit because the family is structurally slippery. Work describing definition extraction notes that contracts rely on defined terms and that detection often uses patterns over well-organized text, while reported results on a separate Supreme Court opinion task put Incite-Base at 80.6% accuracy against GPT-4 at 81.8% (arxiv.org). Those figures are not a threshold for your sample; they are a reminder that a definitions row needs an explicit span check, not just a presence check. Freeze the family, tag the page, fill all four cells, and the later decision has something solid to stand on.

How to audit 500 PDFs by clause family — Legal PDF clause errors

Evidence: source figures and gaps

The evidence most often cited to justify a full-pipeline retrain does not actually measure clause-family performance. The arXiv result on definition_extraction reports Incite-Base at 80.6% accuracy against GPT-4 at 81.8%, but the task there is narrow: extract the term defined by a sentence taken from a Supreme Court opinion. That is a definition-only benchmark, not a hand-labeled recall or precision measurement for indemnity, liability cap, termination, governing law, or payment terms in a 500-contract legal-PDF sample. Treating that figure as proof that every family in your own corpus needs retraining collapses two different questions: can a model spot a definitional sentence in judicial prose, and does your extractor meet your pre-registered threshold on a specific contract clause family?

The same distinction shows up across the named sources. arXiv supplies roughly 80–82% accuracy for that definition-task setup, which is useful as a sanity check on model capability but not as a substitute for family-level labels. ACL Anthology’s ConReader paper describes a Definition Extractor and TDR Solver that use regular expressions to extract definitions from English contracts because contracts are well-organized; that supports a structural check on your own parse output before blaming the model. The dev.to piece on graph-aware retrieval says contracts rely heavily on defined terms and that definition extraction detects those terms using patterns. Read together, these sources converge on the importance of defined-term structure, not on a universal retrain trigger.

Named source What it reports What it cannot establish for your sample Check to run before acting
arXiv definition_extraction result Incite-Base 80.6% vs GPT-4 81.8% accuracy on Supreme Court opinion sentences Recall or precision by contract clause family Compare only against a hand-labeled family slice, not the headline accuracy
ACL Anthology, ConReader Regex definition extraction used because contracts are well-organized Whether your PDF parse preserves the structure those patterns expect Inspect parsed clause boundaries and defined-term cues before retraining
dev.to, graph-aware retrieval Contracts rely heavily on defined terms; patterns detect defined terms Which of your families miss the pre-registered threshold Confirm whether errors cluster around defined-term references

That convergence limit matters for the myth in circulation: “definition-extraction accuracy” is not a clause-family recall measurement. A model can identify a defined term and still miss a liability-cap carveout, misclassify a termination notice provision, or lose a governing-law clause embedded in a boilerplate block. The arXiv figure is a benchmark result on a different text type. It should not be quoted as evidence that the whole 500-contract pipeline is below standard, because it contains no family-level false-negative or false-positive accounting.

The defensible reading is narrower. Use the arXiv, ACL Anthology, and dev.to findings as reasons to examine definition handling and document structure, then let your own labeled family results decide the action. If a family clears the pre-registered recall and precision thresholds, the evidence supports keeping manual review for that family rather than retraining it. If a family misses a threshold, retrain that family. The external sources tell you where to look; they do not tell you which families in your sample failed.

For the 2026 sample, the practical rule is to keep the benchmark in its lane. Cite the definition_extraction figures only as background on definition-task performance, cite ConReader and dev.to only as support for checking defined-term structure, and require every retraining decision to rest on family-level hand labels from your own PDFs. That is what makes the clause-family split defensible: the decision is tied to measured local error, not to a headline number from a different task.

Retrain vs review: cost table winner

The cost comparison has a single winner rule: review first, then retrain only the clause family that fails its pre-registered audit threshold. That rule keeps annotation spend capped by measured family misses instead of spreading labor across the full 500-contract sample. The fixed buckets are the audit protocol, the family list, the reviewer instructions, and the reporting template. The variable buckets are the clauses actually touched: predicted-positive clauses checked by a reviewer, gold labels created for a failing family, engineering time on that family, and re-audit PDFs. A full-pipeline retrain is not the default because the spend is not tied to a measured miss.

Action Trigger Fixed labor Variable labor Time to first clean audit Failure mode
Manual review Audit protocol, reviewer instructions, reporting template Predicted-positive clauses in the audit slice checked by a human; no new gold set for the family At the first audit sign-off for that family Reviewer drift if the clause text is ambiguous or the family boundary is fuzzy
Family retrain Family misses the pre-registered recall or precision threshold Same audit setup plus the engineering harness for that family Gold-label count per failing family × annotation minutes per clause + engineering hours + re-audit PDFs Only after the family clears the same audit threshold on re-audit
Full 500-contract retrain No measured family miss supports this trigger Full-pipeline engineering setup Annotation and re-audit scaled across all families, including families that cleared the threshold Not established by the family audit Spend is detached from the measured defect

For the retrain row, keep the arithmetic literal. Count the gold labels required for the failing family, multiply by the minutes per clause agreed in the annotation instructions, then add the engineering hours for the change and the PDFs pulled for re-audit. Do not scale that bucket to all 500 contracts until the family clears the same audit threshold it failed. The check is simple: if the family is back above the threshold on the re-audit, the retrain is done; if it is still below, the variable bucket stays open for that family only.

For the review row, the variable bucket is the predicted-positive clauses in the audit slice that a human checks, not every clause in every PDF. That distinction is the cost lever. A family that clears the threshold keeps manual review as the control, and the labor stops at the reviewed clauses. A family that fails gets gold labels, engineering, and re-audit. The winner is review first because it spends only where the audit shows a miss; retrain is the exception, not the program default.

Use the table as a sign-off check before any annotation order is placed. Name the action, name the trigger, fill the fixed and variable buckets, and state what a clean audit would look like for that family. If the trigger is not a measured threshold miss, the row does not qualify. If the variable bucket includes families that cleared the audit, the row is mispriced. This is the whole comparison: measured family misses cap the spend, and the cap is what makes the split defensible.

Worked 500-contract audit worksheet

Start the worksheet with the arithmetic, not with labels. The sample is 500 PDFs; the random audit slice is 100 PDFs; the reserve is 400 PDFs. That is 100 + 400 = 500, so the slice is one fifth of the sample and the reserve is four fifths. Draw the 100 before any family-level result is computed, and record the draw method in the protocol. This section is the only place that supplies the 500-to-100 random-slice arithmetic and the copy-usable worksheet columns; use it as the working sheet rather than as a place to import outside performance figures.

Apply the stopping rule before reporting any family rate. If any clause family has fewer than 20 positive examples in the 100-PDF slice, do not publish a recall or precision rate for that family. Instead, label additional PDFs from the remaining 400 until that family reaches at least 20 positives, then recompute from the enlarged labeled set. The check is mechanical: count gold_present = 1 by family, compare with 20, and log the added contract_id values. A thin positive count is a sampling failure, not evidence that the family is easy or safe.

Copy these columns exactly: contract_id, family, page_tag, gold_present, gold_span, predicted_present, predicted_span, match_type, reviewer, dispute_flag, lower_95_recall, lower_95_precision. Fill observed values only. contract_id, family, page_tag, reviewer, and dispute_flag come from the audit record; gold_present, gold_span, predicted_present, and predicted_span come from the human and system outputs being compared; match_type records the observed comparison outcome; lower_95_recall and lower_95_precision are computed from the observed counts for that family. Do not paste literature values into the last two columns. The arXiv result on definition_extraction reports Incite-Base at 80.6% accuracy against GPT-4 at 81.8%, but those figures belong to the evidence discussion, not to a worksheet cell.

Run checkpoint one as completeness. Require zero missing gold_present values, zero missing predicted_present values, zero blank family values, and zero blank reviewer values across every audited row. For each row where gold_present = 1, require a nonblank gold_span; for each row where predicted_present = 1, require a nonblank predicted_span. If a span is genuinely absent because the clause is absent, keep the corresponding present flag at 0 and leave the span field marked as not applicable under the protocol’s label convention. A missing gold label is not a negative label.

Run checkpoint two as internal consistency. A row with gold_present = 0 and predicted_present = 0 should not carry a positive match_type; a row with gold_present = 1 and predicted_present = 1 should identify whether the observed spans match under the protocol’s span rule. Flag disagreements with dispute_flag rather than silently correcting them. The worksheet supports the pre-registered family decision: compute the family’s lower confidence bounds from the observed cells, then compare those bounds with the thresholds fixed in the protocol. Only families that miss the registered recall or precision threshold move to retraining; families that clear it stay on manual review under the family-split rule.

Limits and if/then decision rules

This section is where the family split stops being a preference and becomes a controlled decision. The edge-case limits and the if/then rules below override the family split when the evidence is thin, ambiguous, or unstable. They are the guardrails I apply after the family-level audit and before any retraining decision is allowed to affect the 500-contract sample.

Condition Required action Effect on family split
Fewer than 20 positives in the family Keep manual review Overrides a retrain signal
Confidence interval overlaps the threshold Do not retrain Keeps the family in review
Unresolved text-layer or OCR attribution Freeze the decision Blocks both retrain and review routing until resolved
Sufficient positives and clean spans Apply the threshold rule Family split stands

The if/then rules are narrow. If the lower-bound recall for a family is below 0.90, retrain that family. If the lower-bound precision is below 0.85, retrain that family. If both lower bounds clear those thresholds, route the family to manual review rather than retraining it. These rules use the lower bound, not the point estimate, so a family must clear the threshold with uncertainty accounted for. The rule still wins when the family has sufficient positives and clean spans; that is the ordinary case in which the family split is meant to operate.

There are two stop conditions. If a second 100-PDF audit reverses the family, stop scale-up and resolve the conflict before extending any decision beyond that family. And never retrain the whole 500-contract sample as a single block under this protocol. The decision unit is the family, the trigger is the pre-registered threshold, and the override is an explicit limit. That is the point of the design: a family with sufficient positives and clean spans can be retrained on a defensible signal, while a family with weak support, overlapping uncertainty, unresolved attribution, or a reversed audit stays contained.

What to do next

The steps above reduce to the protocol already fixed in this article: freeze the clause-family list and pre-register the retrain thresholds before sampling; split the 500-contract sample by family rather than pooling; hand-label and audit the random 100-PDF slice by family; compute each family’s lower 95% confidence bounds from the worksheet cells; retrain only families whose lower bound misses a pre-registered threshold, leaving the rest on manual review; and if a second 100-PDF audit reverses a family, stop scale-up and resolve the conflict before extending any decision beyond that family. The specific thresholds and arithmetic live in the sections above and are not restated here.

Frequently Asked Questions

Should the 500-contract sample be treated as one pool when deciding whether to retrain?

Split the 500-contract sample by clause family before deciding to retrain.

Which performance measures determine whether a clause family gets retrained?

Retrain only families whose hand-labeled recall or precision misses a pre-registered threshold.

What example method does the reader rule give for using recall in the retrain decision?

The reader rule gives the example method: retrain if recall.

What must be audited before any retraining, and how large is the slice?

Audit a random 100-PDF slice of the 500 before any retraining.

How must that pre-retraining audit be organized?

The reader rule requires auditing by clause family on a random 100-PDF slice of the 500 before retraining.

What happens to clause families that clear the pre-registered threshold?

Keep manual review for clause families that clear the threshold.

Quick answers

What should be done with the 500-contract sample before deciding to retrain?Split the 500-contract sample by clause family before deciding to retrain.
Which clause families should be retrained under the thesis?Retrain only families whose hand-labeled recall or precision misses a pre-registered threshold.
What does the reader rule give as an example method?The reader rule gives the example method: retrain if recall.
What must be audited before any retraining?Audit a random 100-PDF slice of the 500 before any retraining.
What remains for clause families that clear the threshold?Keep manual review for clause families that clear the threshold.

Also worth reading: Benchmarking NLP privilege-clause extraction in eDiscovery PDFs: Benchmarking NLP privilege-clause extraction in · EU AI Act Article 50: clause-level provenance checks for legal PDFs: EU AI Act Article 50: · Contract clause extraction: CUAD (Contract Understanding Atticus Dataset) 510: Regex Wins: Contract clause extraction: CUAD (Contract

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

Research Methodology & Editorial Standards

We begin by defining the specific objectives the reader needs to accomplish. Primary product documentation and authoritative secondary sources are assembled into a verified research corpus; drafting occurs only after this foundation is in place.

Every quantitative claim is subjected to dual-source verification. Any figure that cannot be independently corroborated is either qualified or omitted.

Published · Last reviewed · Owned by the Legalpdf editorial desk (About, Contact, Privacy).

Related answers