# How to Validate Indemnification Extraction on 500-PDF Software Contracts

Justin Howard · November 10, 2026

> Validate indemnification extraction on 500 software contract PDFs by requiring page numbers and bounding boxes to audit every extracted span.

Every extracted span must carry a page number and bounding box. In software contracts, cross-references and exhibits can move operative language outside the main clause, so an unlocated span cannot be audited against the source PDF. The location is part of the extraction result, not a display convenience.

Because the two stages and five fields are scored separately, precision can be split by failure type: a false positive clause, a false positive field, a false negative clause, or a false negative field. That split is what makes the review queue actionable instead of collapsing every error into one aggregate accuracy figure.

![Every extracted span must carry a page number — How to Validate Indemnification Extraction on](https://static.mm-ais.com/article-images-ai/how-to-validate-indemnification-extracti-ai-f9222418.jpg)

## What 94.2% Clause Accuracy Does Not Mean

A 94.2% aggregate clause-accuracy figure does not make a 500-PDF indemnification review safe. Sirion’s 2026 benchmark reports 94.2% clause extraction accuracy and defines F1 as the harmonic mean of precision and recall, but that benchmark summary contains no indemnity-only recall figure. The count that matters for this decision is therefore zero reported indemnity-only recall values; aggregate accuracy can hide a slice with low recall because a high score on common clauses can offset missed indemnity language.

F1 is a balance, not a substitute for slice-level evidence. Because it is the harmonic mean of precision and recall, a strong precision result can coexist with weak recall, and an aggregate F1 can still look healthy while one indemnity slice is under-recalled. Sirion’s benchmark supplies the definition, not an indemnity-specific recall table, so the reader cannot tell whether the 94.2% figure reflects indemnity clauses or easier clause types.

Superdots frames 500-contract reviews as a move from months to two days. That is throughput evidence: it shows elapsed time and volume, not a precision/recall pair. It does not show that indemnity clauses were extracted correctly, nor that a missed indemnity clause would be caught before auto-clear. Speed is useful only after the extraction metric is tied to the clause family being reviewed.

Harvey’s 2026 precedent argument is qualitative, not numeric: it says generic indemnification examples show possible language, while your own deal history shows what counterparties have accepted in similar transactions. That supports benchmarking against internal precedent, but it is not an accuracy percentage and should not be treated as validation of extraction recall. It is a negotiation source, not a measurement of whether the model found the clause.

The defensible check is therefore separate from all three sources: build a stratified gold set from the same 500 PDFs, score each indemnity slice on its own precision and recall, and auto-clear only when the critical slice meets the recall gate. Until that slice-level score exists, 94.2% aggregate accuracy, two-day throughput, and precedent benchmarking remain supporting context, not clearance evidence.

![What 94.2% Clause Accuracy Does Not Mean — How to Validate Indemnification Extraction on](https://static.mm-ais.com/article-images-ai/how-to-validate-indemnification-extracti-ai-6adc5ff2.jpg)

## Three Routes for 500 Indemnity PDFs

Route selection for a 500-PDF indemnification review should be judged by whether it can support an auto-clear decision, not by whether it produces a tidy summary. A route that cannot expose slice-level failure should not be used to clear a batch. The three routes differ in what they measure, where they break, and whether they can justify clearing documents without human review.

The break-even point is direct: one false-negative repair equals 12 false-positive checks (60 divided by 5). This means a system that reduces missed fields by accepting additional manual checks can remain time-efficient even when its precision appears noisy, because each avoided post-signature repair offsets a dozen pre-signature confirmations.

Applied to a 500-PDF queue, this arithmetic makes extra review a bounded, schedulable cost and missed indemnity fields an unbounded operational risk. Before any batch is auto-cleared, the team should treat reviewer minutes spent on false positives as the cheaper control and reserve auto-clearance for results whose measured performance keeps expected repair time below the cost of added checks.

![Three Routes for 500 Indemnity PDFs — How to Validate Indemnification Extraction on](https://static.mm-ais.com/article-images-pixabay/how-to-validate-indemnification-extracti-49c8d91c.jpg)

## Where the 500-PDF Evidence Breaks

Aggregate benchmarks do not establish that clause-level performance transfers to indemnification language in software PDFs. Until your team measures per-slice recall on its own corpus, any vendor accuracy figure should be treated as an unverified claim rather than evidence that a 500-PDF review can be auto-cleared. The recall gate is only defensible when the documents sampled for validation reflect the same conditions the extraction pipeline will face in production.

On a 500-PDF software-contract review, failures tend to cluster around document conditions that hide, replace, or relocate indemnity text. A stratified gold set drawn from the same PDFs must deliberately include these conditions so that each indemnity slice is tested where the rule is most likely to break, not only on clean, self-contained agreements.

| Edge case | Rule breaks | Rule still wins |
| --- | --- | --- |
| Scanned PDF with OCR noise | OCR confidence is low and no duplicate or corrective extraction pass is run before scoring. | Low-confidence pages are routed for human review and the recall gate is applied to verified text. |
| Amendment chain with superseded indemnity | The pipeline treats each PDF in isolation and scores an outdated clause as current. | Clause versions are linked across the amendment chain and only the operative indemnity is scored. |
| Cross-reference to another document or section | The extracted text stops at the reference and omits obligations defined elsewhere. | Referenced sections or documents are resolved and included before the slice is scored. |

Scanned files are a direct test of text fidelity: if the characters feeding the extractor are unreliable, recall measured on clean PDFs will overstate performance. Amendment chains add a different risk, because the relevant indemnity may be modified or replaced in a later file. Before treating an isolated extraction as complete, check whether related agreements sit across matters, folders, or document systems.

Cross-references create a similar visibility gap when the operative obligation lives outside the extracted span. LlamaIndex’s guidance on long-form document extraction highlights that extraction from complex documents requires careful schema and handling, reinforcing the need to confirm that linked content is captured before scoring.

For the edge cases above, include scanned, amendment-chain, and cross-reference documents in the stratified gold set before scoring; the per-slice recall and precision gates are set out above.

![Where the 500-PDF Evidence Breaks — How to Validate Indemnification Extraction on](https://static.mm-ais.com/article-images-pixabay/how-to-validate-indemnification-extracti-790410e3.jpg)

## Worked 500-PDF Indemnity Gold Set

To make a 500-PDF indemnification review defensible, this section runs the illustrative gold-set arithmetic and specifies the copy-usable worksheet columns for validating extraction performance on the same corpus. From 500 PDFs, draw a sample of 100 stratified by agreement type—SaaS, license, and implementation—so the gold set reflects the contract mix under review. If 80 of the sampled PDFs contain an indemnity clause and each clause is scored across five required fields, the exercise yields 400 field labels adjudicated by two reviewers, creating a stable basis for per-slice measurement rather than a single aggregate pass/fail.

Clause detection is the first checkpoint. With 78 true positives, 2 false negatives, and 4 false positives, recall is 78/80 = 97.5% and precision is 78/82 = 95.1%. These results show that the system can usually find the clause, but clause-level performance does not authorize auto-clearance for every downstream obligation captured within it.

Field-level checkpoints determine whether human routing is required. For the critical-obligation slice, 70 true positives, 10 false negatives, and 7 false positives produce recall of 70/(70+10) = 87.5% and precision of 70/(70+7) = 70/77 ≈ 90.9%. For the procedure slice, 52 true positives, 28 false negatives, and 10 false positives produce recall of 52/(52+28) = 65.0% and precision of 52/(52+10) = 52/62 ≈ 83.9%. Because any critical slice below 95% recall must be routed to human review, the procedure slice cannot be auto-cleared; reviewers should handle those extractions manually before the batch advances.

| Worksheet column | Purpose | Example entry |
| --- | --- | --- |
| Document ID | Trace each label back to a specific PDF in the 500-document corpus. | PDF-0147 |
| Agreement stratum | Preserve stratification across SaaS, license, and implementation contracts. | SaaS |
| Indemnity present (Reviewer A / B / Adjudicated) | Record clause-detection judgments and resolve disagreements. | Yes / Yes / Yes |
| Slice name | Score each indemnity slice separately. | Procedure |
| Extracted value | Capture the system output evaluated for the slice. | Prompt notice required |
| Gold value | Store the adjudicated reference label. | Written notice within stated period |
| Outcome (TP/FP/FN/TN) | Enable per-slice precision and recall calculations. | FN |
| Routing decision | Document whether the slice is auto-cleared or sent to human review. | Route to human |

Use the completed worksheet as the audit record for the stratified sample; the procedure example above shows why a slice that misses the recall gate must remain in human review.

![Worked 500-PDF Indemnity Gold Set — How to Validate Indemnification Extraction on](https://static.mm-ais.com/article-images-pixabay/how-to-validate-indemnification-extracti-3dd7732b.jpg)

## Five If/Then Rules for Recall Gates

These five operational if/then rules convert per-slice recall and precision measurements into concrete review routing decisions for a 500-PDF indemnification review. Apply them only after scoring each indemnity slice separately against a labeled gold set drawn from the same PDF population.

**Rule 1:** If a vendor report lacks indemnity-only recall and precision on your PDF population, then do not auto-clear any clause. Build a stratified gold set from the same PDFs first, label the relevant indemnity slices, and establish a baseline before any automated routing begins.

**Rule 2:** If critical-obligation recall is below 95%, then route every critical slice to human review and do not use the pipeline for compliance flagging. A miss in a critical obligation is a false clear, so this gate prioritizes recall over throughput for the slices that carry the highest review risk.

**Rule 3:** If procedure recall is below 85% but cap recall is at least 95%, then auto-clear only the cap slice and manually review procedure. Do not apply one global threshold across slices; each slice must earn its own routing decision based on its measured performance.

**Rule 4:** If a slice meets its recall gate but its measured precision does not support defensible reliance for your review standard, then treat extracted fields as review aids rather than auto-cleared determinations. Route those outputs to human confirmation so that false positives do not bypass scrutiny under the appearance of a passing recall score.

**Rule 5:** If every indemnity slice meets or exceeds its recall and precision gates on the stratified gold set, then auto-clear only the passing slices and retain a periodic sample review as new PDFs enter the corpus. Passing once is not a permanent clearance; revalidate against newly added documents to confirm that routing decisions remain defensible over time.

## What to do next

| Step | Action | Why it matters |
| --- | --- | --- |
| 1 | Assemble a stratified gold set drawn directly from the same 500-PDF corpus under review. | Ensures validation reflects the actual distribution and language of the contracts being analyzed, rather than an unrepresentative sample. |
| 2 | Score precision and recall separately for each indemnity slice within the gold set. | Reveals slice-level performance that aggregate clause-accuracy figures can obscure. |
| 3 | Compare each critical indemnity slice against the 95% recall threshold and route any slice below it to human review. | Prevents high-risk indemnity obligations from being auto-cleared when extraction coverage is insufficient. |
| 4 | Compare each procedural indemnity slice against the 85% recall threshold and route any slice below it to human review. | Ensures procedural requirements receive appropriate scrutiny when they fail to meet the defined recall standard. |
| 5 | Reject vendor aggregate clause-accuracy as sufficient evidence for auto-clearance decisions. | A single aggregate metric cannot substitute for per-slice precision and recall validation on the corpus under review. |
| 6 | Auto-clear only those indemnity slices that meet their respective per-slice recall thresholds, and document the routing decision for each slice. | Creates a defensible, auditable record that every slice was measured and handled according to the established decision rule. |

## Frequently Asked Questions

**Why must every extracted span carry a page number and bounding box?**

In software contracts, cross-references and exhibits can move operative language outside the main clause, so an unlocated span cannot be audited against the source PDF.

**Is span location only a display convenience?**

The location is part of the extraction result, not a display convenience.

**How does separate scoring make the review queue actionable?**

Because the two stages and five fields are scored separately, precision can be split by failure type: a false positive clause, a false positive field, a false negative clause, or a false negative field.

**Does 94.2% aggregate clause accuracy make a 500-PDF indemnification review safe?**

A 94.2% aggregate clause-accuracy figure does not make a 500-PDF indemnification review safe.

**What indemnity-specific count is missing from Sirion’s 2026 benchmark summary?**

The benchmark summary contains no indemnity-only recall figure, so the count that matters is zero reported indemnity-only recall values.

**Can strong precision coexist with weak recall in an aggregate F1?**

Because F1 is the harmonic mean of precision and recall, a strong precision result can coexist with weak recall.

## Quick answers

| Why must every extracted span include a page number and bounding box? | Every extracted span must carry a page number and bounding box because cross-references and exhibits can move operative language outside the main clause, so an unlocated span cannot be audited against the source PDF. |
| --- | --- |
| How can precision be split when the two stages and five fields are scored separately? | Precision can be split by failure type: a false positive clause, a false positive field, a false negative clause, or a false negative field. |
| Why is splitting errors by failure type beneficial for the review process? | Splitting errors by failure type makes the review queue actionable instead of collapsing every error into one aggregate accuracy figure. |
| Does the aggregate clause-accuracy figure reported in Sirion’s benchmark make a 500-PDF indemnification review safe? | No, the aggregate clause-accuracy figure reported in Sirion’s benchmark does not make a 500-PDF indemnification review safe because the benchmark summary contains no indemnity-only recall figure. |
| Why is F1 not a substitute for slice-level evidence? | F1 is not a substitute for slice-level evidence because it is the harmonic mean of precision and recall, meaning a strong precision result can coexist with weak recall and aggregate accuracy can hide a slice with low recall. |

Also worth reading: **Contract clause extraction: CUAD (Contract Understanding Atticus Dataset) 510: Regex Wins**: [Contract clause extraction: CUAD (Contract](https://legalpdf.io/blog/contract-clause-extraction-cuad-contract-understanding-atticus-dataset-510-regex-wins.php) · **EU AI Act Article 50: clause-level provenance checks for legal PDFs**: [EU AI Act Article 50:](https://legalpdf.io/blog/eu-ai-act-article-50-clause-level-provenance-checks-for-legal-pdfs.php) · **Scanned contracts miss indemnity: 88% vs 98.2% Flag vs Fix-and-Re-Extract**: [Scanned contracts miss indemnity: 88%](https://legalpdf.io/blog/scanned-contracts-miss-indemnity-88-vs-982-flag-vs-fix-and-re-extract.php)

### Related reading

- [Find Indemnification Clauses in Vendor Contracts: 2% to 94.3% Trust vs Rescan](https://legalpdf.io/blog/find-indemnification-clauses-in-vendor-contracts-2-to-943-trust-vs-rescan.php)
- [Public Sector Legal Tech 2026: NLP Clause Extraction at 85% Recall for Compliance](https://legalpdf.io/blog/public-sector-legal-tech-2026-nlp-clause-extraction-at-85-recall-for-compliance.php)
- [Contract clause extraction: CUAD (Contract Understanding Atticus Dataset) 510: Regex Wins](https://legalpdf.io/blog/contract-clause-extraction-cuad-contract-understanding-atticus-dataset-510-regex-wins.php)
- [How to Verify NER Clause-Extraction Benchmarks on 2024 SEC 10-K Risk Factors](https://legalpdf.io/blog/how-to-verify-ner-clause-extraction-benchmarks-on-2024-sec-10-k-risk-factors.php)
- [PDF Clause Extraction Accuracy: Layout-Aware Parsers 2026 SEC Filings F1 vs Latency](https://legalpdf.io/blog/pdf-clause-extraction-accuracy-layout-aware-parsers-2026-sec-filings-f1-vs-latency.php)
- [AI-Powered Video Analysis in eDiscovery Revolutionizing Legal Evidence Extraction](https://legalpdf.io/blog/ai_powered_video_analysis_in_ediscovery_revolutionizing_lega.php)

### Latest

- [CILEX Apprenticeship Agreements: NLP Recall 91% vs Manual Review, 50 PDFs (2026)](https://legalpdf.io/blog/cilex-apprenticeship-agreements-nlp-recall-91-vs-manual-review-50-pdfs-2026.php)
- [EU AI Act Article 50: clause-level provenance checks for legal PDFs](https://legalpdf.io/blog/eu-ai-act-article-50-clause-level-provenance-checks-for-legal-pdfs.php)
- [California AI Laws: How to Meet AB 2013’s January 1, 2026 Training Data...](https://legalpdf.io/blog/california-ai-laws-how-to-meet-ab-2013s-january-1-2026-training-data-disclosure-mandate.php)

Canonical: https://legalpdf.io/blog/how-to-validate-indemnification-extraction-on-500-pdf-software-contracts.php
Markdown: https://legalpdf.io/blog/how-to-validate-indemnification-extraction-on-500-pdf-software-contracts.php/index.md