# How to Verify NER Clause-Extraction Benchmarks on 2024 SEC 10-K Risk Factors

Justin Howard · October 14, 2026

> Verify NER clause-extraction benchmarks on 2024 SEC 10-K risk factors using source-defined terms and live SEC guide checks for accurate, reliable results.

## How It Works

Key terms matter here, and they should come straight from the source documents. The SEC's Small Entity Compliance Guide defines a "resource extraction issuer" as an issuer required to file an annual report under Section 13 or 15(d) of the Exchange Act that engages in the commercial development of oil, natural gas, or minerals. Before relying on any such definition, open the live SEC guide and confirm the current wording, since the disclosure requirements apply to both U.S. and foreign companies meeting the definition regardless of company size. Those are the entity types a NER-based extraction model must handle: issuer names, payment types, totals, and project-level detail.

The verify-before-you-commit check works like this. Take the model's extracted output on a sample 10-K risk-factor section and compare it line by line against the source text. Confirm that each extracted payment type and total matches the filing verbatim, not approximately. The SEC's own guidance notes that disclosure requirements apply to both U.S. and foreign companies meeting the definition, regardless of company size, so your test set should include filings of varying scale to see whether the model's accuracy holds across them.

Compare like-for-like totals when evaluating benchmark results. If one vendor reports accuracy on a curated subset of filings and another reports on the full universe of 2024 10-Ks, the numbers are not comparable. Ask each provider which documents, which entity types, and which scoring method (exact match versus partial span overlap) produced the figure. Exact-match scoring is stricter; partial-overlap scoring inflates apparent accuracy when a model captures most but not all of a payment total or project name.

Finally, run the same sample through every candidate model before committing. The mechanism only protects you if the input is identical across tests: same filing, same risk-factor section, same reference answers. Record precision and recall separately for each entity type, because a model that excels at company names may lag on payment totals, and the disclosure rules treat both as required elements. Confirm the live, complete option before you commit.

![How It Works — How to Verify NER Clause-Extraction Benchmarks](https://static.mm-ais.com/article-images-ai/how-to-verify-ner-clause-extraction-benc-ai-385a199c.jpg)

## Key Factors to Consider

The numbers that matter fall into four buckets, and you should demand all four before you commit. First, the size of the verified key: how many clauses, drawn from how many filings, across how many issuers. A key built from a handful of filings cannot support a claim about "2024 SEC 10-K risk-factor disclosures" as a class. Second, the label schema counts: number of entity types, number of relation types, and the inter-annotator agreement rate on a held-out sample. Third, the scoring counts: true positives, false positives, and false negatives reported separately, not collapsed into a single F1. Fourth, the split counts: how many clauses were held out for testing versus used for tuning. If the test set overlaps the tuning set, the headline number is inflated.

Run one arithmetic check before you trust any table. Recompute precision as true positives divided by the sum of true positives and false positives, and recall as true positives divided by the sum of true positives and false negatives. If the published precision and recall do not reproduce from the published counts, the benchmark has an internal inconsistency and you should not commit to it. The same discipline applies to any headline figure: confirm the counts behind it before treating the number as evidence.

| Criterion | What to verify | Why it changes your decision |
| --- | --- | --- |
| Answer-key provenance | Annotator count, adjudication rule, source document | Determines whether the score reflects the filed text or a paraphrase |
| Label granularity | Entity types, relation types, agreement rate | Coarse labels inflate recall; fine labels expose real errors |
| Corpus composition | Filings, issuers, clause counts per bucket | A narrow corpus cannot support a class-level claim |
| Split discipline | Test clauses held out from tuning | Overlap between tuning and test inflates every metric |

One rule ties these together: compare like-for-like totals and terms. A benchmark reporting 90% recall on paragraph-level labels is not comparable to one reporting 70% recall on sentence-level obligation fields, even if both cite 2024 10-K risk factors. Match the label schema, the clause unit, and the corpus scope before you put two numbers side by side, and treat any mismatch as a reason to withhold commitment rather than to average the results.

![Key Factors to Consider — How to Verify NER Clause-Extraction Benchmarks](https://static.mm-ais.com/article-images-pixabay/how-to-verify-ner-clause-extraction-benc-1841b36f.jpg)

## Common Mistakes

The first pitfall is scoring a model against a benchmark whose answer key was built from a different document version than the one you intend to run. A 10-K risk-factor section is amended, restated, and re-filed; if the benchmark's gold labels came from the original filing and you feed the model an amended exhibit, every entity that shifted position or wording registers as a miss even when the model read the clause correctly. Concrete example: a benchmark keyed to a filing that lists a payment obligation under one project heading, then re-run against a later version where that obligation is folded into a different heading, will show a recall drop that reflects the filing change, not the model. Before you commit, pull the accession number and filing date for both the benchmark key and your target document and confirm they match; if they do not, the comparison is void.

The second pitfall is comparing a precision figure from one benchmark against a recall figure from another and treating the pair as a single accuracy score. Precision and recall answer different questions, and a model tuned for one will look strong on that number while quietly failing the other. The check is mechanical: confirm both numbers come from the same run, the same answer key, and the same label set before you put them side by side. If a vendor reports only one of the two, ask for the confusion matrix behind it — the counts of true positives, false positives, and false negatives let you recompute both figures yourself and catch a number that was calculated on a subset.

A third trap is letting label granularity drift between the benchmark and your use case. A benchmark that tags an entire risk-factor paragraph as one entity will post high scores against a model you intend to use for sentence-level clause extraction, because the model gets credit for bounding a large block rather than the precise clause. Match the label depth first: if your downstream task needs clause-level spans, verify the benchmark's labels are at that depth before you read any score as evidence.

| Check before committing | What to verify |
| --- | --- |
| Document version | Accession number and filing date match between key and target |
| Metric pairing | Precision and recall drawn from the same run and label set |
| Label granularity | Benchmark span depth equals your extraction depth |

Run all three checks before you accept any benchmark number as a reason to commit. Each one is cheap to perform and each one, skipped, converts a clean comparison into a misleading one. The discipline is the same as verifying any live option: confirm the complete, current version, then compare like-for-like totals and terms before you decide.

![Common Mistakes — How to Verify NER Clause-Extraction Benchmarks](https://static.mm-ais.com/article-images-pixabay/how-to-verify-ner-clause-extraction-benc-393c5192.jpg)

## Insider Tactics

The tactic most teams miss: run your model against the benchmark's answer key in reverse. Instead of asking "what did the model extract from this risk-factor section," ask "for every entity in the answer key, did the model find it, and where did it look?" This flips the evaluation from precision-first to recall-first, and it exposes the failure mode that matters most on 10-K risk factors — models that silently skip the second or third clause in a multi-clause disclosure because the language is boilerplate-adjacent. A model can post a strong headline score while missing a meaningful share of the entities that actually carry legal weight, and you will not see that from the aggregate number alone.

Pair that with a diff-by-position check. When a model's output disagrees with the answer key, log whether the miss is an offset error (right text, wrong span boundary) or a true miss (entity absent entirely). These two failure types demand different fixes — offset errors usually mean your tokenization or annotation guidelines are misaligned, while true misses mean the model needs more domain exposure. Teams that only track a single blended score end up retraining models to solve the wrong problem.

On timing: anchor every benchmark run to a fixed document snapshot, and record the snapshot date alongside your results. SEC filings get amended, and risk-factor sections are among the most commonly revised parts of a 10-K. If you benchmark in one week and commit to a vendor in another, re-run the same suite against the same snapshot before you sign — a model that looked ready before an amendment cycle can drift once the disclosure language shifts. The verify-before-you-commit rule applies to the benchmark itself, not just the model: confirm the answer key matches the exact filing version you will deploy against.

Another non-obvious move is to hold out the filings whose risk factors were amended most recently and score them as a separate slice. Models tend to perform worse on freshly rewritten disclosure language because their training data skews toward older, more formulaic phrasing. If your vendor cannot produce per-slice results, that is a signal about how they evaluate internally, and it is worth asking about before you commit.

Finally, time your benchmark run to the filing calendar rather than your procurement calendar. Run your evaluation suite when the newest 10-K risk-factor disclosures are available in volume, so your benchmark reflects current disclosure patterns instead of last cycle's. Then re-verify the live, complete option — same snapshot, same answer key, same slices — immediately before committing, and compare like-for-like totals only. If the two runs differ materially, treat that gap itself as a finding: it tells you how sensitive the model is to document drift, which no static benchmark score will ever show you.

![caught nature bird how they](https://static.mm-ais.com/article-images-pixabay/how-to-verify-ner-clause-extraction-benc-331d8af8.jpg)
caught nature bird how they

## Comparison

Put the four realistic candidates on one ledger — a fine-tuned span-based NER extractor, a token-classification NER model, a few-shot large-language-model extractor, and a pattern-and-rules baseline — and the comparison only becomes honest after every published score is re-expressed on the same basis. Re-expressing is arithmetic, not trust.

Pull the per-label true positives, false positives, and false negatives out of each benchmark report, then rebuild precision as TP ÷ (TP + FP), recall as TP ÷ (TP + FN), and F1 as 2PR ÷ (P + R) from your own precision and recall. Set micro-F1, which weights every entity in the corpus equally, beside macro-F1, which weights every label equally. When macro-F1 sits above micro-F1, the model's edge lives in the rare labels; when micro-F1 leads, the common labels carry the total. A headline score with no counts behind it cannot be set beside a rival that publishes counts, because relaxed boundary matches and exact matches are different totals. Recompute both figures yourself before you put two numbers side by side.

Token-classification NER takes the lead when the target spans are short, flat, and repeated across filings, and when the scoring key reports each label type separately — then macro-F1 is the number that decides. A rules baseline takes the lead on any subset of clauses triggered by a fixed statutory phrase or a defined term, because the string list makes recall cheap to verify. The SEC's small entity compliance guide shows how tightly one definition can pin a single entity type: a resource extraction issuer is an issuer required to file under Section 13 or 15(d) of the Exchange Act that engages in the commercial development of oil, natural gas, or minerals (sec.gov). A 10-K risk-factor key needs that same definitional anchoring before two scores can share a ledger.

Span-based NER wins when entities nest or run past sentence boundaries, which is the common shape of risk-factor text; it loses the lead only when the schema flattens every clause into one label. Few-shot LLM extraction wins when the label set shifts filing to filing and no labeled training set exists, and it gives up the lead once the schema is stable enough for a supervised model to be trained against it.

| Option | Deciding measure | Wins when | Loses the lead when |
| --- | --- | --- | --- |
| Fine-tuned span-based NER | Clause-level exact match, counts recomputed from TP/FP/FN | Entities nest or cross sentence boundaries | The schema collapses every clause to one label |
| Token-classification NER | Per-label macro-F1 | Spans are short, flat, and repeated | Boundary errors sit inside multi-sentence clauses |
| Few-shot LLM extraction | Agreement across repeated prompt runs on the same sampled clauses | The label set changes per filing and training data is absent | A stable schema rewards a supervised model |
| Pattern and rules baseline | Recall against the fixed string list | A defined term or fixed phrasing triggers the clause | Phrasing varies across issuers |

On 2024 SEC 10-K risk-factor disclosures, the decisive measure is clause-level exact match on the same answer key, and the fine-tuned span-based extractor takes the win. Change your pick only when a candidate leads on that measure with that key; a lead on a different total, or on a different key, is not the same comparison. Before committing, re-run the same sample through every candidate and confirm the input is identical across tests.

## What to do next

| Step | Action | Why it matters |
| --- | --- | --- |
| 1 | Open the live benchmark or model report for the NER clause-extraction run and confirm it covers the complete 2024 SEC 10-K risk-factor disclosure set — not a cached excerpt, partial filing, or summary. | The headline benchmark is scoped to 2024 SEC 10-K risk-factor disclosures; the canonical rule says verify the live, complete option before committing. |
| 2 | Match the model’s clause labels and boundaries to the benchmark’s clause-extraction definition before reading F1. | Like-for-like terms require the same clause definition; otherwise the headline benchmark does not transfer. |
| 3 | Reconcile the model’s total clause count and risk-factor section count against the benchmark’s totals for the 2024 SEC 10-K risk-factor corpus. | Totals that do not align mean the comparison is not like-for-like and the accuracy claim may be inapplicable. |
| 4 | Check the live, complete option’s evaluation split and sample size; confirm the F1 is calculated on the full 2024 SEC 10-K risk-factor scope. | A subset can overstate accuracy and violates the live-complete-option rule before any commitment. |
| 5 | Compare the model’s F1 against the headline benchmark only after the 2024 SEC 10-K risk-factor scope and clause-extraction terms match. | Non-matching scope makes the headline benchmark inapplicable to the live option being evaluated. |
| 6 | Before any commitment, re-check the live benchmark page for updates or corrections, then confirm final totals and terms side by side. | The decision rule requires verifying the live, complete option and comparing like-for-like totals and terms. |

## Frequently Asked Questions

**Which SEC publication defines the term 'resource extraction issuer'?**

The SEC's Small Entity Compliance Guide defines a 'resource extraction issuer' as an issuer required to file an annual report under Section 13 or 15(d) of the Exchange Act that engages in the commercial development of oil, natural gas, or minerals.

**Do the disclosure requirements apply only to U.S. companies?**

The SEC's own guidance notes that disclosure requirements apply to both U.S. and foreign companies meeting the definition, regardless of company size.

**What should I do before relying on the SEC's definition of a resource extraction issuer?**

Before relying on any such definition, open the live SEC guide and confirm the current wording.

**What entity types must a NER-based extraction model handle?**

A NER-based extraction model must handle issuer names, payment types, totals, and project-level detail.

**How closely must extracted payment types and totals match the filing?**

Confirm that each extracted payment type and total matches the filing verbatim, not approximately.

**Why should a test set include filings of varying scale?**

Because disclosure requirements apply to both U.S. and foreign companies meeting the definition regardless of company size, your test set should include filings of varying scale to see whether the model's accuracy holds across them.

## Quick answers

| What is a "resource extraction issuer" according to the SEC's Small Entity Compliance Guide? | An issuer required to file an annual report under Section 13 or 15(d) of the Exchange Act that engages in the commercial development of oil, natural gas, or minerals. |
| --- | --- |
| What should you do before relying on the SEC's definition of a resource extraction issuer? | Open the live SEC guide and confirm the current wording. |
| How does the verify-before-you-commit check work for a NER-based extraction model? | Take the model's extracted output on a sample 10-K risk-factor section and compare it line by line against the source text, confirming each extracted payment type and total matches the filing verbatim, not approximately. |
| Why should your test set include filings of varying scale? | Because the SEC's guidance notes that disclosure requirements apply to both U.S. and foreign companies meeting the definition regardless of company size, so you can see whether the model's accuracy holds across them. |
| What should you watch for when comparing benchmark results across vendors? | Compare like-for-like totals, since one vendor may report accuracy on a curated subset of filings while another reports on the full universe. |

Also worth reading: **PDF Clause Extraction Accuracy: Layout-Aware Parsers 2026 SEC Filings F1 vs Latency**: [PDF Clause Extraction Accuracy: Layout-Aware](https://legalpdf.io/blog/pdf-clause-extraction-accuracy-layout-aware-parsers-2026-sec-filings-f1-vs-latency.php) · **AI Clause Extraction: 92% Precision, But Recall Varies by Region**: [AI Clause Extraction: 92% Precision,](https://legalpdf.io/blog/ai-clause-extraction-92-precision-but-recall-varies-by-region.php) · **Automated Clause Extraction: 89% Precision vs Recall in 2026 Colorado Opinions**: [Automated Clause Extraction: 89% Precision](https://legalpdf.io/blog/automated-clause-extraction-89-precision-vs-recall-in-2026-colorado-opinions.php)

### Related reading

- [PDF Clause Extraction Accuracy: Layout-Aware Parsers 2026 SEC Filings F1 vs Latency](https://legalpdf.io/blog/pdf-clause-extraction-accuracy-layout-aware-parsers-2026-sec-filings-f1-vs-latency.php)
- [Automated Clause Extraction: 89% Precision vs Recall in 2026 Colorado Opinions](https://legalpdf.io/blog/automated-clause-extraction-89-precision-vs-recall-in-2026-colorado-opinions.php)
- [AI-Powered Contract Analysis A Deep Dive into Anti-Disparagement Clause Detection and Risk Assessment](https://legalpdf.io/blog/ai_powered_contract_analysis_a_deep_dive_into_anti_disparage.php)
- [Long-Context Contract NLP: Benchmarks Over Leaderboards](https://legalpdf.io/blog/long-context-contract-nlp-benchmarks-over-leaderboards.php)
- [AI-Powered Video Analysis in eDiscovery Revolutionizing Legal Evidence Extraction](https://legalpdf.io/blog/ai_powered_video_analysis_in_ediscovery_revolutionizing_lega.php)
- [Understanding Arkansas Battery in the Third Degree Key Penalties and Aggravating Factors in 2024](https://legalpdf.io/blog/understanding_arkansas_battery_in_the_third_degree_key_penal.php)

### Latest

- [PDF Clause Extraction Accuracy: Layout-Aware Parsers 2026 SEC Filings F1 vs...](https://legalpdf.io/blog/pdf-clause-extraction-accuracy-layout-aware-parsers-2026-sec-filings-f1-vs-latency.php)
- [Automated Clause Extraction: 89% Precision vs Recall in 2026 Colorado Opinions](https://legalpdf.io/blog/automated-clause-extraction-89-precision-vs-recall-in-2026-colorado-opinions.php)
- [Public Agency PDFs: Check DOJ Title II Accessibility Standards and Deadlines](https://legalpdf.io/blog/public-agency-pdfs-check-doj-title-ii-accessibility-standards-and-deadlines.php)

Canonical: https://legalpdf.io/blog/how-to-verify-ner-clause-extraction-benchmarks-on-2024-sec-10-k-risk-factors.php
Markdown: https://legalpdf.io/blog/how-to-verify-ner-clause-extraction-benchmarks-on-2024-sec-10-k-risk-factors.php/index.md