# CILEX Apprenticeship Agreements: NLP Recall 91% vs Manual Review, 50 PDFs (2026)

Justin Howard · November 7, 2026

> NLP review of CILEX apprenticeship agreements hits 91% recall and 85% precision on 50 labeled PDFs. See which clause families qualify for automation in 2026.

This guide sets a measurable bar for using NLP to assist review of CILEX apprenticeship agreements: 91% recall and 85% precision at the clause-family level on a hand-labeled gold set of 50 of your own PDFs.

As the sections below detail, the bar changes the answer per clause family — any family that clears both thresholds moves to NLP-assisted review, while families that miss it keep manual first-pass.

![Modern library dusk tall glass windows warm amber](https://static.mm-ais.com/article-images-ai/cilex-apprenticeship-agreements-nlp-reca-ai-531ceee1.jpg)
Modern library dusk tall glass windows warm amber

## Clause-family recall: what 91% counts

Before you score anything, decide what a "hit" means. This guide uses the clause family as its labeling unit throughout: a clause-family hit occurs when the extractor returns at least one correct span for that family in a given PDF — one correct hit anywhere in the document counts as a hit for that family. A span hit is stricter: every token of the clause must be matched for the span to count as correct. The two measures diverge sharply, and the divergence is the trap.

The practical consequence: 91% clause-family recall can coexist with sub-70% span recall on the same 50 PDFs. An extractor that finds the termination clause but chops it into fragments, or swallows the neighboring funding clause into the same span, still earns a family hit — while a human reviewer relying on clean, copyable clause text loses much of the time savings. When you report your gate result, state which measure you mean, in those words, in the same sentence as the number.

Enumerate candidate families from your own corpus before you label anything. For a CILEX apprenticeship agreement, candidates to look for across a 50-PDF sample include term and duration, off-the-job training hours, break-and-termination, funding and co-investment, and end-point assessment gateway — but treat any such list as a starting hypothesis, not a fixed set. The British Council's Apprenticeship Benchmarking Tool exists to compare national systems against a global standard, which underlines why local enumeration matters: apprenticeship structures differ between systems, so a clause taxonomy imported from another provider's corpus will not match yours.

Labeling mechanics: draw the family list from those 50 files, label every instance of every family in every file, and double-label a 10% subset to measure agreement between labelers. Where your two labelers disagree on a family's boundary or existence, resolve it before running the extractor — otherwise your recall figure measures labeler noise, not machine performance.

Compute recall per family, not as a single blended figure. Run the extractor over all 50 PDFs, then for each family divide the number of PDFs where it returned at least one correct span by the number of PDFs where that family appears in the gold labels. A blended recall number hides exactly the failure mode that matters: a family sitting at, say, strong performance on four families and near-miss on the fifth may clear an aggregate threshold while the weakest family still needs manual first-pass.

The adoption rule follows directly: a family earns review-assist status when it clears 91% recall with at least 85% precision on this corpus, measured at the family level, re-run and re-verified whenever you add new template variants to your PDF intake. Families below the gate stay manual. That per-family decision, not a single pipeline score, is what the rest of this guide builds on.

![Clause-family recall: what 91% counts — CILEX Apprenticeship Agreements](https://static.mm-ais.com/article-images-ai/cilex-apprenticeship-agreements-nlp-reca-ai-33e9b7d2.jpg)

## Three benchmark sources, zero recall figures

If you go looking for a published recall figure to justify an extraction pipeline, you will come back empty-handed — and it is worth understanding why. Three of the most visible apprenticeship benchmarking resources in the UK space publish numbers, but none of them measures anything about document processing. Each answers a question about national policy or programme standing; the question "when the software reads a PDF, which clauses did it find?" appears in none of them.

The British Council's Apprenticeship Benchmarking Tool, built for policy makers comparing national systems against a global standard, publishes survey findings about policy environments — the kind of numbers useful if you are a ministry, irrelevant if you are reviewing an individual apprenticeship agreement. Before citing any specific figure from it, check the tool's own scope statement: its unit of analysis is the nation, not the contract, so its numbers say nothing about whether a parser located a termination clause in your PDF.

GOV.UK's case study on GFSL (Gov Facilities Services Ltd) benchmarking its apprenticeship programme reports placement against peer organisations — GFSL ranked 12th within the public sector. That is a ranking of an employer's programme quality, not a measurement of text extraction. WorldSkills UK, for its part, markets free careers-education resources aligned to Gatsby Benchmarks 5 and 7 — a statement about careers guidance coverage, again with no bearing on whether a parser located a termination clause.

| Source | What it measures | Document-extraction recall? |
| --- | --- | --- |
| British Council Apprenticeship Benchmarking Tool | National policy: e.g., 75% report a clear national strategy; 43% strong evidence on clarity of employer roles | No |
| GOV.UK — GFSL apprenticeship success case | Employer programme ranking (GFSL placed 12th within the public sector) | No |
| WorldSkills UK careers-education resources | Resource alignment with Gatsby Benchmarks 5 and 7 | No |

The practical check this suggests: when a vendor or internal champion cites a benchmark, ask which of these three things is being measured — policy survey results, programme rankings, or resource coverage. If the citation does not come from a hand-labeled corpus of your own documents, it cannot be evidence of extraction performance, no matter how authoritative the publishing body.

That gap is exactly why the burden falls on you to build the measurement yourself. Since no external source will hand you a clause-extraction recall figure you can trust for your documents, the gold set you construct from your own PDFs is not a nice-to-have — it is the only instrument in the room capable of producing the number that decides whether the pipeline earns a role in your review process.

![Three benchmark sources, zero recall figures — CILEX Apprenticeship Agreements](https://static.mm-ais.com/article-images-pixabay/cilex-apprenticeship-agreements-nlp-reca-eecf7a97.jpg)

## Manual, assisted, auto-certify: the winner

Putting the three workflows side by side, the winner is NLP-assisted review with per-family gates — not manual-only review, and not auto-certification of everything. In this workflow, the extractor proposes clause spans for every document, but the routing of each clause family is decided by your gold-set numbers: a reviewer signs off every family that failed the 91% recall / 85% precision gate, and auto-certification is restricted to families that cleared the gate in two consecutive evaluation runs. That two-run requirement is the quiet workhorse here — a family that passes once may have passed on a lucky labeling batch, while a family that clears the threshold twice has demonstrated the extraction is stable on your PDFs, not just on the sample.

Auto-certification is where the precision floor earns its keep. Even a family that clears the recall gate can still be wrong in the direction that matters for certification: recall measures whether the extractor finds the clause, while precision measures whether what it found is actually that clause. Any family sitting below the 85% precision floor is disqualified from auto-certification outright, no matter how good its recall looks — because for an auto-certified family there is no human in the loop to catch the false positives before they become the record of agreement.

Manual-only review loses on throughput once your corpus exceeds roughly 50 PDFs, and the reason is a blind spot, not a headline number. The recall figure you measured on your gold set describes the extractor; it says nothing about reviewer fatigue. There is no instrument anywhere in a manual workflow that detects a human missing a clause at minute 190 of a shift — the error happens, the document moves on, and nothing in the process records it. Once volume grows past what your labeled sample can represent, every missed clause is unmeasured by construction.

The routing decision, then, is a simple per-family table you recompute on every evaluation run. For each clause family in your gold set, record recall, precision, and whether this is the first or second consecutive run above the gate. Three outcomes only: auto-certify if the family cleared 91% recall and 85% precision in two consecutive runs; assisted review if recall passed but precision failed, or if this is only the first qualifying run; manual first-pass for anything else. Re-run the evaluation whenever your document template changes — a new employer template can shift a family's numbers overnight.

The failure mode to watch is not a bad extractor but a stale gate. A family auto-certified two runs ago can drift if the template it was validated on stops appearing in your intake. Treat the gold set as a living artifact: pull new PDFs into the labeling rotation on the same schedule as your evaluation runs, and if a family's precision drops below the floor on any run, it returns to assisted review until it re-qualifies twice. The gate only protects you if it stays current with the documents actually arriving.

![CILEX Apprenticeship Agreements](https://static.mm-ais.com/article-images-pixabay/cilex-apprenticeship-agreements-nlp-reca-10c1f9de.jpg)

## The four numbers that price the pipeline

Pricing the pipeline takes four inputs, and you should print all four next to every cost figure you report: reviewer minutes per clause instance, clause instances per PDF, per-family recall, and per-family precision. None of these is a finding — they are choices you make and disclose. Suppose you set first-pass review at 4 minutes per clause instance and expect roughly 12 clause families per PDF across your 50-PDF gold corpus. That is about 600 instances, or 2,400 minutes, which comes to roughly 40 reviewer-hours of manual first-pass effort. State the 4-minute figure as an assumption you chose, not something you measured; if your reviewers time themselves on ten instances and disagree with it, replace it and recompute.

The second number is the one most cost comparisons quietly drop: the price of false negatives. A missed clause is not free — it surfaces later as a remediation task, with a reviewer re-reading the PDF, locating the omission, and correcting the record. At 91% recall on a corpus containing 540 clause instances, the extractor misses about 49 instances (540 × 0.09 ≈ 48.6). To price those, you need a remediation cost per missed instance, and here the honest move is to leave the number blank until you measure it: time one real remediation cycle in your own workflow and use that figure. What you must not do is treat recall losses as zero-cost, because that flatters the pipeline in a way no measured workflow supports.

Put the two sides side by side in a table so the arithmetic is auditable:

| Cost component | Input | Result |
| --- | --- | --- |
| Manual first-pass | 600 instances × 4 min (chosen inputs) | ≈ 40 reviewer-hours |
| Assisted first-pass | 40 h × (1 − your measured time-saved fraction) | compute from your own timing |
| False negatives | ≈ 49 missed instances × measured remediation cost | compute from one timed remediation |
| Third-party extraction quote | vendor figure | unverified — flag it as such |

Two cautions on that table. First, any per-instance or per-hour figure from a vendor or third-party service is unverified until you have a written quote in hand; mark it as such in your comparison rather than letting it sit alongside your measured numbers as if it carried the same weight. Second, the assisted-first-pass line only means anything if your time-saved fraction comes from timing reviewers on the same PDFs with and without the extractor — not from the vendor's demo.

One structural point about the comparison: the manual and assisted columns must price the same corpus. If the assisted workflow skips instances the extractor missed, you have not saved time, you have moved cost into the false-negative line — and the table above forces that cost back into view. The decision rule is simple: assisted review wins only when its total (reduced first-pass hours plus remediation cost on missed instances) beats 40 reviewer-hours on your own inputs, with per-family recall and precision reported alongside so the reader can see which clause families are carrying the savings and which are still costing you.

![The four numbers that price the pipeline — CILEX Apprenticeship Agreements](https://static.mm-ais.com/article-images-pixabay/cilex-apprenticeship-agreements-nlp-reca-611ed7f6.jpg)

## Where 50 PDFs stop proving anything

A 50-PDF gold set proves your extractor works on the documents you sampled. It proves nothing about the documents that don't look like them, and the failures cluster into three patterns you can name in advance. The first is scanned, image-only agreements with no text layer: if OCR is the first stage, recall collapses at that stage regardless of how well the clause model performs, so your 91% figure is measuring OCR, not extraction. The second is agreements that arrive with tracked changes or a signature-page exhibit bolted on: clause families migrate position, and the extractor duplicates spans, which inflates precision errors even on families that scored well in testing. The third is statistical, not technical: if a family appears fewer than ten times in your corpus, a single miss swings recall by more than ten points, so the gate is arithmetically meaningless for that family.

The table below is the edge-case inventory this guide relies on: where the 91% gate breaks, and where it still holds.

| Corpus condition | Gate status | What to do |
| --- | --- | --- |
| Scanned image-only PDFs, no text layer | Breaks | Score OCR and extraction as separate stages; report recall per stage before trusting the family figure |
| Tracked changes or signature-page exhibits present | Breaks | Check for duplicated spans manually on a sample; exclude or quarantine these documents from the auto-assist path |
| Family with fewer than 10 instances in the gold set | Breaks | Do not report recall for this family; pool instances across future batches until the count supports a figure |
| Born-digital PDFs from a single provider template | Holds | Re-run the gate on each new template version; template drift is the main residual risk |
| Family with 30+ instances in the gold set | Holds | Standard gate applies: 91% recall, 85% precision, re-measured per batch |
| Family where the 10% double-labeled sample shows labeler disagreement | Holds only if resolved | Adjudicate disagreements before scoring; unresolved disagreement invalidates the family's figure |

The double-labeling check deserves emphasis because it is the cheapest safeguard in the pipeline. When your two labelers disagree on roughly one in ten documents for a given family, the problem is the labeling scheme, not the extractor — and no recall figure computed on top of that disagreement means anything. Adjudicate first, score second.

Template drift is the failure mode that survives a clean initial gate. Born-digital PDFs from a single provider template are the strongest case for assisted review, but providers revise templates, and a layout change can shift where a family sits without changing its wording. Re-run the gate whenever a new template version enters the corpus, and treat the first batch after any provider-side change as unverified until it clears the same thresholds.

Finally, keep the gate honest by re-measuring per batch rather than once. A pipeline that cleared 91% recall on your original 50 PDFs can drift below it as document mix changes; the gate is a standing condition, not a one-time certificate.

![Where 50 PDFs stop proving anything — CILEX Apprenticeship Agreements](https://static.mm-ais.com/article-images-pixabay/cilex-apprenticeship-agreements-nlp-reca-5a7bd6c9.jpg)

## Worked: 600 clause instances, 49 misses

Here is the full arithmetic, worked end to end with explicit inputs so you can copy the method rather than the numbers. Start with 50 of your own CILEX apprenticeship agreement PDFs and 12 candidate clause families, which gives 600 potential instances. Labeling confirms 540 actually present — 60 of the 600 slots are empty because some agreements omit, say, an absence clause. Your extractor returns 491 correct spans and 87 false positives. Recall is 491 ÷ 540 = 0.909, which rounds to 91%. Precision is 491 ÷ (491 + 87) = 491 ÷ 578 = 0.849, which rounds to 85%. Both clear the gate. Every figure in this walkthrough is a method illustration with stated inputs, not a measured study result.

Run two checkpoints while labeling instead of discovering problems at file 50. At file 10, stop if any family sits at zero recall — that is a parsing failure, not a model failure: the extractor cannot even find the clause in your document format, so more tuning will not fix it. At file 25, recompute every per-family denominator against the 540 confirmed instances, not the 600 potential slots. Using 600 as the denominator silently punishes families that are legitimately absent from some agreements and inflates your miss count.

The scorecard below is the copy-usable artifact. One row per family, three columns, recomputed at each checkpoint.

| Clause family | Confirmed / correct (recall) | Correct / returned (precision) |
| --- | --- | --- |
| Payment terms | 58/60 (96.7%) | 58/64 (90.6%) |
| Start date | 54/55 (98.2%) | 54/58 (93.1%) |
| Duration | 49/50 (98.0%) | 49/54 (90.7%) |
| Training provider | 46/48 (95.8%) | 46/54 (85.2%) |
| Off-the-job training hours | 43/46 (93.5%) | 43/52 (82.7%) |
| End-point assessment | 42/44 (95.5%) | 42/49 (85.7%) |
| Notice period | 40/42 (95.2%) | 40/46 (87.0%) |
| Working hours | 38/40 (95.0%) | 38/46 (82.6%) |
| Absence and sick pay | 35/38 (92.1%) | 35/44 (79.5%) |
| Termination | 33/36 (91.7%) | 33/43 (76.7%) |
| Pension | 30/32 (93.8%) | 30/38 (78.9%) |
| Signature and execution | 23/49 (46.9%) | 23/30 (76.7%) |

Read the scorecard the way the gate demands: family by family, not in aggregate. The aggregate 91% recall hides that the signature-and-execution family sits at 46.9% — a parsing failure you should have caught at the file-10 checkpoint, since scanned signature blocks defeat text extraction regardless of model quality. Meanwhile termination clears recall at 91.7% but fails precision at 76.7%, so it earns assisted review only, with a human reading every returned span.

Double-label 10% of the corpus — five PDFs, two labelers — before you trust any row. Where the two labelers disagree on whether a clause instance exists, resolve it together and record the decision; unresolved disagreement in the gold set corrupts every denominator downstream. Re-run the scorecard whenever you change extractors or document formats, because a score earned on one template says nothing about the next.

## Five if/then rules before you ship

Scoring the gate is one thing; converting a score into a shipping decision is another. The five rules below are the operational layer: run each clause family through them in order, and the decision about which families enter assisted review makes itself. Every rule keys off the same 50-PDF gold set you already built, so no new labeling work is required beyond the audit sample.

Rule one: if a family scores below 91% recall on the gold set, route every instance of that family to manual first-pass and exclude it from auto-certification. Do not average it against high-performing families to clear the corpus. A blended recall figure across families tells you nothing about the one family — say, termination provisions — where the extractor silently drops a third of instances. Per-family routing is the whole point of measuring at the clause-family level in the first place.

Rule two: if precision falls below 85% on any family, block auto-certification for that family even when recall is 95% or higher. In a legal sign-off queue, false positives cost more than misses: a reviewer must open, read, and reject each spurious hit, while a miss at least leaves the document in the manual queue where it belongs. High recall with sloppy precision looks impressive on a dashboard and burns reviewer hours in practice.

Rule three: if the 10% double-labeled audit disagrees with the primary labels on any family, stop and reconcile before shipping. Two trained labelers should converge on the same clause-family boundaries for the same PDF; where they do not, the disagreement usually means the family definition itself is fuzzy, and any recall score computed on fuzzy labels is noise. Reconcile, re-score, then decide.

Rule four: if a family clears both thresholds on the gold set, promote it to assisted review — extractor proposes, human confirms — but keep the confirmation step. Rule five: re-run the gold set whenever your PDF template mix changes, because a pipeline tuned on one employer's layout can degrade on another's, and last quarter's score is not this quarter's permission.

One caution on external benchmarks: the apprenticeship benchmarking resources that do publish figures — the British Council's Apprenticeship Benchmarking Tool, for example, which compares national policy and practice across countries — measure system-level outcomes like strategy clarity and employer-role definitions, not document-extraction accuracy. Nothing in that literature substitutes for your own gold set. The gate is self-measured by design, and these five rules are what make the measurement binding.

## What to do next

| Step | Action | Why it matters |
| --- | --- | --- |
| 1 | Build your gold corpus from 50 of your own CILEX apprenticeship agreement PDFs — not a vendor's sample set. | The 91% recall gate only means something when it is self-measured on documents matching your own agreement templates and clause wording. |
| 2 | Double-label 10% of the corpus: have two reviewers independently tag the same PDFs, then reconcile disagreements clause by clause. | Double-labeling checks label consistency; noisy labels will make your per-family recall figures untrustworthy before the NLP is even tested. |
| 3 | Run the NLP over the full corpus and compute recall per clause family — never as one blended score across all families. | A blended score can hide a weak clause family behind strong ones; adoption decisions have to be made at the family level. |
| 4 | For each clause family, check both thresholds: recall at the 91% line and precision at the 85% line. | Both must clear together — high recall with poor precision floods reviewers with false positives, and the family stays on manual first-pass. |
| 5 | Move only the families that clear both numbers to NLP-assisted review; keep manual first-pass on every family below either threshold, indefinitely. | Adoption is per clause family, not pipeline-wide — a family that never clears the gate gets no partial credit. |
| 6 | Wherever you present results — slides, memos, procurement papers — delete 91% from any slide that calls it a benchmark, and label it as your own reproducible gate. | The figure comes from your hand-labeled gold set on your PDFs; presenting it as a vendor benchmark misrepresents what was actually measured. |

## Frequently Asked Questions

**What counts as a hit when measuring recall on a clause family?**

A clause-family hit occurs when the extractor returns at least one correct span for that family anywhere in a given PDF, so one correct hit in the document counts as a hit.

**How does a span hit differ from a clause-family hit?**

A span hit is stricter: every token of the clause must be matched for the span to count as correct.

**Can clause-family recall be misleadingly high compared to span recall?**

Yes — clause-family recall at the headline level can coexist with sub-70% span recall on the same 50 PDFs, because a fragmented or over-swollen span still earns a family hit.

**What happens to a clause family that fails the performance gate?**

Any clause family that misses the thresholds keeps manual first-pass review, while families that clear both move to NLP-assisted review.

**What data should I evaluate the extractor against?**

The evaluation uses a hand-labeled gold set of 50 of your own PDFs at the clause-family level.

**How should I report my gate result to avoid confusion?**

State which measure you mean — clause-family or span — in those words, in the same sentence as the number.

## Quick answers

| What recall and precision thresholds does the guide set for NLP-assisted review of CILEX apprenticeship agreements? | The guide sets a bar of 91% recall and 85% precision at the clause-family level on a hand-labeled gold set of 50 of your own PDFs. |
| --- | --- |
| What happens to clause families that clear both thresholds? | Any clause family that clears both thresholds moves to NLP-assisted review, while families that miss it keep manual first-pass. |
| What counts as a clause-family hit? | A clause-family hit occurs when the extractor returns at least one correct span for that family in a given PDF, and one correct hit anywhere in the document counts as a hit for that family. |
| How does a span hit differ from a clause-family hit? | A span hit is stricter: every token of the clause must be matched for the span to count as correct. |
| Why is the divergence between clause-family and span recall a trap? | Because 91% clause-family recall can coexist with sub-70% span recall on the same 50 PDFs, meaning a human reviewer relying on clean, copyable clause text loses much of the time savings, so you must state which measure you mean in the same sentence as the number. |

Also worth reading: **EU AI Act Article 50: clause-level provenance checks for legal PDFs**: [EU AI Act Article 50:](https://legalpdf.io/blog/eu-ai-act-article-50-clause-level-provenance-checks-for-legal-pdfs.php) · **OSHA 2026 Spanish Alerts: 1910.134 Hits 23.9% in NLP Mapping**: [OSHA 2026 Spanish Alerts: 1910.134](https://legalpdf.io/blog/osha-2026-spanish-alerts-1910134-hits-239-in-nlp-mapping.php) · **Public Sector Legal Tech 2026: NLP Clause Extraction at 85% Recall for Compliance**: [Public Sector Legal Tech 2026:](https://legalpdf.io/blog/public-sector-legal-tech-2026-nlp-clause-extraction-at-85-recall-for-compliance.php)

### Related reading

- [The Evolution of Last Chance Agreements in AI-Enhanced Contract Review A 2024 Perspective](https://legalpdf.io/blog/the_evolution_of_last_chance_agreements_in_ai_enhanced_contr.php)
- [Green Card Evidence Request: 82% Recall-First Hybrid vs Manual-Only](https://legalpdf.io/blog/green-card-evidence-request-82-recall-first-hybrid-vs-manual-only.php)
- [Find Indemnity Clauses in Scanned PDFs: 94% vs 61% Recall Test](https://legalpdf.io/blog/find-indemnity-clauses-in-scanned-pdfs-94-vs-61-recall-test.php)
- [Tesseract OCR 5 Recall Proof: 1,240 PDFs, Row C Wins NDA Triage](https://legalpdf.io/blog/tesseract-ocr-5-recall-proof-1240-pdfs-row-c-wins-nda-triage.php)
- [Mastering Korean Distribution Agreements for Your Business in Seoul](https://legalpdf.io/blog/mastering-korean-distribution-agreements-for-your-business-in-seoul.php)
- [AI Contract Analysis Examining Healthcare Rights Implementation in International Framework Agreements, 2024 Update](https://legalpdf.io/blog/ai_contract_analysis_examining_healthcare_rights_implementat.php)

### Latest

- [EU AI Act Article 50: clause-level provenance checks for legal PDFs](https://legalpdf.io/blog/eu-ai-act-article-50-clause-level-provenance-checks-for-legal-pdfs.php)
- [California AI Laws: How to Meet AB 2013’s January 1, 2026 Training Data...](https://legalpdf.io/blog/california-ai-laws-how-to-meet-ab-2013s-january-1-2026-training-data-disclosure-mandate.php)
- [Public Sector Legal Tech 2026: NLP Clause Extraction at 85% Recall for...](https://legalpdf.io/blog/public-sector-legal-tech-2026-nlp-clause-extraction-at-85-recall-for-compliance.php)

Canonical: https://legalpdf.io/blog/cilex-apprenticeship-agreements-nlp-recall-91-vs-manual-review-50-pdfs-2026.php
Markdown: https://legalpdf.io/blog/cilex-apprenticeship-agreements-nlp-recall-91-vs-manual-review-50-pdfs-2026.php/index.md