Measure legal-agent rework, not just output volume

How can a legal team measure agent rework rather than output volume?

Count comparable reviewed tasks, record their disposition, and measure correction effort and error severity. Report the denominator, review coverage, and task mix beside every rate. Draft count or words produced cannot show whether an agent reduced the team's total work.

Define the unit and outcomes before counting

Use a stable task unit such as one contract issue summary reviewed against a defined standard. Record whether the task was accepted on first review, accepted after correction, rejected, or left unreviewed. NIST's GAI profile cautions against generalizing capabilities from narrow or anecdotal assessments. [S1]

• First-pass acceptance rate = tasks accepted without substantive correction / reviewed tasks.
• Rework rate = reviewed tasks requiring substantive correction or replacement / reviewed tasks.
• Correction effort = human minutes spent repairing substantive defects.
• Review coverage = reviewed completed drafts / all completed drafts.
• Critical error rate = reviewed tasks containing a defined critical error / reviewed tasks.

Synthetic cohort: forty reviewed summaries

A fictional team reviews 40 comparable summaries: 28 pass initially, 8 need correction, and 4 are rejected and replaced. First-pass acceptance is 28/40 = 70%; rework is 12/40 = 30%. The team spends 96 minutes correcting the eight and 120 minutes replacing the four, giving 216 correction/replacement minutes, or 5.4 minutes per reviewed task.

• If 10 other drafts remain unreviewed, coverage is 40/50 = 80%; their quality is unknown.
• Do not label the unreviewed ten as successful or failed.
• Report the 12 affected tasks separately from the 216 minutes; frequency and burden differ.
• Preserve the cohort definition so the next month is comparable.

Separate friction from consequential defects

A formatting repair and a fabricated authority should not cancel each other in an average score. Define severity before scoring, use examples, and have reviewers calibrate disagreements. Also distinguish a model defect from missing input, changed scope, outdated instructions, or reviewer preference.

• Critical: a defect that could materially mislead a decision or cause an unauthorized external action.
• Substantive: an incorrect or unsupported proposition requiring professional correction.
• Minor: presentation or clerical work that does not change meaning.
• Unresolved: reviewers lack evidence to assign an outcome.
• Track both the source of the defect and the step that detected it.

Compare total effort fairly

A proposed pilot compares similar tasks with and without the agent, measuring preparation, review, correction, and completion time. Avoid claiming causation from a single before-and-after week with a changed task mix. Include abandoned work and exception handling; otherwise the apparent improvement may merely move work out of the measured queue.

• Report medians and ranges alongside totals when task durations vary.
• Separate waiting time from active human effort.
• Hold review criteria constant or explain the change.
• Present savings as measured, estimated, or unestablished.
• Keep accuracy and critical-error measures visible even if speed improves.

Scope and limitations

Original measurement definitions and invented data. The metrics are proposals, not industry benchmarks, validated savings, or measures of legal-service quality by themselves.

No production telemetry or measured legal outcome was analyzed. The arithmetic is reproducible, but the example is synthetic and cannot establish ROI.

Sources

1. NIST AI 600-1 — Generative Artificial Intelligence Profile — GAI risk and evaluation guidance; Measures 2.3/2.5 address capability evidence, generalization limits, and source verification. Checked 2026-11-05

Explore further

These metrics can make a workflow assessment with Agentic Attorney more concrete: https://agenticattorney.com/ . No performance claim about that platform is derived from this invented cohort. Explore Agentic Attorney.