Executive Summary
A client's document extraction system worked on straightforward documents but became unreliable on complex tables, scanned pages, and unfamiliar layouts, often producing outputs that looked complete while values were missing, duplicated, or assigned to the wrong row. GenAI Protos evaluated OCR, document parsing, layout-aware, and vision-language approaches on a de-identified sample of the client's documents to pinpoint where extraction failed, why, and which approach warranted further development.
01Client Requirement
The engineering team needed to understand why extraction was failing and whether a different approach would improve reliability. Spot checks had revealed problems, but the team lacked a consistent way to compare tools or estimate how much human review each option would require. A readable output was not necessarily an accurate one.
- The selected approach and the evidence supporting that choice
- Detect missing, duplicated, transposed, and incorrectly attributed content
- Check whether tables retained their rows, columns, and relationships
- Trace extracted values back to the source page and region
- Establish when the system should return a result or send it for review
- Support a decision on the next development stage
02Our Approach
We started with the documents that caused problems. Working with the client, we assembled a de-identified test set grouped by document type, layout complexity, and scan quality, and labeled the expected values and table structures, including where each item appeared in the source.
We then ran selected extraction approaches against the same test set and compared results by document group, so strong performance on clean pages would not hide failures on difficult ones. Each error was classified: a missing value was recorded separately from a correct value attached to the wrong field or table row. This showed whether an approach struggled with reading text, understanding layout, or preserving relationships.
We also tested confidence thresholds where available, along with rules for flagging incomplete or inconsistent results, and measured both the errors that escaped these checks and the volume sent to human review. The engagement remained a focused evaluation, with reusable test assets and documented findings.
- Exact and normalized field accuracy
- Omission, duplication, and misattribution rates
- Preservation of table rows and cells
- The proportion of documents with every required field correct
- Processing time and estimated cost
03Outcome
The evaluation gives the client a clear, evidence-based view of which extraction approach performs best for each document group, where failures remain, and how much human review is required at the tested thresholds. The client receives:
If no approach meets the agreed requirements, the finding is recorded along with the conditions that would need to change before further investment.
- The selected approach and the evidence supporting that choice
- Accuracy and failure rates for the most difficult document groups
- Incorrect results that passed without being flagged
- Review volume, latency, and estimated processing cost
- A go/no-go recommendation for a bounded next stage
- A reusable, labeled test set for future regression checks