Document Intelligence System
Turning a continuous stream of Indian corporate filings into structured data that downstream analytics and AI products can actually depend on.
Can filings that follow no common format become data you can trust?
- · Overview
- · The problem
- · Why it was uncertain
- · How the system evolved
- · Working with the customer
- · Key engineering decisions
- · Outcome
- · Lessons
Overview
A client needed to turn a continuous stream of Indian corporate filings into structured data that downstream analytics and AI products could actually depend on. The difficulty was never reading text out of a PDF. It was determining whether any system could handle the diversity of these documents reliably enough to sit in production.
DistrictD had not built a system for this problem before. There was no internal architecture to follow and no prior evidence that the problem was tractable at the required reliability. So it started as an experiment rather than a project: a short exploratory engagement with one engineer, meant to answer a feasibility question.
What began as a short exploratory effort evolved into a long term engineering engagement as each iteration demonstrated that broader classes of documents could be handled reliably.
The problem
Public companies do not publish information in a common format. They publish in their own format.
A single week of filings might include:
- ·A two page regulatory disclosure with one table
- ·An annual report running several hundred pages
- ·A DRHP or RHP prospectus with risk factors, promoter backgrounds, and competitive landscape sections
- ·An investor presentation where the numbers live inside charts rather than text
- ·A concall transcript structured as speaker attributed conversation
- ·A credit rating report with agency specific layout conventions
- ·A scanned document with no text layer at all
Layout varies as much as length. Some companies publish vertical tables, some horizontal. Some tables have ruled borders, some have none. Some critical values exist only inside embedded images. Two companies filing the same regulatory disclosure will structure it differently, and both are valid.
Traditional extraction pipelines assume documents follow predictable patterns and degrade gracefully when they do not. These documents break both assumptions. There is no dominant pattern to encode, and silent degradation is the worst available failure mode, because a wrong number that parses cleanly is more dangerous than no number at all.
Why it was uncertain
The project was not handed over with a solution to implement. The open question was whether reliable automation was possible across this document ecosystem at all.
That uncertainty shaped how the work was staffed and run. Rather than committing a team to an unproven architecture, the engagement began as a single engineer exploration with a short initial scope. The first deliverable was not a system. It was evidence.
Every extension of the timeline was earned by a demonstration that the previous limitation had been removed.
How the system evolved
The architecture was never designed up front. Each version exists because reality exposed a specific limitation in the version before it.
v0.1 — Raw extraction baseline
Everything treated as either plain text or a table. Deliberately crude, built for coverage rather than quality, to prove that content could be retrieved at all.
What it revealed. the document diversity problem, immediately. A single generic extractor was never going to hold.
v0.2 — Library combination
pdfminer, pdfplumber, and camelot combined to classify text and tables and to segregate document sections.
Where it broke. camelot was excellent at table detection and too aggressive with it. Page borders and margins were flagged as tables, while borderless tables were missed entirely. Structurally the output was defensible. Semantically it was noise.
v0.3 — LLM patched onto library output (failed)
An LLM was introduced to fix labeling and classification. A Microsoft instruction following model produced inconsistent output, Qwen performed better, and Mistral was also evaluated in house.
Why it failed architecturally rather than model wise. the extraction output and the LLM's context had not been designed for each other. Bolting a language model onto a pipeline built for a different consumer does not produce a language model pipeline. This was the point where it became clear the architecture had to be rebuilt rather than extended.
v1.0 — Markdown first pipeline
The pivot was to stop feeding the LLM raw library output, and instead convert the PDF into layout aware markdown first, using PyMuPDF4LLM and PyMuPDF-Layout, preserving column structure and reading order. The LLM's job became labeling and restructuring a document that already carried spatial context.
Why it mattered. this addressed the model's visual blindness. It was no longer inferring structure from flattened text, because the structure arrived with the input.
What remained. model confidence scores were unreliable. The system was frequently confident and wrong, which meant self reported confidence could not be used as a quality gate.
v1.1 — Chunking with overlap
Annual reports and DRHPs run to hundreds of pages. Sending them whole was impossible on context limits and undesirable on attention degradation.
Where naive chunking broke. table continuity. A table spanning pages 7 to 9 split across a chunk boundary would be registered as two unrelated tables. Chunks also produced truncated paragraphs and duplicate tables.
The fix. an overlapping pages strategy, where each chunk carries the previous chunk's final page so incomplete structures are detectable, paired with aggregation that merges tables by (title, header) key rather than concatenating them.
A second failure surfaced here. hallucination driven by prompt complexity. Each additional logical branch in the prompt added another surface for the model to contradict itself. A two pass review approach, returning output to the model for correction, did not meaningfully help. Constraining and simplifying the prompt did.
v2.0 — OCR branch
Testing against client provided documents produced a new failure class. Three of four PDFs extracted acceptably. The fourth, a scanned filing with no text layer, returned nothing usable.
OCR was not a drop in solution. Integrating pytesseract surfaced two further problems. It was substantially slower than native extraction, which required page level detection so only pages that needed OCR received it. And on low resolution scans it returned confident gibberish. Upscaling images before OCR recovered readable text from borderline scans. For pages too degraded to recover, the pipeline detects the gibberish and reports it rather than emitting it.
The principle that came out of this. on low quality input, naive OCR is worse than no OCR, because it converts a visible gap into an invisible error.
v2.1 — Per page quality scoring and audit reporting
A PageQualityScorer evaluates each page using real word ratios, gibberish pattern matching, and financial anchor detection, assigning a CLEAN, DEGRADED, or GIBBERISH rating that is carried into the output JSON and a per page report.
This turned a black box pipeline into a debuggable one. It also became the routing signal for the next version.
v3.0 — VLM branch and adaptive routing
Presentation style filings exposed the ceiling of text based extraction. Charts carry their values in labels and axes, and OCR reads them in bounding box order rather than visual reading order, so the numbers came out in the wrong relationships.
Vision language models could interpret those pages semantically. During research, MegaParse, built on DocLayNet, and Docling were evaluated for layout aware region classification, and VLM behaviour was tested in isolation before integration, to understand its output format independently of the pipeline.
The result is quality gated multimodal routing. Clean text native pages take the fast path and never invoke a VLM. Degraded pages route through OCR with upscaling and gibberish filtering. Chart and presentation pages route to the VLM. Extraction is non destructive, so original markdown is preserved alongside VLM enriched content.
Working with the customer
Every one to two weeks, the current state of the system was presented directly to a senior stakeholder on the client side.
These were not status updates. They were engineering reviews covering where the system performed well, where it failed, the technical reason behind each failure, and the architectural direction proposed for the next iteration. Real production documents came back from those sessions and became the next round of test cases. The scanned filing that forced OCR and the presentation charts that forced the VLM pivot both entered the project this way.
Regular reviews with senior stakeholders validated each iteration against real operational requirements, which kept the system evolving around production needs rather than laboratory benchmarks. It also meant scope was expanded on evidence. Each extension followed a demonstration that a previously unhandled class of documents now worked.
Key engineering decisions
Markdown as the intermediate representation. Rather than passing raw text to the model or passing the PDF directly, the pipeline commits to a layout aware markdown layer in between. This gives the model spatial context, and gives the system a stable, inspectable artifact to debug against when extraction goes wrong.
Route by page quality, not by document type. Document level classification is unreliable, because a single filing can contain native text, scans, and charts. Routing is decided per page by the quality scorer, which also keeps expensive VLM calls off the majority of pages that do not need them.
Never trust self reported confidence. Neither library confidence nor model confidence proved to correlate with correctness. Quality had to be assessed by independent heuristics against the output, not requested from the component that produced it.
Validate before storing, and repair before discarding. Structured output is enforced against a strict Pydantic schema with explicit semantic rules: one semantic value per table cell, original wording preserved in narrative fields, paraphrase confined to summary sub fields. Malformed model responses go through a json-repair pass before being rejected. Parsing successfully was never treated as evidence of being correct.
Idempotency by content hash. Downloads and extractions are keyed on SHA-256 hashes, so re running produces the same output and duplicate filings are skipped. Combined with resume logic built in the first week, this made retries safe and long test cycles survivable.
Surface uncertainty instead of hiding it. Pages that could not be extracted reliably are flagged in the output rather than silently returning degraded content. Downstream consumers need to know what to trust, not just receive JSON.
Outcome
What began as a short exploratory effort became a production document intelligence system that handles native text, scanned, and chart heavy filings through a single adaptive pipeline, with per page quality reporting and asynchronous chunk processing for large documents.
The structured output it produces is now the foundation of a broader product direction: content level alerts, announcement based screening, full text search across filings, and a structured data feed suitable for external API consumers. The extraction layer is intentionally schema agnostic, which means layout changes in future filings are absorbed by the model layer rather than breaking hard coded rules, the failure mode that makes conventional scrapers expensive to maintain.
The architecture continues to expand as new document categories and edge cases appear in real usage. It is not finished, and the design assumes it will not be.
Lessons
I started this project believing document extraction was a parsing problem. It is a systems problem. The hardest parts were not extracting information. They were deciding which strategy to apply, recognising when an extraction could not be trusted, and designing something that could absorb document types nobody had seen yet.
Build the crude version first. The primitive extractor produced nothing usable, and it was the fastest way to learn what actually made this problem hard. No amount of upfront design would have surfaced the camelot border problem or the table continuity problem as quickly as a working baseline did.
Every logical branch in a prompt is a surface for hallucination. Elaborate, condition heavy prompts consistently lost to simpler, more constrained ones. Complexity that would be routine in code behaves very differently inside a model's instructions.
OCR is a pipeline, not a toggle. Usable OCR required page detection, quality assessment, preprocessing, and post processing filtering. Enabling it without those four is a reliability regression rather than a feature.
Auditability is a feature, not an afterthought. The per page report changed the character of the system more than any single extraction improvement. It converted a black box into something a user could evaluate and an engineer could debug.
Models are one component. Reliability came from the architecture around them: the routing, the validation, the aggregation, the quality gates, and the ability to report that a specific page could not be trusted rather than guessing. That is the thing I would carry into any AI system I build now.
Client and system details are described at the level of architecture and engineering process. Specific commercial and competitive material from the engagement is intentionally not reproduced here.