How can I reliably validate structured JSON extracted by AI from 100+ page industrial catalogs?

I am a mechanical design engineer working with factory automation equipment in Japan.

I regularly use technical catalogs from manufacturers such as IAI, SMC, THK, Mitsubishi Electric, and others. These catalogs can easily contain hundreds of pages, with specifications distributed across tables, model-code explanations, footnotes, diagrams, and compatibility charts.

I am currently experimenting with using AI to extract the information I need from these catalogs into structured JSON so that it can later be searched and reused for engineering tasks such as component selection and compatibility checking.

The main problem is reliability.

A typical workflow today looks like this:

  1. One AI/agent reads the catalog and generates structured JSON.

  2. A second agent reviews the extracted result.

  3. Sometimes I also ask another model, such as Claude, Grok, or DeepSeek, to independently check the result.

This helps, but I do not think it solves the fundamental problem.

Two models can agree and still be wrong.

For example, they may both:

  • read the wrong row or column in a table;

  • miss a footnote;

  • confuse specifications between very similar model numbers;

  • overlook an operating condition;

  • interpret a diagram incorrectly;

  • or repeat the same plausible but unsupported assumption.

For engineering use, “several models agree” is therefore not enough.

What I would like to build is a verification process where every extracted value can be traced back to authoritative evidence in the original catalog.

Ideally, a JSON field would contain not only the value:

{
  "max_speed": 500
}

but something closer to:

{
  "max_speed": {
    "value": 500,
    "unit": "mm/s",
    "source_page": 183,
    "source_section": "Speed / Acceleration Specifications",
    "source_text": "...",
    "verification_status": "verified"
  }
}

I am considering an architecture where:

  • one agent performs extraction;

  • another verifier independently searches the original PDF for evidence for each field;

  • deterministic checks validate things such as units, ranges, model-code relationships and compatibility rules;

  • fields without direct evidence are flagged rather than accepted;

  • and a manually verified sample is used as a gold dataset to measure the real error rate.

My main questions are:

1. What is the best way to store field-level provenance for data extracted from large technical PDFs?

2. How would you design an independent verification agent so that it does not simply inherit the first agent’s mistakes?

3. What evaluation metrics would you use to quantify the reliability of the resulting engineering dataset?

4. Is model consensus useful here, or is it better to treat models only as evidence-finding tools and rely on deterministic verification wherever possible?

5. Has anyone built a similar pipeline for technical catalogs, standards, manuals, medical documents, legal documents, or other sources where a plausible extraction is not good enough?

My eventual goal is not just to make an AI that can “read a catalog.”

I want to build a system where an engineer can ask something like:

“Are this servo motor, amplifier, power cable and encoder cable compatible?”

and receive not only an answer, but also the exact manufacturer evidence behind that answer and a clear warning whenever the system cannot verify something with sufficient confidence.

I would be very interested in hearing how people working on evals, retrieval, document extraction, or tool-using agents would approach this problem.