A valid object can still describe the wrong event
A financial-news extractor can return a perfectly shaped object with the wrong company, an unsupported amount, or a speculative event marked as completed. The downstream pipeline can parse it without discovering that the business meaning has changed.
Structured Outputs helps models return data that follows a supported schema. This makes integration easier. It does not remove the need to check the extracted facts against the source or decide whether the result is useful for a particular task.
Make the data contract explicit
JSON Schema lets an application specify required properties and how it handles extra fields. A practical extraction contract also needs an agreed meaning for unknown values, identifiers, units, dates, and evidence references.
Keep the first contract small. If the workflow needs an event type, company, status, and source span, validate those fields before adding a dozen speculative attributes. Handle refusals, incomplete responses, and unavailable evidence as explicit application states.
Score meaning separately from structure
Schema validity answers whether the object fits the interface. Task scoring answers whether it contains the right information. Keep those measurements separate so an improvement in formatting cannot hide a decline in extraction quality.
For an event extractor, compare event types, participant roles, evidence spans, and factual status with reviewed examples. Report strict matches and justified relaxed matches separately. Inspect omissions as well as invented events: both can make a monitoring system miss what matters.
Use failures to guide the next change
Save a fixed evaluation split and a small set of difficult regression cases. Run them when changing a model, prompt, retrieval step, or normalization rule. Slice the results by the categories your team needs to trust rather than relying on a single headline score.
If most failures concern company resolution, improve identifiers and source context before changing the whole architecture. If the schema passes but supporting evidence is weak, revise evidence selection and review. The aim is a workflow that produces useful, traceable records at an acceptable operating cost.
Sources & further reading
Written by Gilles Jacobs, AI & data engineering consultant at Exverbai.
