A complete object can be wrong
A response may parse as JSON and still name the wrong participant, select an unsupported category, or omit a required event. Treat format validity, schema conformance, and task correctness as separate checks. The JSON Schema object reference explains structural constraints; it cannot establish that a value is true in the source material.
Specify the matching rule
The open-source model evaluation project uses fixed splits, schema validation, strict and relaxed extraction scoring, and regression tests. Its reference task comes from financial event-extraction research. Matching rules make the difference between a boundary error and a wrong extraction visible. Results remain specific to the declared task and data.
Choose checks for the decision
An email-routing pilot needs an appropriate destination, not merely a valid department label. Financial reporting needs statements that agree with source values. Clinical writing adds specialist review, while compliance analysis needs evidence supporting an interpretation. These examples motivate different acceptance criteria rather than a universal evaluation score.
Retain examples that explain failure
Review false positives, omissions, ambiguous inputs, and unsupported claims. Keep accepted references and scoring rules stable when comparing changes, and record intentional changes to the task. Those are general recommendations for evaluation design. A passing format check is useful evidence about a narrow property; it is not proof that a whole workflow is correct or ready for production.
Sources & further reading
Written by Gilles Jacobs.
