Context
An extraction model can return perfectly valid JSON while selecting the wrong event, missing a participant, or inventing an unsupported relationship. Evaluation therefore needs a task definition as well as a parser. Gilles’s independent open-source follow-up to SENTiVENT packages public financial language data for contemporary model evaluation. It builds on the original event-extraction research while remaining a separate engineering contribution. The original publications and their authors retain their academic attribution.
Approach
The work includes dataset packaging, fixed data splits, schema validation, strict and relaxed F1 scoring, and regression tests. Explicit matching rules make comparisons easier to interpret: a boundary mismatch is a different failure from an unsupported event. The historical research account explains the underlying extraction problem. Reviewing errors against shared public reference material lets readers inspect the evaluation target and understand how matching rules influence a result.
Quality controls
Fixed splits help keep model comparisons stable. Schema checks identify malformed results, while extraction scoring tests agreement with reference annotations under declared rules. Regression tests protect those rules when the toolkit changes. The article on structured-output evaluation separates these layers and explains why a single aggregate score can hide important errors. Results on this dataset do not establish performance in unrelated domains, production workloads, or employee coaching outcomes.
Outcomes
The delivered outcome is a public dataset and evaluation toolkit that make the extraction task reusable and its scoring inspectable. Coaching-agent work raises different judgment questions, illustrating why a benchmark’s acceptance criteria should follow the intended use rather than travel unchanged between tasks.
- Published fixed splits and structured evaluation support.
- Implemented strict and relaxed scoring, schema checks, and regression tests.
