Decisions with Intent

Decision models

Decision models put a narrow language task behind a small API: choose an option, score an answer or test a condition. They suit software that needs a decision it can use directly, without generating a paragraph of text.

Recent releases make this an interesting option for classification. TypeSafe’s Jev and System One Models return typed decisions with confidence estimates. OpenAI’s Decisions API supports choices and rubric scores. Microsoft-Decision-1 scores supplied options, including classification labels.

I spent two working days applying this approach to a cascading classifier for customer-service intents. After cleaning the annotations and running controlled comparisons, the configuration I selected more than doubled accuracy on the monitored benchmark. The full results, including where optimization fell short, are below.

The classification task

An intent describes why a customer contacts a service team. Identifying it helps route work and shows which requests account for the workload. Across customer-service clients, I see taxonomies of roughly 5–20 main intents and 40–70 sub-intents: a broad reason for contact, followed by a more specific one.

I built a cascading classifier: the first stage predicts the main intent from the interaction text; the second chooses a sub-intent within that predicted branch. A mistake in the first stage can rule out the right answer in the second, so I scored the complete path. This is a natural fit for decision models: the answers are known, the software needs bounded choices, and the volume makes inference efficiency matter. The hard part is choosing the right labels when a conversation contains several topics or an unclear request.

The previous approach

The previous system used general-purpose LLMs with structured outputs. It supplied interaction text and requested labels in a schema defined in code. LangChain’s structured-output support connects the model call to a declared output type; Pydantic models define fields and validate the returned data.

This is a useful baseline. It produces results the application can consume and catches invalid values. A valid response can still contain the wrong intent, though. Schema compliance and classification quality need separate checks.

A brief inference note: providers can enforce a schema by blocking tokens that would break it as the LLM generates its answer. That constrains the response’s shape, not whether its judgment is correct. OpenAI’s explanation of constrained decoding covers the mechanics. A framework’s structured-output method may instead use tool calling or validation, depending on the provider and settings.

What I changed

I began with the client’s annotations, cleaned the dataset and checked label consistency. I fixed the training, development and evaluation partitions so each configuration faced the same examples. Training data could inform candidate instructions; development scores could guide selection. Evaluation labels stayed out of instruction generation.

For model selection, I reviewed results on related decision tasks. Microsoft’s published comparison reports leading accuracy across 36 benchmarks and 2.5× faster inference than H2O-Lightning-4B v1.1, its measured runner-up. The comparison includes OpenAI Decisions, with Jev accuracy results added later. These vendor-reported results made Decision-1 worth testing locally. They are separate from this engagement’s measurements; I did not run a three-way client-data benchmark.

I used DSPy’s MIPROv2 optimizer to search candidate instructions and selected the configuration by development-set macro-F1. MIPROv2 can optimize instructions and example selection; here the search concerned instructions, with no change to the model’s weights. This gave prompt changes an explicit objective and a repeatable comparison.

I saved predictions to compare runs without repeating inference. Reports covered accuracy, macro precision, recall and F1, confusion matrices and manual error analysis. Macro averages give each represented class equal weight. I counted failed predictions as errors and scored the complete intent/sub-intent path. Reading the mistakes helped identify where the next change needed investigation.

Results

The selected configuration produced the following changes against the previous system on the same 54 evaluation examples. I had inspected this set repeatedly, so I treat it as a monitored benchmark. These results show progress on known cases; they do not establish performance on unseen traffic. The systems may also split conversations differently.

The accuracy gain is approximately 2.1×. Macro-F1 improved by about 28%. All gains below are relative to the previous system. These comparisons do not isolate the contribution of the model change from prompt optimization.

Another Decision-1 baseline scored slightly higher on both accuracy and macro-F1 on this set. I kept the configuration chosen on development scores as the reported result instead of switching after seeing the evaluation scores. This is a useful reason to check an optimized setup against a baseline.

In two working days, I had cleaned data, saved comparisons and errors the team could inspect. The next step is to test fresh conversations, review the remaining mistakes and measure inference cost and latency under the intended workload before choosing a deployment configuration.

• End-to-end cascade accuracy: approximately 2.1× the previous result.
• Macro precision: approximately 27% higher.
• Macro recall: approximately 26% higher.
• Macro-F1: approximately 28% higher.