Choose the task before the hardware
Start with one task: classify correspondence, extract fields, or retrieve an internal document. Define an acceptable result and the review it needs. Local inference is an architecture choice to evaluate against that task. Gilles has used self-hosted models in proofs of concept; this article does not claim a production on-premises platform or a universal cost advantage.
Follow the whole data path
Hosting a model internally can keep inference within a chosen environment, but embeddings, logs, backups, and document processing also need attention. The llama.cpp security guidance discusses sensitive data, untrusted inputs, and tenant isolation. PII masking is one complementary control; it does not guarantee complete anonymization or replace permissions.
Measure accepted work
Compare quality, waiting time, retries, review effort, and operating cost on representative inputs. Include maintenance and realistic utilization when comparing hosting options. The PagedAttention research explains why growing attention caches matter for serving throughput, without settling the economics of a particular workload. Test concurrent requests and long inputs rather than relying on a headline token rate.
Keep a reviewed reference
Use the same inputs and acceptance rules for each candidate. Structured model evaluation provides an open-source example of explicit reference data and repeatable scoring. A pilot should end with measured tradeoffs and an operating owner. These are evaluation recommendations; choosing local hosting alone does not establish better privacy, quality, or cost.
Sources & further reading
Written by Gilles Jacobs.
