Start with the work you need done
Local AI has become a serious architecture option for organisations that need control over sensitive documents, predictable access or a specialised workflow. The useful question is precise: which tasks can an internally hosted model perform well enough, at an acceptable operating cost? That question works for a small consultancy processing confidential files as well as a large enterprise serving several departments.
Choose a bounded task first: classify incoming correspondence, extract fields from documents, draft an answer from approved material or transcribe an internal meeting. Establish what a correct result looks like and how much review it needs. A model that produces fluent text but increases correction time has failed the business case, even if its token throughput looks impressive.
Control comes from the whole system
Running inference on your own hardware can keep prompts and outputs within your environment, provided the surrounding application does too. Map document ingestion, embeddings, search, monitoring and backups. An external embedding endpoint or verbose remote log can undermine the intended boundary. Treat model licences and update provenance as deployment decisions, alongside the choice of weights.
Local hosting also needs identity, document permissions, isolation and an owner for patching. The llama.cpp security guidance explicitly discusses sensitive data, untrusted inputs and tenant isolation. An internal endpoint does not automatically give each employee appropriate access to every document. Retrieval should enforce permissions before evidence reaches the model.
Price accepted work, including the queue
Compare hardware amortisation, electricity, maintenance and engineering time with realistic hosted usage. Include retries, fallback calls and human review on both sides. The most useful denominator is an accepted document or resolved task. A lightly used server may be an expensive choice; a steady batch workload can make its capacity much easier to use efficiently.
Memory planning must include the model, context and concurrent requests. The vLLM PagedAttention research shows why managing the growing attention cache matters for serving throughput. Quantisation can change the hardware requirement, but supported formats depend on the engine and device. Recheck task accuracy after compression, particularly for extraction, multilingual material and tool use. Measure waiting time under load, alongside generation speed.
A pilot that supports a decision
Build a representative evaluation set with ordinary cases, difficult layouts and examples where the system should abstain. Compare a local candidate with the current workflow and, where permitted, a hosted reference. Keep prompts, scoring and inputs consistent. Separate document errors from retrieval errors and model errors so the next investment addresses the actual constraint.
The pilot should end with a deployment recommendation, an operating owner and explicit acceptance criteria. Small organisations can begin with one internal workflow; larger organisations can test a shared service with department-specific access. Hybrid routing is another option when the data policy permits it. The valuable outcome is an architecture that earns its place through reliable work, controllable data flows and a cost model the business can maintain.
Sources & further reading
Written by Gilles Jacobs, AI & data engineering consultant at Exverbai.
