Self-hosted LLM for business: what it takes

Self-hosted, on-premise, local and private each answer a different question, and marketing uses them interchangeably. Deciding which one you actually need is most of the requirements work.

Organisations that start from “it must be on-premise” often settle, once the data flows are drawn, for a single-tenant deployment with a defined model path, because that is what their obligations turn out to require.

The open-weight families that matter in mid-2026 are Meta's Llama 4, Alibaba's Qwen, Mistral, DeepSeek, Google's Gemma and OpenAI's gpt-oss. Capability is no longer the blocker it was two years ago. For document work, drafting and internal question-answering, current open-weight models are competent, and the gap to frontier hosted models rarely decides the outcome.

There is no best model, only a best result on your tasks. Shortlist two or three, run your own documents and questions through them, and let that decide.

Two published reference points, both from OpenAI's own model cards: gpt-oss-20b runs within 16GB of memory, which is workstation territory, and gpt-oss-120b runs on a single 80GB GPU, meaning one H100-class card rather than a cluster. Larger open models need multi-GPU servers.

Those figures are for running the model. Serving it to a team is a different sizing exercise, because concurrent users multiply memory and throughput requirements. Renting GPUs in your own cloud account is still self-hosting in the sense that matters, and it is how most businesses start.

We give no cost figures. Hardware prices, rental rates and utilisation vary too much for a number that would transfer to your situation, and a specific figure here would be decoration.

Two tools dominate and they solve different problems. Ollama installs as a single binary, manages model downloads and quantisation, and exposes an OpenAI-compatible API, which makes it the right tool for evaluating models and building prototypes. vLLM is the production server, built for throughput under concurrent load and multi-GPU serving, and it exposes the same API shape.

The common pattern in 2026 is to prototype on Ollama and serve production traffic on vLLM. Because both speak the OpenAI API shape, application code rarely changes between them.

This is the part that surprises teams. A served model is an endpoint that completes text. A system a business can rely on needs several more layers, and that is where most of the cost and most of the risk live.

Self-hosting moves responsibility. It does not remove risk, and a badly run private server is less secure than a well-run vendor service.

Owning the deployment means owning security patching for the whole stack, model updates and the regression testing that goes with them, backups and tested recovery, monitoring, incident response, secrets management and egress policy. Each needs a named owner before production rather than after. Operational capability, not hardware, is routinely the binding constraint.

Build it yourself when the constraints are hard, the workload sits close to a bare model, and you have the engineering capacity to own the stack above.

Buy a private deployment when what you need is the whole system inside a controlled boundary rather than a model endpoint. Marella is a system of that second kind. It ships as a self-hostable, Docker-based package run in a customer-controlled environment as a single-tenant deployment, and its model and provider paths are configurable rather than fixed to one API. A fully local model configuration is architecture-specific and gets verified per deployment for the proposed models and workflow.

What you stop building is the system itself: answers over your own documents, cited to the exact passage with the source opened alongside, and fact-checked against those sources after generation with unsupported claims flagged.