“It depends on your requirements” is not an answer you can put a number against. If you don’t know which choices push the price up and which pull it down, there’s no way to guess what your own build would actually cost.
If you found this page searching “local LLM deployment cost” or “on-premise generative AI cost,” a vague answer isn’t what you’re after — you want the actual variables that move the number. This piece breaks those down one at a time.
The foundation: deployment form changes the upfront-to-monthly ratio entirely
Before model specs, the first thing that moves the cost structure of a local (private) LLM is how you run it. That single choice changes the ratio between upfront cost and monthly cost completely.
- Self-hosted (on your premises) — GPU hardware installed in your own building. The equipment becomes your capital asset (capex) from day one, and ongoing monthly cost is mostly operations and maintenance. This tends to be cheapest at sustained, heavy, predictable usage, provided your facility can handle the electrical and cooling load.
- Managed hosting — a vendor houses and operates the hardware for you. Power, cooling, UPS, connectivity, and monitoring roll into a monthly fee (opex), which removes the upfront weight. You keep hardware ownership.
- Shared infrastructure — no hardware purchase at all. You rent GPU/VRAM capacity, metered by the unit — a pure opex model. This fits usage that’s still light or unproven.
None of the three is cheapest in every case — usage pattern is what flips the answer. Heavy, sustained usage lets self-hosting’s upfront cost pay down into a lower cost per unit of use over time; usage that’s still uncertain makes the metered pricing of shared infrastructure the more rational way to wait and see. The full comparison, with monthly ranges for each, is on the Private LLM Infrastructure page.
The reason this single choice changes the whole structure is simple: the same model, running for the same purpose, generates cost at different times and under different line items depending on which of the three you pick. That’s why it needs deciding before model size does.
Model size moves cost for two separate reasons
Once deployment form is settled, model size is next — and the common mistake is treating it as one ladder where bigger is simply better. In practice there are two separate reasons to move up in size, sitting on different axes. Conflating them means paying for capacity you don’t actually need.
Axis 1: the accuracy floor. The smallest model that still produces business-usable answers.
- 32B-class (dense, ~32GB in FP8) sits at the practical floor for business use.
- 70B-class (dense, ~70GB in FP8) is the default and recommended starting point — not “bigger to be safe,” but tied to a concrete benchmark: one machine, sized to approximate a company-wide ChatGPT-equivalent. For most companies, that’s the answer to “what’s the first machine we stand up.”
Axis 2: the concurrency ceiling. Not about answer quality — about how many simultaneous users or requests the deployment can serve.
- 122B-A10B (MoE, 10B active, ~125GB in FP8) and 400B-class (MoE, 17B active, ~400GB in FP8) exist mainly to raise that ceiling, not to make any single answer more accurate than the 70B default already does.
Why concurrency pushes model size up for reasons that have nothing to do with accuracy
A single machine has a fixed amount of GPU memory and compute. When several people query it at once, the server has to hold each conversation’s context (its KV cache) in memory simultaneously while batching requests together. As concurrent sessions grow, they compete for that same fixed resource — no single answer gets less accurate, but responses slow down, or there’s a hard ceiling on how many sessions the deployment can carry before service quality degrades.
Moving to a larger MoE tier — 122B-A10B, or the ~400GB 400B-class — is how that ceiling gets raised. MoE (Mixture of Experts) architectures carry a much larger total parameter count while keeping the “active” parameters used to generate any single token modest — 10B or 17B. That larger total capacity gives the deployment more memory and routing headroom to handle a wider variety of simultaneous requests, while the compute cost of producing any one token stays in roughly the same range as the 70B baseline. What you’re paying for at that tier isn’t a smarter answer to one request — it’s headroom to serve more requests, of more varied kinds, at the same time. Working out whether the actual problem is answer quality or throughput, before sizing up, is what keeps you from paying for the wrong axis.
One model sits outside this ladder entirely: 35B-A3B (MoE, 3B active) is scoped for a triage or guardrail role — lightweight routing and safety checks — not for carrying a business’s primary answering traffic on its own.
Quantizing further lowers the hardware bill. Here’s why we don’t go below FP8
Once model size is set, quantization is the next lever. The lower you quantize, the less GPU memory the same model needs, and the lower the hardware bill. Technically, you can push all the way down to 4-bit/INT4.
Our standard configurations hold FP8 as the floor and don’t go lower by default. The reason isn’t theoretical caution — it’s measured. In our own testing, quantizing below FP8 produces a measurable accuracy drop on Japanese-language business tasks. Lower quantization would lower the hardware bill, but we don’t ship it as a default recommendation.
That’s less a blanket conservative policy than a line drawn against a measured tradeoff: hardware savings weighed against a documented accuracy cost. The option to quantize further still exists for use cases with genuinely lower accuracy requirements — but that’s a call to make against your own measured results, not the first variable to reach for when trying to cut cost.
Starting small and scaling later — what gets redone, and what carries over
Committing to a full 70B-class build on day one isn’t the only path. Starting smaller and scaling up as usage justifies it is a legitimate option — and it’s worth knowing in advance what that scaling step redoes, and what it doesn’t.
What carries over — the RAG configuration and retrieval index, the evaluation dataset, and the integration layer. Our builds are designed around a standard, OpenAI-compatible inference interface, so changing model size or hosting form doesn’t require rewriting how your applications talk to the model. The documents and questions used for evaluation are model-size-agnostic too, so the same evaluation set carries straight into testing the next tier.
What gets redone — the fine-tuned weights themselves, and the hardware or hosting arrangement. Fine-tuning is tied to a specific base model’s weights, so a tuning result produced for a 32B model doesn’t transfer to a 70B model — it has to be redone against the new base. Similarly, moving from validating on shared infrastructure to provisioning self-hosted hardware means redoing the hosting decision and procurement each time the tier changes.
Knowing that boundary in advance — what’s reusable and what isn’t — makes it easier to decide how small the first step can safely be. The 30-day Private LLM PoC is built around exactly that: testing against your own documents and workload before any hardware purchase, starting from the 70B-class default and moving the tier up or down from there based on measured accuracy and latency rather than a guess.
One more thing worth separating out: everything above is the cost of the platform the model runs on. If the goal extends into the application layer — wiring the model into existing business systems, or building workflows beyond RAG — that’s a separate line item, covered under AI-Powered Development Services. Infrastructure and application are different estimates.
LLL’s pricing (for reference)
With those variables in mind, here’s what we actually quote. Figures below are in USD and reflect what we quote clients directly — not internal cost figures.
| Engagement | Price | Notes |
|---|---|---|
| 30-day PoC | $6,000 ($3,000 for teams of 50 or fewer) | Fully credited toward the full build if you continue |
| Full build (single use case) | $8,000–$22,000, scale-dependent | Environment build, RAG, integration with existing systems, and training |
| Ongoing maintenance & operations | $300–$1,300/month | Tiered by GPU count |
| Managed hosting | $300–$1,600/month | Tiered by power draw. Power, cooling, UPS, connectivity, and monitoring included |
| Shared infrastructure | From $300/month per 24GB VRAM unit | No hardware purchase |
The hardware itself — the GPU server — isn’t in that table. GPU and server-memory pricing has been moving quickly enough that we don’t publish a fixed equipment price. As a rough sense of scale: a full 8-GPU build sized for the 70B-class default sits in the low six figures, in USD, as a complete build — that’s a scale reference, not a quote. Facility costs (power, cooling, UPS) vary enough by building that they’re priced individually after a site visit.
Summary: the order for building your own number
Broken into an order you can actually work through, “it depends on your requirements” becomes:
- Pick a deployment form — self-hosted, managed, or shared changes the upfront-to-monthly ratio entirely. Full detail on the Private LLM Infrastructure page.
- Size the accuracy floor and the concurrency ceiling as separate questions — whether you land on 32B, 70B, 122B-A10B, or 400B-class depends on whether the problem is answer quality or request volume.
- Hold FP8 as the quantization floor — going lower saves on hardware, but our own testing shows a measurable accuracy cost on Japanese-language business tasks.
- If starting small, separate what carries over from what gets redone — RAG, evaluation data, and the integration layer transfer; fine-tuning and hardware procurement get redone at each tier change.
Run your own workload through those four and you’ll have a real sense of what the number looks like for you. A desk estimate only goes so far, though — when an actual figure is needed, the 30-day Private LLM PoC tests it against your own documents before any hardware gets purchased.