“Self-host it if you’re worried about security, use the API if you’re not” is a decision rule most teams end up walking back. The reason is simple: there are actually three configurations to choose from, not two, and the decision isn’t a single fork — it’s a sequence of smaller decisions that only make sense in a particular order.
Plenty of companies that have already started using generative AI get stuck at the same next step: should we build a private LLM, or is API access to a frontier model — OpenAI, Anthropic, Google — enough? This piece lays out how to actually decide, using the three configurations that exist in practice and the order in which each decision gets made.
Why “on-prem is secure, cloud is convenient” doesn’t settle anything
That binary breaks down because “secure” and “convenient” aren’t doing real work as descriptors. A private LLM isn’t automatically secure if network boundaries and access control are sloppy. A frontier API isn’t automatically insecure if you contractually limit what gets sent and configure data-retention settings — for a lot of workloads, that’s enough governance.
What actually matters isn’t security versus convenience. It’s where the data in your prompt goes, and whose servers process it. Looked at through that lens, there are three real options.
The choice is three configurations, not two
In practice, enterprise generative AI usage settles into one of three patterns:
- Private LLM — the model runs on infrastructure you control, or that a vendor operates on your behalf. Prompts and generated responses are processed inside your own environment and are not sent to a public AI service.
- Frontier API — you use a provider’s model (OpenAI, Anthropic, Google) directly via API. Prompts are actually transmitted to that provider’s servers. Governance here means tuning data-retention settings, limiting what gets sent, and spelling out the terms in a contract.
- Hybrid — sensitive material is routed to the private path, and everything else goes to the frontier model.
Each of the three moves data differently, and that difference is the actual reason to pick one over another. We cover this same breakdown in the data privacy FAQ on our AI Development Services page. Which of the three fits starts with classifying what would actually flow into a prompt — which is the next question, and the order matters.
The order: classify the data first
Starting the decision from a hardware budget or a vendor’s pitch deck gets the sequence backwards. The order that actually works looks like this:
- Inventory what could end up in a prompt — internal documents, customer data, source code, contracts: anything that could be subject to a confidentiality obligation you can’t hand to a third party.
- Check whether any of it shouldn’t reach a public AI provider’s servers. If none of it should, a frontier API alone is probably sufficient. If some of it shouldn’t, a hybrid setup that routes only the sensitive slice becomes the relevant option. If most of the workload falls into that category, a private LLM is worth evaluating seriously.
- Only then does infrastructure sizing enter the picture. Starting from “we should have our own AI” as a general instinct, and retrofitting a data-sensitivity justification afterward, tends to produce either overspending or underprovisioning.
Followed in this order, whether to build a private LLM at all stops being a security-policy debate and becomes close to a direct consequence of the data classification. The harder judgment calls come after that — once you’ve chosen a private LLM, what you actually have to decide next.
Once you choose a private LLM, two more decisions follow
Say the data classification points toward a private LLM. From here, the decision splits into two separate questions: how to run it, and how to size it. Treating these as one decision is where sizing tends to go wrong.
How to run it — three deployment forms
There’s no single “correct” deployment form for a private LLM. Each of the three has a usage level where it’s the cheapest option.
- Self-hosted (on your premises) — GPU hardware installed in your own building. The equipment becomes your capital asset from day one. This tends to be cheapest at sustained, heavy, predictable usage, provided your facility can support the electrical and cooling load.
- Managed hosting — a vendor houses and operates the hardware on your behalf, with power, cooling, UPS, connectivity, and monitoring included in a monthly fee. You keep hardware ownership without taking on the facilities burden.
- Shared infrastructure — no hardware purchase at all. You rent GPU/VRAM capacity, metered by the unit. This fits usage that’s still light or unproven.
None of the three wins at every scale — usage pattern determines which is cheapest. The full comparison is on our Private LLM Infrastructure page.
How to size it — the accuracy floor and the concurrency ceiling are different axes
The most common mistake in sizing a model is treating it as one ladder where bigger is simply better. In practice, there are two separate reasons to move up in size, and they sit on different axes.
Axis 1: the accuracy floor. This is about the smallest model that still produces business-usable answers.
- 32B-class (dense) sits at the practical floor for business use. Going smaller than that tends to run into accuracy problems that make the model unreliable for real work.
- 70B-class (dense) is the default and recommended starting point. That’s not “bigger to be safe” — it’s tied to a specific, concrete benchmark: one machine, sized to approximate a company-wide ChatGPT-equivalent. For most companies, that’s the answer to “what’s the first machine we stand up.”
Axis 2: the concurrency ceiling. This isn’t about answer quality — it’s about how many simultaneous users or requests the deployment can handle.
- 122B-A10B (MoE, 10B active) and 400B-class (MoE, 17B active) exist mainly for higher concurrent usage, not as an additional accuracy upgrade over the 70B default.
Conflating these two axes is where sizing goes wrong — reaching for a concurrency-tier model because accuracy feels insufficient, or reaching for an accuracy-tier upgrade because concurrent load has grown. The useful first move is separating which problem you actually have: is it answer quality, or throughput?
One more tier sits outside this ladder entirely. 35B-A3B (MoE, 3B active) is scoped for a triage or guardrail role — routing requests or running lightweight safety checks — not as an answering model. Its small active-parameter count makes it fast and cheap for that narrow job, but it isn’t sized or positioned to carry a business’s primary Q&A traffic on its own.
How far to quantize — why FP8 is the floor
Quantizing further than FP8 — down to 4-bit/INT4, for instance — is technically possible and would lower the hardware bill. Our standard configurations don’t go below FP8 by default.
The reason is empirical rather than theoretical. In our testing, quantizing below FP8 produces a measurable accuracy drop on Japanese-language business tasks. That’s not a blanket conservative policy — it’s a line drawn from measured results, weighed against real hardware savings that the line leaves on the table.
Measure before you commit: a PoC before the hardware order
The three decisions above — configuration (private / API / hybrid), deployment form (self-hosted / managed / shared), and model sizing (accuracy floor, concurrency ceiling, quantization floor) — aren’t decisions that can be fully closed out with general reasoning alone. They depend on how your specific workload behaves.
A full GPU build is a real capital commitment, and the range moves with a volatile equipment market. Committing to that before measuring accuracy or latency on your own workload is a real risk. That’s what a 30-day Private LLM PoC exists to remove: starting from the 70B-class default, it tests whether scaling down to 32B or up to the 122B-A10B or 400B-class tiers is warranted, based on measured accuracy and latency against your own documents rather than a generic benchmark — and the resulting report also names which deployment form fits your usage pattern.
Summary: the order for deciding
Working through the private-LLM-or-API question in the actual order, rather than a blunt binary, looks like this:
- Inventory what could end up in a prompt, and check whether any of it shouldn’t reach a public AI service. That alone settles most of whether private LLM, frontier API, or hybrid is the right foundation.
- If a private LLM is in scope, treat “how to run it” (self-hosted / managed / shared) and “how to size it” (accuracy floor, concurrency ceiling, quantization floor) as separate decisions, not one.
- Measure against your own workload before committing to hardware, rather than sizing off general reasoning alone.
Followed in that order, you land on a configuration that actually fits your data and your workload — not one borrowed from “on-prem is secure, cloud is convenient.”