Generative AI PoC to Production | What's Actually Missing

The PoC Worked. Here’s What’s Actually Missing Before It Goes to Production

The PoC worked. Months later, it’s still not in production. This is where a lot of generative AI projects stall — not because the technology failed, but because “we confirmed it works” doesn’t actually answer the next question, and nobody defined in advance what would.

This is written for a reader who has already run a PoC. It skips the explanation of what a PoC is. What follows is specifically what’s missing to move a stalled PoC into production.

”Governance matters” doesn’t move a stalled PoC

Most articles about taking generative AI to production land, eventually, on “put a governance framework in place.” That’s not wrong. But for a reader who’s already run a PoC, it’s not enough information. Governance mattering isn’t news. What’s actually useful is knowing specifically what was missing from the PoC’s design that left the go/no-go decision unanswerable in the first place.

There’s a real gap in how much value companies actually get from AI investment — a gap we checked against primary sources (BCG and McKinsey’s 2025 studies) in an earlier piece, “60% of Companies That Invested in AI Got Zero Financial Value.” That piece was about why the gap exists, read through the survey data. This one picks up from there: it’s about how to move a specific, already-stalled PoC forward, worked out at the level of what the PoC’s design should have produced.

Everything from here on is not a finding from BCG or McKinsey’s research. It’s an operational pattern LLL sees directly in designing private LLM PoCs and taking them into production — read it as that, not as a conclusion either study reached.

A PoC that didn’t end in a decidable form can’t move forward

The usual reason a PoC stalls isn’t a technology shortfall. It’s that the PoC left behind proof that something worked, but not the material needed to decide whether it should go to production.

The demo succeeded, the accuracy looked reasonable — most PoCs get that far. The real test is whether the PoC’s own deliverables can answer these:

What accuracy bar counts as “cleared to go to production” — and was that threshold written down before testing started, or decided after looking at the results?
Was accuracy measured against vendor demo data, or against your own documents and your own question patterns?
Is the configuration you’re planning for production (deployment form, model size, where the data goes) the same one the PoC actually tested?
Is there a concrete cost estimate for the full build, derived from the PoC’s own results?

If a PoC gets marked “done” without answers to those four, the production conversation starts over from zero. Stakeholders reconvene, re-litigate whether “it worked” is enough to proceed, and that recurring re-litigation is usually what “stalled” actually looks like from the inside.

Three things that need to be built into the PoC’s design from the start

The difference between a PoC that leads somewhere and one that stalls isn’t found in a post-mortem — it’s set at the design stage. These are the three deliverables LLL fixes into a private LLM PoC’s design before testing begins.

1. The evaluation set is built from your own documents and your own questions — not a benchmark

Generative AI demos tend to look strong against generic public benchmarks or vendor-supplied sample data. That doesn’t guarantee the same accuracy against your actual business documents and your actual question patterns. If the PoC’s evaluation set wasn’t built from your own material, the accuracy number it produced isn’t something you can safely act on for a production decision.

2. “Feasible / conditionally feasible / not feasible” criteria, written down before testing starts

What accuracy threshold counts as “go”? How much latency is tolerable? Decide that after seeing the results, and the numbers get interpreted by whoever’s in the room that day. That threshold has to be agreed in writing before testing starts.

Build that into the design, and the PoC’s outcome comes out as one of three things: feasible, conditionally feasible, or not feasible. A “not feasible” result is not a failed PoC — it’s the PoC doing its job. Leaving the criteria vague to avoid that outcome is exactly what causes a stall later on.

3. Deployment form, model size, and a cost estimate, included as PoC deliverables

A PoC that ends at “we confirmed it works” forces the production conversation to start from a blank page. Which of self-hosted, managed, or shared infrastructure actually fits your usage level? What model class does the workload need? What does a full build cost? If the PoC’s deliverables don’t already answer those, there’s another research phase waiting after the PoC — before production planning can even begin.

Decide at the design stage that the PoC will produce written answers to those three, and moving to production stops being a new investigation — it becomes executing a recommendation the PoC already made. LLL’s 30-Day Private LLM PoC is built specifically to produce a written report covering all three.

Decide where the data goes before the PoC starts — not after

A legal or security question surfacing right before launch — “are we actually allowed to send this data to a third-party API?” — is another common way a project stalls. Most of the time, the root cause is that the PoC left the data question open and built something that technically worked first.

Using generative AI means choosing between three configurations, each with a genuinely different answer to where the data in a prompt actually goes:

Private LLM — inference completes inside infrastructure you control, or that a vendor operates on your behalf
Frontier API — prompts go to a provider’s servers: OpenAI, Anthropic, Google
Hybrid — sensitive material routed to the private path, everything else to the frontier model

The full breakdown of how to decide between the three is in Private LLM or API — Three Configurations, and an Order to Decide Them In.

What matters here is timing: settle that decision before the PoC starts, and set up the PoC’s own evaluation environment to already match the configuration planned for production. Run the PoC on a convenient shortcut instead — a sample set that avoids sensitive material, for instance — and production planning inherits an extra step: redoing the work properly, on real data, in the real configuration. That’s often the actual mechanism behind the gap between “PoC done” and “in production.” The AI Development page covers the same three-way split under its data privacy FAQ.

Model size: validate accuracy and concurrent capacity as two separate axes

A common failure mode in model sizing is treating accuracy and throughput as the same problem. They’re not — they sit on different axes.

The accuracy floor is set by model class and quantization floor. A 32B-class model is a practical floor for business use; 70B-class is a reasonable default sized to approximate a company-wide ChatGPT-equivalent — accuracy headroom generally grows with class. Quantization works the same way: going below FP8 (to 4-bit, for instance) produces a measurable accuracy drop on Japanese-language business tasks, based on our testing.
The concurrency ceiling is set by how many users and how much request volume the deployment needs to handle. That’s where model architecture matters more directly — 122B-A10B for heavier concurrent usage, or a 35B-A3B tier built specifically for a triage/guardrail role rather than as an answering model.

If a PoC validates accuracy on a smaller, cheaper model and separately estimates throughput against a different, larger model that was never actually tested, the model that was validated and the model planned for production have quietly diverged. That gap tends to surface right before launch — “the model we validated accuracy on can’t handle production concurrency,” or “the model that can handle the load doesn’t clear the accuracy bar.” A PoC needs to test both accuracy and latency/throughput against the same model configuration planned for production, not a stand-in for either. Model sizing is covered in more depth on Private LLM Infrastructure.

What actually gets a stalled PoC moving

When a PoC has stalled, the fix is usually neither “run the PoC again” nor “start a governance initiative.” What’s missing is documenting these four things as PoC deliverables:

Measured accuracy and latency against an evaluation set built from your own documents and questions
“Feasible / conditionally feasible / not feasible” criteria, written down before testing started
A production recommendation covering deployment form, model size, and cost estimate
Validation run against the same data flow and the same model configuration planned for production

Governance and human review still matter after this. Routing high-stakes outputs through a human checkpoint, and guardrails on generated content, are ongoing requirements once something is live in production. But that layer sits on top of a PoC that already produced a decision — it doesn’t substitute for one. Stacking governance discussion on top of a PoC that never produced decidable material doesn’t move the production decision forward on its own.

Turning a PoC Into a Production Decision

Where this shows up in what LLL builds: a 30-day PoC built to test whether an investment holds up on your own data, private LLM infrastructure, and AI development work more broadly.

30-Day Private LLM PoC Private LLM Infrastructure AI Development Services