There is a fairy in the bottom-right corner of this page
Right now, a small fairy is floating in the bottom-right corner of this article. Her name is Pixie. Talk to her and she talks back.
(Go ahead and try. She usually answers in two or three seconds.)
This fairy does not use any external AI service. What you type is never sent anywhere. A single machine we manage receives it, thinks about it, and answers.
Which means we are the ones paying her electricity bill.
Why would anyone go to that trouble? That is what this article is about.
It started with “so how good is it, really?”
Running an AI inside a server your own company manages. It is called a private LLM, and we had known this option existed for a while.
What we could not tell was how capable it actually was. You can read every explainer article out there and still not know whether it will hold up in real work. That is something you can only settle by running it.
What pushed us over the line was money.
These days, every system we build comes with the question “can you add AI to this too?” We are glad to be asked. The catch is that AI APIs bill you for what you use. On client projects the customer covers that cost, which is simple enough. Our own products are a different story. You build the feature once, but the money keeps going out for as long as it runs.
The same goes for AI coding subscriptions. One seat is nothing. Multiply it by the team and the number stops being funny.
So we were stuck in a loop. You cannot tell whether it is good enough without trying it — but trying it costs money.
At that point, putting one on our own desk started to look like the sensible move.
The short answer: it does almost everything
This was the biggest surprise.
Most of what an external AI service does, a private one does too.
When people hear “self-hosted,” they tend to picture a stripped-down version. At least so far, that has not been our experience.
- Write, summarise, and tidy up prose
- Translate
- Pull the one thing you need out of a long thread
- Read photos and screenshots
- Take a PDF, Word file, or Excel sheet as-is and answer questions about it
- Answer from your own internal documents rather than from general knowledge
- Get built into your own apps and services as a feature
We had been picturing something closer to “a slightly clever search box that only handles text,” so the first time we handed it an image and it simply read it, we felt a little silly.
Of course, put side by side with the very best models, there is a gap. For anything that means designing something intricate from scratch, we still lean on the strong external services. But most of what we want to hand off day to day is already covered by the AI on our own hardware. That was the honest takeaway.
That said, this was not a matter of switching it on and having it work well.
Which model to use, how to configure it, how to feed it our internal documents — all of that had to be worked out piece by piece, and getting the environment right took real effort. Although the process was genuinely fun. Study your gear, tune it, go into the dungeon and see how it holds up. It felt like a hack-and-slash game.
That part deserves its own article, so we will write up the build separately.
What changed was not the quality — it was how often we try things
More than raw capability, what mattered was that experimenting became free.
It sounds obvious, but: however much you use it, the bill does not grow.
Before, ten attempts meant paying for ten attempts, so there was always a beat where you asked yourself “is this worth trying?” That beat came up every single time.
Now it doesn’t.
The clearest effect has been on our own products. When someone thinks “it would be fun to let the AI handle this bit,” they can just try it. Before, the idea would run into a cost conversation immediately, and it usually died right there.
The electricity bill, by the way, has come in far below what we braced for. A pleasant surprise.
What we actually have it doing
Here is some of what is running today.
Guide on the website The fairy from the top of this article. She answers “what services do you offer?” and “who are you, exactly?” Nobody has to sit and watch it, and she does not complain about being asked at two in the morning.
Turning conversations into tasks We connected the private LLM to the chat app we built for ourselves, and it picks the to-dos out of the flow of conversation. Things that used to disappear into “right, that’s decided then” now actually stay on the list.
Closing the distance between languages Japanese, English, and Malay all fly around our team, so translation comes up daily. There is a lot of it — exactly the kind of quietly accumulating work that adds up on an external API.
Beyond those, we hand over smaller jobs too: simple chatbots, a first-pass “is this okay to post?” check.
Lined up together, they have two things in common.
The first is that none of them work without handing over real internal information. Customer names, figures, things that are not public yet. Not one character of it can go to an outside service.
The second is that none of them need intelligence level 100. What mattered was zero versus one. No translation at all, versus a translation that reads reasonably well. Nobody looking at it, versus a first pass that exists. That gap is the big one, and everything past it — “but smarter” — is honestly a bonus.
There turned out to be far fewer jobs that only the very best AI can do. Once that clicked, the range of things we were willing to hand over widened quickly.
(The chat app itself is something we built, but that is another article.)
It is good for privacy, too
The real obstacle to putting AI into your work is not the technology. It is deciding what you are allowed to feed it. Customer names, quoted prices, drawings that have not been published, HR conversations. A great deal of the work you would most like help with involves information that cannot leave the building.
Plenty of companies respond by banning AI at work. That only stops it inside the company, though.
Someone up against a deadline will reach for their own phone and have an AI do the job there. There is no malice in it. We already live in an age where AI is simply part of how people work. And the company has no way of knowing what left the building.
It is the same as a smoking ban. Prohibition does not make something disappear. It makes it invisible.
A private LLM is, we think, a real answer to that, and the heavier the information a company handles, the more it is worth. A listed company needs to be able to explain afterwards who put what in. A law or accounting firm has client confidentiality. A public body has to decide where residents’ data may sit. In healthcare, where a mistake carries real weight, the first step never gets taken at all. None of these are stuck on how to use AI. They are stuck on where to put it.
We still have not moved everything in-house
To be clear, we have not stopped using external AI.
We still pay for external AI for development work. For the hard problems, asking the strongest model is simply faster. We use both.
So this is not an argument about which one is right.
Work that can leave goes outside. Work that cannot stays here.
What matters is who gets to draw that line. If you are the one drawing it, you can move it whenever you like.
We are also quietly tuning a coding agent that runs on the private LLM, with the eventual goal of developing entirely on our own hardware.
”We won’t send it out” is not something willpower can enforce
A quick word on the mechanics. You sometimes see the explanation “our policy is not to transmit externally.” Good intentions have never held that line. It needs to be impossible, not discouraged.
What we do is simple enough: there is a single checkpoint between the user and the AI. Everything about what passes and what does not is decided there. The important part is that the checkpoint sits on our side, which is also why the rules can be changed later.
Roughly, these are the things we have settled:
- Outbound traffic is blocked by default. Anything allowed through is named explicitly
- Who asked what is recorded — not to catch anyone out, but so we can answer if asked
- Each use has a defined scope, and the AI does not wander outside it
- If it is not in the documents, the answer is “that is not covered here”
That last one is quietly excellent. You can configure it not to bluff. It is the most practical answer we have found to the worry everyone raises: “won’t it confidently make things up?”
With all that said, honestly: no setup reduces the risk to zero. Rather than promising perfect safety, it is more realistic to decide in advance what goes out and how far.
In closing
You may be thinking this all sounds too big for your company. We started with a single machine. We did add a UPS so it would not drop during a power cut, but that is about the extent of it.
Last of all, do talk to the fairy in the bottom-right corner.
The point about this fairy is that we can leave her open to anyone, for free. If she ran on a pay-per-request API, every conversation would add to a bill, and we could not afford to be so generous. She runs on electricity, so we can leave the door open.
For the few seconds it takes her to reply, nothing you wrote leaves our building. That really is all there is to it.
Except that we have watched a lot of companies fail to get started on AI purely because that one thing was never settled. It seems like a waste.
If this article has left you wondering what it would look like at your own company, the private LLM infrastructure page may be worth a look too. And you can always get in touch — we would be glad to talk it through.