Build the system, not the prompt: how I approach an LLM project

Most LLM projects start in the wrong place. Someone opens a playground, writes a clever prompt, gets a good answer, and decides the hard part is done. Then it meets real traffic and falls apart: slow, expensive, wrong in ways the demo never showed.

The prompt is the easy 10%. The system around it — the part that decides what the model sees, what happens when it's wrong, what it costs, and how fast it answers — is the actual job. Here's the order I work in, and why each step sits where it does.

1. Start from the constraints, not the model

Before I think about which model or what prompt, I write down the constraints, because they decide everything else:

• How fast does this have to answer? "Under a second, on a phone call" is a completely different system from "within a few seconds, in a web app."
• What does it cost per call, and how many calls a day? Cheap-and-frequent and expensive-and-rare pull in opposite directions.
• What happens when it's wrong? A wrong answer in a draft is fine. A wrong answer that tells someone the wrong loan balance is not.

These numbers are the spec. A voice bot that has to reply in 800 ms can't use the same approach as one that can think for five seconds, no matter how good the prompt is. Pick the model and the architecture after you know the limits, not before.

2. Define the loop and the contract first

Every LLM system is a loop: something comes in, the model does a unit of work, something goes out, repeat. Before any prompting, I nail down the shape of that loop and the contract at each step — exactly what goes in and exactly what must come out.

"Come out as good text" is not a contract. "Come out as JSON with these three fields, or it's rejected and retried" is. The moment you commit to a strict output shape, the model stops being a chat partner and becomes a component you can build on. Everything downstream — parsing, validation, the next step — depends on that contract holding.

So I decide: what's the unit of work per iteration, what's the exact input, what's the exact output, and what does "done" mean. Get that wrong and no prompt will save you.

3. Map where state, latency, and cost actually live

Now I draw the pipeline and mark three things on it: where state lives, where time is spent, and where money is spent. They're rarely where you'd guess.

In a voice loop, most of the felt latency isn't the model — it's waiting to decide the person finished talking. In an agent loop, most of the cost isn't the smart step — it's the same cheap call running fifty times. You can't see any of this from the prompt. You have to look at the whole path and ask, at each hop: what's being held here, what are we waiting on, what are we paying for?

This map is what tells you where to spend effort. Optimizing the wrong hop is the most common waste in these systems.

4. Decide deterministic vs. model-driven, per step

This is the judgment call that separates a system that holds up from one that's flaky. For every step in the loop, I ask: does this need the model, or can plain code do it?

The model is the most expensive, slowest, least predictable component you have. Use it only where you actually need its judgment. Routing, validation, formatting, state transitions, "has this been said already" — that's code. Understanding a messy human sentence, deciding what to say — that's the model. Push as much as you can onto the deterministic side. Every step you hand to code is one fewer place the system can hallucinate, stall, or surprise you.

5. Instrument before you optimize

I don't tune anything until I can measure it. That means, per call: tokens in and out, cost in real money, and the time each stage took — tagged by which part of the system made the call.

This isn't bureaucracy, it's the difference between fixing problems and guessing at them. Once the numbers are there, the expensive call names itself, the slow stage names itself, and a change that triples your output length shows up the next day instead of on the invoice. Optimizing before you can see is just rearranging code and hoping.

6. Build the fallback ladder

The model will be wrong, return junk, or not respond. A system that assumes it won't is a demo. So before I call anything finished, I design what happens at each failure: the JSON didn't parse, so retry with a stricter instruction; the first approach failed, so fall back to a simpler one; the model is stuck, so escalate or hand off. In voice, "the answer isn't ready yet" gets a short filler so the line never goes silent.

The ladder is most of the engineering. The happy path is short. The interesting work is everything that happens when the happy path doesn't.

7. Ship, measure on real traffic, iterate

Real inputs are weirder than anything you'll imagine at your desk. So I get it in front of real traffic early — even a narrow slice — and watch the numbers from step 5. Where does it stall, what does it cost, where is it wrong. Then I change one thing, measure again, and keep what helped. That loop — ship, measure, adjust — is the whole job once a first version exists. Not big rewrites. Small changes against real data.

The point

The model is one unreliable component inside a system you design around it. The work is the constraints, the contract, the map, the deterministic/model split, the instrumentation, and the fallbacks. Do those well and a mediocre prompt holds up in production. Skip them and the best prompt in the world won't.

---

Neville James Achieng builds LLM and voice systems in Nairobi. github.com/Neville777.