Building LLM software lives between two worlds

Classic software is deterministic. You separate storage from compute, data from process, and you get repeatable behavior. LLM apps add a probabilistic core into that tidy stack. They sit between data science and traditional engineering. If you live on one side, you have to pick up habits from the other.

Below is a short field guide: what changes, what actually moves the needle, why projects fail, and how to run a tight build-and-evaluate loop that makes this stuff reliable.

Old vs. probabilistic software (mindset + architecture)

In classic systems you can assume the same input yields the same output. With LLMs, the same input can produce different outputs and can degrade as the surrounding context changes. Treat the model like a powerful but flaky coworker with a short-term memory.

A practical split that works:

• Database: source of truth. Facts, policies, user state, and logs live outside the model.
• LLM: orchestrator and sometimes worker. It plans, routes, and synthesizes. It calls tools and fetches data, then drafts an answer.
• Context window: working memory. Put only what is needed for this step and keep durable state elsewhere.
• External memory: long-term recall. Use retrieval over curated stores for knowledge and history.

Design as if you were building around a flaky RPC: add retries, timeouts, idempotency, and guardrails. Keep boundaries sharp. Reliability comes from the model together with the rules and infrastructure around it.

What actually moves the needle in LLM apps

Most wins come from making the surrounding system legible and controlled. Swapping to a bigger model is usually a later optimisation.

• Structured outputs. Ask for JSON that matches a schema, validate it, and retry on failure. This shrinks a big class of downstream errors.
• High-quality retrieval. Clean your corpus, chunk by meaning, add headings and metadata, deduplicate near-duplicates, and store provenance so answers can cite sources. Add query rewriting and reranking before you consider a model change.
• Tight prompts and templates. For multi-step deliverables (reports, emails, forms), give explicit formats and examples. The less the model guesses, the better it behaves.
• Slice-aware thinking. Identify fragile cases early: dates, prices, IDs, tables, multi-hop instructions, and policy-guarded content. Write targeted checks for them.
• Observability from day one. Log traces, tokens, latency, cost, retrieved chunks, and tool calls. If you cannot see it, you cannot improve it.

Why projects fail (recurring anti-patterns)

• Tools-first, evals-later. Teams buy frameworks and dashboards and only then ask what “good” means. Start with concrete failure modes from real transcripts.
• Letting the model be memory. Packing long histories into prompts causes drift, leakage, and cost spikes. Move stable facts to external stores and fetch them.
• Retrieval blind spots. Poor chunking, noisy text, stale indices, or missing metadata lead to hallucinations that are actually data issues.
• Eval mismatch with humans. If your score does not track user satisfaction, you will optimize the wrong thing.
• Security gaps in agent flows. Untrusted inputs plus tool access plus the ability to change state is dangerous. Reduce capabilities, isolate tools, and require approval for sensitive actions.
• Over-automation too early. Start with a narrow job and a deterministic workflow. Add autonomy only where it earns its keep.

Shipping well: evals, retrieval, and structured outputs

Focus on the process. The goal is agreement between your evaluation and what users consider “good.”

1. Build an error taxonomy. Read real sessions. Label failures in plain language: wrong fact, missing citation, format error, bad tone, unsafe action, wrong tool, stale data. This becomes your checklist.

2. Write binary checks for each error. Prefer pass/fail over vague scores. Examples: “JSON matches schema,” “All IDs present,” “Every citation corresponds to retrieved text,” “No future-dated claims,” “No hidden instructions executed.”

3. Separate retrieval and generation. Measure retrieval precision/recall independently from answer faithfulness/correctness. If retrieval is weak, fix the index and queries first. If generation is unfaithful, tighten prompts, add constrained decoding, or post-validate facts against retrieved text.

4. Mix offline and online signals.

   Offline: a small golden set, synthetic expansions for edge cases, and adversarial tests for slices you know are brittle.

   Online: shadow runs on real traffic, pre/post rollouts with deltas on quality, latency, and cost. Block launches if key slices regress.

5. Enforce structured outputs. Define the schema before you prompt. Validate strictly. On failure, repair or reprompt with the error message. For long outputs, use incremental or partial validation to avoid throwing everything away.

6. Keep retrieval practical. Use semantic units as the primary chunk boundary and fixed size as a secondary constraint. Prefer fewer, higher-quality chunks to many tiny ones. Attach titles, sections, dates, and IDs. Rewrite queries to include constraints the user implied but did not state. Rerank with a lightweight model before you send context to the LLM.

Operator wisdom: where teams get real lift

• Narrow the job-to-be-done. A focused assistant with a strong template usually beats a general agent. Define success in one sentence the team agrees on.
• Design for CAIR: confidence in results. Show citations, show what was retrieved, and make correction easy. Users forgive minor misses when they can inspect and fix.
• Own budgets. Put ceilings on tokens, context size, and tool calls. Track cost per successful task; cost per call misses whether the work succeeded.
• Fail safe, then fail fast. If a tool call is risky, require confirmation. If retrieval is empty, return a graceful fallback rather than guessing.
• Make state explicit. Keep a typed state object across steps: user intent, constraints, retrieved evidence, intermediate outputs, and decisions taken. Keep durable state out of prompts.
• Ship with a kill switch and feature flags. Roll out by cohort. If a new prompt or retriever hurts a slice, turn it off without a redeploy.
• Continuously prune. Remove unused tools, stale prompts, and dead branches in the flow. The smaller the surface, the fewer surprises.

LLM applications combine a probabilistic engine with deterministic scaffolding. Data people need software discipline around contracts, observability, and rollouts. Software people need evaluation, retrieval quality, and slice-aware thinking. Reliable products emerge where those practices meet.