Harbor Software shipped its first AI-powered product in mid-2020. Five years later, we have built and maintained inference pipelines, recommendation engines, document processing systems, procurement intelligence platforms, and autonomous agent workflows for clients ranging from Series A startups to Fortune 500 enterprises. Along the way, we accumulated a set of hard-won lessons that no conference talk or blog post could have prepared us for. This is not a retrospective about technology choices. It is about the patterns that separate AI products that survive from those that collapse under the weight of their own complexity.

Lesson 1: The Model Is Never the Hard Part

Every prospective client conversation starts the same way. They want to talk about which model to use, whether GPT-4 is better than Claude, whether they need to fine-tune, whether they should train from scratch. These are reasonable questions, and they account for roughly 15% of the actual difficulty of shipping an AI product.

The hard parts, consistently, across every project we have delivered:

• Data plumbing. Getting the right data into the right format, from the right sources, with the right freshness, at the right cost. We have spent more cumulative engineering hours on ETL pipelines, data validation, and schema reconciliation than on model selection and prompt engineering combined. One enterprise client had 14 separate data systems that needed to feed into a single recommendation engine. The integration layer took 4 months. The model layer took 3 weeks. The data sources included a mainframe system from 1998 that exposed data via nightly CSV dumps, a modern REST API with rate limits of 100 requests per minute, and a Salesforce instance with custom objects nested 6 levels deep. Reconciling entity IDs across these systems—where the same customer might be identified by an account number, an email address, or a CRM ID depending on the source—consumed more debugging time than any model-related work.
• Evaluation infrastructure. How do you know the system is working? Not “working” in the sense of returning a 200 status code, but producing outputs that are actually correct, useful, and safe. Building robust evaluation suites—with human-labeled ground truth, automated regression tests, and statistical significance testing—is unglamorous work that determines whether your product improves or stagnates after launch. We have seen teams that skipped evaluation and spent 6 months “improving” a system that was actually getting worse, because they had no baseline to compare against. The evaluation suite is your compass; without it, you are navigating by feel.
• Edge case management. ML models are probabilistic. They will produce unexpected outputs on inputs that are technically valid but fall outside the training distribution. The difference between a demo and a product is 10,000 hours of edge case handling: input sanitization, output validation, graceful degradation, user-facing error messages that are actually helpful, and fallback paths that preserve the user experience when the model fails. One client’s document extraction system worked perfectly on the 500 sample documents we tested with, but failed on 12% of production documents because real-world documents included watermarks, handwritten annotations, multi-column layouts, and scanned pages that were rotated 90 degrees. Each edge case required specific handling—not a model fix, but an engineering fix in the pre-processing pipeline.
• Operational infrastructure. Monitoring, alerting, model versioning, A/B testing, canary deployments, rollback procedures, cost tracking, rate limiting. None of this is intellectually challenging. All of it is essential. We have seen teams that built extraordinary models fail to ship because they could not operate them reliably in production. A model that crashes at 3 AM and nobody knows until customers complain at 9 AM is worse than a mediocre model with excellent monitoring and automatic recovery.

The lesson is not that models do not matter. It is that the model is a component in a system, and the system’s reliability is determined by its weakest component. If your model is 95% accurate but your data pipeline drops 10% of inputs silently, your effective accuracy is 85.5%. If your evaluation suite does not catch regressions, every model update is a gamble. The teams that succeed invest proportionally across the entire stack, not disproportionately in the model.

Lesson 2: Ship a Heuristic First, Then Add ML

This is the single most counterintuitive lesson, and we learned it by violating it repeatedly in our first two years. When a client comes to us with an AI problem, our first question is: can we solve 80% of this with rules?

A concrete example. A logistics company wanted to classify incoming customer support tickets into 23 categories. The obvious approach: fine-tune a text classifier, build a training pipeline, deploy an inference endpoint, monitor for drift. We estimated 8 weeks of work.

Instead, we spent 3 days building a keyword-based classifier. We analyzed 5,000 historical tickets, identified the 50 most discriminative keywords and phrases for each category, and wrote a scoring function that weighted keyword matches by position (subject line matches weighted 3x body matches) and frequency. This heuristic classifier achieved 71% accuracy on a held-out test set.

We shipped it. The client started using it. Over the next 4 weeks, the client’s support team corrected misclassifications, generating a clean labeled dataset of 12,000 tickets. We then fine-tuned a DistilBERT classifier on this labeled data, achieving 89% accuracy. Total time from kickoff to ML-powered classification: 7 weeks—comparable to the original estimate—but with a critical difference: the client had a working (if imperfect) system from week one, and the ML model was trained on their data, corrected by their team, reflecting their actual category definitions rather than our assumptions about what those definitions should be.

The heuristic-first approach delivers three things that the ML-first approach does not:

1. Immediate value. A 71% accurate system deployed today is more valuable than a 89% accurate system deployed in 8 weeks, because it starts saving time immediately. The client’s support team was spending 4 hours per day manually routing tickets. Even an imperfect classifier reduced that to 1.5 hours from day one.
2. Clean training data. Human corrections on the heuristic’s output produce high-quality labeled data that is tailored to the actual distribution of inputs, not a synthetic or generic dataset. The corrections also surface ambiguous categories that the client had not clearly defined—categories where even human classifiers disagreed 30% of the time. This led to a category taxonomy revision that improved both human and ML classification accuracy.
3. A performance baseline. When you ship the ML model, you know exactly how much better it is than the simple approach. This makes ROI conversations concrete rather than hypothetical. The ML model improved accuracy from 71% to 89%—an 18 percentage point improvement—which reduced daily manual routing time from 1.5 hours to 35 minutes.

We have used this pattern on document extraction (regex first, then NER models), anomaly detection (statistical thresholds first, then autoencoders), and content moderation (keyword lists first, then classifiers). It works every time. The heuristic is never the final product, but it is always the right starting point. Teams that skip the heuristic phase typically spend weeks building ML infrastructure before validating that the problem is worth solving at all.

Lesson 3: Prompt Engineering Is Software Engineering

When LLMs became the default building block for AI products in 2023, we initially treated prompts as configuration—strings that lived in config files and were tweaked ad hoc. This was a mistake that cost us roughly 6 weeks of cumulative debugging time across three projects before we corrected course.

Prompts are code. They have the same properties as code: they can be buggy, they can regress, they interact with each other in unexpected ways, and they need version control, testing, and review. We now apply the same engineering discipline to prompts that we apply to any other software artifact:

# prompt_registry.py
from dataclasses import dataclass
from typing import Dict
import hashlib

@dataclass(frozen=True)
class Prompt:
    name: str
    version: str
    template: str
    model: str
    temperature: float
    max_tokens: int
    
    @property
    def fingerprint(self) -> str:
        content = f"{self.template}{self.model}{self.temperature}"
        return hashlib.sha256(content.encode()).hexdigest()[:12]

PROMPTS: Dict[str, Prompt] = {
    "classify_ticket": Prompt(
        name="classify_ticket",
        version="3.1.0",
        template="""Classify the following support ticket into exactly one category.

Categories: {categories}

Ticket subject: {subject}
Ticket body: {body}

Respond with only the category name, nothing else.""",
        model="claude-sonnet-4-20250514",
        temperature=0.0,
        max_tokens=50,
    ),
}

Every prompt is versioned with semantic versioning. Template changes that alter behavior bump the minor version. Model or parameter changes bump the major version. We run evaluation suites against prompt changes the same way we run test suites against code changes. A prompt PR requires the same review rigor as a code PR: does the change improve accuracy on the evaluation set? Does it introduce regressions on known edge cases? Is the cost impact acceptable?

The practical impact of this discipline was measurable. Before we implemented prompt versioning, we had 3 production incidents caused by prompt changes that introduced regressions. After implementing the prompt registry and mandatory evaluation, we had zero prompt-related incidents over 14 months. The evaluation suite catches regressions before they reach production, and the version history lets us pinpoint exactly when and why a prompt’s behavior changed. When a model provider updates their model, we re-run our evaluation suite against the new model version, and the fingerprint system automatically detects that the prompt-model combination has changed, triggering a review.

The teams that treat prompts as magic incantations—tweaking words here and there based on intuition—invariably end up with fragile, untestable systems that break in unpredictable ways when the underlying model is updated. The teams that treat prompts as versioned, tested software artifacts build systems that are maintainable, debuggable, and improvable over time.

Lesson 4: Cost Awareness Must Be Architectural, Not Operational

In 2023, one of our clients racked up $47,000 in LLM API costs in a single month. The system was working correctly—it was just calling Claude 3 Opus for every single request, including trivial ones that a much cheaper model (or a cached response) could have handled. The fix was architectural: we built a request router that classified incoming requests by complexity and routed them to the appropriate model tier.

class ModelRouter:
    def __init__(self):
        self.tiers = {
            "simple": {"model": "claude-haiku", "cost_per_1k": 0.00025},
            "standard": {"model": "claude-sonnet", "cost_per_1k": 0.003},
            "complex": {"model": "claude-opus", "cost_per_1k": 0.015},
        }
    
    def route(self, request: dict) -> str:
        input_tokens = estimate_tokens(request["prompt"])
        requires_reasoning = request.get("requires_reasoning", False)
        has_code = request.get("contains_code", False)
        
        if input_tokens < 200 and not requires_reasoning:
            return "simple"
        elif has_code or requires_reasoning:
            return "complex"
        else:
            return "standard"

After deploying the router, monthly costs dropped to $11,200—a 76% reduction—with no measurable impact on output quality, because 68% of requests were genuinely simple and did not need the most capable model. We have since refined this pattern with response caching (identical inputs get cached responses, reducing API calls by an additional 30-40%), semantic caching (similar-enough inputs get cached responses, using embedding similarity with a 0.97 threshold), and adaptive routing (the router learns from user feedback which request types need which model tier).

The cache layer deserves specific mention because the implementation details matter. We use a two-tier cache: an in-memory LRU cache (using Python's functools.lru_cache with a 10,000-entry limit) for exact-match caching, and a Redis-backed semantic cache for approximate-match caching. The exact-match cache has a hit rate of 23% on production traffic—meaning nearly a quarter of all requests are served from cache with zero API cost and sub-millisecond latency. The semantic cache adds another 12% hit rate for requests that are not identical but semantically equivalent (different wording, same intent). Together, these caches reduce our effective API cost by 35% on top of the savings from model routing.

The architectural lesson: cost controls must be designed into the system, not bolted on after the bill arrives. Every AI system we build now includes a cost tracking layer from day one, with per-request cost attribution, daily budget alerts, and automatic fallback to cheaper models when budgets are exhausted. This is not premature optimization. It is responsible engineering. LLM costs scale linearly with usage, and linear cost scaling on exponential usage growth will bankrupt you. We have a client whose usage grew 8x over 6 months; without the routing and caching infrastructure, their monthly API bill would have grown from $11,200 to $89,600. With it, the bill grew to $28,400—still significant, but manageable.

Lesson 5: Your Evaluation Suite Is Your Most Valuable Asset

More valuable than your model. More valuable than your prompts. More valuable than your data pipeline. Your evaluation suite is the single artifact that determines whether your system is improving or degrading over time.

We maintain evaluation suites for every AI system we operate. Each suite consists of:

• Golden datasets: 200-500 hand-labeled examples per task, reviewed quarterly, covering the full distribution of inputs including edge cases. These are labeled by domain experts, not annotators, because domain expertise matters for ambiguous cases. Labeling is expensive—typically $5,000-$10,000 per golden dataset, including expert time and review cycles—but it is the foundation that everything else rests on. Cheap labels produce cheap evaluations that miss real problems.
• Automated metrics: Accuracy, precision, recall, F1, latency percentiles (p50, p95, p99), cost per request, error rate. These run on every commit to the AI components. We use a custom evaluation harness that runs the full golden dataset against the current system, compares results against the baseline, and generates a report that includes both aggregate metrics and per-example results. The per-example results are crucial for debugging: when accuracy drops by 2%, you need to know which examples regressed, not just that something got worse.
• Regression tests: Specific inputs that caused production incidents in the past. If the system ever gets these wrong again, the test fails. We have accumulated 340 regression test cases across all projects. Each one represents a real production failure that we never want to repeat. The regression suite is append-only—we never remove cases, even if they seem redundant, because removing a regression test is how regressions recur.
• Statistical significance testing: When comparing two prompt versions or model configurations, we do not eyeball the accuracy numbers. We run paired bootstrap tests with 10,000 iterations to determine whether the difference is statistically significant at p < 0.05. This has prevented us from shipping "improvements" that were actually within the noise margin at least a dozen times. A 0.5% accuracy improvement that is not statistically significant is not an improvement—it is noise, and shipping it adds complexity without adding value.
• Human evaluation loops: For tasks where automated metrics are insufficient (summarization quality, response helpfulness, tone appropriateness), we run periodic human evaluations where domain experts rate system outputs on a 1-5 Likert scale. These evaluations happen monthly and are calibrated against inter-rater reliability scores to ensure consistency across evaluators.

Building evaluation infrastructure is not exciting. It does not demo well. Nobody at a conference wants to hear about your test harness. But it is the thing that prevents your AI product from slowly degrading into unreliability, and it is the thing that gives you confidence to ship improvements quickly. Teams without robust evaluation infrastructure are flying blind—they ship changes and hope for the best. Teams with robust evaluation infrastructure ship changes and know whether they helped.

Lesson 6: Explain the Uncertainty

Every AI system produces uncertain outputs. The question is whether you hide that uncertainty from users or make it visible and actionable. We have learned, consistently, that transparency about uncertainty increases user trust rather than decreasing it.

In our document classification system, we show confidence scores alongside classifications. When confidence is below 0.85, we flag the classification as "needs review" and surface it to a human reviewer. Users initially expected this would undermine their confidence in the system. The opposite happened: because the system honestly reported when it was uncertain, users trusted its high-confidence classifications more. Net trust went up, not down. Usage analytics confirmed this: users who saw confidence scores were 34% more likely to accept high-confidence classifications without manual verification, and 2.8x more likely to catch and correct errors on low-confidence classifications.

The pattern generalizes. When an LLM-powered system generates a response, include the sources it drew from. When a recommendation engine suggests an item, explain why ("purchased by customers who also bought X"). When an anomaly detector flags a transaction, show the specific features that triggered the flag and the historical baseline for comparison. These explanations do not need to be technically sophisticated. They need to give the human enough context to make an informed decision about whether to trust the output.

Systems that present AI outputs as infallible binary decisions ("this is fraud" / "this is not fraud") create two failure modes: users over-trust the system and miss real issues, or users lose trust entirely after a few visible mistakes and stop using the system. Systems that present AI outputs as recommendations with confidence levels and explanations give users the agency to apply their own judgment, which produces better outcomes and more sustainable adoption.

What Changed Between Year One and Year Five

In year one, we optimized for model accuracy. We spent weeks squeezing out incremental improvements on benchmark metrics, experimenting with architectures, tuning hyperparameters. The models we built were good, but the systems around them were fragile.

By year three, we optimized for system reliability. We invested in monitoring, testing, deployment automation, and operational tooling. Our models were no longer the best possible for any given task, but our systems were dependable. Clients stopped experiencing outages and silent failures. The shift was not a conscious strategic decision—it was forced on us by production incidents that made it clear that a 93% accurate model in a reliable system outperforms a 97% accurate model in a fragile one.

By year five, we optimize for time to value. How quickly can we get something useful into a client's hands? How quickly can we iterate based on their feedback? How quickly can we detect and fix problems? The specific technology matters less than the velocity of the feedback loop. A system built with a mediocre model but deployed in 2 weeks with solid evaluation will outperform a system built with a state-of-the-art model deployed in 3 months without evaluation, because the first system has had 10 weeks of production data and feedback that the second system lacks.

Five years of building AI products taught us that the technology is the easy part. The hard parts are understanding the problem deeply enough to build the right thing, engineering the system robustly enough to run it in production, and building the feedback loops to improve it continuously. Every team figures this out eventually. The ones that figure it out early build better products and waste less time on technical dead ends.

If you are starting an AI project today, begin with the evaluation suite. Define what success looks like before you write a single line of model code. Ship a heuristic, collect data, then add ML. Treat prompts as code. Build cost awareness into the architecture. And explain the uncertainty to your users. These lessons took us five years and millions of API calls to internalize. Hopefully they save you some of that time.