Most test suites are theater. They pass, the CI badge is green, everyone feels good, and then a customer reports a bug that slipped through 400 tests without triggering a single failure. After eight years of writing tests across ML platforms, SaaS products, and security tools, I have developed strong opinions about what separates tests that catch real bugs from tests that exist only to inflate coverage metrics. This post is a practical guide to writing the former.

The Coverage Trap

Let me start with the most common failure mode: optimizing for code coverage. A team sets a coverage target (80% is the usual number), engineers write tests to hit that target, and the test suite grows to hundreds of tests that each exercise a code path but verify almost nothing meaningful.

Here is a real example from a codebase I inherited. The function calculates a discount based on user tier and purchase history:

def calculate_discount(user, cart):
    if user.tier == "gold" and cart.total > 100:
        return 0.15
    elif user.tier == "silver" and cart.total > 50:
        return 0.10
    elif len(user.purchase_history) > 20:
        return 0.05
    return 0.0

The existing test suite had four tests, one for each branch, achieving 100% coverage:

def test_gold_discount():
    user = User(tier="gold", purchase_history=[])
    cart = Cart(total=150)
    assert calculate_discount(user, cart) == 0.15

def test_silver_discount():
    user = User(tier="silver", purchase_history=[])
    cart = Cart(total=75)
    assert calculate_discount(user, cart) == 0.10

def test_loyalty_discount():
    user = User(tier="bronze", purchase_history=["x"] * 21)
    cart = Cart(total=30)
    assert calculate_discount(user, cart) == 0.05

def test_no_discount():
    user = User(tier="bronze", purchase_history=[])
    cart = Cart(total=30)
    assert calculate_discount(user, cart) == 0.0

100% line coverage. 100% branch coverage. And the test suite misses at least four bugs:

1. What happens when a gold user has 25+ purchases and a cart over $100? They should arguably get the best applicable discount, but the code returns 0.15 (the first match) without checking if the loyalty discount combined with tier discount would be more favorable. The requirement said “apply the highest applicable discount” but the implementation applies the first matching discount.
2. What happens when cart.total is exactly 100 for a gold user? The condition is > 100, not >= 100. This is a classic off-by-one that no amount of happy-path testing reveals. The product spec said “purchases of $100 or more” which implies >=.
3. What happens when user.tier is None or an unexpected string like "GOLD" (uppercase)? The code silently returns 0.0, which might be correct or might be a data integrity bug masquerading as a valid result. In the production database, 0.3% of users had uppercase tier values due to a migration issue, and they were all silently denied discounts they were entitled to.
4. What happens when cart.total is negative (refund scenario)? The function happily applies a 15% discount to a negative total, which in some contexts means the user pays more rather than less. The function should reject negative totals or handle them as a special case.

The problem is not that the tests are wrong. Each test correctly verifies what it claims to verify. The problem is that the tests were designed to cover code paths, not to catch bugs. There is a fundamental difference between “this code path executes without errors” and “this code path produces correct results for the range of inputs it will encounter in production.”

Boundary Testing: Where Bugs Actually Live

The single highest-ROI testing technique is boundary testing. Bugs cluster at boundaries: the edge of a valid range, the transition between two states, the point where one condition stops being true and another starts. If you only test the middle of each range, you will miss the majority of real-world bugs.

For the discount function above, boundary tests look like this:

import pytest

@pytest.mark.parametrize("total,expected", [
    (99.99, 0.0),     # Just below threshold
    (100.00, 0.0),    # Exactly at threshold (reveals the > vs >= bug)
    (100.01, 0.15),   # Just above threshold
])
def test_gold_discount_boundary(total, expected):
    user = User(tier="gold", purchase_history=[])
    cart = Cart(total=total)
    assert calculate_discount(user, cart) == expected

@pytest.mark.parametrize("history_length,expected", [
    (19, 0.0),   # Just below loyalty threshold
    (20, 0.0),   # Exactly at threshold (reveals > vs >= bug)
    (21, 0.05),  # Just above threshold
])
def test_loyalty_discount_boundary(history_length, expected):
    user = User(tier="bronze", purchase_history=["x"] * history_length)
    cart = Cart(total=30)
    assert calculate_discount(user, cart) == expected

This is not a novel technique. Boundary value analysis has been in testing textbooks since the 1970s. But in practice, fewer than 10% of the test suites I review include systematic boundary tests. Engineers default to “pick a representative value from the middle of the range” because it is faster to write and easier to reason about. The result is test suites that verify the obvious cases and miss the subtle ones.

A practical rule of thumb: for every conditional with a numeric comparison, write three tests: one below the boundary, one at the boundary, and one above the boundary. For string comparisons, test the exact match, a near-miss, and an empty or null input. This mechanical approach catches a disproportionate number of real bugs relative to the effort involved.

Property-Based Testing: Let the Machine Find Edge Cases

Writing boundary tests by hand works when you can enumerate the boundaries. But for functions with complex input spaces, manual enumeration misses cases. Property-based testing (PBT) automates the search for failing inputs by defining properties that should always hold, then generating thousands of random inputs to verify them.

Hypothesis (Python) and fast-check (JavaScript) are the two mature PBT libraries. Here is how you would test the discount function with Hypothesis:

from hypothesis import given, settings, assume
from hypothesis.strategies import floats, text, integers, sampled_from

@given(
    tier=sampled_from(["gold", "silver", "bronze", "platinum", None, ""]),
    total=floats(min_value=-1000, max_value=100000, allow_nan=False),
    history_length=integers(min_value=0, max_value=1000)
)
@settings(max_examples=5000)
def test_discount_properties(tier, total, history_length):
    user = User(tier=tier, purchase_history=["x"] * history_length)
    cart = Cart(total=total)
    discount = calculate_discount(user, cart)

    # Property 1: Discount is always between 0 and 1
    assert 0.0 <= discount <= 1.0

    # Property 2: Discount should never be applied to negative totals
    if total < 0:
        assert discount == 0.0, f"Discount {discount} applied to negative total {total}"

    # Property 3: Higher tiers should never get worse discounts
    # (for the same cart and history)
    if tier == "gold":
        silver_user = User(tier="silver", purchase_history=user.purchase_history)
        silver_discount = calculate_discount(silver_user, cart)
        assert discount >= silver_discount

When I ran this against the original function, Hypothesis found the negative total bug in 47 examples. It also found that a platinum-tier user (a tier that exists in the database but was not handled in the function) gets zero discount, which is arguably incorrect business logic. The property-based test did not need me to anticipate these specific edge cases. I defined the properties that should hold, and the tool found inputs that violated them.

The key insight with PBT is choosing the right properties. Common property patterns that catch real bugs:

• Roundtrip properties: deserialize(serialize(x)) == x for any valid input. This catches serialization bugs that happy-path tests miss. We use this extensively for our API payloads: serializing a request object to JSON and back should produce an identical object.
• Monotonicity: If input A is “greater” than input B, output A should be “greater” than or equal to output B. Catches ordering bugs and boundary errors.
• Idempotency: f(f(x)) == f(x) for operations that should be idempotent. Catches state mutation bugs. We use this for all our API PUT endpoints: applying the same update twice should produce the same result as applying it once.
• Commutativity: f(a, b) == f(b, a) when order should not matter. Catches subtle ordering dependencies.
• No-crash: The function should not throw an unhandled exception for any valid input. This is the simplest property and catches a surprising number of bugs, especially null pointer exceptions and type errors on unexpected input shapes.

Testing Error Paths: The Most Neglected Category

In my experience, error handling code has 3-5x more bugs per line than happy-path code. The reason is straightforward: happy-path code is exercised constantly during development and manual testing, while error paths are exercised rarely or never until they encounter real failures in production.

Specific patterns for testing error paths:

Test that errors propagate correctly. If a database query fails, does the API return a 500 with a useful error message, or does it return a 200 with a null body? This is shockingly common: developers write try/catch blocks that swallow exceptions and return default values, turning hard failures into silent data corruption. We found a bug last quarter where a payment service returned {"status": "success", "amount": null} when the payment gateway was down, because the error handler caught the connection timeout and returned a default response object.

// Bad: swallows the error
async function getUser(id: string): Promise<User | null> {
  try {
    return await db.users.findById(id);
  } catch (e) {
    return null;  // Database is down, but caller thinks user doesn't exist
  }
}

// Test that catches this
test('getUser propagates database errors', async () => {
  db.users.findById = jest.fn().mockRejectedValue(
    new Error('connection refused')
  );
  await expect(getUser('abc')).rejects.toThrow('connection refused');
});

Test timeout behavior. If your function calls an external service with a 5-second timeout, verify that the timeout actually fires and that the function handles it correctly. I have seen production incidents caused by timeouts that were configured but never tested, where the timeout handler had a bug that caused a different exception than the one the caller expected. Test that your timeout produces the specific error type your retry logic expects.

Test partial failures. If your function processes a batch of 100 items and item #47 fails, what happens? Does it abort the entire batch? Continue processing and report the failure? Silently skip the failed item? The answer should be deliberate and tested, not accidental.

@pytest.mark.parametrize("failing_index", [0, 49, 99])
def test_batch_processor_handles_individual_failures(failing_index):
    items = [valid_item() for _ in range(100)]
    items[failing_index] = invalid_item()

    result = process_batch(items)

    assert result.total_processed == 99
    assert result.total_failed == 1
    assert result.failures[0].index == failing_index
    assert result.failures[0].error is not None

Test resource cleanup on failure. If your function opens a database connection, then fails during processing, does it close the connection? Resource leaks under error conditions are a common source of production outages. The function works fine for hours, then a transient error causes a connection leak, and the connection pool slowly fills up until the service cannot handle any requests. Test that resources are cleaned up even when the function throws.

Test Doubles: Mocks, Stubs, and When Each Is Appropriate

The mock-versus-integration-test debate generates more heat than light. The practical answer is: use mocks for things you do not own, use real implementations for things you do own, and use fakes for things that are too slow or expensive to use in tests.

Mock external services. Your test suite should not break because Stripe’s API is down. Mock the HTTP client that calls Stripe. But mock at the HTTP boundary, not deep inside your code. If you mock your PaymentService class, you are not testing the integration between your code and the payment service client library. If you mock the HTTP client that PaymentService uses, you test everything except the actual HTTP call. The distinction matters because client library bugs are a real source of production issues.

Use real databases. SQLite in-memory or TestContainers for PostgreSQL/MySQL. Mocking your database layer is the single most common cause of tests that pass in CI and fail in production. The database has behavior (constraint enforcement, transaction isolation, type coercion, collation rules) that mocks do not replicate. A test that verifies a unique constraint violation by checking that your code calls db.insert() proves nothing. A test that actually inserts two rows with the same unique key and verifies the error handling proves something real. We switched from mock-based database tests to TestContainers-based tests and immediately found 7 bugs that the mocked tests had been hiding for months.

Use fakes for expensive operations. If you are testing a function that calls an ML model, use a fake model that returns deterministic results. The fake should implement the same interface as the real model but return hardcoded outputs for known inputs. This is faster than mocking because the fake lives in-process and does not require configuring mock return values for every test case.

class FakeEmbeddingModel:
    """Deterministic fake for testing. Returns consistent embeddings
    based on input hash so tests are reproducible."""

    def embed(self, text: str) -> list[float]:
        # Produce a deterministic 384-dim vector from the input
        seed = hash(text) % (2**32)
        rng = random.Random(seed)
        return [rng.gauss(0, 1) for _ in range(384)]

    def embed_batch(self, texts: list[str]) -> list[list[float]]:
        return [self.embed(t) for t in texts]

Mutation Testing: Measuring Test Quality, Not Quantity

If code coverage tells you how much code your tests execute, mutation testing tells you how much code your tests actually verify. Mutation testing works by making small changes (mutations) to your source code and checking whether your test suite detects each change. If a mutation survives (tests still pass after the change), your tests are not verifying that part of the code effectively.

Tools like mutmut (Python), Stryker (JavaScript/TypeScript), and pitest (Java) automate this process. Here is what a mutation testing report reveals:

# Original code
if user.age >= 18:
    return "eligible"

# Mutation: changed >= to >
if user.age > 18:
    return "eligible"

# If your tests still pass with this mutation,
# you are not testing the boundary at age=18

When we introduced mutation testing to VibeGuard’s test suite, we had 92% line coverage and a mutation score of 61%. That means 39% of our code could be subtly changed without any test failing. After a focused week of writing boundary tests and property tests for the surviving mutations, we raised the mutation score to 84% and caught three latent bugs in the process: a comparison that should have been <= instead of <, a null check that used loose equality instead of strict equality, and a loop that iterated to length instead of length - 1.

Mutation testing is slow (it runs your entire test suite once per mutation, and a typical codebase generates thousands of mutations), so we run it nightly rather than on every commit. We use Stryker with a mutation budget of 60 minutes, which covers our most critical modules. The insights it provides are invaluable for identifying weak spots in your test suite. If you have never run a mutation testing tool on your codebase, I guarantee the results will be humbling. The gap between “all tests pass” and “all tests actually verify something” is consistently larger than teams expect.

The Testing Pyramid Is Wrong (For Most Teams)

The traditional testing pyramid says: many unit tests, fewer integration tests, even fewer end-to-end tests. This made sense when unit tests were fast and integration tests required spinning up physical servers. In 2024, with TestContainers, in-memory databases, and sub-second Docker startup times, the cost of integration tests has dropped dramatically.

Our current testing strategy at Harbor Software is closer to a diamond: a moderate number of unit tests for pure functions and algorithms, a large number of integration tests that exercise real database and service interactions, a moderate number of end-to-end tests for critical user journeys, and property-based tests that cross-cut all levels.

The unit tests catch logic bugs in isolated functions. The integration tests catch interaction bugs between components. The end-to-end tests catch deployment and configuration bugs. And the property-based tests catch edge cases that none of the other categories would find because no human thought to test that specific input combination.

This approach results in a test suite that takes 4 minutes to run (compared to 45 seconds for a unit-test-only suite) but catches approximately 3x more regressions before they reach production. We measured this by tracking the number of bugs found in production over two six-month periods: one with a unit-test-heavy approach and one with our current diamond approach. Production bugs dropped from 2.1 per month to 0.7 per month. The 3.5-minute additional CI time pays for itself many times over when a single production bug costs 2-4 hours of incident response.

Write fewer tests that verify more. Test boundaries, not midpoints. Test errors, not just success. Let machines find edge cases you would never think of. And measure your test quality by mutation score, not coverage percentage. Your customers will thank you by not filing bug reports.