Testing Fundamentals

• 1: Test Architecture
• 2: What to Test - and What Not To
• 3: Pipeline Test Strategy
• 4: Getting Started
• 5: Defect Feedback Loop
• 6: Test Types
• 6.1: Component Tests
• 6.2: Contract Tests
• 6.3: End-to-End Tests
• 6.4: Integration Tests
• 6.5: Static Analysis
• 6.6: Smoke Tests
• 6.7: Unit Tests
• 7: Patterns
• 7.1: API Provider
• 7.2: API Consumer
• 7.3: Scheduled Job
• 7.4: User Interface
• 7.5: Event Consumer
• 7.6: Event Producer
• 7.7: CLI Tool or Library
• 7.8: Stateful Service
• 8: Applied Testing Strategies
• 8.1: Pre-Ship Checklist
• 8.2: Cross-Cutting Concerns
• 9: Test Feedback Speed
• 10: Testing Antipatterns
• 11: Testing Glossary

Phase 1 - Foundations | Scope: Team

Continuous delivery requires that trunk always be releasable, which means testing it automatically on every change. A collection of tests is not enough. You need a test architecture: different test types working together so the pipeline can confidently deploy any change, even when external systems are unavailable.

Testing Goals for CD

Your test suite must meet these goals before it can support continuous delivery.

Goal	Target	How to Measure
Fast	CI gating tests < 10 minutes; full acceptance suite < 1 hour	CI gating suite duration; full acceptance suite duration
Deterministic	Same code always produces the same result	Flaky test count: 0 in the gating suite
Catches real bugs	Tests fail when behavior is wrong, not when implementation changes	Defect escape rate trending down
Independent of external systems	Pipeline can determine deployability without any dependency being available	External dependencies in gating tests: 0
Test doubles stay current	Contract tests confirm test doubles match reality	All contract tests passing within last 24 hours
Coverage trends up	Every new change gets a test	Coverage percentage increasing over time

In This Section

Page	What You’ll Learn
Test Architecture	The principles behind a CD test suite: the pyramid, the trophy, and determinism
What to Test	Which boundaries matter and how to eliminate external dependencies from your pipeline
Pipeline Test Strategy	What tests run where in a CD pipeline and how contract tests validate test doubles
Getting Started	Audit your current suite, fix flaky tests, and decouple from external systems
Defect Feedback Loop	Trace defects to their origin and prevent entire categories of bugs
Test Types	Definitions of each test type and where it runs in the pipeline
Patterns	How to fully test eight common component patterns, from API providers to stateful services
Applied Testing Strategies	Cross-cutting concerns and a pre-ship checklist for any component
Test Feedback Speed	Why suite speed matters and the targets that follow from cognitive limits
Testing Antipatterns	Common testing antipatterns and how to get an existing suite back on track
Testing Glossary	Definitions for testing terms as this site uses them

The Ice Cream Cone: What to Avoid

An inverted test distribution, with too many slow end-to-end tests and too few fast unit tests, is the most common testing barrier to CD.

The ice cream cone makes CD impossible. Manual testing gates block every release. End-to-end tests take hours, fail randomly, and depend on external systems being healthy. For the test architecture that replaces this, see Pipeline Test Strategy and the Testing reference.

Next Step

Automate your build process so that building, testing, and packaging happen with a single command. Continue to Build Automation.

---

Content contributed by Dojo Consortium, licensed under CC BY 4.0. Additional concepts drawn from Ham Vocke, The Practical Test Pyramid, and Toby Clemson, Testing Strategies in a Microservice Architecture.

---

Related Content

• Flaky Tests - Symptom of non-deterministic tests that destroy pipeline trust
• High Coverage, Ineffective Tests - Symptom where coverage metrics mask poor test quality
• Refactoring Breaks Tests - Symptom of white-box tests that assert on implementation details
• Slow Test Suites - Symptom caused by an inverted test pyramid or missing test doubles
• Environment-Dependent Failures - Symptom of tests coupled to external systems
• Inverted Test Pyramid - Anti-pattern where too many slow E2E tests replace fast unit tests
• Pressure to Skip Testing - Anti-pattern where testing is treated as optional under deadline pressure

1 - Test Architecture

A test architecture that lets your pipeline deploy confidently, regardless of external system availability, is a core CD capability. The Test Types pages cover each type in depth.

A CD pipeline’s job is to force every artifact to prove it is worthy of delivery. That proof only works when test changes ship with the code they validate. If a developer adds a feature but the corresponding tests arrive in a later commit, the pipeline approved an artifact it never actually verified. That is not a CD pipeline. It is a CI pipeline with a deploy step. Tests and production code must always travel together through the pipeline as a single unit of change.

Beyond the Test Pyramid

The test pyramid says: write many fast unit tests at the base, fewer integration tests in the middle, and only a handful of end-to-end tests at the top. The underlying principle is sound - lower-level tests are faster, more deterministic, and cheaper to maintain.

The principle behind the shape

The pyramid’s shape communicates a principle: prefer fast, deterministic tests that you fully control. Tests at the base are cheap to write, fast to run, and reliable. Tests at the top are slow, expensive, and depend on systems outside your control. The more weight you put at the base, the faster and more reliable your pipeline becomes - to a point. We also have the engineering goal of achieving the most functional coverage with the fewest number of tests. Every test costs money to maintain and adds time to the pipeline.

The testing trophy

The testing trophy, popularized by Kent C. Dodds, rebalances the pyramid by putting component tests at the center. Where the pyramid emphasizes unit tests at the base, the trophy argues that component tests give you the most confidence per test because they exercise realistic user behavior through a component’s public interface while still using test doubles for external dependencies.

The trophy also makes static analysis explicit as the foundation. Linting, type checking, and formatting catch entire categories of defects for free - no test code to write or maintain.

Both models agree on the principle: keep end-to-end tests few and focused, and maximize fast, deterministic coverage. The trophy simply shifts where that coverage concentrates. For teams building component-heavy applications, the trophy distribution often produces better results than a strict pyramid.

Teams often miss this underlying principle and treat either shape as a metric. They count tests by type and debate ratios - “do we have enough unit tests?” or “are our integration tests too many?” - when the real question is:

Can our pipeline determine that a change is safe to deploy without depending on any system we do not control?

A pipeline that answers yes can deploy at any time - even when a downstream service is down, a third-party API is slow, or a partner team hasn’t shipped yet. That independence is what CD requires, and it is the reason the pyramid favors the base.

What this looks like in practice

A test architecture that achieves this has three responsibilities:

1. Fast, deterministic tests - unit, component, and contract tests - run on every commit using test doubles for external dependencies. They give a reliable go/no-go signal in minutes.
2. Acceptance tests validate that a deployed artifact is deliverable. Acceptance testing is not a single test type. It is a pipeline stage that can include component tests, load tests, chaos tests, resilience tests, and compliance tests. Any test that runs after CI to gate promotion to production is an acceptance test.
3. Integration tests validate that contract test doubles still match the real external systems. They run in a dedicated test environment with versioned test data, on demand or on a schedule, providing monitoring rather than gating.

The anti-pattern: the ice cream cone

Most teams that struggle with CD have inverted the pyramid - too many slow, flaky end-to-end tests and too few fast, focused ones. Manual gates block every release. The pipeline cannot give a fast, reliable answer, so deployments become high-ceremony events.

Test Architecture

A test architecture is the deliberate structure of how different test types work together across your pipeline to give you deployment confidence. Use the table below to decide what type of test to write and where it runs. This is not a comprehensive list. It shows how common tests impact pipeline design and how teams should structure their suites. See the Pipeline Reference Architecture for a complete quality gate sequence.

The critical insight: everything that blocks merge is deterministic and under your control. Acceptance tests gate production promotion after verifying the deployed artifact. Everything that involves real external systems runs post-deployment. This is what gives you the independence to deploy any time, regardless of the state of the world around you.

Acceptance tests can include non-deterministic activities (load, chaos, resilience), but the gate decision is still deterministic: it fires on a documented pass/fail threshold - a performance budget, an error-rate ceiling, a required compliance check - not on the raw variability of the measurement. That is different from gating on a flaky test whose pass/fail flips for reasons unrelated to the change, which the Do Not list below warns against.

Pre-merge vs post-merge

The table maps to two distinct phases of your pipeline, each with different goals and constraints.

Pre-merge (before code lands on trunk): Run unit, component, and contract tests. These must all be deterministic and fast. Target: under 10 minutes total. This is the quality gate that every change must pass. If pre-merge tests are slow, developers batch up changes or skip local runs, both of which undermine continuous integration.

Post-merge (after code lands on trunk, before or after deployment): Re-run the full deterministic suite against the integrated trunk. Then run acceptance tests, E2E smoke tests, and synthetic monitoring post-deploy. Integration tests run separately in a test environment, on demand or on a schedule. Target: under 60 minutes for the full post-merge cycle.

Why re-run pre-merge tests post-merge? Two changes can each pass pre-merge independently but conflict when combined on trunk. The post-merge run catches these integration effects.

If a post-merge failure occurs, the team fixes it immediately. Trunk must always be releasable.

This post-merge re-run is what teams traditionally call regression testing: running all previous tests against the current artifact to confirm that existing behavior still works after a change. In CD, regression testing is not a separate test type or a special suite. Every test in the pipeline is a regression test. The deterministic suite runs on every commit, and the full suite runs post-merge. A green run means the artifact has been regression-tested against every behavior the suite encodes - no more and no less, which is why the suite’s coverage of prior behavior is what makes the signal trustworthy.

good practices

Do

• Run tests on every commit. If tests do not run automatically, they will be skipped.
• Keep the deterministic suite under 10 minutes. If it is slower, developers will stop running it locally.
• Fix broken tests immediately. A broken test is equivalent to a broken build.
• Delete tests that do not provide value. A test that never fails and tests trivial behavior is maintenance cost with no benefit.
• Test behavior, not implementation. Use a black box approach - verify what the code does, not how it does it. As Ham Vocke advises: “if I enter values x and y, will the result be z?” - not the sequence of internal calls that produce z. Avoid white box testing that asserts on internals.
• Use test doubles for external dependencies. Your deterministic tests should run without network access to external systems.
• Validate test doubles with contract tests. Test doubles that drift from reality give false confidence.
• Treat test code as production code. Give it the same care, review, and refactoring attention.
• Run automated accessibility checks on every commit. WCAG compliance scans are fast, deterministic, and catch violations that are invisible to sighted developers. Treat them like security scans: automate the detectable rules and reserve manual review for subjective judgment. See Accessibility testing for the full three-tier strategy and pipeline placement.

Do Not

• Do not tolerate flaky tests. Quarantine or delete them immediately.
• Do not gate your pipeline on flaky, non-deterministic test signals. E2E and integration test failures - pass/fail that flips for reasons unrelated to the change - should trigger review or alerts, not block deployment. (An acceptance gate that fires on a deterministic threshold, like a performance budget, is not this: the gate decision is stable even when the underlying measurement varies.)
• Do not couple your deployment to external system availability. If a third-party API being down prevents you from deploying, your test architecture has a critical gap.
• Do not write tests after the fact as a checkbox exercise. Tests written without understanding the behavior they verify add noise, not value.
• Do not test private methods directly. Test the public interface; private methods are tested indirectly.
• Do not share mutable state between tests. Each test should set up and tear down its own state.
• Do not use sleep/wait for timing-dependent tests. Use explicit waits, polling, or event-driven assertions.
• Do not let unit or component tests depend on a shared or external database or service. A real engine the team controls and isolates per test - a per-test testcontainer, or a transaction that rolls back at teardown - is fine in-band and stays deterministic. A shared, mutable database, or any service the team does not control, is not: that reintroduces non-determinism, so categorize the test as integration or end-to-end and run it post-deployment, not as a pre-merge gate.
• Do not make exploratory or usability testing a release gate. These activities are continuous and inform product direction; they are not a pass/fail checkpoint before deployment.

Related Content

• ACD - How acceptance criteria make testing the constraint that governs agent-generated code
• Testing Fundamentals - Establishing testing practices as part of CD migration
• High Coverage but Ineffective Tests - When tests pass but do not catch real defects

Additional concepts drawn from Ham Vocke, The Practical Test Pyramid, and Toby Clemson, Testing Strategies in a Microservice Architecture.

2 - What to Test - and What Not To

Three principles determine what belongs in your test suite and what does not.

If you cannot fix it, do not test for it

You should never test the behavior of services you consume. Testing their behavior is the responsibility of the team that builds them. If their service returns incorrect data, you cannot fix that, so testing for it is waste.

What you should test is how your system responds when a consumed service is unstable or unavailable. Can you degrade gracefully? Do you return a meaningful error? Do you retry appropriately? These are behaviors you own and can fix, so they belong in your test suite.

This principle directly enables the pipeline test strategy. When you stop testing things you cannot fix, you stop depending on external systems in your pipeline. Your tests become faster, more deterministic, and more focused on the code your team actually ships.

Test interfaces first

Most integration failures originate at interfaces, the boundaries where your system talks to other systems. These boundaries are the highest-risk areas in your codebase, and they deserve the most testing attention. But testing interfaces does not require integrating with the real system on the other side.

When you test an interface you consume, the question is: “Can I understand the response and act accordingly?” If you send a request for a user’s information, you do not test that you get that specific user back. You test that you receive and understand the properties you need - that your code can parse the response structure and make correct decisions based on it. This distinction matters because it keeps your tests deterministic and focused on what you control.

Use contract mocks, virtual services, or any test double that faithfully represents the interface contract. The test validates your side of the conversation, not theirs.

Frontend and backend follow the same pattern

Both frontend and backend applications provide interfaces to consumers and consume interfaces from providers. The only difference is the consumer: a frontend provides an interface for humans, while a backend provides one for machines. The testing strategy is the same.

Test frontend code the same way you test backend code: validate the interface you provide, test logic in isolation, and verify that user actions trigger the correct behavior. The only difference is the consumer (a human instead of a machine).

For a frontend:

• Validate the interface you provide. The UI contains the components it should and they appear correctly. This is the equivalent of verifying your API returns the right response structure.
• Test behavior isolated from presentation. Use your unit test framework to test the logic that UI controls trigger, separated from the rendering layer. This gives you the same speed and control you get from testing backend logic in isolation.
• Verify that controls trigger the right logic. Confirm that user actions invoke the correct behavior, without needing a running backend or browser-based E2E test.

This approach gives you targeted testing with far more control. Testing exception flows - what happens when a service returns an error, when a network request times out, when data is malformed, becomes straightforward instead of requiring elaborate E2E setups that are hard to make fail on demand.

Test Quality Over Coverage Percentage

Code coverage tells you which lines executed during tests. It does not tell you whether the tests verified anything meaningful. A test suite with 90% coverage and no assertions has high coverage and zero value.

Better questions than “what is our coverage percentage?”:

• When a test fails, does it point directly to the defect?
• When we refactor, do tests break because behavior changed or because implementation details shifted?
• Do our tests catch the bugs that actually reach production?
• Can a developer trust a green build enough to deploy immediately?

Why coverage mandates are harmful

When teams are required to hit a coverage target, they write tests to satisfy the metric rather than to verify behavior. This produces:

• Tests that exercise code paths without asserting outcomes
• Tests that mirror implementation rather than specify behavior
• Tests that inflate the number without improving confidence

The metric goes up while the defect escape rate stays the same. Worse, meaningless tests add maintenance cost and slow down the suite.

Instead of mandating a coverage number, set a coverage floor (see Getting Started) and focus team attention on test quality: mutation testing scores, defect escape rates, and whether developers actually trust the suite enough to deploy on green.

Related Content

• High Coverage, Ineffective Tests - When coverage metrics mask poor test quality
• Refactoring Breaks Tests - Tests that assert on implementation details instead of behavior
• Code Coverage Mandates - The anti-pattern of mandating coverage targets
• Test Doubles - Patterns for isolating dependencies in tests
• Contract Tests - Verifying that test doubles match reality

3 - Pipeline Test Strategy

Everything that blocks deployment must be deterministic and under your control. Everything that involves external systems runs asynchronously or post-deployment. This gives you the independence to deploy any time, regardless of the state of the world around you.

Tests Inside the Pipeline

These tests run on every commit and block deployment if they fail. They must be fast, deterministic, and free of external dependencies.

Every test in this pipeline uses test doubles for anything that crosses the component boundary into a system the team does not control: third-party APIs, downstream services owned by other teams, message brokers. No in-band test calls a shared or external service. A real engine the team owns and isolates per test - a database in a per-test testcontainer, for example - is permitted in-band because it stays deterministic. This means:

• A downstream outage cannot block your deployment. Your pipeline runs the same whether external systems are healthy or down.
• Tests are deterministic. The same code always produces the same result.
• The suite is fast. No network latency, no waiting for external systems to respond.

Why re-run tests post-merge?

Two changes can each pass pre-merge independently but conflict when combined on trunk. The post-merge run catches these integration effects. If a post-merge failure occurs, the team fixes it immediately. Trunk must always be releasable.

Tests Outside the Pipeline

These tests involve real external systems and are therefore non-deterministic. They never block deployment. Instead, they validate assumptions and monitor production health.

How Contract Tests Validate Test Doubles

The pipeline’s deterministic tests depend on test doubles to represent external systems. But test doubles can drift from reality. An API adds a required field, changes a response format, or deprecates an endpoint. Contract tests close this gap.

1. Pipeline tests use test doubles that encode your assumptions about external APIs - response schemas, status codes, error formats.
2. Contract tests run on a schedule and send real requests to the actual external APIs.
3. Contract tests compare the real response against what your test doubles return. They check structure and types, not specific data values.
4. When a contract test passes, your test doubles are confirmed accurate. The pipeline’s deterministic tests are trustworthy.
5. When a contract test fails, the team is alerted. They update the test doubles to match the new reality, then re-run component tests to verify nothing breaks.

This design means your pipeline never touches external systems, but you still catch when external systems change. You get both speed and accuracy.

Consumer-driven contracts

When the external API is owned by another team in your organization, you can go further with consumer-driven contracts. Instead of your team polling their API on a schedule, both teams share a contract specification (using a tool like Pact):

• You (the consumer) define the requests you send and the responses you expect.
• They (the provider) run your contract as part of their build. If a change would break your expectations, their build fails before they deploy.
• Your test doubles are generated from the contract, guaranteeing they match what the provider actually delivers.

This shifts contract validation from “detect and react” to “prevent.” See Contract Tests for implementation details.

Summary: All Stages at a Glance

The Testing reference provides detailed documentation for each test type, including code examples and anti-patterns.

Related Content

• Testing Reference - Full reference for each test type
• Test Doubles - Patterns for stubs, mocks, fakes, spies, and dummies
• Contract Tests - Consumer-driven and provider contracts in detail
• Test Feedback Speed - The cognitive science behind the 10-minute target
• Pipeline Architecture - The pipeline structure these tests feed into
• Pipeline Reference Architecture - Reference diagrams for single-team and multi-team pipelines

4 - Getting Started

Starting Without Full Coverage

Teams often delay adopting CI because their existing code lacks tests. This is backwards. You do not need tests for existing code to begin. You need one rule applied without exception:

Every new change gets a test. We will not go lower than the current level of code coverage.

Record your current coverage percentage as a baseline. Configure CI to fail if coverage drops below that number. This does not mean the baseline is good enough. It means the trend only moves in one direction. Every bug fix, every new feature, and every refactoring adds tests. Over time, coverage grows organically in the areas that matter most: the code that is actively changing.

Do not attempt to retrofit tests across the entire codebase before starting CI. That approach takes months and delivers no incremental value. It also produces low-quality tests written by developers who are testing code they did not write and do not fully understand.

Quick-Start Action Plan

If your test suite is not yet ready to support CD, use this focused action plan to make immediate progress.

1. Audit your current test suite

Assess where you stand before making changes.

Actions:

• Run your full test suite 3 times. Note total duration and any tests that pass intermittently (flaky tests).
• Count tests by type: unit, integration, functional, end-to-end.
• Identify tests that require external dependencies (databases, APIs, file systems) to run.
• Record your baseline: total test count, pass rate, duration, flaky test count.
• Map each test type to a pipeline stage. Which tests gate deployment? Which run asynchronously? Which tests couple your deployment to external systems?

Output: A clear picture of your test distribution and the specific problems to address.

2. Fix or remove flaky tests

Flaky tests are worse than no tests. They train developers to ignore failures, which means real failures also get ignored.

Actions:

• Quarantine all flaky tests immediately. Move them to a separate suite that does not block the build.
• For each quarantined test, decide: fix it (if the behavior it tests matters) or delete it (if it does not).
• Common causes of flakiness: timing dependencies, shared mutable state, reliance on external services, test order dependencies.
• Target: zero flaky tests in your main test suite.

3. Decouple your pipeline from external dependencies

This is the highest-leverage change for CD. Identify every test that calls a real external service and replace that dependency with a test double.

Actions:

• List every external service your tests depend on: databases, APIs, message queues, file storage, third-party services.
• For each dependency, decide the right test double approach:
  • In-memory fakes for databases (e.g., an in-memory repository, or SQLite/H2 standing in for the production engine). Fastest, but they do not exercise real SQL semantics.
  • A team-controlled real engine in a per-test testcontainer when the production query plan, constraints, or migrations matter. This is a real database, not a fake, but it stays deterministic because the team pins the version and isolates state per test, so it runs in-band.
  • HTTP stubs for external APIs the team does not control (e.g., WireMock, nock, MSW).
  • Fakes for message queues, email services, and other infrastructure.
• Replace the dependencies in your unit and component tests.
• Move the original tests that hit real services into a separate suite. These become your starting contract tests or E2E smoke tests.

Output: A test suite where everything that blocks the build is deterministic and runs without network access to external systems.

4. Add component tests for critical paths

If you do not have component tests that exercise your whole service in isolation, start with the most critical paths.

Actions:

• Identify the 3-5 most critical user journeys or API endpoints in your application.
• Write a component test for each: boot the application, stub external dependencies, send a real request or simulate a real user action, verify the response.
• Each component test should prove that the feature works correctly assuming external dependencies behave as expected (which your test doubles encode).
• Run these in CI on every commit.

Output: Component tests covering your critical paths, running in CI on every commit.

5. Set up contract tests for your most important dependency

Pick the external dependency that changes most frequently or has caused the most production issues. Set up a contract test for it.

Actions:

• Write a contract test that validates the response structure (types, required fields, status codes) of the dependency’s API.
• Run it on a schedule (e.g., every hour or daily), not on every commit.
• When it fails, update your test doubles to match the new reality and re-verify your component tests.
• If the dependency is owned by another team in your organization, explore consumer-driven contracts with a tool like Pact.

Output: One contract test running on a schedule, with a process to update test doubles when it fails.

6. Adopt TDD for new code

Once your pipeline tests are reliable, adopt TDD for all new work. TDD is the practice of writing the test before the code. It ensures every piece of behavior has a corresponding test.

The TDD cycle

1. Red: Write a failing test that describes the behavior you want.
2. Green: Write the minimum code to make the test pass.
3. Refactor: Improve the code without changing the behavior. The test ensures you do not break anything.

Why TDD matters for CD

• Every change is automatically covered by a test
• The test suite grows proportionally with the codebase
• Tests describe behavior, not implementation, making them more resilient to refactoring
• Developers get immediate feedback on whether their change works

TDD is not mandatory for CD, but teams that practice TDD consistently have significantly faster and more reliable test suites.

How to start: Pick one new feature or bug fix this week. Write the test first, watch it fail, write the code to make it pass, then refactor. Do not try to retroactively TDD your entire codebase. Apply TDD to new code and to any code you modify.

Output: Team members practicing TDD on new work, with at least one completed red-green-refactor cycle.

Related Content

• Pipeline Test Strategy - Where these tests fit in your pipeline
• Flaky Tests - Symptom of non-deterministic tests
• Test Doubles - Patterns for isolating dependencies
• Contract Tests - Verifying test doubles match reality
• No Contract Testing Between Services - The anti-pattern this action plan addresses
• A Large Codebase Has No Automated Tests - Starting testing on a brownfield codebase

5 - Defect Feedback Loop

Treat every test failure as diagnostic data about where your process breaks down, not just as something to fix. When you identify the systemic source of defects, you can prevent entire categories from recurring.

Two questions sharpen this thinking:

1. What is the earliest point we can detect this defect? The later a defect is found, the more expensive it is to fix. A requirements defect caught during example mapping costs minutes. The same defect caught in production costs days of incident response, rollback, and rework.
2. Can AI help us detect it earlier? AI-assisted tools can now surface defects at stages where only human review was previously possible, shifting detection left without adding manual effort.

Trace Every Defect to Its Origin

When a test catches a defect (or worse, when a defect escapes to production) ask: where was this defect introduced, and what would have prevented it from being created?

Defects do not originate randomly. They cluster around specific causes. The CD Defect Detection and Remediation Catalog documents over 30 defect types across eight categories, with detection methods, AI opportunities, and systemic fixes for each.

For the complete catalog covering all defect categories (including product and discovery, dependency and infrastructure, testing and observability gaps, and more) see the CD Defect Detection and Remediation Catalog.

Build a Defect Feedback Loop

You need a process that systematically connects test failures to root causes and root causes to systemic fixes.

1. Classify every defect. When a test fails or a bug is reported, tag it with its origin category from the tables above. This takes seconds and builds a dataset over time.
2. Look for patterns. Monthly (or during retrospectives), review the defect classifications. Which categories appear most often? That is where your process is weakest.
3. Apply the systemic fix, not just the local fix. When you fix a bug, also ask: what systemic change would prevent this entire category of bug? If most defects come from integration boundaries, the fix is not “write more integration tests.” It is “make contract tests mandatory for every new boundary.” If most defects come from untested edge cases, the fix is not “increase code coverage.” It is “adopt property-based testing as a standard practice.”
4. Measure whether the fix works. Track defect counts by category over time. If you applied a systemic fix for integration boundary defects and the count does not drop, the fix is not working and you need a different approach.

The Test-for-Every-Bug-Fix Rule

Every bug fix must include a test that reproduces the bug before the fix and passes after. This is non-negotiable for CD because:

• It proves the fix actually addresses the defect (not just the symptom).
• It prevents the same defect from recurring.
• It builds test coverage exactly where the codebase is weakest: the places where bugs actually occur.
• Over time, it shifts your test suite from “tests we thought to write” to “tests that cover real failure modes.”

Advanced Detection Techniques

As your test architecture matures, add techniques that catch defects before manual review:

For more examples of mapping defect origins to detection methods and systemic corrections, see the CD Defect Detection and Remediation Catalog.

Related Content

• Systemic Defect Fixes - Detailed reference for each defect category
• High Coverage, Ineffective Tests - When tests pass but do not catch real defects
• Refactoring Breaks Tests - Tests that break on implementation changes
• Retrospectives - Where defect pattern review fits in the improvement cycle
• Metrics-Driven Improvement - Using defect escape rate as a key metric

6 - Test Types

Definitions for the test types used throughout this site. Each page covers what the type is, when it runs in the pipeline, what it asserts on, and what it does not.

The list isn’t exhaustive and the boundaries between types aren’t crisp in every codebase. Use these definitions as shared vocabulary for the rest of the testing section, especially Applied Testing Strategies and Testing Antipatterns.

6.1 - Component Tests

Definition

Verification of a coherent structural unit (such as an entire microservice, UI component, or self-contained subsystem) against its specific contract and internal logic, while keeping interactions beyond that component’s boundary mocked or stubbed.

Scope & Boundaries

Broader than a unit test, but strictly narrower than an end-to-end (E2E) integration test. It tests the interplay of multiple internal classes/modules working together within that component. Out-of-process network calls and external downstream services are replaced by API wire-level stubs or in-memory equivalents (e.g., WireMock, MSW, ephemeral test containers).

Core Characteristics

Validates state management, internal workflows, data transformations, and edge-to-edge behavior within a bounded context without taking dependencies on third-party uptime or network latency.

Good Practices

• Mock only at boundary borders: Exercise the component’s internal routing, controllers, domain models, and data mappers together; only mock external HTTP APIs, message brokers, or remote databases.
• Use ephemeral infrastructure: Use fast, disposable local resources (e.g., local SQLite/Postgres in Docker, local WireMock) to mirror real component runtime characteristics.
• Versioned, repeatable test data.
• Verify contract-to-state workflows: Validate that boundary inputs result in the correct local state changes and expected outgoing network payloads.

Anti-Patterns

• E2E scope creep: Allowing the test to call live third-party services or dependent microservices instead of wire-level stubs.
• Re-testing granular unit logic: Writing dozens of micro-permutations of input edge cases at the component level instead of covering them in fast unit tests.
• Leaky test harness state: Failing to purge in-memory databases or reset wire stubs between runs, leading to non-deterministic test flakiness.

When to Avoid

They overlap heavily with other layers when the component is:

• Thin CRUD with no middleware to speak of. Provider contract verification against a booted app plus sociable unit tests of the domain cover most of what a component test would. Keep one per critical flow as smoke coverage; skip exhaustive component coverage.
• Utility libraries are effectively multiple small components in as a single consumable dependency.
• Pure transformation logic. Parsers, calculators, scheduling math. Unit tests give better coverage per unit of effort.

If you’re choosing between an extra component test and an extra unit test for the same behavior, the unit test is cheaper to write, run, and maintain. Component tests earn their keep at the seams between layers, not in repeating ground that unit tests already cover.

Examples

Backend Service

A component test for a REST API, exercising the full application stack with the downstream inventory service replaced by a test double:

describe("POST /orders", () => {
  it("should create an order and return 201", async () => {
    // Arrange: mock the inventory service response
    httpMock("https://inventory.internal")
      .onGet("/stock/item-42")
      .reply(200, { available: true, quantity: 10 });

    // Act: send a request through the full application stack
    const response = await request(app)
      .post("/orders")
      .send({ itemId: "item-42", quantity: 2 });

    // Assert: verify the public interface response
    expect(response.status).toBe(201);
    expect(response.body.orderId).toBeDefined();
    expect(response.body.status).toBe("confirmed");
  });

  it("should return 409 when inventory is insufficient", async () => {
    httpMock("https://inventory.internal")
      .onGet("/stock/item-42")
      .reply(200, { available: true, quantity: 0 });

    const response = await request(app)
      .post("/orders")
      .send({ itemId: "item-42", quantity: 2 });

    expect(response.status).toBe(409);
    expect(response.body.error).toMatch(/insufficient/i);
  });
});

Frontend Component

A component test exercising a login flow with a stubbed authentication service:

describe("Login page", () => {
  it("should redirect to the dashboard after successful login", async () => {
    mockAuthService.login.mockResolvedValue({ token: "abc123" });

    render(<App />);
    await userEvent.type(screen.getByLabelText("Email"), "ada@example.com");
    await userEvent.type(screen.getByLabelText("Password"), "s3cret");
    await userEvent.click(screen.getByRole("button", { name: "Sign in" }));

    expect(await screen.findByText("Dashboard")).toBeInTheDocument();
  });
});

Accessibility Verification

Component tests already exercise the UI from the actor’s perspective, making them the natural place to verify that interactions work for all users. Accessibility assertions fit alongside existing assertions rather than in a separate test suite.

This is the second of three tiers in the Accessibility testing strategy: static-analysis linting catches structural violations in source, component tests catch the rendered-only ones (computed contrast, focus order, keyboard operability), and manual audits cover the subjective remainder.

// accessibility scanner setup

describe("Checkout flow", () => {
  it("should be completable using only the keyboard", async () => {
    render(<CheckoutPage />);

    await userEvent.tab();
    expect(screen.getByLabelText("Card number")).toHaveFocus();

    await userEvent.type(screen.getByLabelText("Card number"), "4111111111111111");
    await userEvent.tab();
    await userEvent.type(screen.getByLabelText("Expiry"), "12/27");
    await userEvent.tab();
    await userEvent.keyboard("{Enter}");

    expect(await screen.findByText("Order confirmed")).toBeInTheDocument();

    const results = await accessibilityScanner(document.body);
    expect(results).toHaveNoViolations();
  });
});

Connection to CD Pipeline

Component tests run after unit tests in the pipeline, but before longer running acceptance tests, and provide the broadest fast, deterministic feedback:

1. Local development: run before committing. Deterministic scope keeps them fast enough to run locally without slowing the development loop.
2. PR verification: CI executes the full suite; failures block merge.
3. Trunk verification: the same tests run on the merged HEAD to catch conflicts.

They should always halt the CD pipeline on failure.

6.2 - Contract Tests

Definition

Verification that two separate systems (such as an API provider and its consumers, or a message publisher and subscriber) adhere to a shared, agreed-upon formal specification, the “contract”, without requiring both services to run simultaneously in an integrated environment.

Scope & Boundaries

Targets only the boundary interface: request payloads, query parameters, HTTP headers, response schemas, status codes, or message formats. It does not test internal business logic, database state, or deep end-to-end user journeys; it solely verifies compatibility with the interface schema (e.g., OpenAPI/Swagger, AsyncAPI, Protobuf).

Characteristics

Fast, independent execution across pipelines, prevents breaking schema changes before deployment, and eliminates the need for expensive, flaky end-to-end integration environments in applications with proper domain separation.

Good Practices:

• Strict schema adherence: Explicitly define nullability, enums, required fields, and format constraints rather than relying on loose, open schemas.
• Automated breaking-change detection: Integrate tools like oasdiff or buf breaking into CI to catch backwards-incompatible schema changes on pull requests.
• Generate stubs directly from contracts: Use contract-driven mock engines (e.g., Prism, Microcks) so mock behavior automatically updates whenever the contract changes.

Anti-Patterns:

• Testing business logic via contracts: Attempting to verify authorization rules, complex workflows, or algorithmic computations inside a contract test.
  • Example: testing that a request for a user return a specific user instead of the expected user object.
• Hand-crafted, unverified mock fixtures: Manually updating JSON response stubs in consumer code repos without validating them against the live contract artifact.
• All-or-nothing megaspecs: Coupling unrelated domains or multiple service interfaces into a single monolithic contract document that cannot be versioned or evolved independently.

Validating the Contract

A contract test only proves your code matches the contract - not that the contract still matches the real system. Integration tests close that gap by running against the real dependency. How tightly that loop closes depends on collaboration level: low collaboration means scheduled integration tests against the real system with no shared tooling; high collaboration adds a hosted specification server (Pact, Pacto), a specification fetched from a shared or provider repository at build time, or a provider webhook that triggers the consumer’s CI when the contract changes.

Consumer and Provider Perspectives

A contract has two sides asking different questions. The consumer asks whether the fields, types, and status codes it depends on still exist. Consumer tests assert only on the subset of the API the consumer actually uses, not the whole response - following Postel’s Law, be liberal in what you accept and conservative in what you send.

The provider asks whether its changes will break any consumer. A provider test runs every published consumer expectation against the real implementation, catching a removed field, a changed type, or altered error behavior before a consumer deploys and discovers the break.

Contract-First Development

The interface is defined as a formal artifact - an OpenAPI, Protobuf, or AsyncAPI spec - before either side writes an implementation. Consumer and provider teams build independently against that artifact, then verify conformance to the spec rather than to each other’s code. Works best for new APIs and parallel development, where there’s no existing implementation to write consumer-driven contracts against.

Examples

A consumer contract test using a consumer-driven contract tool:

describe("Order Service - Inventory Provider Contract", () => {
  it("should receive stock availability in the expected format", async () => {
    // Define what the consumer expects from the provider
    await contractTool.addInteraction({
      state: "item-42 is in stock",
      uponReceiving: "a request for item-42 stock",
      withRequest: { method: "GET", path: "/stock/item-42" },
      willRespondWith: {
        status: 200,
        body: {
          // Only assert on fields the consumer actually uses
          available: matchType(true),   // boolean
          quantity: matchType(10),      // integer
        },
      },
    });

    // Exercise the consumer code against the mock provider
    const result = await inventoryClient.checkStock("item-42");
    expect(result.available).toBe(true);
  });
});

A provider verification test that runs consumer expectations against the real implementation:

describe("Inventory Service - Provider Verification", () => {
  it("should satisfy all registered consumer contracts", async () => {
    await contractBroker.verifyProvider({
      provider: "InventoryService",
      providerBaseUrl: "http://localhost:3001",
      brokerUrl: "https://contract-broker.internal",
      providerVersion: process.env.GIT_SHA,
    });
  });
});

A contract-first schema validation test verifying a provider response against an OpenAPI spec:

// The OpenAPI document is the source of truth. Validate the whole response
// against the named schema rather than hand-checking individual fields - a
// field-by-field check drifts from the spec the moment the spec changes.
const validator = openApiValidator(openApiSpec);

describe("GET /stock/:id - OpenAPI contract", () => {
  it("should return a response conforming to the published schema", async () => {
    const response = await fetch("http://localhost:3001/stock/item-42");
    const body = await response.json();

    expect(response.status).toBe(200);

    // Asserts structure, types, required fields, and additionalProperties
    // rules exactly as the OpenAPI schema declares them.
    const result = validator.validate(body, "StockResponse");
    expect(result.errors).toEqual([]);
  });
});

Connection to CD Pipeline

Contract tests run after unit tests in the pipeline:

1. Local development: run before committing. Deterministic scope keeps them fast enough to run locally without slowing the development loop.
2. PR verification: CI executes the full suite; failures block merge.
3. Trunk verification: the same tests run on the merged HEAD to catch conflicts.

They should always halt the CD pipeline on failure.

6.3 - End-to-End Tests

Definition

Verification of a complete end-to-end user or business transaction through the entire deployed application stack, matching the perspective and experience of an actual user or external consumer.

Scope & Boundaries

Encompasses the entire system topology—from the frontend UI or external API gateway through all internal microservices, asynchronous workers, live queues, databases, and necessary third-party sandbox integrations.

Core Characteristics

Highest real-world confidence, highest execution cost, slowest run time, and highest vulnerability to environment or network-induced flakiness.

Good Practices

• Restrict to critical revenue/operational paths: Focus E2E coverage strictly on non-negotiable user journeys (e.g., user registration, primary checkout, key ingest pipelines).
• Automate environment provisioning: Deploy ephemeral, on-demand preview environments to run E2E suites and tear them down immediately upon completion.
• Implement resilient element selection: Select UI elements using accessibility roles or stable data attributes (e.g., data-testid) rather than fragile CSS classes or absolute XPath selectors.

Anti-Patterns

• Using E2E tests for regression safety nets: Relying on E2E suites to catch regressions that could have been detected upstream in unit, component, or contract stages (the “inverted testing pyramid”).
• Arbitrary thread sleeps: Adding fixed pauses (e.g., sleep(5)) to wait for asynchronous events rather than using explicit, condition-driven polling.
• Accepting flaky tests: Rerunning failing E2E tests until they turn green rather than quarantining and fixing the underlying timing or state issues immediately.

Weaknesses & Challenges

• High Flakiness and Low Signal-to-Noise Ratio: Non-deterministic failures are common. Network blips, browser rendering lag, race conditions in asynchronous frontend frameworks, and transient third-party service outages often cause false-negative test failures that erode developer trust.
• Poor Root-Cause Localization: When an E2E test fails with a generic error (e.g., TimeoutError: Element #confirmation-banner not found), finding the source of the issue requires combing through client logs, gateway routes, backend microservice traces, and database state to determine what actually broke.
• Environment Maintenance & Resource Cost: E2E suites typically demand fully integrated staging or preview environments. Keeping these environments populated with realistic test data, configured with active credentials, and synchronized across dozens of microservices is notoriously resource-intensive.
• Prohibitive Execution Times: Running full browser automation or multi-service distributed flows can take anywhere from tens of minutes to several hours. This latency breaks continuous delivery flow, encouraging teams to defer testing to late-stage, batch-processed pipelines rather than getting instant feedback on change.
• Tight Coupling to Volatile UI/API Layouts: Small cosmetic changes (like modifying class names, reordering markup, or tweaking a multi-step user flow) often break brittle E2E tests even though the underlying business capability remains completely functional.

Examples

import { test, expect } from '@playwright/test';

test('user can complete entire purchase flow', async ({ page }) => {
  // Navigates real UI against a fully deployed environment
  await page.goto('https://checkout.staging.example.com');
  await page.fill('#username', 'test_user');
  await page.fill('#password', 'SecurePass123!');
  await page.click('button[type="submit"]');

  // Add item to cart and initiate purchase
  await page.click('button[data-item="product-42"]');
  await page.click('#cart-checkout');
  await page.fill('#card-element', '4242424242424242');
  await page.click('#submit-payment');

  // Confirms response propagated across API, async billing, and UI rendering
  await expect(page.locator('.order-confirmation')).toHaveText(/Order #\d+ Confirmed/);
});

When to Use / Avoid

Use them for:

• Happy-path validation of critical business flows that cannot be verified any other way (e.g., a payment flow that depends on a real payment provider).
• Entangled domain workflows that span multiple deployables and cannot be isolated within a single component test.

They are the most expensive test type to write, run, and maintain. Use them sparingly.

Avoid for:

• Edge cases, error handling, or input validation. Those scenarios belong in unit or component tests.

Connection to CD Pipeline

E2E tests should only run in the pipeline as part of the longer running acceptance tests if they can be made dependable and deterministic. Otherwise, they should be run on a schedule and not act as a delivery decision.

6.4 - Integration Tests

Definition

Verification that two or more distinct architectural subsystems or external dependencies interact correctly across their transport layer and data boundaries.

Scope & Boundaries

Broader than a component test because it explicitly validates communication with real external systems (such as a database, message queue, cache, or filesystem), but narrower than a full end-to-end test because it targets a specific integration boundary rather than complete multi-service user workflows.

Core Characteristics

Detects driver/dialect mismatches, schema serialization issues, connection pooling misconfigurations, and ORM/SQL query errors that mock-heavy tests overlook.

Good Practices

• Use disposable, ephemeral infrastructure: Spin up real databases and queues using container tooling (e.g., Testcontainers) rather than using shared, persistent static environments.
• Verify transport-level error handling: Intentionally test connection timeouts, pool exhaustion, network blips, and transaction rollbacks.
• Run tests against clean boundary state: Truncate tables, flush caches, and clear queues between test runs to guarantee deterministic execution.

Anti-Patterns

• Using shared remote environments: Pointing integration test suites to shared dev/staging databases, causing data collisions and race conditions between concurrent CI jobs.
• Testing business permutations: Testing dozens of conditional logic branches through real databases instead of pushing that logic down to fast unit tests or leveraging component tests.
• Ignoring production parity: Testing against an SQLite in-memory database locally when production runs Postgres, masking dialect, constraint, and indexing differences.

Weaknesses & Challenges

• Infrastructure Orchestration Overhead: Requires managing real databases, message brokers, and caches inside the test execution context. Maintaining container definitions (e.g., Docker/Testcontainers) and keeping schema migrations up to date adds operational burden to developers.
• Test Isolation and State Contamination: When tests write real rows to a database or publish messages to an active broker, dirty state from one test can bleed into another. Cleaning, truncating, or rolling back transactions between runs adds latency and complexity.
• Slow Pipeline Feedback Cycles: Because integration tests involve real I/O, network socket handshakes, and disk writes, they are orders of magnitude slower than in-memory unit tests. Over-relying on them severely bloats commit-stage feedback loops.
• Local vs. Production Discrepancies: Test-specific database configurations, lightweight containerized replicas, or local mocks often mask subtle production issues—such as database clustering behavior, regional latency, connection pool limits, or privilege boundaries.

Examples

import pytest
from sqlalchemy import create_engine
from myapp.storage import OrderRepository, Order

# Runs against an actual ephemeral Postgres container, not an in-memory mock
def test_order_repository_persists_and_retrieves_roundtrip(real_pg_session):
    repo = OrderRepository(session=real_pg_session)
    new_order = Order(order_id=101, customer_id="cust_abc", total=49.99)

    repo.save(new_order)
    retrieved = repo.find_by_id(101)

    assert retrieved is not None
    assert retrieved.customer_id == "cust_abc"
    assert retrieved.total == 49.99

Connection to CD Pipeline

Integration tests should only run in the pipeline as part of the longer running acceptance tests if they can be made dependable and deterministic. Otherwise, they should be run on a schedule and not act as a delivery decision.

6.5 - Static Analysis

Definition

Static analysis (also called static testing) evaluates non-running code against rules for known good practices, inspecting source, configuration, and dependency manifests to catch errors, complexity, and security issues before the code ever runs.

Scope & Boundaries

Analysis runs against source code, configuration files, and dependency manifests at rest - no application starts and no test doubles are needed. Scope is the entire codebase, not a single unit, component, or transaction.

Characteristics

Seconds-scale execution (the fastest test category), fully deterministic, and codebase-wide scope, with no external dependencies except the network calls a dependency scanner makes to a vulnerability database.

Good Practices

• Run it everywhere feedback is possible: IDE plugins, pre-commit hooks, and CI each catch issues before the next stage makes them more expensive to fix.
• Customize the ruleset: default rules are a starting point; add rules for patterns that keep coming up in code review.
• Enforce it as a gate: treat lint, type, and security findings as build-breaking, the same as a failing test.

Anti-Patterns

• Disabling rules instead of fixing code: suppressing linter warnings or ignoring security findings erodes the value of static analysis over time.
• Skipping ruleset customization: default rules are a starting point, not a ceiling, for patterns specific to the codebase.
• Running static analysis only in CI: by the time CI reports a formatting error, the developer has context-switched; IDE and pre-commit feedback catch it sooner.
• Ignoring dependency vulnerabilities: known CVEs in dependencies are a direct attack vector and should break the build.
• Treating static analysis as optional: if developers can bypass the checks, they will.

When to Run It

• In the IDE: real-time feedback as developers type, via editor plugins and language server integrations.
• On save: format-on-save and lint-on-save catch issues immediately.
• Pre-commit: hooks prevent problematic code from entering version control.
• In CI: the full suite of static checks runs on every PR and on the trunk after merge, verifying that earlier local checks were not bypassed.

Static analysis is always applicable. Every project, regardless of language or platform, benefits from linting, formatting, and dependency scanning.

Examples

Linting

A .eslintrc.json configuration enforcing test quality rules:

{
  "rules": {
    "no-disabled-tests": "warn",
    "require-assertions": "error",
    "no-commented-out-tests": "error",
    "valid-assertions": "error",
    "no-unused-vars": "error",
    "no-console": "warn"
  }
}

Type Checking

Statically typed languages catch type mismatches at compile time, eliminating entire classes of runtime errors. Java, for example, rejects incompatible argument types before the code runs:

public static double calculateTotal(double price, int quantity) {
    return price * quantity;
}

// Compiler error: incompatible types: String cannot be converted to double
calculateTotal("19.99", 3);

Dependency Scanning

Dependency scanning tools scan for known vulnerabilities:

$ npm audit
found 2 vulnerabilities (1 moderate, 1 high)
  moderate: Prototype Pollution in lodash < 4.17.21
  high:     Remote Code Execution in log4j < 2.17.1

Types of Static Analysis

Accessibility Linting

Accessibility linting catches deterministic WCAG violations the same way a security scanner catches known vulnerability patterns. Automated checks cover structural issues (missing alt text, invalid ARIA attributes, insufficient contrast ratios, broken heading hierarchy) while manual review covers subjective aspects like whether alt text is actually meaningful.

Linting is the first of three tiers. For how it fits with component-test DOM scans and manual audits across the pipeline - and the caveat that automated checks catch only a fraction of WCAG criteria - see Accessibility testing.

An accessibility checker configuration running WCAG 2.1 AA checks against rendered pages:

{
  "defaults": {
    "standard": "WCAG2AA",
    "timeout": 10000,
    "wait": 1000
  },
  "urls": [
    "http://localhost:1313/docs/",
    "http://localhost:1313/docs/foundations/testing-fundamentals/test-architecture/"
  ]
}

An accessibility scanner test asserting that a rendered component has no violations:

// accessibility scanner setup (e.g. import scanner and extend assertions)

it("should have no accessibility violations", async () => {
  const { container } = render(<LoginForm />);
  const results = await accessibilityScanner(container);
  expect(results).toHaveNoViolations();
});

Connection to CD Pipeline

Static analysis is the first gate in the CD pipeline, providing the fastest feedback:

1. IDE / local development: plugins run in real time as code is written.
2. Pre-commit: hooks run linters, formatters, and accessibility checks on changed components, blocking commits that violate rules.
3. PR verification: CI runs the full static analysis suite (linting, type checking, security scanning, dependency auditing, accessibility linting) and blocks merge on failure.
4. Trunk verification: the same checks re-run on the merged HEAD to catch anything missed.
5. Scheduled scans: dependency and security scanners run on a schedule to catch newly disclosed vulnerabilities in existing dependencies.

Because it requires no running code and no external dependencies, static analysis is the cheapest, fastest gate. A mature CD pipeline treats its failures like any other test failure: they break the build.

6.6 - Smoke Tests

Definition

A lightweight, automated verification run executed immediately after code is deployed to a live target environment (staging, pre-production, or production) to verify that the environment is healthy, operational, and capable of handling traffic before fully shifting user load.

Scope & Boundaries

Strictly limited to high-level system vitality. It checks that critical infrastructure components (processes, routing, database connectivity, secret access, core endpoints) are alive and reachable. It explicitly avoids deep workflow testing, exhaustive edge-case permutations, or long-running operational flows.

Core Characteristics

Fast (seconds to 2–3 minutes max), strictly non-destructive/read-only, high criticality, and directly tied to deployment orchestration (triggers immediate automated rollback or stops traffic migration if it fails).

Examples

import requests
import sys

BASE_URL = "https://checkout.internal.net"

def run_smoke():
    # 1. Deep health check (probes DB connection, cache, and queue availability)
    health = requests.get(f"{BASE_URL}/healthz/ready", timeout=5)
    assert health.status_code == 200, f"Health check failed: {health.text}"
    assert health.json().get("database") == "UP"

    # 2. Key static config / version check
    version = requests.get(f"{BASE_URL}/version", timeout=3)
    assert version.status_code == 200
    assert version.json().get("commit_sha") == sys.argv[1]

    # 3. Non-destructive read query against a core endpoint
    catalog = requests.get(f"{BASE_URL}/api/v1/products/featured", timeout=5)
    assert catalog.status_code == 200
    assert len(catalog.json()) > 0

if __name__ == "__main__":
    run_smoke()

A Java sociable unit test exercising real domain logic through its public interface. The collaborators (the pricing policy and the order model) are real objects, not mocks, and the test asserts on the observable outcome - the computed total - rather than on which methods were called:

@Test
public void appliesBulkDiscountWhenQuantityReachesThreshold() {
    // Arrange: real collaborators, no test doubles - this is pure in-process logic
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(12)));

    // Act
    Money total = pricing.totalFor(order);

    // Assert: the observable result, not the sequence of internal calls
    // 12 * 20.00 = 240.00, less 15% = 204.00
    assertEquals(money("204.00"), total);
}

@Test
public void chargesFullPriceBelowTheThreshold() {
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(9)));

    assertEquals(money("180.00"), pricing.totalFor(order));
}

Good Practices

• Design for idempotency and read-only behavior: Keep smoke tests non-destructive so they can run safely against live production environments without corrupting customer data, charging credit cards, or sending false operational emails.
• Target deep health endpoints: Use application endpoints that actively verify connectivity to backend resources (database reads, Redis caches, downstream dependency reachability) rather than shallow /ping endpoints that only return 200 OK from the web server process.
• Automate immediate rollback gates: Tie smoke test outcomes directly into your CD pipeline (e.g., progressive delivery, canary, or blue/green deployments). If the smoke suite fails, the deployment halts and rolls back automatically without human intervention.
• Verify deployment identity: Assert that the deployed system is serving the exact build artifact, tag, or Git commit hash intended for the deployment to catch caching or orchestration misconfigurations. Anti-Patterns
• Mutating real production data: Creating synthetic test users, modifying real records, or generating phantom financial transactions without rigorous isolation or synthetic-data isolation strategies.
• Bloating into a full regression suite: Packing dozens of detailed integration checks into the smoke run, slowing pipeline execution from seconds into tens of minutes and defeating the purpose of a fast sanity gate.
• Ignoring downstream read timeouts: Setting long or infinite request timeouts that cause the deployment pipeline to hang indefinitely when an infrastructure route or firewall rule is misconfigured.
• Running exclusively in staging: Skipping smoke tests in production under the false assumption that passing staging guarantees environment-specific configs, network policies, and IAM roles are correctly wired in production. Weaknesses & Challenges
• Coarse-Grained Blind Spots: Smoke tests only verify that the front door is open and core pipes are connected; they cannot catch subtle business regressions, boundary calculation errors, or minor UI rendering glitches.
• Risk of Production State Side Effects: If write checks are included, synthetic data can leak into operational reports, analytics dashboards, or customer views, requiring custom clean-up routines that are prone to failure.
• Permissions and Security Boundaries: Probing internal services and operational health endpoints in locked-down production environments often requires elevated network routes or secure service tokens that must be strictly audited and maintained.
• Handling Transient Startup Latency: Newly launched containers, warm-up caches, or JIT compilation can cause false-positive smoke failures immediately following a rollout if proper readiness probes and retry loops are not configured.

Connection to CD Pipeline

Smoke tests are run after every deploy to validate the deploy. They are also used to trigger auto-rollback in the pipeline if they don’t pass.

6.7 - Unit Tests

Definition

Verification of the smallest testable piece of code—typically a single function, method, or class—in complete isolation from the rest of the application, network, file system, or external services. Test doubles are used where needed.

Scope & Boundaries

Execution runs entirely in-memory. All external dependencies (databases, APIs, message brokers, system clocks) are replaced with test doubles (stubs, mocks, or fakes).

Characteristics

Millisecond execution speeds, highly deterministic (zero flakiness), and pinpoint failure localization.

Good Practices:

• Test public behavior, not implementation details: Assert on return values and visible side effects rather than internal private state or execution paths.
• Strict isolation: Keep all tests in-memory; mock or stub out network, disk I/O, database, and system time to ensure sub-millisecond execution.
• Single assertion concept: Each test should verify one specific behavior or edge case to maintain pinpoint failure localization.

Anti-Patterns

• Over-mocking: Mocking domain entities, data transfer objects, or language primitives instead of purely external/infrastructure boundaries.
• Testing private methods: Forcing visibility or coupling tests to internal helper methods, which causes refactoring resistance without increasing behavioral confidence.
• Inter-test dependencies: Letting execution order matter or sharing mutable global state between test cases.

Solitary vs. sociable unit tests

A solitary unit test replaces all collaborators with test doubles. A sociable unit test allows real in-process collaborators while still replacing any external I/O. Both styles are unit tests as long as no real external dependency is involved.

When to Run Them

• During development: run the relevant subset of unit tests continuously while writing code. TDD (Red-Green-Refactor) is the most effective workflow.
• On every commit: use pre-commit hooks or watch-mode test runners so broken tests never reach the remote repository.
• In CI: execute the full unit test suite on every pull request and on the trunk after merge to verify nothing was missed locally.

Unit tests are the right choice when the behavior under test can be exercised without network access, file system access, or database connections. If you need any of those, you likely need a component test or an end-to-end test instead.

Examples

// castArray.test.js
describe("castArray", () => {
  it("should wrap non-array items in an array", () => {
    expect(castArray(1)).toEqual([1]);
    expect(castArray("a")).toEqual(["a"]);
    expect(castArray({ a: 1 })).toEqual([{ a: 1 }]);
  });

  it("should return array values by reference", () => {
    const array = [1];
    expect(castArray(array)).toBe(array);
  });

  it("should return an empty array when no arguments are given", () => {
    expect(castArray()).toEqual([]);
  });
});

A Java sociable unit test exercising real domain logic through its public interface. The collaborators (the pricing policy and the order model) are real objects, not mocks, and the test asserts on the observable outcome - the computed total - rather than on which methods were called:

@Test
public void appliesBulkDiscountWhenQuantityReachesThreshold() {
    // Arrange: real collaborators, no test doubles - this is pure in-process logic
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(12)));

    // Act
    Money total = pricing.totalFor(order);

    // Assert: the observable result, not the sequence of internal calls
    // 12 * 20.00 = 240.00, less 15% = 204.00
    assertEquals(money("204.00"), total);
}

@Test
public void chargesFullPriceBelowTheThreshold() {
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(9)));

    assertEquals(money("180.00"), pricing.totalFor(order));
}

Connection to CD Pipeline

Unit tests run in the earliest stages of the CD pipeline and provide the fastest feedback loop:

1. Local development: watch mode reruns tests on every save.
2. Pre-commit: hooks run the suite before code reaches version control.
3. PR verification: CI runs the full suite and blocks merge on failure.
4. Integrated change verification: CI reruns tests on the merged HEAD to catch integration issues.

They should always halt the CD pipeline on failure.

7 - Patterns

Each page in this subsection covers one component pattern. The structure is the same on every page so you can scan-compare:

1. What needs covered - the layers of testing the pattern typically benefits from.
2. Positive test cases - common success behaviors worth testing.
3. Negative test cases - common failure modes that produce production incidents.
4. Test double validation - how the doubles in pipeline tests stay honest.
5. Pipeline placement - where each test type tends to run.
6. Example - a short code sample illustrating one of the harder cases for that pattern.

These are recommended starting points, not exhaustive lists or required gates. Real components have details these pages don’t capture; ignore items that don’t apply, and add items the pattern doesn’t mention but your component clearly needs. The goal is to prompt the conversation, not to constrain it.

API provider, API consumer, scheduled job, and user interface are covered in depth. Event consumer, event producer, CLI/library, and stateful service are deliberately briefer sketches: the same six principles apply, the same checklist still prompts useful questions, and the test double validation model is the same. Use the briefer sketches as a starting point and expand the depth in your own runbooks for the patterns your services actually use.

The patterns

• API provider - a backend service exposing an HTTP/gRPC/GraphQL API and owning its own data.
• API consumer - the above, plus outbound calls to other services. The most failure-prone pattern.
• Scheduled job - a service triggered on a cron, queue, or external scheduler.
• User interface - a UI that renders data and accepts user interaction.
• Event consumer - a service that consumes messages from a broker.
• Event producer - a service that produces messages to a broker.
• CLI tool or library - a binary or package consumed by other developers.
• Stateful service - a service that maintains long-lived in-memory state.

7.1 - API Provider

A backend service that exposes an HTTP/gRPC/GraphQL API and owns its own data. No outbound calls to other services in your control.

What needs covered

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Documented endpoints: return the expected shape and status for valid input.
• Auth: succeeds for valid credentials and tokens.
• Pagination, filtering, sorting: all return the documented results.
• Idempotency: idempotent operations are idempotent; non-idempotent operations create exactly one record.
• Success-path side effects: events emitted and audit log entries happen on the success path.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Malformed body: bad JSON, missing required fields, wrong types, extra fields handled per the documented policy (reject vs. ignore).
• Out-of-range values: negatives where positives are expected, oversize strings, unicode edge cases.
• Auth failures: missing token, expired token, valid token with insufficient scope, valid token for a different tenant.
• Authorization boundaries: user A cannot read or modify user B’s resources.
• Resource not found: referenced IDs don’t exist, return 404 not 500.
• Concurrency: two writes to the same resource at once, optimistic-lock conflict handled with the documented status code.
• Persistence failure: DB unavailable, deadlock, constraint violation. The error envelope is correct and no partial state is committed.
• Rate limiting and request size limits: both enforce as documented.
• Idempotency under retry: same idempotency key within the window returns the original result, not a duplicate write.

Test double validation

Doubles in this pattern are mostly around persistence. Two layers keep them honest:

1. Adapter integration tests run against a real instance of your production database engine (the same major version, same extensions). If component tests use an in-memory SQLite shim while production runs Postgres, the shim is the lie. The adapter integration test exercises every query and migration against a Postgres testcontainer in CI.
2. Provider-side contract tests verify the API still satisfies every published consumer expectation. See Consumer and Provider Perspectives. Provider verification is where you discover that a “harmless” field rename broke a consumer before that consumer deploys.

Pipeline placement

• Unit + sociable unit tests: pre-commit and CI Stage 1.
• Adapter integration tests against testcontainers: CI Stage 1 if fast, Stage 2 otherwise.
• Component tests: CI Stage 1.
• Provider-side contract verification: CD Stage 1 (Contract and Boundary Validation).

Example: component test

A flow-oriented component test for an order-placement endpoint. The full app is assembled with an in-memory order repository and an in-memory event bus. The test drives the assembled component through its HTTP handlers and asserts on observable outcomes (status, persisted state, emitted event):

• Java
• C#
• JavaScript

@SpringBootTest
@AutoConfigureMockMvc
class OrderPlacementTest {

  @Autowired MockMvc mvc;
  @Autowired InMemoryOrderRepo orderRepo;
  @Autowired InMemoryEventBus events;

  @Test
  void places_order_with_valid_payment_creates_order_and_emits_OrderPlaced() throws Exception {
    var body = """
      { "items": [{"sku": "A1", "qty": 2}], "paymentToken": "pm_ok" }
      """;

    var result = mvc.perform(post("/orders")
        .header("Authorization", "Bearer tok_valid")
        .contentType(APPLICATION_JSON)
        .content(body))
      .andExpect(status().isCreated())
      .andReturn();

    var orderId = JsonPath.<String>read(result.getResponse().getContentAsString(), "$.id");
    assertThat(orderRepo.findById(orderId)).isPresent();
    assertThat(events.published()).anyMatch(e ->
        e.type().equals("OrderPlaced") && e.orderId().equals(orderId));
  }
}

public class OrderPlacementTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient client;
    private readonly InMemoryOrderRepo orderRepo = new();
    private readonly InMemoryEventBus events = new();

    public OrderPlacementTests(WebApplicationFactory<Program> factory)
    {
        client = factory.WithWebHostBuilder(b => b.ConfigureServices(s =>
        {
            s.AddSingleton<IOrderRepo>(orderRepo);
            s.AddSingleton<IEventBus>(events);
        })).CreateClient();
    }

    [Fact]
    public async Task Places_order_with_valid_payment_creates_order_and_emits_OrderPlaced()
    {
        client.DefaultRequestHeaders.Authorization = new("Bearer", "tok_valid");
        var body = new { items = new[] { new { sku = "A1", qty = 2 } }, paymentToken = "pm_ok" };

        var response = await client.PostAsJsonAsync("/orders", body);

        response.StatusCode.Should().Be(HttpStatusCode.Created);
        var created = await response.Content.ReadFromJsonAsync<OrderCreated>();
        orderRepo.FindById(created!.Id).Should().NotBeNull();
        events.Published.Should().Contain(e =>
            e.Type == "OrderPlaced" && e.OrderId == created.Id);
    }
}

import request from "supertest";
import { buildApp } from "./app.js";
import { InMemoryOrderRepo } from "./test/in-memory-order-repo.js";
import { InMemoryEventBus } from "./test/in-memory-event-bus.js";

test("places order with valid payment creates order and emits OrderPlaced", async () => {
  const orderRepo = new InMemoryOrderRepo();
  const events = new InMemoryEventBus();
  const app = buildApp({ orderRepo, events });

  const res = await request(app)
    .post("/orders")
    .set("Authorization", "Bearer tok_valid")
    .send({ items: [{ sku: "A1", qty: 2 }], paymentToken: "pm_ok" });

  expect(res.status).toBe(201);
  expect(orderRepo.findById(res.body.id)).toBeDefined();
  expect(events.published).toContainEqual(
    expect.objectContaining({ type: "OrderPlaced", orderId: res.body.id })
  );
});

The test asserts on what a real caller can observe, not on private methods or call sequences inside the controller.

7.2 - API Consumer

Same as API provider, plus outbound HTTP/gRPC calls to services the team does not own (or does own but deploys independently). This is the most failure-prone pattern in distributed systems and gets the most testing attention.

What needs covered

Everything from the API provider pattern, plus:

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Outbound call: constructs the right URL, headers, body, auth, and timeout.
• Success response: parsed correctly, including optional fields and unknown fields per Postel’s Law.
• Multi-call composition: multiple downstream calls in sequence or parallel produce the documented composite response.
• Caching: returns the cached value within TTL and refreshes after.
• Trace context: propagates downstream.

Negative test cases

Common cases to consider, not an exhaustive list. The bulk of the negative testing happens here, and it’s where most production incidents originate. Drive each failure mode through a client double that simulates it.

• Timeout (downstream exceeds configured deadline): the deadline enforces; the upstream caller gets the documented response (e.g., 504); no partial state is committed. Use a client double that delays past the deadline.
• Connection refused: retry policy executes the documented count and backoff; falls over to fallback or returns an error. Use a client double that rejects the connection.
• 5xx responses (500, 502, 503): retry only on retryable codes. Use a client double that returns 5xx.
• 4xx responses (400, 401, 403, 404, 409, 422, 429): each maps to documented behavior; 4xx generally not retried; 429 respects Retry-After. Use a client double that returns each code.
• Slow response within timeout: performance-budget assertions hold if the service has SLO commitments. Use a client double that delays within the deadline.
• Malformed response body: the response is rejected, not silently coerced. Use a client double that returns a truncated or wrong-type body.
• Schema drift (extra or missing fields): extra fields tolerated; missing required fields detected with a clear error. Use a client double that returns a drifted body.
• Wrong status code (200 with error body, 500 with success body): the client trusts the status code, not the body. Use a client double that returns mismatched status and body.
• Circuit open: the circuit opens under sustained failure; fast-fails subsequent calls; recovers on a half-open probe. Use a client double that sustains failures.
• Partial multi-call failure: compensation, rollback, or documented partial-success behavior. First client double succeeds, second fails.

Test double validation

This is where the “doubles need tests” rule lives or dies. Four layers:

1. Consumer-side contract tests run in the pipeline on every commit using doubles. They pin the request the consumer sends and the response shape the consumer depends on. Contract artifacts are published to a broker. Fast, deterministic, blocks the build.
2. Adapter integration tests exercise the outbound HTTP client against the real dependency in a controlled state - typically a testcontainer running an in-house service the team owns. They verify the adapter code correctly speaks the protocol: serialization, deserialization, header handling, timeout behavior, error mapping. The test asserts the adapter’s correctness, not the dependency’s behavior: if the test asks for a user, it validates that the response parses into a valid User, not which user was returned. For third-party dependencies the team can’t run in a controlled state, run these tests out-of-band on a schedule. WireMock loaded with provider-supplied fixtures is a useful complement but functions more like a contract test against recorded shapes than an integration test against the live protocol.
3. Provider-side contract verification runs in the provider’s pipeline. The provider executes every consumer’s published contract against the real provider implementation. Breaking changes are caught at the source before the provider deploys.
4. Post-deploy integration check runs periodically against the real downstream in a non-production environment. Same fixtures used in contract tests. Catches drift in fields the contract didn’t pin, version skew, environment differences. Failures trigger review, not a build break. See Out-of-Pipeline Verification.

For third-party APIs you do not control, there is no provider verification step. The post-deploy check against the live (or sandbox) API is the only mechanism keeping doubles honest. Run it more often than for in-house dependencies. Daily at minimum.

The anti-pattern to avoid: stubbing the third-party SDK directly. Always wrap third-party clients in a thin adapter the team owns, then double the adapter. This is called out explicitly as Mocking what you don’t own and is the single most common source of “but it worked in tests” incidents.

Pipeline placement

Same as the API provider pattern, plus:

• Consumer-side contract tests: pre-commit and CI Stage 1.
• Adapter integration tests for the outbound HTTP client against an in-house dependency the team controls (a testcontainer running the team’s own service in a known state): CI Stage 1 or Stage 2.
• Adapter integration tests against a third-party API or a service owned by another team: out-of-band on a schedule, never in-band. The risk of a flaky external service blocking deploys outweighs any in-band coverage benefit, and adapter tests with WireMock fixtures already cover the team’s adapter code.
• Resilience component tests with fault injection: CI Stage 1.
• Post-deploy integration checks against real downstreams: out of pipeline, on a schedule.

Example: fault injection at the client double

A negative-path test for downstream timeout. The payment client double simulates a slow response, the test asserts the deadline enforces and the upstream caller gets the documented error envelope:

• Java
• C#
• JavaScript

@SpringBootTest
@AutoConfigureMockMvc
class PaymentTimeoutTest {

  @Autowired MockMvc mvc;
  @Autowired InMemoryOrderRepo orderRepo;
  @MockBean PaymentsGateway payments;

  @Test
  void returns_504_when_payment_service_exceeds_deadline() throws Exception {
    when(payments.charge(any())).thenAnswer(inv -> {
      Thread.sleep(50);
      throw new UpstreamTimeoutException("payments");
    });

    var body = """
      { "items": [{"sku": "A1", "qty": 1}], "paymentToken": "pm_ok" }
      """;

    mvc.perform(post("/orders")
        .header("Authorization", "Bearer tok_valid")
        .contentType(APPLICATION_JSON)
        .content(body))
      .andExpect(status().isGatewayTimeout())
      .andExpect(jsonPath("$.error.code").value("UPSTREAM_TIMEOUT"));

    assertThat(orderRepo.all()).isEmpty();
  }
}

public class PaymentTimeoutTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient client;
    private readonly InMemoryOrderRepo orderRepo = new();
    private readonly Mock<IPaymentsGateway> payments = new();

    public PaymentTimeoutTests(WebApplicationFactory<Program> factory)
    {
        payments.Setup(p => p.ChargeAsync(It.IsAny<ChargeRequest>()))
            .Returns(async () =>
            {
                await Task.Delay(50);
                throw new UpstreamTimeoutException("payments");
            });

        client = factory.WithWebHostBuilder(b => b.ConfigureServices(s =>
        {
            s.AddSingleton<IOrderRepo>(orderRepo);
            s.AddSingleton(payments.Object);
        })).CreateClient();
    }

    [Fact]
    public async Task Returns_504_when_payment_service_exceeds_deadline()
    {
        client.DefaultRequestHeaders.Authorization = new("Bearer", "tok_valid");
        var body = new { items = new[] { new { sku = "A1", qty = 1 } }, paymentToken = "pm_ok" };

        var response = await client.PostAsJsonAsync("/orders", body);

        response.StatusCode.Should().Be(HttpStatusCode.GatewayTimeout);
        var error = await response.Content.ReadFromJsonAsync<ErrorEnvelope>();
        error!.Error.Code.Should().Be("UPSTREAM_TIMEOUT");
        orderRepo.All().Should().BeEmpty();
    }
}

test("returns 504 when payment service exceeds deadline", async () => {
  const slowPayments = {
    charge: () => new Promise((_, reject) => {
      setTimeout(() => reject(new TimeoutError("payments")), 50);
    })
  };
  const orderRepo = new InMemoryOrderRepo();
  const app = buildApp({ orderRepo, payments: slowPayments, deadlineMs: 30 });

  const res = await request(app)
    .post("/orders")
    .set("Authorization", "Bearer tok_valid")
    .send({ items: [{ sku: "A1", qty: 1 }], paymentToken: "pm_ok" });

  expect(res.status).toBe(504);
  expect(res.body.error.code).toBe("UPSTREAM_TIMEOUT");
  expect(orderRepo.all()).toHaveLength(0);
});

The test verifies three things at once: the documented status code, the structured error body the API contract promises, and that no partial state was committed.

7.3 - Scheduled Job

A job that runs on a cron, queue, or external scheduler. Reads from data sources, writes reports or updates state. Often has no inbound API surface. The entrypoint is the scheduler.

This pattern has two test design challenges that the API provider and API consumer patterns don’t have: time and data volume.

What needs covered

Process startup matters more here than for an API service, because scheduled jobs typically have non-trivial startup behavior (config loading, secret resolution, lock acquisition) that a component test with the SUT in-memory can bypass. The right shape is many component tests for behavior, plus one or two tests that invoke the actual deployed binary the scheduler will invoke.

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• End-to-end run: with representative input, produces the expected output (report file, database update, message published).
• Idempotency: running the job twice for the same logical period produces the same result, not duplicates.
• Checkpointing: a job that processes a stream resumes from the last checkpoint, not from scratch.
• Time windows: “yesterday’s data” computes correctly for various reference times, especially around DST, month boundaries, and year boundaries.
• Empty input: zero records produces a valid empty report, not an error.
• Output format: the report or message conforms to the documented schema.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Source unavailable: DB down, source API returning 5xx. Verify the job fails cleanly with a documented exit code/status, doesn’t write partial output, and is safely re-runnable.
• Sink unavailable: destination DB or message broker rejects writes. Verify no source state changes (e.g., “marked as processed”) happen if the sink fails.
• Partial-write failure: half the batch writes successfully, then the connection drops. Verify the next run reprocesses the failed half without duplicating the successful half. This is where idempotency keys, transactional outboxes, or compensating reads earn their keep.
• Slow job: job exceeds its expected runtime. Verify it surfaces as alertable, doesn’t silently overlap with the next scheduled run, and that the lock prevents concurrent execution.
• Malformed source data: null where non-null was expected, wrong type, encoding issues. Verify the bad record is logged with enough context to investigate, and the job decides per its policy: skip, dead-letter, or fail the whole run. The choice is design; the test pins it.
• Time-zone bugs: the job runs at 02:30 UTC for a “daily” report. What does it do on the day clocks shift? Test it. Use the injected clock so the test deterministically simulates the boundary.
• Concurrent run: the previous run hadn’t finished when the next was triggered. Verify the lock prevents overlap or, if overlap is acceptable, that the work is partitioned correctly.
• Crash mid-run: kill -9 in the middle of processing. Verify on restart the job resumes from a consistent state.
• Schema drift on source: a new field appears or a field changes type. Verify per the contract policy.

Test double validation

Three classes of doubles need validation, each through a different mechanism:

1. The injected clock. Every in-band test that depends on “now” uses an injected clock. Validate it with one out-of-band check that runs against the real system clock, exercises a known time-window calculation, and confirms the production wiring of the clock dependency is correct. This catches the “tests use UTC, prod uses container local time” class of bug.
2. Source and sink gateways. Same model as the API consumer pattern. Adapter integration tests in the pipeline exercise each gateway against a real source/sink container or WireMock. Contract tests pin the shape. Post-deploy integration checks confirm the doubles still match the real systems on a schedule.
3. The scheduler trigger. The doubled trigger in component tests must match what the real scheduler invokes. Verify with a post-deploy integration check that runs the real scheduler against a deployed instance in a non-prod environment and confirms the entrypoint is found, the cron expression fires at the expected times, environment variables and secrets resolve, and the concurrency policy holds. This is the test that catches “passed in CI, didn’t run in prod because the cron expression had a typo.”

Pipeline placement

• Unit and component tests: CI Stage 1.
• Adapter integration tests for the source and sink adapters: CI Stage 1 or Stage 2.
• Contract tests for each source and sink: CI Stage 1.
• Component tests of the deployed binary (small set): CI Stage 1 or Stage 2.
• Real-clock and real-scheduler integration check: out of pipeline, scheduled, against a non-prod environment.
• Post-deploy: a synthetic invocation of the job in production that verifies it ran, processed records, and met its SLO.

Example: time-window logic with an injected clock

A test that pins the daily-report window calculation around a DST boundary. The clock is injected so the test deterministically simulates the moment of interest. source and sink are field-level fakes set up in the test class with seeded data for 2026-03-08 and 2026-03-09.

• Java
• C#
• JavaScript

@Test
void daily_report_run_after_DST_spring_forward_uses_correct_window() {
  Clock fixedClock = Clock.fixed(
      Instant.parse("2026-03-09T07:30:00Z"),
      ZoneOffset.UTC);
  ReportJob job = new ReportJob(fixedClock, source, sink);

  job.run();

  Report emitted = sink.lastReport();
  assertThat(emitted.windowStart())
      .isEqualTo(Instant.parse("2026-03-08T05:00:00Z"));
  assertThat(emitted.windowEnd())
      .isEqualTo(Instant.parse("2026-03-09T05:00:00Z"));
  assertThat(emitted.recordsProcessed())
      .isEqualTo(source.recordsForDay("2026-03-08"));
}

[Fact]
public void Daily_report_run_after_DST_spring_forward_uses_correct_window()
{
    var fixedClock = new FakeClock(DateTimeOffset.Parse("2026-03-09T07:30:00Z"));
    var job = new ReportJob(fixedClock, source, sink);

    job.Run();

    var emitted = sink.LastReport();
    emitted.WindowStart.Should().Be(DateTimeOffset.Parse("2026-03-08T05:00:00Z"));
    emitted.WindowEnd.Should().Be(DateTimeOffset.Parse("2026-03-09T05:00:00Z"));
    emitted.RecordsProcessed.Should().Be(source.RecordsForDay("2026-03-08"));
}

test("daily report run after DST spring forward uses correct window", () => {
  const fixedClock = { now: () => new Date("2026-03-09T07:30:00Z") };
  const job = new ReportJob({ clock: fixedClock, source, sink });

  job.run();

  const emitted = sink.lastReport();
  expect(emitted.windowStart).toEqual(new Date("2026-03-08T05:00:00Z"));
  expect(emitted.windowEnd).toEqual(new Date("2026-03-09T05:00:00Z"));
  expect(emitted.recordsProcessed).toBe(source.recordsForDay("2026-03-08"));
});

A separate out-of-band check runs the deployed binary against the real system clock once, to verify the production wiring of the clock dependency matches the doubled clock used here.

7.4 - User Interface

A UI that renders data and accepts user interaction. Talks to one or more backend APIs.

What needs covered

UI component tests run in a real browser engine (Chromium, Firefox, WebKit) driven by Playwright, with the team’s existing unit-testing framework (Vitest, Jest, or whatever is already in the project) as the runner. In-memory renderer shortcuts like JSDOM are rejected: they trade accuracy for speed and produce false greens around layout, focus, event timing, Intersection Observer, and animations - exactly the surface where UI bugs live. Playwright’s headless Chromium starts in milliseconds and runs the suite fast enough to use as the default. Backends are stubbed at the network layer with page.route so the same fixtures drive component tests today and end-to-end smoke tests later.

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Critical flows: a user can complete each documented critical flow via keyboard and via mouse.
• Forms: accept valid input, submit, and show success.
• Loading states: render while the backend is in flight.
• Empty, populated, and overflow states: all render correctly.
• Internationalization: the UI renders with longer translations and right-to-left scripts.
• Responsive layouts: render at the documented breakpoints.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Backend errors: for every API call the UI makes, what does the user see for 4xx, 5xx, network failure, timeout? Test each. The most common UI bug is “spins forever on error.”
• Form validation: required fields, format errors, length limits, cross-field rules. Each shows a specific, actionable message that’s announced to screen readers.
• Authentication expiry: token expires mid-session. Verify the user is sent through the documented re-auth flow, not silently dropped.
• Permission denied: the user navigates to a page they cannot access. Verify the documented response (redirect, “not authorized,” etc.).
• Stale data: a list rendered, then a delete on another tab, then the user clicks the deleted item. Verify the documented refresh or error behavior.
• Slow network: every interaction has a documented behavior at 3G speeds. Verify with throttled fixtures.
• Concurrent edit: two users editing the same record. Verify the optimistic-lock UX behaves as documented.
• Browser back button: the back button is a public interface. Test it.
• Accessibility violations: automated WCAG scan in component tests catches missing labels, contrast failures, ARIA misuse on every commit. Don’t defer to quarterly audits.

Test double validation

Backend doubles in component tests must match the real backends. Same mechanism as the API consumer pattern: the UI is a consumer, every backend it talks to is a provider. Consumer-driven contracts run on every commit; provider verification runs in the backend’s pipeline. Post-deploy E2E smoke tests against the real backend close the loop on drift the contract didn’t pin.

Because UI component tests run in a real browser engine, there is no renderer-level double to validate. The browser is the production renderer, just headless. The remaining gap is between the stubbed backend and the real backend, which the out-of-band E2E suite covers. Out-of-band failures trigger review, not a build break.

Pipeline placement

• Unit tests (rendering, composition): CI Stage 1.
• Component tests in headless browser (including a11y assertions): CI Stage 1.
• Visual regression: CI Stage 1 if fast, CI Stage 2 if slow.
• Consumer-side contract tests for each backend: CI Stage 1.
• E2E happy-path smoke tests against real backends: post-deploy, in a production-like environment, blocking the rollout but not the build.
• Real user monitoring + synthetic transactions: continuously in production.

Example: UI component test for an error path

A flow-oriented test for the checkout error path. Playwright drives a headless browser; the backend is stubbed at the network layer with page.route; the team’s existing unit-testing framework (Vitest, JUnit, xUnit) runs the test. The assertion: the user sees a documented error message and the spinner does not get stuck.

• Java
• C#
• JavaScript

@Test
void shows_error_and_clears_spinner_when_checkout_fails_with_500() {
  try (Playwright playwright = Playwright.create();
       Browser browser = playwright.chromium().launch()) {
    Page page = browser.newPage();

    page.route("**/api/checkout", route ->
        route.fulfill(new Route.FulfillOptions()
            .setStatus(500)
            .setContentType("application/json")
            .setBody("{\"error\":{\"code\":\"INTERNAL\"}}")));

    page.navigate("http://localhost:3000/checkout");
    page.getByRole(AriaRole.BUTTON,
        new Page.GetByRoleOptions().setName("Place order")).click();

    assertThat(page.getByRole(AriaRole.ALERT))
        .containsText("Something went wrong, please try again");
    assertThat(page.getByRole(AriaRole.STATUS)).not().isVisible();
  }
}

[Fact]
public async Task Shows_error_and_clears_spinner_when_checkout_fails_with_500()
{
    using var playwright = await Playwright.CreateAsync();
    await using var browser = await playwright.Chromium.LaunchAsync();
    var page = await browser.NewPageAsync();

    await page.RouteAsync("**/api/checkout", route => route.FulfillAsync(new()
    {
        Status = 500,
        ContentType = "application/json",
        Body = "{\"error\":{\"code\":\"INTERNAL\"}}"
    }));

    await page.GotoAsync("http://localhost:3000/checkout");
    await page.GetByRole(AriaRole.Button, new() { Name = "Place order" })
        .ClickAsync();

    await Expect(page.GetByRole(AriaRole.Alert))
        .ToContainTextAsync("Something went wrong, please try again");
    await Expect(page.GetByRole(AriaRole.Status)).Not.ToBeVisibleAsync();
}

import { test, expect, beforeAll, afterAll } from "vitest";
import { chromium } from "playwright";

let browser;

beforeAll(async () => { browser = await chromium.launch(); });
afterAll(async () => { await browser.close(); });

test("shows error and clears spinner when checkout fails with 500", async () => {
  const page = await browser.newPage();

  await page.route("**/api/checkout", route =>
    route.fulfill({
      status: 500,
      contentType: "application/json",
      body: JSON.stringify({ error: { code: "INTERNAL" } }),
    })
  );

  await page.goto("http://localhost:3000/checkout");
  await page.getByRole("button", { name: /place order/i }).click();

  await expect(page.getByRole("alert"))
    .toContainText(/something went wrong, please try again/i);
  await expect(page.getByRole("status")).not.toBeVisible();
});

The test exercises the rendered DOM the way a real user would. Intercepting at the network layer with page.route keeps the same fixtures reusable when the component test gets promoted to an end-to-end smoke test against the real backend.

7.5 - Event Consumer

A consumer of messages from Kafka, SQS, RabbitMQ, Pub/Sub, or similar. Reads messages, processes them, often updates state and produces downstream messages. The “public interface” is the topic or queue and the schema of messages on it.

This pattern has problems the API provider and API consumer patterns don’t have: ordering, replay, poison messages, dead-letter queues, and delivery semantics (at-most-once, at-least-once, exactly-once-with-effort).

What needs covered

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Well-formed message: produces the expected state change and the documented downstream events.
• Batch processing: processes per documented policy.
• Replay from offset: reproduces the same end state.
• Documented schema versions: are accepted.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Malformed message: routes to the DLQ with a correlation ID; the consumer survives.
• Duplicate delivery: absorbed by idempotency.
• Out-of-order delivery: follows the documented behavior.
• Mid-batch downstream failure: the offset is left uncommitted.
• Schema-version skew: handled per the documented policy.
• Slow downstream: applies backpressure rather than OOM.
• Consumer-group rebalance during processing: no in-flight messages are stranded.

Test double validation

The broker double in component tests is validated by adapter integration tests against a real broker container the team controls (Kafka in Docker, ElasticMQ for SQS, Redpanda in Docker). The test exercises the broker client adapter against that controlled instance and asserts the adapter speaks the protocol correctly - it does not assert anything about which messages the broker returns or in what order; that is the broker’s behavior, not the adapter’s. Schema registry double is validated by contract tests pinning each version, plus a post-deploy check against the real registry. Post-deploy synthetic publishes a known message to the real topic in a non-prod environment.

Pipeline placement

Handler unit tests and component tests run in CI Stage 1; adapter integration tests against a team-controlled broker container in CI Stage 1 or Stage 2; adapter integration tests against a managed broker the team can’t pin to a known state run out-of-band on a schedule, alongside the post-deploy synthetic.

Example: idempotency under duplicate delivery

Money.usd takes minor units (cents); 4250 represents $42.50.

• Java
• C#
• JavaScript

@Test
void same_message_processed_twice_creates_one_payment_record() {
  PaymentEvent event = new PaymentEvent(
      "evt-9f12", OrderId.of("ord-001"), Money.usd(4250));
  PaymentRepo repo = new InMemoryPaymentRepo();
  PaymentEventHandler handler = new PaymentEventHandler(repo);

  handler.handle(event);
  handler.handle(event);

  assertThat(repo.findByEventId("evt-9f12")).hasSize(1);
  assertThat(repo.totalForOrder(OrderId.of("ord-001"))).isEqualTo(Money.usd(4250));
}

[Fact]
public void Same_message_processed_twice_creates_one_payment_record()
{
    var evt = new PaymentEvent("evt-9f12", OrderId.Of("ord-001"), Money.Usd(4250));
    var repo = new InMemoryPaymentRepo();
    var handler = new PaymentEventHandler(repo);

    handler.Handle(evt);
    handler.Handle(evt);

    repo.FindByEventId("evt-9f12").Should().HaveCount(1);
    repo.TotalForOrder(OrderId.Of("ord-001")).Should().Be(Money.Usd(4250));
}

test("same message processed twice creates one payment record", () => {
  const event = new PaymentEvent(
    "evt-9f12", OrderId.of("ord-001"), Money.usd(4250));
  const repo = new InMemoryPaymentRepo();
  const handler = new PaymentEventHandler(repo);

  handler.handle(event);
  handler.handle(event);

  expect(repo.findByEventId("evt-9f12")).toHaveLength(1);
  expect(repo.totalForOrder(OrderId.of("ord-001"))).toEqual(Money.usd(4250));
});

7.6 - Event Producer

The producer side, often paired with the Event consumer pattern in the same service. After a state change, the service publishes a message that downstream consumers depend on.

The hard problems differ from the consumer side: atomicity with persistence (did the DB row commit and the message publish?), exactly-once semantics that require an outbox or two-phase commit, and downstream consumer dependence on schema, routing key, and headers.

What needs covered

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• State change: produces the correct message on the correct topic with the correct routing key, headers, and schema version.
• Outbox drain: drains in order.
• Redelivery: does not reorder.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• DB commits but broker fails: the message stays in the outbox and emits on the next drain. No event lost.
• Broker accepts but DB rolls back: nothing is emitted. No phantom events.
• Broker unavailable for an extended period: the outbox accumulates with bounded growth and alerts at a threshold.
• Breaking schema change: fails provider-side contract verification before shipping.

Test double validation

The broker double in component tests is validated against a real broker container the team controls in adapter integration tests. The test asserts the adapter publishes with the right routing key, headers, and serialization - it does not assert which messages downstream consumers happen to read or in what order; those are downstream concerns. Provider-side contract verification runs in this service’s pipeline against every consumer’s published expectations.

Pipeline placement

Outbox component tests and routing tests run in CI Stage 1; adapter integration tests against a team-controlled broker container in CI Stage 1 or Stage 2; adapter integration tests against a managed broker the team can’t pin run out-of-band on a schedule. Provider-side contract verification in CD Stage 1; post-deploy synthetic state change verifies the message arrives with the expected shape.

7.7 - CLI Tool or Library

A binary (CLI) or package (library) consumed by other developers. The “public interface” is the CLI invocation surface (argv, stdin, stdout, stderr, exit code) or the library’s exported API.

The pattern is different because the consumer is a developer or another program, not a user clicking a button. Cross-platform behavior, semantic versioning, and backward compatibility matter more than they do for a service.

What needs covered

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Valid arguments: produce documented stdout output, no stderr, and exit code 0.
• Pipe-friendly mode: produces machine-readable output (JSON/NDJSON) when stdout is not a TTY.
• Library API: returns documented values for valid input.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Bad arguments: exit with the documented non-zero code and structured stderr.
• Help text: reachable via --help.
• Large input: does not OOM.
• Interrupt (Ctrl-C, SIGTERM): runs cleanup and flushes or rolls back partial output.
• Invalid arguments to the library: throws the documented error type.
• Public symbol removed or renamed: the API-surface test fails the build.

Test double validation

File system doubles validated by integration tests against the real FS in a temp directory. Subprocess doubles validated by tests that actually spawn the subprocess on each supported OS. Doctests validate README examples against the real binary or library on every build.

Pipeline placement

Unit and component tests run in CI Stage 1 on every supported OS; API surface diff and doctests in CI Stage 1; cross-platform integration tests in CI Stage 2 if slow.

7.8 - Stateful Service

A service that maintains long-lived in-memory state: caches, in-memory aggregates, leader-elected coordinators, websocket gateways, real-time engines, sticky-session servers.

The hard problems are concurrency, recovery, and unbounded growth. Stateful services fail in ways stateless services do not.

What needs covered

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• State transitions: follow the documented machine.
• Restart: state rebuilds and behavior matches pre-restart.
• Replication lag under expected load: stays within budget.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Crash mid-write: consistent state on restart. No torn writes.
• Network partition: minority replicas step down with documented reconciliation on heal.
• Slow replication: applies backpressure rather than silent divergence.
• Memory pressure: evicts oldest entries per policy without OOM.
• Idle long-running connections: close cleanly with documented reconnect behavior.
• Concurrent state mutations: serialize without lost updates.

Test double validation

Persistence doubles validated by adapter integration tests against the real production engine. Consensus library doubles validated by cluster tests against a multi-node testcontainer setup. Soak tests run out of pipeline against a deployed instance to catch slow leaks and unbounded growth.

Pipeline placement

State machine unit tests, recovery component tests, and single-node concurrency tests run in CI Stage 1; cluster tests with real consensus library in CI Stage 2; soak and chaos tests out of pipeline.

8 - Applied Testing Strategies

A practical guide for fully testing eight common component patterns. Builds on the test-type definitions in Architecting Tests for CD and the deterministic-pipeline model used throughout this site.

This is a set of recommended patterns to consider when designing a test suite, not a prescriptive checklist. The patterns describe shapes of components teams commonly build; the lists of positive cases, negative cases, and pipeline placements are common things to consider for that shape, not an all-inclusive set. Use them as a starting point for the conversation about what your component actually needs.

That said, three goals apply to every pattern:

1. Cover the positive paths - the component does what it should under expected inputs.
2. Cover the negative paths - the component fails safely, predictably, and observably under bad inputs, broken dependencies, and adverse conditions.
3. Validate the test doubles - every double used to keep deterministic tests fast must be backed by a non-deterministic check that the double still matches reality.

If the third point is missing, the first two lie to you over time.

How to use this section

• New to the patterns? Start with Cross-cutting principles below, then Patterns.
• Auditing a component before ship? Jump to the Pre-ship checklist.
• Looking for a specific concern that crosses every pattern (authn, migrations, fixtures, observability, perf, mutation testing, flake handling, time budgets)? See Cross-cutting concerns.
• Existing suite needs rework first? See Testing Antipatterns.

Terminology

Two phrases that look similar but mean different things:

• Adapter integration test (Toby Clemson’s “integration test”): a narrow test of a single boundary adapter (HTTP client, DB query layer, message-broker client) exercised against the real external dependency or a high-fidelity stand-in. Pins the adapter’s protocol behavior - serialization, deserialization, headers, error mapping - not the behavior of the dependency itself. Runs in-band only when the team has full control over the dependency (typically a per-test testcontainer) and the test is fully deterministic; otherwise runs out-of-band on a schedule.
• Out-of-band integration check (this site’s Integration Tests): runs out-of-band on a schedule or post-deploy against real external systems. Confirms that doubles used by in-band tests still match reality. Failures trigger review, not a build break.

When this section says bare “integration test,” it’s the gateway flavor unless qualified.

Cross-cutting principles

Six principles apply to every pattern. The first three are short pointers to pages that own the topic; the last three are unique to this section.

1. In-band tests are deterministic; out-of-band checks confirm reality

In-band tests run in the commit-to-deploy pipeline and gate the build. They must be deterministic, which means test doubles replace anything that crosses the component boundary - downstream services, message brokers, schedulers, browsers talking to real backends. Out-of-band checks run on a schedule or post-deploy against the real systems those doubles stand in for. They confirm the doubles still match reality. Failures trigger review or rollback, not a build break. See the architecture in Architecting Tests for CD.

2. Test doubles need their own tests

Every double is traceable to a contract test pinning its claims and an out-of-band check confirming the claims still hold. The mechanics live in Test Doubles.

3. Test through the public interface

Public methods for classes; HTTP routing for services; rendered DOM for UIs; the entrypoint the scheduler invokes for jobs. See Component Tests. Reflection, package-private back doors, and asserting on private state are tested-the-wrong-thing in disguise.

4. Sociable unit tests dominate; solitary unit tests are the narrow exception

Domain logic in a real system lives in how behaviors collaborate, not in any single class. A sociable unit test drives the actual collaborators that implement a domain operation - validators, domain services, repositories backed by an in-memory or testcontainer double - and asserts on the observable outcome of that operation: the response, the persisted state, the event emitted. That is the bulk of the suite. Solitary unit tests are reserved for genuinely complex pure logic with no collaborators worth wiring up - pricing math, parsers, scheduling arithmetic.

Organize the suite around domain operations (“place an order,” “cancel a subscription within the grace period”), not around the classes or methods that happen to implement them. Tests written this way survive refactoring, catch bugs that live in the interactions between collaborators, and document what the component does to a stakeholder who can’t read the code. Tests written one-class-at-a-time with mocks for every collaborator do none of that.

5. Negative paths get equal weight

For every “it works” test, ask: malformed input, dependency timeout, dependency 500, dependency 200-with-malformed-body, slow response, partial write, duplicate request, missing or wrong authn/authz. Negative paths are where production incidents come from.

6. Name tests in domain terms, not implementation terms

A test name is documentation. places_order_with_valid_payment_creates_order_and_emits_OrderPlaced survives refactoring; OrderService.processPayment_returns_PaymentResult does not. The translation rule: if the name only makes sense to someone who has read the code, rewrite it. Highest-ROI change a team can make to an existing suite without any new infrastructure. For more on what to avoid, see Testing Antipatterns.

Related Content

• Architecting Tests for CD - the section overview, with the do/do-not list and the architecture diagram.
• Testing Antipatterns - common testing anti-patterns and a migration guide for teams whose suite needs rework.
• Test Doubles - types, when to use, anti-patterns.
• Contract Tests - consumer-driven and contract-first approaches.
• Component Tests - exercising a deployable through its public interface with doubles for everything outside the boundary.
• Integration Tests - the post-deploy check that keeps the deterministic pipeline honest.
• Pipeline Reference Architecture - quality gates sequenced by defect detection priority.

The layered approach (unit, integration, component, contract, end-to-end) this section builds on comes from Toby Clemson, Testing Strategies in a Microservice Architecture.

8.1 - Pre-Ship Checklist

Use this as a set of prompts for a quick self-audit, not a list of gates that must all pass. Items that don’t apply to a component can be ignored; items the list doesn’t mention but your component clearly needs should be added. Walk back to the pattern or cross-cutting concern that needs attention for any item that prompts a “we should fix that.”

• The bulk of the suite is sociable unit tests that exercise how behaviors collaborate to deliver a domain operation. Solitary unit tests are reserved for genuinely complex pure logic.
• Tests are organized around domain operations, not around classes or methods. Test names read as something a stakeholder would recognize.
• Every public-interface contract (inbound and outbound) has a contract test running in the pipeline.
• Classes are tested through their public methods only. No reflection, no test-only visibility relaxations, no asserting on private state.
• Every consumed external dependency is wrapped in a gateway the team owns; doubles are of the gateway, not of the third-party library.
• Every boundary adapter has an adapter integration test against the real dependency or a high-fidelity stand-in (testcontainer, WireMock with provider fixtures).
• The bulk of testing runs in-band in the pipeline and gates the build; out-of-band checks against real systems run on a schedule and trigger review on failure, never a build break.
• Every test double has a corresponding non-deterministic check that exercises the real dependency on a schedule or post-deploy.
• Every documented failure mode has a negative test.
• Every error response has a test that verifies the error envelope, status code, and any side effects (or absence thereof).
• Time, randomness, and the network are injected, not called directly. No sleep in tests. Use bounded polling or a fake clock.
• All deterministic tests run pre-commit and in CI Stage 1, and fail the build on failure.
• All post-deploy integration checks run out of pipeline and trigger review on failure, never blocking a commit.
• Pipeline gates map to defect sources from the Systemic Defect Fixes catalog. If a defect category has no automated check, that’s a known risk.
• Authn and authz are tested across every protected endpoint, not as one-offs per feature.
• Database migrations are tested forward, backward (where supported), and on representative data volume against the production engine.
• Fixtures are generated from the schema or built through Object Mother / builder helpers, not inline literals.
• Failure-path tests assert on observability (metric incremented, structured log emitted with correlation ID), not just the response.
• Per-endpoint perf budgets exist for hot paths; load tests gate production promotion; soak tests run out of pipeline.
• Flaky tests are quarantined with a dated owner and time-boxed remediation. No permanent quarantine list.
• The deterministic suite respects the pattern’s time budget (under 5 to 8 minutes per component, under 10 minutes total).

8.2 - Cross-Cutting Concerns

The patterns describe testing organized by component shape. The concerns below cut across all patterns and deserve dedicated coverage in any non-trivial system.

Authn and authz testing

Authentication and authorization deserve dedicated, exhaustive coverage. They are a major source of high-impact incidents and the failure modes are predictable:

• Tenant isolation: tenant A’s queries never return tenant B’s data. Test every read path. Multi-tenant SaaS bugs are almost always missing isolation tests.
• Scope or role escalation: a token with read:orders cannot perform write:orders. Test the matrix of scope and endpoint.
• Expired tokens: rejected even if cached locally. Clock-skew tolerance is a property of the verifier, not a license to skip the test.
• Forged tokens: signature validation actually validates. The classic JWT alg: none bug still ships periodically.
• Missing auth: every protected endpoint returns 401, never 500 (information leak) and never 200 (catastrophic).
• Service-to-service auth: machine identities respected, mTLS validated, token-swapping attacks detected.

The pattern: a parameterized test that takes (endpoint, method, expected-status-when-no-token, expected-status-when-wrong-scope) and runs across every endpoint in the OpenAPI or schema definition. New endpoints are covered automatically.

Database migrations

Migrations have their own discipline. For every migration:

• Forward on representative data: produces the expected schema and data.
• Backward (where supported): returns to the previous schema with no data loss. Expand-contract migrations may not roll back; that’s a design choice the test pins.
• Forward + backward + forward: idempotent.
• Time on production-scale data: budget assertion. A 30-minute migration on a 50M-row table needs a different deploy strategy than a 30-second one.
• Under traffic: the expand-contract pattern doesn’t break in-flight transactions.

Test against the real production database engine and version using testcontainers. SQLite-against-Postgres is a frequent source of “passed in CI, broke at 02:00 in prod” incidents.

Test data and fixtures

Fixtures rot faster than the code that uses them. Two principles keep them honest:

1. Generate fixtures from the schema, not by hand. When the schema is the source of truth (Avro, OpenAPI, SQL DDL, Protobuf), generate fixture builders from it. A type change breaks the build, not production.
2. Use Object Mother or builder patterns, not raw inline literals. A test that says placeOrder(buildValidOrder().withItem("A1", 2).build()) survives a schema change because the builder updates centrally. A test with 30 lines of raw JSON inline does not.

Avoid shared global fixtures that tests mutate. Each test creates the state it needs, names what is essential about that state, and discards the rest.

Observability as a tested artifact

Logs, metrics, and traces are part of a service’s contract with operators. If an alert depends on a metric, the test for the failure path should assert the metric is emitted. If a runbook depends on a structured log line, the test should assert the line is produced with the right fields and correlation ID.

The pattern: in component tests, attach a metrics collector and a log capture to the assembled component. Failure-path tests assert three things at once:

1. The response status is correct.
2. The error metric is incremented with the right labels.
3. The structured log line is emitted with correlation ID, error code, and any fields the runbook depends on.

This prevents silent regressions where the code “works” but the operator can’t see what’s happening when it doesn’t.

Accessibility testing

For any pattern that renders a user interface, accessibility is a functional requirement, not a finishing touch, and it belongs in the same in-band / out-of-band split as every other concern on this page. The dividing line is the one the whole test architecture uses: deterministic checks gate the build; subjective judgment runs continuously and never blocks.

The governing rule: automate the deterministic rules, reserve human judgment for the rest. A large share of WCAG success criteria are machine-checkable - missing alt attributes, invalid or contradictory ARIA, unlabeled form controls, insufficient color contrast, broken heading hierarchy, a missing document language. Those are deterministic and belong in the pipeline. The remainder - whether alt text is meaningful, whether the screen-reader narrative makes sense, whether a flow is actually operable with a keyboard or a switch device - cannot be settled by a tool and must not be faked with one.

Three tiers, mapped to pipeline placement:

1. Static analysis (in-band, blocks build). Accessibility linting catches structural violations in source without rendering: missing alt text, ARIA misuse, label associations, heading order. It runs in the IDE, pre-commit, and CI, exactly like any other static check. Cheapest and fastest; treat high-severity findings as build-breaking, the same as a security finding.
2. Component tests against the rendered DOM (in-band, blocks build). Some violations exist only in the rendered output: contrast computed after CSS resolves, focus order, dynamic ARIA state, keyboard operability. A scanner assertion inside a component test (expect(results).toHaveNoViolations()) plus explicit keyboard-navigation assertions cover these deterministically, on every commit. The user interface pattern shows the full shape.
3. Manual audit and assistive-technology testing (out-of-band, never a gate). Real screen-reader passes, keyboard-only walkthroughs, and expert review of whether the experience is coherent. This is continuous and informs the backlog; like exploratory testing, it is not a pass/fail checkpoint and must not gate a deploy.

The caveat that keeps tiers 1 and 2 honest: automated checks detect only a fraction of WCAG success criteria - industry estimates commonly land between a third and a half, depending on the tool and the page. A green automated scan means “no detectable violations,” not “accessible.” Wiring a scanner into the build is necessary and high-value, but a team that reads a passing scan as proof of accessibility has the same false-confidence problem as a team that reads high line coverage as proof of correctness. The deterministic tiers shrink the manual surface; they do not remove it.

This mirrors observability as a tested artifact above: the machine-verifiable part of a human contract gets pinned in the deterministic suite, and the judgment part stays with people.

Performance and load testing

Three classes of perf tests, each with a different home in the pipeline:

1. Per-endpoint perf budgets in component tests. Simple latency assertion under no load (assertThat(p99).isLessThan(50ms)). Catches algorithmic regressions cheaply. Fits in CI Stage 1 if the assertions are tight and the runtime is stable.
2. Load tests in acceptance. k6, Gatling, or Locust against a deployed instance. Validate p99 latency, throughput, and error rate at expected production load. Gates production promotion.
3. Soak tests out of pipeline. Long-running load to catch memory leaks, file handle leaks, and slow drift. Scheduled, non-blocking.

A perf regression that breaches a documented budget should block deploy. A regression within budget but worse than baseline should generate a finding for review, not a build failure: noisy alerts get ignored.

Mutation testing

Coverage % tells you what code ran. Mutation testing tells you whether the tests would have failed if the code had been wrong. Tools (Stryker for JS, PIT for Java) systematically change operators, return values, and conditionals, then re-run the test suite. Surviving mutants are tests that didn’t catch the mutation.

Each surviving mutant is one of three things:

• A real test gap. Add a flow-oriented test that would have failed when the mutation was applied.
• An equivalent mutant, semantically identical to the original. Mark and move on.
• A trivially equivalent mutant (logging change, assertion message tweak). Configure the tool to skip.

Mutation testing is too slow to run on every commit. Run it nightly or weekly on the highest-value modules. Treat it as a periodic audit of test quality, not a gating check.

Flake handling protocol

A flaky test is a known unknown. Three rules keep flakes from rotting the suite:

1. Quarantine on detection. First flake gets the test moved to a quarantine lane that doesn’t block the build. Don’t ignore it; don’t keep failing builds for unrelated reasons.
2. Time-boxed remediation. Quarantined tests have a deadline (e.g., five business days) and an owner. After the deadline, fix or delete. No silent quarantine.
3. Track the cause. Most flakes share root causes: timing, shared state, network, ordering. The fix is usually structural (eliminate the timing dependency) rather than local (add a longer sleep).

A suite with a permanent quarantine list has lost its CD-ready quality. See also Tests Randomly Pass or Fail.

Cost and time budgets

Empirical starting points for in-band test budgets, based on typical service complexity. Adjust for your codebase, language, framework, and the size of the component under test.

The total CD pipeline in-band suite under 10 minutes is the gating constraint at the team level. The first lever for hitting that budget is parallel execution: the suite should fan out across cores or runners, not run serially. Parallelism only works when tests are independent of each other - no shared mutable state, no ordering dependencies, no global fixtures that one test mutates and another reads. Decoupling tests is a prerequisite for speed, not an optimization on top of it.

If a component’s tests still can’t fit the budget after the suite is running in parallel, the goal is to remediate the underlying cause - slow component startup, oversize fixtures, expensive setup duplicated per test, hidden serialization through a shared resource - not to declare the budget unreachable. While the remediation is underway, moving the offending tests out-of-band on a schedule is a reasonable stopgap so the in-band suite stays fast. Out-of-band placement here is a temporary mitigation, not the destination: those tests should come back in-band once the underlying speed issue is fixed.

9 - Test Feedback Speed

Why speed has a threshold

The 10-minute CI target and the preference for sub-second unit tests are not arbitrary. They are long-standing conventions in CD practice, and they align with how human cognition handles interrupted work. When a developer makes a change and waits for test results, three things determine whether that feedback is useful: whether the developer still holds the mental model of the change, whether they can act on the result immediately, and whether the wait is short enough that they do not context-switch to something else.

Research on task interruption and working memory consistently shows that context switches are expensive. Gloria Mark’s research at UC Irvine found that it takes an average of 23 minutes for a person to fully regain deep focus after being interrupted during a task, and that interrupted tasks take twice as long and contain twice as many errors as uninterrupted ones.¹ If the test suite itself takes 30 minutes, the total cost of a single feedback cycle approaches an hour - and most of that time is spent re-loading context, not fixing code.

The cognitive breakpoints

Jakob Nielsen’s foundational research on response times identified three thresholds that govern how users perceive and respond to system delays: 0.1 seconds (feels instantaneous), 1 second (noticeable but flow is maintained), and 10 seconds (attention limit - the user starts thinking about other things).² These thresholds, rooted in human perceptual and cognitive limits, apply directly to developer tooling.

Different feedback speeds produce fundamentally different developer behaviors:

The conventional 10-minute CI target lines up with the boundary between “developer waits and acts on the result” and “developer starts something else and pays a full context-switch penalty.” Below 10 minutes, feedback is actionable. Above 10 minutes, feedback becomes an interruption. The number itself is an established CD convention rather than a figure the cognitive research produces directly, but DORA’s research on continuous integration converges on the same target: tests should complete in under 10 minutes to support the fast feedback loops that high-performing teams depend on.³

What this means for test architecture

These cognitive breakpoints should drive how you structure your test suite:

Local development (under 1 second). Unit tests for the code you are actively changing should run in watch mode, re-executing on every save. At this speed, TDD becomes natural - the test result is part of the writing process, not a separate step. This is where you test complex logic with many permutations.

Pre-push verification (under 2 minutes). The full unit test suite and the component tests for the component you changed should complete before you push. At this speed, the developer stays engaged and acts on failures immediately. This is where you catch regressions.

CI pipeline (under 10 minutes). The full deterministic suite - all unit tests, all component tests, all contract tests - should complete within 10 minutes of commit. At this speed, the developer has not yet fully disengaged from the change. If CI fails, they can investigate while the code is still fresh.

Post-deploy verification (minutes to hours). E2E smoke tests and integration test validation run after deployment. These are non-deterministic, slower, and less frequent. Failures at this level trigger investigation, not immediate developer action.

When a test suite exceeds 10 minutes, the solution is not to accept slower feedback. It is to redesign the suite: replace E2E tests with component tests using test doubles, parallelize test execution, and move non-deterministic tests out of the gating path.

Impact on application architecture

Test feedback speed is not just a testing concern - it puts pressure on how you design your systems. A monolithic application with a single test suite that takes 40 minutes to run forces every developer to pay the full context-switch penalty on every change, regardless of which module they touched.

Breaking a system into smaller, independently testable components is often motivated as much by test speed as by deployment independence. When a component has its own focused test suite that runs in under 2 minutes, the developer working on that component gets fast, relevant feedback. They do not wait for tests in unrelated modules to finish.

This creates a virtuous cycle: smaller components with clear boundaries produce faster test suites, which enable more frequent integration, which encourages smaller changes, which are easier to test. Conversely, a tightly coupled monolith produces a slow, tangled test suite that discourages frequent integration, which leads to larger changes, which are harder to test and more likely to fail.

Architecture decisions that improve test feedback speed include:

• Clear component boundaries with well-defined interfaces, so each component can be tested in isolation with test doubles for its dependencies.
• Separating business logic from infrastructure so that core rules can be unit tested in milliseconds without databases, queues, or network calls.
• Independently deployable services with their own test suites, so a change to one service does not require running the entire system’s tests.
• Avoiding shared mutable state between components, which forces integration tests and introduces non-determinism.

If your test suite is slow and you cannot make it faster by optimizing test execution alone, the architecture is telling you something. A system that is hard to test quickly is also hard to change safely - and both problems have the same root cause.

The compounding cost of slow feedback

Slow feedback does not just waste time - it changes behavior. When the suite takes 40 minutes, developers adapt:

• They batch changes to avoid running the suite more than necessary, creating larger and riskier commits.
• They stop running tests locally because the wait is unacceptable during active development.
• They push to CI and context-switch, paying the full rebuild penalty on every cycle.
• They rerun failures instead of investigating, because re-reading the code they wrote an hour ago is expensive enough that “maybe it was flaky” feels like a reasonable bet.

Each of these behaviors degrades quality independently. Together, they make continuous integration impossible. A team that cannot get feedback on a change within 10 minutes cannot sustain the practice of integrating changes multiple times per day.⁴

Sources

Further reading

• Build Duration - Measuring and improving CI pipeline speed
• Integration Frequency - How feedback speed affects integration cadence
• Development Cycle Time - End-to-end flow from change to production

10 - Testing Antipatterns

Most teams arrive at this section with a test suite that doesn’t match the Applied Testing Strategies guide. This page covers the failure modes that show up most often and the migration moves that get a suite back on track.

Common testing anti-patterns

Each entry below is a smell that the suite is testing the wrong thing, will erode trust over time, or will block refactoring instead of enabling it.

Reflection to reach private members

Using reflection (or language-equivalent escape hatches: @VisibleForTesting-only public access, friend classes, internal exposed only for tests) to read or invoke private members from a test. This couples the test to the exact internal structure of the class, breaks every time the implementation is refactored, and tests something the caller cannot observe, meaning the test can pass while the actual public behavior is broken.

If a private behavior is worth testing, it’s reachable through a public method that exercises it. If no public method exercises it, the private code is dead and should be deleted. Reflection in tests is a signal that either the design needs adjustment (the class is too large and a collaborator wants to come out) or the test is aimed at the wrong abstraction level.

Testing private methods directly

Same root cause as the reflection anti-pattern, but achieved by making methods package-private, protected, or otherwise reachable through a side door specifically so tests can call them. The method’s accessibility is now distorted by the test, not by the design. Drive private logic through the public method that uses it, or extract it into a collaborator with its own public surface and test that collaborator through its public interface.

One test class per production class, one test per method

Tests organized as a mirror of the production code structure, such as OrderServiceTest with testProcessPayment, testValidateOrder, testEmitEvent, produce a suite that documents the implementation and dies on contact with refactoring. Organize tests by behavior. An OrderPlacement test class with places_order_with_valid_payment, rejects_order_when_payment_declined, holds_order_when_inventory_unavailable is what survives, what reads well, and what catches integration bugs between methods.

Tests that mirror the implementation

A test that asserts “method A is called, then method B is called, then method C is called with these arguments” is testing the implementation, not the behavior. The same outcome could be achieved by a different sequence of calls, and if the test fails when the sequence changes but the outcome doesn’t, the test is wrong, not the code. Assert on observable outcomes (returned value, persisted state, emitted event, response status) and use mocks/spies sparingly, only for outbound interactions that are themselves part of the contract.

Mocking what you don’t own

Stubbing a third-party SDK, ORM, HTTP client, or cloud SDK directly in tests. The double is now a claim about a library the team has no control over and incomplete knowledge of. When the library updates or the team upgrades versions, the doubles are silently wrong and the tests still pass. Wrap third-party clients in a thin gateway the team owns, then double the gateway.

Doubles without validating tests

Any test double that has no corresponding mechanism (contract test, adapter integration test, post-deploy integration check) keeping it honest is a lie waiting to be discovered in production. If a double exists and there’s no traceable answer to “how would we know if this stopped matching reality?” that double is a known risk and should be tracked as one.

Over-mocking

Replacing every collaborator with a mock so the test sees only the system under test in isolation. The test now mirrors the implementation: every refactor that moves a method between collaborators breaks tests that didn’t fail for any production reason. Only mock what’s necessary to keep the test deterministic. Real in-process collaborators - value objects, domain models, in-memory repositories - belong in the test, not behind a mock.

Complex mock setup

If a single test needs dozens of lines to set up its mocks, the system under test probably has too many dependencies for one unit of behavior. Setup complexity is a smell pointing at the production design, not at the test. Refactor the production code (extract a collaborator, narrow the interface, push concerns into separate classes) before adding more mocks.

Sleeping in tests

Thread.sleep, await sleep(500), and friends to “wait for” an asynchronous operation. Sleeps are either too short (flaky) or too long (slow), and they ratchet upward over time as people debug flakes. Use the framework’s built-in waiting primitives (Awaitility, waitFor from Testing Library, eventually blocks) that poll until a condition is true with a bounded timeout. If the system under test depends on real wall-clock time, inject a fake clock. Never sleep.

Shared mutable state between tests

Tests that depend on the order they run in, or that leak state through static singletons, shared databases without per-test isolation, or module-level caches. Each test should set up the state it needs and tear it down (or use a fresh isolated context). Order-dependent suites fail randomly when run in parallel and produce “works on my machine” failures that erode trust in the suite.

Skipping or muting tests instead of fixing them

A muted test is a known bug in the test or in the system, hidden. Either fix it now, delete it, or open a ticket and put a deadline on it. Suites with a steady population of @Ignore/@skip/xit decorations end up with a steady population of latent bugs.

Test code held to lower standards than production code

Copy-pasted setup blocks, string-typed assertions on JSON fragments, magic numbers, no abstractions, no review. Tests are production code. They’re how the team learns whether the system works. Refactor them, deduplicate them, name them well, and review them as carefully as the code they protect.

Testing through the UI when the same behavior is testable lower in the stack

UI tests are the slowest and most fragile layer. Pushing logic-only assertions into UI tests because “that’s where we’re set up to test” produces a brittle, slow suite that becomes a tax on every change. Test logic where the logic lives. Reserve UI tests for things that can only be observed at the UI layer.

“We’ll add tests later”

Tests added after the code is already in production, written by someone who didn’t write the code, asserting only what the code currently does, are not tests of the system’s intended behavior. They’re a snapshot of the current implementation, including its bugs. The team learns nothing from them and refactoring becomes risky in exactly the way tests are supposed to prevent. Tests written alongside the code (or before it, TDD-style) are the only ones that document intent.

Migrating an existing suite

The right first move depends on what the suite looks like now. Five common starting points and the first three steps for each:

If most coverage is end-to-end Selenium or Cypress against real backends

1. Inventory the flows the E2E suite exercises. Pick the top five that fail most often.
2. Build component tests for those flows. Double the backend through the gateway the team owns.
3. Once those component tests are green and the doubles they rely on are backed by a contract test plus an out-of-band check that is actually running and watched, delete the corresponding E2E tests. Don’t keep both: duplicated coverage doubles the maintenance cost without doubling the confidence. Until that out-of-band validation is in place and monitored, keep one real-integration smoke test per flow - the component test’s confidence rests on doubles, and deleting the last real-integration signal before anything proves those doubles still match reality just moves the risk somewhere you can’t see it.

If most “unit” tests mock third-party SDKs

1. Identify the third-party clients (HTTP, DB, cloud SDKs). For each, define a thin gateway interface owned by the team.
2. Replace direct SDK use in production code with the gateway. Tests now double the gateway, which the team controls.
3. Add adapter integration tests against the real dependency (testcontainer, sandbox account). The doubles are now backed by reality.

If line coverage is high but production keeps breaking

1. Run mutation testing on a high-traffic module. Most surviving mutants are tests that didn’t catch the mutation.
2. For each surviving mutant, add a flow-oriented test that would have caught it. Don’t add a test of the specific mutation: add the test of the behavior the mutation breaks.
3. Repeat module by module, prioritized by production incident frequency. Coverage % won’t change much. Defect-finding will.

If the suite has six figures of tests and runs for 90 minutes

1. Move tests that need a database or downstream into an integration lane on a different cadence (post-merge or scheduled), not the pre-commit gate.
2. Convert sociable unit tests to component tests where they exercise complete flows. Delete redundant unit-level duplicates.
3. Set a budget: deterministic suite under 10 minutes. Non-conforming tests get reviewed; if they can’t be made fast, they move to acceptance or get deleted.

If there are no tests at all

1. Don’t try to retrofit unit tests for existing code. You’ll write tests that pin the current bugs.
2. Start with a small set of component tests for the highest-value flows. They double as characterization tests for legacy behavior.
3. As the team changes code, write tests for the change first. The test base grows organically with the change set, and the parts of the code that change most are the parts that get tests soonest.

The pattern across all five: don’t try to convert the whole suite at once. Move flow by flow, module by module. The test that matters next is the one for the change you’re about to make.

Related Content

• Applied Testing Strategies - the patterns this page is helping teams migrate toward.
• Architecting Tests for CD - the section overview, with the do/do-not list this page expands on.
• Test Double - the glossary entry covering the five flavours and when to use each.

11 - Testing Glossary

These definitions reflect how this site uses each term. They are not universal definitions - other communities may use the same words differently.

Acceptance Tests

Automated tests that verify a system behaves as specified. Acceptance tests exercise user workflows in a production-like environment and confirm the implementation matches the acceptance criteria. They answer “did we build what was specified?” rather than “does the code work?” They do not validate whether the specification itself is correct - only real user feedback can confirm we are building the right thing.

In CD, acceptance testing is a pipeline stage, not a single test type. It can include component tests, load tests, chaos tests, resilience tests, and compliance tests. Any test that runs after CI to gate promotion to production is an acceptance test.

Referenced in: CD Testing, Pipeline Reference Architecture

Adapter Integration Test

A narrow test of a single boundary adapter - the team’s own HTTP client, database query layer, message-broker client, file-system adapter, or similar - exercised against either the real external dependency or a high-fidelity stand-in like a testcontainer running the production engine. (Legacy name from Toby Clemson: “gateway integration test.”)

What the test is for

The test asserts that the adapter correctly speaks the protocol: that it serializes the request the way the dependency expects, parses the response shape correctly, maps errors to the right exception types, propagates headers, enforces timeouts, and handles transactional semantics.

What the test is not for

It does not test the behavior of the dependency itself. If the adapter asks for a user, the test validates that the response parses into a valid User object - not which user comes back, not the dependency’s own business rules, not anything that the dependency owns. The dependency’s correctness is the dependency’s problem; the adapter’s job is to speak the protocol faithfully. Conflating the two produces brittle tests that fail on unrelated changes to the dependency’s data or logic.

Pipeline placement

Runs in-band only when both conditions hold:

1. The team has full control over the dependency - a database, broker, or service the team owns and can pin to a known version, typically via a per-test testcontainer.
2. The test is fully deterministic against that controlled instance.

For everything else - third-party APIs, services owned by another team, dependencies whose state the team can’t reset between runs - the test runs out-of-band on a schedule. Out-of-band placement is the default for any adapter test that touches a system outside the team’s full control. Failures trigger review, not a build break. Pulling these tests in-band is the most common cause of flaky pipelines.

Distinguishing from neighboring test types

Different from a broader end-to-end test: an adapter integration test isolates one boundary adapter, not a flow across multiple components. Different from a contract test at the same boundary: contract tests pin shape against doubles in the pipeline; adapter integration tests pin protocol against the real dependency.

Referenced in: API Consumer, API Provider, Applied Testing Strategies, Antipatterns, Event Consumer, Event Producer, Scheduled Job, Stateful Service

API Surface Test

A test that pins the public-facing API of a library or CLI - the exported symbols, their signatures, the documented arguments and exit codes. Typically a snapshot: the current public surface is captured to a file, and any diff fails the build. Catches accidental breaking changes (a renamed function, a removed flag, a tightened type) before they reach consumers. Distinct from a contract test, which pins the wire boundary between two services; an API surface test pins the source-level boundary between a library and its callers.

Referenced in: CLI Tool or Library

Black Box Testing

A testing approach where the test exercises code through its public interface and asserts only on observable outputs - return values, state changes visible to consumers, or side effects such as messages sent. The test has no knowledge of internal implementation details. Black box tests are resilient to refactoring because they verify what the code does, not how it does it. Contrast with white box testing.

Referenced in: CD Testing, Unit Tests

Cluster Test

A test that exercises a stateful service across multiple nodes - replication, leader election, consensus, partition tolerance - against a real multi-node setup, typically via testcontainers running the production consensus library. Cluster tests catch behavior that only appears under a real cluster: split-brain, slow followers, leader transitions, partition reconciliation. Deterministic enough to run in-band but slower than single-node component tests, so usually relegated to a later CI stage.

Referenced in: Stateful Service

Component Test

See Component Tests.

Referenced in: Component Tests, End-to-End Tests, Tests Randomly Pass or Fail, Unit Tests

Contract Test

See Contract Tests.

Referenced in: API Consumer, API Provider, Contract Tests, Event Consumer, Event Producer

Cross-OS Test Matrix

A CI configuration that runs the existing test suite on each supported operating system rather than a separate test type. The matrix catches platform-specific behavior single-OS tests can’t: path separators, line endings, signal-handling differences, locale defaults, file-system case sensitivity. Required for any deployable consumed across multiple OSes - CLI tools, libraries, cross-platform desktop or mobile apps.

Referenced in: CLI Tool or Library

Deployed-binary Test

A test that invokes the actual deployed artifact - the same binary, container image, or package the scheduler, orchestrator, or operator will invoke in production - and asserts on observable behavior at startup or first invocation. Catches what in-process component tests bypass: configuration loading, secret resolution, signal handling, exit codes, lock acquisition, dependency-version mismatches. Usually a small set; the bulk of behavior is tested in component tests against an in-memory assembled app.

Referenced in: CLI Tool or Library, Scheduled Job

Doctest

An executable test extracted from documentation - typically the README or inline code samples - that runs the documented examples against the real binary or library and fails the build if the examples are broken. Doctests close the gap between “the docs say X works” and “X actually works in the latest build”. Most languages have framework support: Python’s doctest module, Rust’s #[doc] attribute, and Markdown-based runners for Node and Java.

Referenced in: CLI Tool or Library

In-Band Test

A test that runs in the delivery pipeline as part of the commit-to-deploy flow. In-band tests must be deterministic, which means test doubles replace anything that crosses the component boundary - downstream services, message brokers, schedulers, browsers talking to real backends. Failures block the build or the deployment.

The bulk of any project’s test suite is in-band: unit tests, component tests, contract tests, and adapter integration tests against team-controlled dependencies (testcontainers running an engine the team pins). Adapter integration tests against third-party services or shared environments run out-of-band on a schedule, not in-band. They give a deterministic go/no-go signal in minutes.

Contrast with out-of-band tests, which run on a schedule against real systems and never gate the build.

Referenced in: Applied Testing Strategies, Architecting Tests for CD

Out-of-Band Test

A test that runs outside the delivery pipeline on a schedule or post-deploy, exercising real external systems. Out-of-band tests are non-deterministic by design (they depend on the real world) and never gate a commit or merge. Failures trigger review, alerts, or rollback decisions.

Out-of-band checks are how teams confirm that the doubles used by in-band tests still match reality. Examples: post-deploy integration tests against the real downstream, synthetic monitoring of production, scheduled smoke checks against a sandbox API.

Referenced in: Applied Testing Strategies, Architecting Tests for CD, Integration Tests

Soak Test

A long-running test that exercises a deployed service for hours or days under representative load to catch behavior that only appears with time: memory leaks, unbounded growth, replication-lag drift, slow-burn resource exhaustion. Soak tests are out-of-band by design - they don’t fit a pre-merge budget. Failures trigger review, not a build break. Often paired with chaos testing (deliberate fault injection during the soak) to validate recovery behavior over time.

Referenced in: Stateful Service

Sociable Unit Test

A unit test that allows real collaborator objects to participate - for example, a service object calling a real domain model or value object - while still replacing any external I/O (network, database, file system) with test doubles. The “unit” being tested is a behavior that spans multiple in-process objects. When the scope expands to the entire public interface of a frontend component or backend service, that is a component test.

Referenced in: Unit Tests, Component Tests

Solitary Unit Test

A unit test that replaces all collaborators with test doubles and exercises a single class or function in complete isolation. Contrast with sociable unit test, which allows real collaborator objects while still replacing external I/O.

Referenced in: Unit Tests

Synthetic Monitoring

Automated scripts that continuously execute realistic user journeys or API calls against a live production (or production-like) environment and alert when those journeys fail or degrade. Unlike passive monitoring that watches for errors in real user traffic, synthetic monitoring proactively simulates user behavior on a schedule - so problems are detected even during low traffic periods. Synthetic monitors are non-deterministic (they depend on live external systems) and are never a pre-merge gate. Failures trigger alerts or rollback decisions, not build blocks.

Referenced in: Architecting Tests for CD, End-to-End Tests

TDD (Test-Driven Development)

A development practice where tests are written before the production code that makes them pass. TDD supports CD by ensuring high test coverage, driving simple design, and producing a fast, reliable test suite. TDD feeds into the testing fundamentals required in Phase 1.

Referenced in: CD for Greenfield Projects, Integration Frequency, Inverted Test Pyramid, Small Batches, TBD Migration Guide, Trunk-Based Development, Unit Tests

Test Double

A stand-in object that replaces a real production dependency during testing. The term comes from the film industry’s “stunt double”: just as a stunt double replaces an actor for dangerous scenes, a test double replaces a costly or non-deterministic dependency to make tests fast, isolated, and reliable.

Test doubles let you:

• Remove non-determinism by replacing network calls, databases, and file systems with predictable substitutes.
• Control test conditions by forcing specific states, error conditions, or edge cases that would be hard to reproduce with real dependencies.
• Increase speed by eliminating slow I/O.
• Isolate the system under test so failures point at the code being tested, not at an external dependency.

Types of test doubles

Choosing the right double

• Use a stub when you need to supply data but don’t care how it was requested.
• Use a spy when you need to verify call arguments or call count.
• Use a mock when the interaction itself is the primary thing being verified.
• Use a fake when you need realistic behavior but can’t use the real system.
• Use a dummy when a parameter is required by the interface but irrelevant to the test.

Test doubles are heaviest in the early pipeline stages (unit, component, contract tests) where deterministic speed is the priority. They thin out as you move through the pipeline; end-to-end tests use no doubles by design. The guiding principle from Justin Searls: “Don’t poke too many holes in reality.” Use a double when you must, and prefer the real implementation when it’s fast and deterministic.

Doubles are only as good as the contract they encode. Every double in the suite should trace to a contract test pinning its claims and an out-of-band check confirming the claims still hold. See the Antipatterns page for the failure modes of unvalidated doubles.

Referenced in: Antipatterns, Applied Testing Strategies, Component Tests, Contract Tests, Unit Tests

Virtual Service

A test double that simulates a real external service over the network, responding to HTTP requests with pre-configured or recorded responses. Unlike in-process stubs or mocks, a virtual service runs as a standalone process and is accessed via real network calls, making it suitable for component testing and end-to-end testing where your application needs to make actual HTTP requests against a dependency. Service virtualization tools can create virtual services from recorded traffic or API specifications. See Test Doubles.

Referenced in: Component Tests, End-to-End Tests, Testing Fundamentals

White Box Testing

A testing approach where the test has knowledge of and asserts on internal implementation details - specific methods called, call order, internal state, or code paths taken. White box tests verify how the code works, not what it produces. These tests are fragile because any refactoring of internals breaks them, even when behavior is unchanged. Avoid white box testing in unit tests; prefer black box testing that asserts on observable outcomes.

Referenced in: CD Testing, Unit Tests