technical skills - testing without mocks - foundational patterns

src: Foundational patterns

Narrow Tests

Broad tests, such as end-to-end tests, tend to be slow and brittle. They’re complicated to read and write, often fail randomly, and take a long time to run. Therefore:

Instead of using broad tests, use narrow tests. Narrow tests check a specific function or behavior, not the system as a whole. Unit tests are a common type of narrow tests.

When testing infrastructure, use Narrow Integration Tests. When testing pure logic, use the Logic Patterns. When testing code that has infrastructure dependencies, use Nullables.

To ensure your code works as a whole, use State-Based Tests and Overlapping Sociable Tests.

State-Based Tests

Mocks and spies result in “interaction-based” tests that check how the code under test uses its dependencies. However, they can be hard to read, and they tend to “lock in” your dependencies, which makes structural refactorings difficult. Therefore:

Use state-based tests instead of interaction-based tests. A state-based test checks the output or state of the code under test, without any awareness of its implementation. For example, given the following production code:

// Production code to describe phase of moon (JavaScript)
import * as moon from "astronomy";
import { format } from "date_formatter";
 
export function describeMoonPhase(date) {
  const visibility = moon.getPercentOccluded(date);
  const phase = moon.describePhase(visibility);
  const formattedDate = format(date);
  return `The moon is ${phase} on ${formattedDate}.`;
}

A state-based test would pass in a date and check the result, like this:

// State-based test of describeMoonPhase() (JavaScript)
import { describeMoonPhase } from "describe_phase";
 
it("describes phase of moon", () => {
  const dateOfFullMoon = new Date("8 Dec 2022");    // a date when the moon was actually full
  const description = describeMoonPhase(dateOfFullMoon);
  assert.equal(description, "The moon is full on December 8th, 2022.";
});

In contrast, an interaction-based test would check how each dependency was used, like this:

// Interaction-based test of describeMoonPhase() (JavaScript and fictional mocking framework)
const moon = mocker.mockImport("astronomy");
const { format } = mocker.mockImport("date_formatter");
const { describeMoonPhase } = mocker.importWithMocks("describe_phase");
 
it("describes phase of moon", () => {
  const date = new Date();    // specific date doesn't matter
 
  mocker.expect(moon.getPercentOccluded).toBeCalledWith(date).thenReturn(999);
  mocker.expect(moon.describePhase).toBeCalledWith(999).thenReturn("PHASE");
  mocker.expect(format).toBeCalledWith(date).thenReturn("DATE");
 
  const description = describeMoonPhase(date);
  mocker.verify();
  assert.equal(description, "The moon is PHASE on DATE");
};

State-based tests naturally result in Overlapping Sociable Tests. To use state-based tests on code with infrastructure dependencies, use the Nullability Patterns.

Overlapping Sociable Tests

Tests using mocks and other test doubles isolate the code under test by replacing its dependencies. This requires broad tests to confirm that the system works as a whole, but we don’t want to use broad tests. Therefore:

When testing the interactions between an object and its dependencies, use the code under test’s real dependencies. Don’t test the dependencies’ behavior, but do test that the code under test uses its dependencies correctly. This happens naturally when using State-Based Tests.

For example, the following test checks that describeMoonPhase uses its Moon and format dependencies correctly. If they don’t work the way describeMoonPhase thinks they do, the test will fail.

// Example of sociable tests (JavaScript)
 
// Test code
it("describes phase of moon", () => {
  const dateOfFullMoon = new Date("8 Dec 2022");
  const description = describeMoonPhase(dateOfFullMoon);
  assert.equal(description, "The moon is full on December 8th, 2022.";
};
 
// Production code
describeMoonPhase(date) {
  const visibility = moon.getPercentOccluded(date);
  const phase = moon.describePhase(visibility);
  const formattedDate = format(date);
  return `The moon is ${phase} on ${formattedDate}.`;
}

Write Narrow Tests that are focused on the behavior of the code under test, not the behavior of its dependencies. Each dependency should have its own thorough set of Narrow Tests. For example, don’t test all phases of the moon in your describeMoonPhase() tests, but do test them in your Moon tests. Similarly, don’t check the intricacies of date formatting in your describeMoonPhase tests, but do test them in your format(date) tests.

In addition to checking how your code uses its dependencies, sociable tests also protect you against future breaking changes. Each test overlaps with dependencies’ tests and dependents’ tests, creating a strong linked chain of tests. This gives you the coverage of broad tests without their speed and reliability problems.

For example, imagine the dependency chain LoginController → Auth0Client → HttpClient:

• The LoginController tests checks that LoginController is correct, including how it uses Auth0Client. (Auth0Client in turn runs HttpClient, but that isn’t explicitly checked by the LoginController tests.)
• The Auth0Client tests check that Auth0Client is correct, including how it uses HttpClient.
• The HttpClient tests check that HttpClient is correct, including using Narrow Integration Tests to check how it communicates with HTTP servers.
• Together, they ensure the whole chain is checked. Even if HttpClient and its tests are changed intentionally, if that change breaks Auth0Client, its tests would fail (and possibly the LoginController tests, too). Changing Auth0Client’s behavior would similarly break the LoginController tests.

In contrast, if the LoginController tests stubbed or mocked out Auth0Client, the chain would be broken. Changing Auth0Client’s behavior would not break the LoginController tests, because nothing would check how LoginController used the real Auth0Client.

To avoid manually constructing the entire dependency chain, use Parameterless Instantiation with Zero-Impact Instantiation. To isolate tests from changes in dependencies’ behavior, use Collaborator-Based Isolation. To prevent your tests from interacting with external systems and state, use Nullables. To catch breaking changes in external systems, use Paranoic Telemetry. For a safety net, use Smoke Tests.

Smoke Tests

Overlapping Sociable Tests are supposed to cover your entire system. But nobody’s perfect, and mistakes happen. Therefore:

Write one or two end-to-end tests that make sure your code starts up and runs a common workflow. For example, if you’re coding a web site, check that you can get an important page.

Don’t rely on smoke tests to catch errors. Your real test suite should consist of Narrow, Sociable tests. If the smoke tests catch something the rest of your tests don’t, fill the gap with more narrow tests.

Zero-Impact Instantiation

Overlapping Sociable Tests instantiate their dependencies, which in turn instantiate their dependencies, and so forth. If instantiating this web of dependencies takes too long or causes side effects, the tests could be slow, difficult to set up, or fail unpredictably. Therefore:

Don’t do significant work in constructors. Don’t connect to external systems, start services, or perform long calculations. For code that needs to connect to an external system or start a service, provide a connect() or start() method. For code that needs to perform a long calculation, consider lazy initialization. (But even complex calculations aren’t likely to be a problem, so profile before optimizing.)

Parameterless Instantiation

Overlapping Sociable Tests require your whole dependency tree to be instantiated, but multi-level dependency chains are difficult to set up in tests. Dependency injection (DI) frameworks work around the problem, but we don’t want to require such magic. Therefore:

Ensure all classes have a constructor or factory that doesn’t take any parameters. This factory (or constructor) should have sensible defaults that set up everything the object needs, including instantiating its dependencies. You can make these defaults overridable if desired. (If your language doesn’t support overridable defaults, use method overloading or an Options object, as shown in the Signature Shielding pattern.)

For some classes, particularly Value Objects, a parameterless factory isn’t a good idea in production, because people could forget to provide a necessary value. For example, an immutable Address class should be constructed with its street, city, and so forth. Providing a default city could result in addresses that seemed to work, but actually had the wrong city.

In that case, provide a test-specific factory method with overridable defaults. Choose defaults that make the class work in as many situations as possible, and use a name such as createTestInstance() to indicate that it’s only for tests. In your tests, pass in every parameter your test cares about, rather than relying on default values. That way, changes to the factory won’t break your tests.

// Test-specific factory using named, optional parameters (JavaScript)
class Address {
  // Production constructor
  constructor(street, city, state, country, postalCode) {
    this._street = street;
    this._city = city;
    //...etc...
  }
 
  // Test-specific factory
  static createTestInstance({
    street = "Address test street",
    city = "Address test city",
    state = State.createTestInstance(),
    country = Country.createTestInstance(),
    postalCode = PostalCode.createTestInstance(),
  } = {}) {
    return new Address(street, city, state, country, postalCode);
  }
}

This test-specific factory method is easiest to maintain if it’s located in the production code next to the real constructors. However, if you don’t want test-specific code in production, or if the logic gets complicated, you can use the Object Mother pattern to put it in a test-only helper module instead.

Signature Shielding

As you refactor your application, method signatures will change. If your code is well-designed, this won’t be a problem for production code, because most methods will only be used in a few places. But tests can have many duplicated method and constructor calls. When you change those methods or constructors, you’ll have a lot of busywork to update the tests. Therefore:

Provide helper functions to instantiate classes and call methods. Have these helper functions perform any setup your tests need rather than using your test framework’s before() or setup() functions.

Make the helper functions take optional parameters for customizing your setup and execution. If convenient in your programming language, return multiple optional values as well. This will allow you to expand your helper functions without breaking existing tests.

// Optional parameters and multiple return values (JavaScript)
 
// Example test
it("uses hosted page for authentication", () => {
  const { url } = getLoginUrl({       // Use the helper function
    host: "my.host",
    clientId: "my_client_id",
    callbackUrl: "my_callback_url"
  });
 
  assert.equal(url, "https://my.host/authorize?response_type=code&client_id=my_client_id&callback_url=my_callback_url");
});
 
// Example helper function
function getLoginUrl({
  host = "irrelevant.host",           // Optional parameters
  clientId = "irrelevant_client_id",
  clientSecret = "irrelevant_secret",
  connection = "irrelevant_connection"
  username = "irrelevant_username",
  callbackUrl = "irrelevant_url",
} = {}) {
  const client = new LoginClient(host, clientId, clientSecret, connection);
  const url = client.getLoginUrl(username, callbackUrl);
 
  return { client, url };             // Multiple return values
}

If you’re using a language without support for optional parameters, use method overloading or an “Options” object. If you’re using a language without support for multiple return values, you can return a simple data structure.

// Optional parameters and multiple return values (Java)
 
// Example tests
@Test
public void usesHostedPageForAuthentication() {
  GetLoginUrlResult actual = getLoginUrl(new GetLoginUrlOptions()   // Use the helper function and Options object
    .withHost("my.host")
    .withClientId("my_client_id")
    .withCallbackUrl("my_callback_url")
  );
  assert.equal(actual.url, "https://my.host/authorize?response_type=code&client_id=my_client_id&callback_url=my_callback_url");
}
 
// Helper function using Options object and return data structure
private GetLoginUrlResult getLoginUrl(GetLoginUrlOptions options) {
  LoginClient client = new LoginClient(options.host, options.clientId, options.secret, options.connection);
  String url = client.getLoginUrl(options.username, options.callbackUrl);
  return new GetLoginUrlResult(client, url);
}
 
// Options object
private static final class GetLoginUrlOptions {
  public String host = "irrelevant.host";
  public String clientId = "irrelevant_client_id";
  public String clientSecret = "irrelevant_secret";
  public String connection = "irrelevant_connection";
  public String username = "irrelevant_username";
  public String callbackUrl = "irrelevant_url";
 
  GetLoginUrlOptions withHost(String host) {
    this.host = host;
    return this;
  }
 
  GetLoginUrlOptions withClientId(String clientId) {
    this.clientId = clientId;
    return this;
  }
 
  GetLoginUrlOptions withCallbackUrl(String url) {
    this.callbackUrl = url;
    return this;
  }
}
 
// Return data structure
private static final class GetLoginUrlResult {
  public LoginClient client;
  public String url;
 
  public GetLoginUrlResult(LoginClient client, String url) {
    this.client = client;
    this.url = url;
  }
}