How to Make CI Fast and Cheap with Test Impact Analysis

For teams making frequent changes to a single codebase, CI quickly becomes a bottleneck. It's common for tests on a large codebase to take 30 minutes, or even hours to run. Developers need this feedback as fast as possible to iterate quickly.

Teams typically do one of two things to speed up their CI pipelines:
- Pay for faster machines
- Pay for more machines (parallelism & test splitting)

A powerful, yet lesser-known option is to use Test Impact Analysis.

‍

What is Test Impact Analysis (TIA)?

TIA is an approach for identifying which tests need to be run in response to a set of source code changes.

It speeds up CI pipelines by reducing the number of tests which run on a given PR. For example, if you can determine that an incoming change to the source code only impacts one of your test files, you don't need to re-run your entire suite to validate the change.

This can lead to dramatic speedups, especially for large teams working on a single codebase, since a typical change has a relatively small impact.

More details can be found in this article by Paul Hammant.

How Tach Uses TIA

Tach is a tool that lets you enforce boundaries between modules, as well as define strict interfaces for a given module.
Since it understands the boundaries between your modules along with their dependencies, it can easily figure out the impact of your changes.

When you run tach test:
- Tach uses git to determine which files in your project have changed (relative to main by default)
- These files are mapped onto your configured modules
- Following the module dependency graph, Tach collects all modules impacted by the changes
- Tach analyzes the imports in your test files
- Only affected test files are collected and run (using pytest)

Example: 8x Test Speedup for FastAPI

To see this in action, we'll use FastAPI as an example.

‍
Since FastAPI uses pytest, we can use tach test as a drop-in replacement and immediately see a speed improvement.

‍

1. Install Tach

‍

> pip install tach

‍

2. Set up module boundaries
‍

‍> tach mod

‍
More documentation on how this works is available here.

‍

3. Make a change

For the purposes of this walkthrough, I'm going to make a trivial change in fastapi/background.py
‍

‍

4. Run 'tach test'

‍

> tach test

Given the change we just made, we know that we don't need to re-run all of our tests. It would be more efficient to only re-run the tests which are impacted by a change to the fastapi.background module.
‍

Running tach test takes care of this automatically:
‍

‍

Compared to running the entire test suite, tach test is 8x faster!

‍

Conclusion

Test Impact Analysis is a powerful way to significantly speed up your team's CI pipeline.

Using Tach, you can get these benefits with minimal configuration. Even further, Tach can help preserve clean module boundaries in your application automatically through linting and CI checks.

If there’s a feature you’d like to see in Tach, feel free to create an issue or join the discussion on Discord!