Testcontainers and Reuse

The backing application for this site is made of nearly fifty modules, about forty of which test against a database. The basic structure for a feature looks like this:

• lib-core - basic services and functionality, like provenance and orchestration support
• lib-testsupport - a library that depends on lib-core that provides, of all things, support for tests
• lib-foo - basic interfaces for an operation related to "foo" services
• operation-foo or service-foo - an operation that is internal orchestration like a command, or something that has capability for accepting information (ingress) or sending information elsewhere (egress)

This creates a build that's pretty well-suited, conceptually, for parallel builds: there are roughly twenty pairs of libraries and attached operations or services, so you can extrapolate that most builds can run roughly a dozen or more build threads at a time.

... except for Docker. Most of those modules work with databases, and I find I prefer to connect to an actual datastore rather than mock services, and Docker was struggling to create instances for each of those modules: it needed to, because each module needed its own schema, and thus Testcontainers ended up starting a fresh container for each one.

Testcontainers is really nice, when it works for you.

Testcontainers' JDBC URL, for example, is one of the nicest features in Java testing for databases. Use jdbc:tc:postgresql:18://test in your test configuration and every test gets a real Postgres instance, started on demand, thrown away afterwards. No fixtures, no shared database server, no "works on my machine," because it's actually using the destination database instead of using a shim or replacement or, well, hopes and dreams.

But in Docker, it can scale poorly under load. The container lives as long as the JVM that started it, and in a multi-module Maven build every module's tests run in their own JVM. A 40-module project starts 40 Postgres containers per build. Run the modules in parallel (with mvnd, or -T) and you start several at once, and sooner or later Docker Desktop, already sharing a VM with a CPU-bound build, fails to get one of them accepting connections in time. The build dies in Flyway with "Unable to obtain connection from database", in a module that didn't even get changed.

It's rather alarming, and frustrating.

The usual answer is to stop paying for parallelism: build with plain mvn, one module at a time. This application's build went from about three minutes to eight that way¹; that's a lot of time poured down the drain. It's worth it when the alternative is failing builds, or skipping tests, but one hopes for an something better.

Reuse is indeed possible

Testcontainers has a reuse feature: mark a container reusable, opt in on your machine, and instead of starting a new one it finds the running one with the same configuration. With the JDBC URL it's one parameter:

jdbc:tc:postgresql:18://test?TC_REUSABLE=true

Now every module, and every build, shares one database... but two things go wrong at once.

First up are migrations. Each module's tests only have that module's migrations on the classpath: the core library has migrations, the blog module inherits the core's migrations and adds some of its own, and the server uses them all from the classpath. Once a module with more migrations than another has run, the next one with fewer fails Flyway's validation: applied migration not resolved locally. You can tell Flyway to ignore that, but you've now got tests running against a schema they don't know about. That's an invalid test, and one that shouldn't be trusted.

The next problem is leftovers. Tests that clean up after themselves are the minority. Of my 200-or-so Spring test classes, 43 truncated their tables between tests; the rest relied, without saying so, on starting each module with an empty database². Share one across modules and builds and they start finding each other's rows: duplicate usernames, counts that are off by the last module's data, the occasional unique-constraint failure that only happens on the third build of the day.

We didn't need or want a shared database. We wanted a database for each module, which gave us the isolation we had before.

A database per JVM, in one container

Postgres will happily hold hundreds of databases, and creating an empty one takes milliseconds. So:

• one reusable container for the whole build (and the next one, and the next one, and the...);
• in it, a fresh database for each test JVM, created when the JVM first needs it;
• dropped when the JVM exits.

Every module still starts from an empty database. Nothing is shared but the server process, which means Docker doesn't gag at the barrage of containers.

In a Spring Boot project the place to do this is an EnvironmentPostProcessor in your shared test library. It sees the jdbc:tc: URL, takes its place with a real one, and nothing else in the test setup has to change³:

public class SharedTestDatabaseEnvironmentPostProcessor 
    implements EnvironmentPostProcessor, Ordered {
    private static final Pattern TC_URL = 
        Pattern.compile("^jdbc:tc:postgresql:([^:/]+)://");

    @Override
    public int getOrder() {
        return Ordered.LOWEST_PRECEDENCE;
    }

    @Override
    public void postProcessEnvironment(
        ConfigurableEnvironment env, 
        SpringApplication app
    ) {
        String url = env.getProperty("spring.datasource.url");
        if (url == null) return;
        Matcher tc = TC_URL.matcher(url);
        if (!tc.find()) return;
        var db = SharedTestDatabase.forThisJvm(tc.group(1));
        env.getPropertySources().addFirst(
            new MapPropertySource("sharedTestDatabase", 
                Map.of(
                    "spring.datasource.url", db.url(),
                    "spring.datasource.username", db.username(),
                    "spring.datasource.password", db.password()
                )
            )
        );
    }
}

Register it in META-INF/spring.factories (in Boot 4 the key is org.springframework.boot.EnvironmentPostProcessor), and the interesting part is creating the database:

public final class SharedTestDatabase {
    public record Database(
        String url, 
        String username, 
        String password
    ) {}

    private static Database database;

    private SharedTestDatabase() {}

    public static synchronized Database forThisJvm(
        String version
    ) {
        if (database == null) {
            database = create(version);
        }
        return database;
    }

    private static Database create(String version) {
        var container = new PostgreSQLContainer(
            DockerImageName.parse("postgres:" + version)
        )
            .withReuse(true)
            .withCommand(
                "postgres", 
                "-c", 
                "max_connections=1000", 
                "-c", 
                "fsync=off"
            );
        // Parallel JVMs would each find 
        // none running and each start one.
        var lockFile = Path.of(
            System.getProperty("java.io.tmpdir"), 
            "test-postgres.lock"
        );
        try (
            var channel = FileChannel.open(
                lockFile, 
                StandardOpenOption.CREATE, 
                StandardOpenOption.WRITE
            );
            var lock = channel.lock()
        ) {
            container.start();
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }

        var name = "test_" + 
            ProcessHandle.current().pid() + 
            "_" + 
            UUID.randomUUID().toString().substring(0, 8);
        try (
            var conn = connect(container); 
            var st = conn.createStatement()
        ) {
            dropStale(st);
            st.execute("CREATE DATABASE \"" + name + "\"");
        } catch (SQLException e) {
            throw new IllegalStateException(
                "Couldn't create the test database", e
            );
        }
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            try (
                var conn = connect(container); 
                var st = conn.createStatement()
            ) {
                st.execute(
                    "DROP DATABASE IF EXISTS \"" + 
                    name + 
                    "\" WITH (FORCE)"
                );
            } catch (SQLException ignored) {
                // The next JVM drops it: 
                // by then its process has gone.
            }
        }));
        return new Database(
            "jdbc:postgresql://" + 
            container.getHost() + 
            ":" + 
            container.getFirstMappedPort() + 
            "/" + 
            name,
            container.getUsername(), 
            container.getPassword()
        );
    }

    private static Connection connect(
        PostgreSQLContainer container
    ) throws SQLException {
        return DriverManager.getConnection(
            container.getJdbcUrl(), 
            container.getUsername(), 
            container.getPassword()
        );
    }

    /**
     * Drops the databases of test JVMs 
     * that died before dropping their own. 
     */
    private static void dropStale(Statement st) throws SQLException {
        var names = new ArrayList<String>();
        try (
            var rs = st.executeQuery(
                "SELECT datname FROM pg_database WHERE datname LIKE 'test\\_%'"
            )
        ) {
            while (rs.next()) {
                names.add(rs.getString(1));
            }
        }
        for (var name : names) {
            if (ownerGone(name)) {
                st.execute(
                    "DROP DATABASE IF EXISTS \"" + name + "\" WITH (FORCE)"
                );
            }
        }
    }

    /** 
     * Whether the JVM a database was made for, 
     * whose pid its name carries, has gone. 
     */
    private static boolean ownerGone(
        String database
    ) {
        var rest = database.substring("test_".length());
        int cut = rest.indexOf('_');
        try {
            long pid = Long.parseLong(
                cut < 0 ? rest : rest.substring(0, cut)
            );
            return ProcessHandle
                .of(pid)
                .map(p -> !p.isAlive())
                .orElse(true);
        } catch (NumberFormatException e) {
            return true;
        }
    }
}

A few details:

• The file lock: on a cold start, with no container running, parallel modules would each look for the reusable container, find none, and each start their own. Holding an OS file lock around start() lets the first create it and the rest find it.
• max_connections: each JVM keeps a connection pool per cached Spring context, and mvnd runs several JVMs at once. Postgres' default of 100 can run out pretty early.
• fsync=off: it's a test database. Durability is the one thing it doesn't need.
• The pid in the name: a JVM killed mid-test never runs its shutdown hook. Put the process id in the database's name, and the next JVM can drop any test_* database whose process no longer exists, without ever touching one that's still in use.
• Ordering: if something else in your startup reads spring.datasource.url (I had a post-processor that loads settings from the database, and skipped jdbc:tc: URLs on purpose), make sure it runs before this one, or it starts behaving differently in tests.

It degrades to what you had

Reuse only happens where it's enabled, in ~/.testcontainers.properties:

testcontainers.reuse.enable=true

(or TESTCONTAINERS_REUSE_ENABLE=true in the environment). Anywhere it isn't, CI included, withReuse(true) is ignored: the container belongs to the JVM and goes with it, exactly as the jdbc:tc: URL behaved. Same tests, same isolation, one container per module. Nobody has to opt in for the build to keep working, although then you pay the price for the flotillae of container instances.

The outcome

This changed two things: first, it bought time. The same full build, executed in three ways:

Build	Time
mvn, a container per module	471 s
mvn, one shared container	435 s
mvnd, one shared container, from cold	172 s

The sequential build barely moves: container start-up was never most of its time. The win is that parallel builds work again, and while three minutes of a build (for example) is still expensive, it's nowhere near as bad as eight and a half minutes.

It also took out another set of problems: a test that waited two seconds for an asynchronous callback can pass for months (ask me how I know.) Under the parallel build it didn't get two seconds of CPU, so the deadlines in use had to be pessimistic, up to thirty seconds; the loop ends as soon as the callback has run, so a healthy build is no slower, and only a starved one waits. If you use to parallel builds, look for those first: any fixed timeout in a test is a bet on how busy the machine is.

But the other win for this single-container, multiple databases wasn't just time: it was reliability. As the prior paragraph shows, I'd been accepting the possibility of a failed build to save five minutes of build time, until the system got large enough that it failed parallel builds most of the time - and since I put in the reusable container, I've had no database-acquisition failures in the builds at all, and I save the build time.