Introducing paid integration tests

Very often systems that you work on connect with external services. Some of the interactions with those are paid, some are free. Testing the free integrations is straightforward - you write tests and execute them every time you want. The only constraint here can be rate limits set by different providers. When it comes to the paid ones, things are more complicated (unless you have an unlimited budget of course). The simplest solution is to get around without paid ones, but sometimes they are required, especially for critical paths. When your expenses are limited, you cannot freely execute paid tests whenever you want. It must be regulated somehow by a chosen strategy.

Strategies

One of the simplest is to run those tests manually and check if the monthly budget has not been exceeded. And then do it again and again. However, this does not scale up and sounds ridiculous in the era of automation. Let's proceed to the automated strategy. By marking chosen tests, we can designate when we want to execute them. There are several options here:

• CRON build - run test suites every some determined period e.g. one week, ten days, etc.
• the main branch build - execute tests when something is merged into it
• PR build - run test suites when new changes are pushed into a PR build

Depending on the established budget, you need to choose the best options that fit your needs. In our case, the main build combined with CRON every two weeks does the job well. Enough with the theory. Let's do some real-life coding. As I work mainly with Java, I will show you how to tag and filter tests using the most popular testing framework, JUnit 5.

Marking and filtering tests

We have two approaches here, one of them is to use the @Tag annotation. Example test:

@Test
@Tag("AWS_S3") 
void upload_file_with_size_exceeding_1gb_succeeds() { }

Marked tests can be filtered in the Maven Surefire plugin configuration. Sample plugin registration:

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.20.1</version>
    <configuration>
        <groups>AWS_S3</groups>
    </configuration>
</plugin>

When we want to run those suites dynamically for example: in the CI, then we must pass the groups parameter.  For instance: mvn test -Dgroups=AWS_S3. For Gradle users, I recommend the guide for testing: https://docs.gradle.org/current/userguide/java_testing.html

The second option for marking tests is to create and use custom annotations. The key element here will be EnabledIfSystemProperty decoration. Here is the example:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@EnabledIfSystemProperty(named = "ENABLE_PAID_TESTS", matches = "(?i)true")
public @interface Paid {
}

Having the annotation declared we can decorate a test using it:

@Paid
@Test
public void upload_file_with_size_exceeding_1gb_succeeds() { }

In the above example, we used EnabledIfSystemProperty which means that decorated tests will be executed only if the ENABLE_PAID_TESTS variable is set to true. Some other conditions that can help us achieve the expected result are:

• EnabledIf
• EnabledIfSystemProperty
• DisabledIf
• DisabledIfEnvironmentVariable

I prefer the second approach because it's more readable for me, and configurable than the first one. We can also produce a more concise version of this annotation by adding the @Test annotation:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@EnabledIfSystemProperty(named = "enable.paid.tests", matches = "(?i)true")
@Test
public @interface PaidTest {
}

I hope these techniques will help you with introducing paid tests as they helped us.