In this post, we will introduce and explain two ways of testing software using Elasticsearch as an external system dependency. We'll cover tests using mocks as well as integration tests, show some practical differences between them, and give some hints on where to go for each style.
Good tests for system confidence
A good test is a test that increases the confidence of every person involved in the process of creating and maintaining an IT system. Tests aren't meant to be cool, fast, or to artificially increase code coverage. Tests play a vital role in ensuring that:
- What we want to deliver is going to work in production.
- The system satisfies the requirements and the contracts.
- There won't be regressions in the future.
- Developers (and other involved team members) are confident that what they have created will work.
Of course, this doesn't mean that tests can't be cool, fast, or increase code coverage. The faster we can run our test suite, the better. It's just that in the pursuit of reducing the overall duration of the testing suite, we should not sacrifice the reliability, maintainability and confidence the automated tests give us.
Good automated tests make the various team members more confident:
- Developers: they get to confirm that what they're doing works (even before the code they work on leaves their machine).
- Quality assurance team: they have less to test manually.
- System operators and SREs: are more relaxed, because the systems are easier to deploy and maintain.
Last but not least: the architecture of a system. We love when systems are organized, easy to maintain, and the architecture is clean and serves its purpose. However, sometimes we might see an architecture which sacrifices too much for the excuse known as "it's more testable this way". There's nothing wrong with being very testable – only when the system is written primarily to be testable instead of serving the needs justifying its existence, we see a situation when the tail wags the dog.
Two kinds of tests
There are many ways the tests can be seen, and thus classified. In this post I'll focus on only one aspect of dividing the tests: using mocks (or stubs, or fakes, or ...) vs. using real dependencies. In our case the dependency is Elasticsearch.
Tests using mocks are very fast because they don't need to start any external dependencies and everything happens only in memory. Mocking in automated testing is when fake objects are used instead of real ones to test parts of a program without using the actual dependencies. This is the reason they're needed and why they shine in any fast-detection-net tests, e.g. validation of input. There's no need to start a database and make a call to it only to verify that negative numbers in a request aren't allowed, for example.
However, introducing mocks has several implications:
- Not everything and every time can be mocked easily, hence mocks have impact on the architecture of the system (which sometimes is great, sometimes not so much).
- Tests running on mocks might be fast, but developing such tests can take quite some time because the mocks deeply reflecting the systems they mimic usually aren't given for free. Someone who knows how the system works needs to write the mocks the proper way, and this knowledge can come from practical experience, studying documentation, and so on.
- Mocks need to be maintained. When your system depends on an external dependency, and you need to upgrade this dependency, someone has to ensure that the mocks mimicking the dependency also get updated with all the changes: breaking, documented and undocumented (which can also have an impact on our system). This becomes especially painful when you want to upgrade a dependency but your (using only mocks) test suite can't give you any confidence that all the tested cases are guaranteed to work.
- It takes discipline to ensure that the effort goes towards developing and testing the system, not the mocks.
For these reasons many people advocate going exactly the opposite direction: never use mocks (or stubs, etc.), but rely solely on real dependencies. This approach works very nicely in demos or when the system is tiny and has only a few test cases generating huge coverage. Such tests can be integration tests (roughly speaking: checking a part of a system against some real dependencies) or end-to-end tests (using all real dependencies at the same time and checking the behavior of the system at all ends, while playing user workflows which define the system as usable and successful). A clear benefit of using this approach is that we also (often unintentionally) verify our assumptions about the dependencies, and how we integrate them against the system we're working on.
However, when tests are using only real dependencies, we need to consider the following aspects:
- Some test scenarios don't need the actual dependency (e.g. to verify the static invariants of a request).
- Such tests usually aren't run in whole suites at developers' machines, because waiting for feedback would take too much time.
- They require more resources at CI machines, and it might take more time to tune things to not waste time & resources.
- It might not be trivial to initialize dependencies with test data.
- Tests with real dependencies are great for cordoning code before major refactoring, migration or dependency upgrade.
- They are more likely to be opaque tests, i.e. not being detailed about the internals of the system under test, but taking care of their results.
The sweet spot: use both
Instead of testing your system with just one kind of test, you can rely on both kinds where it makes sense and try to improve your usage of both of them.
- Run mock-based tests first because they are much faster, and only when all are successful, run slower dependency tests only after.
- Choose mocks for scenarios where external dependencies aren't really needed: when mocking would take too much time the code should be massively altered just for that; rely on external dependencies.
- There is nothing wrong in testing a piece of code using both approaches, as long as it makes sense.
Example of SystemUnderTest
For the next sections we're going to use an example which can be found here. It's a tiny demo application written in Java 21, using Maven as the build tool, relying on Elasticsearch client and using Elasticsearch's latest addition, using ES|QL (Elastic's new procedural query language). If Java is not your programming language, you should still be fine to understand the concepts we're going to discuss below and translate them to your stack. It's just using a real code example makes certain things easier to explain.
The BookSearcher helps us handle search and analyze data, being books in our case (as demonstrated in one of the previous posts).
- It requires Elasticsearch exactly in version
8.15.xas its only dependency (seeisCompatibleWithBackend()), e.g. because we're not sure if our code is forward-compatible, and we're sure it's not backwards compatible. Before upgrading Elasticsearch in production to a newer version, we shall first bump it in the tests to ensure that the behaviour of the System Under Test remains the same. - We can use it to search for the number of books published in a given year (see
numberOfBooksPublishedInYear). - We might also use it when we need to analyze our data set and find out the 20 most published authors between two given years (see
mostPublishedAuthorsInYears).
public class BookSearcher {
private final ElasticsearchClient esClient;
public BookSearcher(ElasticsearchClient esClient) {
this.esClient = esClient;
if (!isCompatibleWithBackend()) {
throw new UnsupportedOperationException("This is not compatible with backend");
}
}
private boolean isCompatibleWithBackend() {
try (ResultSet rs = esClient.esql().query(ResultSetEsqlAdapter.INSTANCE, """
show info
| keep version
| dissect version "%{major}.%{minor}.%{patch}"
| keep major, minor
| limit 1""")) {
if (!rs.next()) {
throw new RuntimeException("No version found");
}
return rs.getInt(1) == 8 && rs.getInt(2) == 15;
} catch (SQLException | IOException e) {
throw new RuntimeException(e);
}
}
public int numberOfBooksPublishedInYear(int year) {
try (ResultSet rs = esClient.esql().query(ResultSetEsqlAdapter.INSTANCE, """
from books
| where year == ?
| stats published = count(*) by year
| limit 1000""", year)) {
if (rs.next()) {
return rs.getInt("published");
}
} catch (SQLException | IOException e) {
throw new RuntimeException(e);
}
return 0;
}
public List<MostPublished> mostPublishedAuthorsInYears(int minYear, int maxYear) {
assert minYear <= maxYear;
String query = """
from books
| where year >= ? and year <= ?
| stats first_published = min(year), last_published = max(year), times = count (*) by author
| eval years_published = last_published - first_published
| sort years_published desc
| drop years_published
| limit 20
""";
try {
Iterable<MostPublished> published = esClient.esql().query(
ObjectsEsqlAdapter.of(MostPublished.class),
query,
minYear,
maxYear);
List<MostPublished> mostPublishedAuthors = new ArrayList<>();
for (MostPublished mostPublished : published) {
mostPublishedAuthors.add(mostPublished);
}
return mostPublishedAuthors;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public record MostPublished(
String author,
@JsonProperty("first_published") int firstPublished,
@JsonProperty("last_published") int lastPublished,
int times
) {
public MostPublished {
assert author != null;
assert firstPublished <= lastPublished;
assert times > 0;
}
}
}Test with mocks to start with
For creating the mocks used in our tests we're going to use Mockito, a very popular mocking library in Java ecosystem.
We might begin with the following, to have mocks reset before each test:
public class BookSearcherMockingTest {
ResultSet mockResultSet;
ElasticsearchClient esClient;
ElasticsearchEsqlClient esql;
@BeforeEach
void setUpMocks() {
mockResultSet = mock(ResultSet.class);
esClient = mock(ElasticsearchClient.class);
esql = mock(ElasticsearchEsqlClient.class);
}
}As we said before, not everything can be easily tested using mocks. But some things we can (and probably even should). Let's try verifying that the only supported version of Elasticsearch is 8.15.x for now (in the future we might extend the range once we confirm our system is compatible with future versions):
@Test
void canCreateSearcherWithES_8_15() throws SQLException, IOException{
// when
when(esClient.esql()).thenReturn(esql);
when(esql.query(eq(ResultSetEsqlAdapter.INSTANCE), anyString())).thenReturn(mockResultSet);
when(mockResultSet.next()).thenReturn(true).thenReturn(false);
when(mockResultSet.getInt(1)).thenReturn(8);
when(mockResultSet.getInt(2)).thenReturn(15);
// then
Assertions.assertDoesNotThrow(() -> new BookSearcher(esClient));
}We can verify similarly (simply by returning a different minor version), that our BookSearcher is not going to work with 8.16.x yet, because we're not sure if it's going to be compatible with it:
@Test
void cannotCreateSearcherWithoutES_8_15() throws SQLException, IOException {
// when
when(esClient.esql()).thenReturn(esql);
when(esql.query(eq(ResultSetEsqlAdapter.INSTANCE), anyString())).thenReturn(mockResultSet);
when(mockResultSet.next()).thenReturn(true).thenReturn(false);
when(mockResultSet.getInt(1)).thenReturn(8);
when(mockResultSet.getInt(2)).thenReturn(16);
// then
Assertions.assertThrows(UnsupportedOperationException.class, () -> new BookSearcher(esClient));
}Now let's see how we can achieve something similar when testing against a real Elasticsearch. For this we're going to use Testcontainers' Elasticsearch Module, which has only one requirement: it needs access to Docker, because it runs Docker containers for you. From a certain angle Testcontainers is simply a way of operating Docker containers, but instead of doing that in your Docker Desktop (or similar), in your CLI, or scripts, you can express your needs in the programming language you know. This makes fetching images, starting containers, garbage-collecting them after tests, copying files back and forth, executing commands, examining logs, etc. possible directly from your test code.
The stub might look like this:
@Testcontainers
public class BookSearcherIntTest {
static final String ELASTICSEARCH_IMAGE = "docker.elastic.co/elasticsearch/elasticsearch:8.15.0";
static final JacksonJsonpMapper JSONP_MAPPER = new JacksonJsonpMapper();
RestClientTransport transport;
ElasticsearchClient client;
@Container
ElasticsearchContainer elasticsearch = new ElasticsearchContainer(ELASTICSEARCH_IMAGE);
@BeforeEach
void setupClient() {
transport = // setup transport here
client = new ElasticsearchClient(transport);
}
@AfterEach
void closeClient() throws IOException {
if (transport != null) {
transport.close();
}
}
}In this example we rely on Testcontainers' JUnit integration with @Testcontainers and @Container, meaning we don't need to worry about starting Elasticsearch before our tests and stopping it after. The only thing we need to do is to create the client before each test and close it after each test (to avoid resource leaks, which could impact bigger test suites).
Annotating a non-static field with @Container means, that a new container will be started for each test, hence we don't have to worry about stale data or resetting the container's state. However, with many tests, this approach might not perform well, so we're going to compare it with alternatives in one of the next posts.
Note:
By relying on
docker.elastic.co(Elastic's official Docker image repository), you avoid exhausting your limits on Docker hub.
It is also recommended to use the same version of your dependency in your tests and production environment, to ensure maximum compatibility. We also recommend being precise with selecting the version, for this reason, there is nolatesttag for Elasticsearch images.
Connecting to Elasticsearch in tests
Elasticsearch Java client is capable of connecting to Elasticsearch running in a test container even with security and SSL/TLS enabled (which are default for versions 8.x, that's why we didn't have to specify anything related to security in the container declaration.) Assuming the Elasticsearch you're using in production has also TLS and some security enabled, it is recommended to go for the integration test setup as close to production scenario as possible, and therefore not disabling them in tests.
How to obtain data necessary for connection, assuming the container is assigned to field or variable elasticsearch:
elasticsearch.getHost()will give you the host on which the container is running (which most of the time will be probably"localhost", but please don't hardcode this as sometimes, depending on your setup, it might be another name, therefore the host should always be obtained dynamically).elasticsearch.getMappedPort(9200)will give the host port you have to use to connect to Elasticsearch running inside the container (because every time you start the container, the outside port is different, so this has to be a dynamic call as well).- Unless they were overwritten, the default username and password are
"elastic"and"changeme"respectively. - If there was no SSL/TLS certificate specified during the container setup, and the secured connectivity is not disabled (which is the default behaviour from versions 8.x), a self-signed certificate is generated. To trust it (e.g. like cURL can do) the certificate can be obtained using
elasticsearch.caCertAsBytes()(which returnsOptional<byte[]>), or another convenient way is to getSSLContextusingcreateSslContextFromCa().
The overall result might look like this:
BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("elastic", "changeme"));
// Create a low level rest client
RestClient restClient = RestClient.builder(new HttpHost(elasticsearch.getHost(), elasticsearch.getMappedPort(9200), "https"))
.setHttpClientConfigCallback(httpClientBuilder ->
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
.setSSLContext(elasticsearch.createSslContextFromCa())
)
.build();
// The RestClientTransport is mainly for serialization/deserialization
RestClientTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
// The official Java API Client for Elasticsearch
ElasticsearchClient client = new ElasticsearchClient(transport);Another example of creating an instance of ElasticsearchClient can be found in the demo project.
Note:
For creating client in production environments please refer to the documentation.
First integration test
Our very first test, verifying that we can create BookSearcher using Elasticsearch version 8.15.x, might look like this:
@Test
void canCreateClientWithContainerRunning_8_15() {
Assertions.assertDoesNotThrow(() -> new BookSearcher(client));
}As you can see, we don't need to set up anything else. We don't need to mock the version returned by Elasticsearch, the only thing we need to do is to provide BookSearcher with a client connected to a real instance of Elasticsearch, which has been started for us by Testcontainers.
Integration tests care less about the internals
Let's do a little experiment: let's assume that we have to stop extracting data from the result set using column indices, but have to rely on column names. So in the method isCompatibleWithBackend instead of
return rs.getInt(1) == 8 && rs.getInt(2) == 15;we are going to have:
return rs.getInt("major") == 8 && rs.getInt("minor") == 15;When we re-run both tests we'll notice, that the integration test with real Elasticsearch still passes without any issues.
However, the tests using mocks stopped working, because we mocked calls like rs.getInt(int), not rs.getInt(String). To have them passing, we now have to either mock them instead, or mock them both, depending on other use cases we have in our test suite.
Integration tests can be a cannon to kill a fly
Integration tests are capable of verifying the behaviour of the system, even if external dependencies aren't needed. However, using them this way is usually a waste of execution time and resources. Let's look at the method mostPublishedAuthorsInYears(int minYear, int maxYear). The first two lines are as follows:
assert minYear <= maxYear;
String query = // here goes the queryThe first statement is checking a condition, which doesn't depend on Elasticsearch (or any other external dependency) in any way. Therefore, we don't need to start any containers to merely verify, that if the minYear is greater than maxYear, an exception is thrown.
A simple mocking test, which is also fast and not resource-heavy is more than enough to ensure that. After setting up the mocks, we can simply go for:
BookSearcher systemUnderTest = new BookSearcher(esClient);
Assertions.assertThrows(
AssertionError.class,
() -> systemUnderTest.mostPublishedAuthorsInYears(2012, 2000)
);Starting a dependency, instead of mocking, would be wasteful in this test case because there's no chance of making a meaningful call for this dependency.
However, to verify the behaviour starting with String query = ..., that the query is written correctly, gives results as expected: the client library is capable of sending proper requests and responses, there are no syntax changes and so it's way easier to use an integration test, e.g.:
@BeforeEach
void setupDataInContainer() {
// here we initialise data in the Elasticsearch running in a container
}
@Test
void shouldGiveMostPublishedAuthorsInGivenYears() {
var systemUnderTest = new BookSearcher(client);
var list = systemUnderTest.mostPublishedAuthorsInYears(1800, 2010);
Assertions.assertEquals("Beatrix Potter", list.get(12).author(), "Beatrix Potter was 13th most published author between 1800 and 2010");
}This way, we can rest assured that when we feed our data to Elasticsearch (in this or any future version we choose to migrate to), our query is going to give us exactly what we expected: the data format didn't change, the query is still valid, and all the middleware (clients, drivers, security, etc.) will to continue to work. We don't have to worry about keeping the mocks up to date, the only change needed to ensure compatibility with e.g. 8.15 would be changing this:
static final String ELASTICSEARCH_IMAGE = "docker.elastic.co/elasticsearch/elasticsearch:8.15.0";The same happens if you decide to e.g. use good old QueryDSL instead of ES|QL: the results you receive from the query (regardless of the language) should still be the same.
Use both approaches when needed
The case of the method mostPublishedAuthorsInYears illustrates that a single method can be tested using both methods. And perhaps even should be.
- Using only mocks means we have to maintain the mock and have zero confidence when upgrading our system.
- Using only integration tests would mean that we're wasting quite a lot of resources, without needing them at all.
Let's recap
- Using both mocking and integration tests with Elasticsearch is possible.
- Use mocking tests as fast-detection-net and only if they pass successfully, start tests with dependencies (e.g. using
./mvnw test '-Dtest=!TestInt*' && ./mvnw test '-Dtest=TestInt*'or Failsafe and Surefire plugins). - Use mocks when testing the behaviour of your system ("lines of code") where integration with external dependencies doesn't really matter (or could even be skipped).
- Use integration tests to verify your assumptions about and integration with external systems.
- Don't be afraid to test using both approaches – if it makes sense – according to the points above.
One could make an observation, that being so strict about the version (in our case 8.15.x) is too much. Using just the version tag alone could be, but please be aware that in this post it serves as the representation of all other features that might change between the versions.
In the next installment, we'll look at ways of initialising Elasticsearch running in a test container, with test data sets. Let us know if you built anything based on this blog or if you have questions on our Discuss forums and the community Slack channel.
