Code Coverage for your Golang System Tests
To better understand which parts of our Golang applications are covered by test suites, we wanted to have a coverage report that aggregates data from all of our automated tests. However, we couldn't find any information on how to generate coverage reports from system tests of Golang applications. We found a solution using the Golang toolchain and this is a guide of how we did it. The screenshot below from Codecov.io shows an example test coverage report from our Packetbeat product.
Having a code coverage report for unit tests is common practice and very well supported by the Go toolchain. The challenge with services which highly depend on the environment, like the operating system or other services, is that unit tests can only cover certain cases. Sometimes writing a simple test that runs the binary and compares the output with the expected output is much easier and faster. In addition, it reproduces what is happening when the binary is executed. Unfortunately in most cases these system tests do not count to the coverage report which also makes it hard to identify which parts of the code were actually run by executing the binary.
Our Code Coverage Challenge
At Elastic, we develop the Beats products using Golang. Packetbeat is an open source project written in Golang that is designed to provide real‑time analytics for web, database, and other network protocols. During the implementation of Packetbeat, we faced the challenge that the most valuable tests were the system tests that executed the binary and checked the expected output. These tests allow us to generate pcap files, feed it through Packetbeat and see if the outcome is as expected. Like this we are able to get files from users with problems and can fully reproduce the problem. In case we fixed a bug, we didn’t have any insights if the specific test covered the code changes except through introducing lots of log messages. Based on this we looked for a simpler solution that allowed us to identify which lines were executed by system tests.
Our Test Coverage Reports
During the implementation of the test code coverage for the Packetbeat project we faced different challenges which we managed to overcome. Now all our Beats automatically generate a coverage report for unit and system tests. These coverage reports are generated by our continuous integration system for each pull request and are published. An excerpt of one of these reports can be seen below. The report does not only show us which lines were executed and which not, but also the number of times each line was executed. This can be seen by the small number 34 in the screenshot, which means the line above was executed 34 times. The red lines were not executed by the unit or system tests.
System Test Coverage Guide
The goal of this blog post is to have a guide on how to implement a coverage report for system tests in your own Golang project. If you just want to see it working check out our Packetbeat and Filebeat repositories. Inside the Makefile the command full-coverage generates all the reports and html outputs. To try it out, clone one of the repositories and run the command make full-coverage inside the repository.
The paragraphs below give a step-by-step description how you can implement a coverage report for system tests in your own Golang project.
Step 1: Generating the Binary
To run system tests, a compiled binary of the application is required. This binary is then executed in different environments with different configurations. Golang offers a unique way to generate a coverage binary instead of the default binary generated by go build. The binary for the code coverage which is generated writes a unique counter after every line of code and checks how many times this counter was called after the binary was executed. More technical details on how this works can be found on the go cover documentation.
When go test is executed, this coverage binary is automatically generated again and disposed of afterwards. Golang also allows to generate this coverage binary with the following command:
go test -c -covermode=count -coverpkg ./...
The file generated is automatically called packetname.test. To specify a different value, the param -o followed by the binary name can be used. The -c flag is used to generate the test binary and -covermode=count./... at the end of the command makes sure the coverage binary is generated for all sub packages under the same path, but not imported packages. If you only want to have the coverage for specific packages, you can list them here separated with a comma. For more details run go test -help.
Step 2: Create Main Test File
Now that we know how to generate the binary, we have to make sure the binary will execute as expected. There are some requirements to your code base for the binary to work as expected. The first one is that there is at least one *_test.go file in the package. Otherwise the binary is not generated. I recommend to create the file main_test.go or a testfile with the same filename where your func main() {} is in.
In our case main_test.go file has the following content:
package main
// This file is mandatory as otherwise the filebeat.test binary is not generated correctly.
import (
"testing"
"flag"
)
var systemTest *bool
func init() {
systemTest = flag.Bool("systemTest", false, "Set to true when running system tests")
}
// Test started when the test binary is started. Only calls main.
func TestSystem(t *testing.T) {
if *systemTest {
main()
}
}
The file defines a systemTest flag and contains a single test case which calls the main function. Running the test binary starts to execute the tests. In our case this means TestSystem is called as this is the only test. Running TestSystem means calling main() which starts the application as the normal binary would. This means running the test binary is identical to running the normal binary except that the lines executed are tracked. To prevent this test from running when the unit tests are running, the command line flag systemTest was added. In case it is not set, the main() function will not be called. To run the system tests, the flag must be set by appending -systemTest during the execution of the test binary.
Step 3: Global Command Line Flags
In order to define different test environments for the system tests, using the command line flags is often quite important. Before the implementation of the system test coverage we used FlagSet to keep flags as local as possible. Unfortunately with the coverage binary these didn't work as expected. For the test binary, all flags must be known before the execution of the tests. If a flag is defined during the execution of a test, the test binary will complain during startup that the flag is not defined. An example is shown below:
$ ./packetbeat.test -undefinedFlag flag provided but not defined: -undefinedFlag
To make the command line flags work with the test binary, make sure that the required flags are defined as part of the main FlagSet and are defined outside methods and functions. It can be either done directly during the assignment of variables or as part of the init() functions which are called before starting the test. Here an example from Filebeat:
var configDirPath string
// Init config path flag
func init() {
flag.StringVar(&configDirPath, "configDir", "", "path to additional filebeat configuration directory with .yml files")
}
To ensure your flags work correctly be sure to follow the rule above. Also make sure that your application flags do not conflict with the golang test flags. Verify that all your flags are part of the test binary by running testbinary -h. A list similar to the example below will be printed out. This list will include the golang test flags.
$ ./packetbeat.test -h
Usage of ./packetbeat.test:
-I string
file
-N Disable actual publishing for testing
-O Read packets one at a time (press Enter)
-c string
Configuration file (default "/etc/beat/beat.yml")
-cpuprofile string
Write cpu profile to fil
...
Step 4: Test your Binary
To see if your binary works as expected, you can execute it manually and generate coverage data. For Packetbeat, it looks like this:
packetbeat.test -systemTest -c config.yml -test.coverprofile coverage.cov
The -systemTest flag is used to start the main test as was discussed before. The -c config.yml flag is a Packetbeat specific flag which is required. Make sure to put all application specific flags before the test flags. The flag -test.coverprofile coverage.cov is a standard flag of the test binary and defines to which file the coverage output should be written. After running the binary with the command above, a file coverage.cov is generated in the same directory where the binary was executed.
If no coverage.cov was generated then check if the binary actually started or if there were any errors in the output. The content generated in the .cov files should look similar to:
mode: count github.com/elastic/packetbeat/sniffer/pfring_stub.go:27.43,29.2 1 0 github.com/elastic/packetbeat/sniffer/pfring_stub.go:31.32,32.2 0 0 github.com/elastic/packetbeat/packetbeat.go:72.13,82.2 1 1 github.com/elastic/packetbeat/packetbeat.go:84.47,87.2 1 1 ...
Step 5: Setup System Tests
Additional code is needed to test the binary with various inputs and validate the outputs. In our case we created a small system test framework in Python. Any other language that can execute binaries and that fits you could be used for this.
In the case of Packetbeat and Filebeat, our test framework generates a new directory for each test where the log output of the test binary is recorded, where the input and output of the test binary are copied, and where the coverage report is stored. This makes it really simple to debug a failing test as all the information exists. The code of our test framework and the tests can be found here.
To get started with your system test framework, the first tests can also be executed manually and the outputs can be verified. Based on what you learn from the manual testing it can be better evaluated what the requirements are for the system test framework. For Packetbeat the system test framework was already in place before the system test coverage implementation as these tests were also run without the coverage. To add the coverage test binary only a change of the command line flags was necessary. If you already have a system test framework in place, it can probably be adapted to implement the coverage. If not, find the best way for you and your team to implement these executions.
Step 6: Collect and Convert Coverage
After running the test binary for all system tests, many different coverage.cov files exist. For the overall coverage report all the coverage.cov files from the system tests must be collected and merged into one file. This can be done with the following commands:
mkdir coverage echo 'mode: count' > ./coverage/system.cov tail -q -n +2 ./coverage/*.cov >> ./coverage/system.cov
The coverage reports are stored in the coverage directory which is created first. A file system.cov is generated which will contain all coverage reports. On top of each coverage file, the mode must be listed. In all our cases we use mode:count. As all reports also have this line on top, all the content of the .cov files is aggregated ignoring the first line. In our case we use the command tail -q -n +2 for this, which reads the full file starting at the second line.
To make this file more human readable, Golang provides the go cover tool which can convert the output to an HTML file. To generate the HTML file, run the following command:
go tool cover -html=./coverage/system.cov -o coverage/system.html
This generates the full coverage report in the file system.html and makes it available to view in the browser.
Conclusion
In our case we now have a framework that runs our system tests on Windows, Linux and OS X and directly reports the coverage. The test binary created above can be executed in all cases where the normal binary is used, and it also generates a coverage report. It can even be used to run an application with some test traffic for a few minutes and then check what parts of the applications were executed.
Microservice Environment Coverage Reports
As Golang is a popular language for Microservices, having coverage reports for binaries offers an additional opportunity. Microservices are a suite of independently deployable services. To test if multiple Microservices work together as intended, a full environment with multple Microservices has to be started. This environment is then tested by sending requests to the services and monitoring the output. Then the output is compared with the expected outcome. Unfortunately the only indication on which parts of each Microservice were executed during these tests are logging and monitoring information. No detailed insight into which lines of code were executed exists. With the coverage test binary it is possible to gain these insights.
Instead of testing one binary as we described in the guide above, during the Microservice environment tests multiple binaries are executed. With the coverage test binary, it is possible to have a coverage report for each Microservice running in the test environment. The only change needed to generate the report is to create a coverage test binary for each Microservice. This binary is then run inside the test environment instead of the standard binary. After the tests the coverage reports from each Microservice must be collected and processed as described in Step 6. This method allows to gain detailed insights into which services were run and which lines of code were executed during the tests. Based on this information each Microservice can be improved
Multiple Coverage Files
In our environment we run unit, integration, and system tests which all report coverage. For each test type we generate a separate coverage and html file. To also have a report which combines all these individual reports, we run the same command to aggregate the coverage files and generate a full.cov coverage file. This gives us a complete overview of which lines of code are covered by our different test suites. If you run the coverage report for larger applications or for longer times, be aware that the coverage reports can get quite large.
Next Steps
We use the same test framework for all of our Beats projects. Next we plan to refactor and extract the testing framework part so improvements from one project don't have to be copied to all the other projects. This could make it possible that the framework can also be used for other Golang projects to provide system test coverage and make the setup simpler. Contributions here are more than welcome.
If you have questions, contact us in the Elastic forum under the Beats category.