smallest dog

Test Driven Development for a Kernel

One of the most challenging, but also most rewarding, programming challenges I’ve taken on is kernel development. However, I’m often frustrated with the lack of good tooling for it.

Given this problem, I did what any programmer with too much time on their hands would do: I tried to solve a problem most people don’t care about.

Table of Contents

1. Geting a consistent build environment
2. Tooling
3. Automatically shutting down test builds
4. Reporting test failures
5. Running on a CI server
6. Support future work

Getting a consistent build environment

The first step towards automated testing is having reproducible builds.

To this end, I used Docker to make a consistent build environment and wrote a handy script to simplify using it.

This was very much biased towards awooOS, but it was a good start.

Tooling

All of the test infrastructure I’ve seen for kernel testing is highly specialized for the kernel in question, because the typical frameworks used for userland software don’t work well in a kernel.

I don’t expect my framework to magically stop this from happening, but I only want to do it once.

As such, I wanted the testing framework to be generic enough to be used throughout the entire kernel, including supporting services and libraries.

I settled on a mixture of unit testing and runtime assertions, using a custom framework. (Which is contained in one C file and one header file.)

Here’s an example unit test for badmalloc, a basic memory allocator with no way to free memory:

// Tests that badmalloc() zeros memory before handing it off.
TestResult *test_badmalloc_initializes_buffer()
{
    // Specify the number of bytes of RAM to allocate.
    size_t length = 100;

    // Allocate the buffer.
    uint8_t *buffer = badmalloc(length);

    // Check that every byte of the buffer is set to zero.
    for (size_t i = 0; i < length; i++) {
        if (buffer[i] != 0) {
            // If something isn't zero, the test fails.
            TEST_RETURN(TEST_FAILURE, "badmalloc() did not initialize entire buffer to null bytes!");
        }
    }

    // If we get here, everything was zeroed out and the test passes.
    TEST_RETURN(TEST_SUCCESS, "badmalloc() initializes entire buffer to null bytes.");
}

And here’s an example assertion, which verifies the bootloader is multiboot compliant:

// Assertions for the Hardware Abstraction Layer.
TestResult *test_hal_assertions()
{
    // Setup for tests with assertions.
    TEST_HAS_ASSERTIONS();

    // Assert that the magic number passed to the kernel is the one
    // provided by multiboot-compatible bootloaders.
    // If the assertion fails, it calls `TEST_RETURN(TEST_FAILURE, ...)`
    TEST_ASSERT(hal_get_magic() == MULTIBOOT_MAGIC);

    // The assertions all passed. Do some miscellaneous cleanup.
    TEST_ASSERTIONS_RETURN();
}

There’s a bit of code that handles setting up the tests, and you have to manually add each one. But it’s a significant improvement over other kernel testing frameworks I’ve worked with or created in the past.

Automatically shutting down test builds

One of the big barriers to doing proper Continuous Integration (CI) with a kernel is having the system shut down automatically after tests are run, but only when being tested.

To resolve this, I made my existing script that generated build information generate macros that let you do:

/*
 * AWOO_BUILD_TYPE_NUMBER is a number representing the current build
 * type.
 *
 * AWOO_TEST_BUILD is what that value will be when it's a test build.
 */
if (AWOO_BUILD_TYPE_NUMBER == AWOO_TEST_BUILD) {
    // If it's a test build, shut down immediately.
    hal_hard_shutdown();
}

Reporting test failures

Alright, so at this point the operating system prints information about test failures. But in order to automate it, we need to pass that information to the outside world.

At first this one had me pretty stumped — how could I possibly pass information from a virtual machine to the host system without writing some kind of driver?!

Turns out, if you use QEMU, you can attach a debug device to a virtual machine which lets you specify a nonzero exit code.

So, if any tests fail and it’s a test build, it uses that debug device to exit with a nonzero status code. If all of the tests pass, it does a clean(er) shutdown that results in a zero status code.

With this, we have the ability to do ./bin/make clean test and have it:

1. build a test ISO,
2. run the ISO in a headless QEMU VM,
3. print test information to the serial port (the terminal), and
4. exit with a status code indicating whether tests passed.

Now we just need to automate it!

Running on a CI server

Since I had a Docker setup already in place that could do everything I needed, all I needed was a CI server that could run a docker image.

Travis CI has support for Docker, so this was actually rather simple.

To speed up the build process, I put the image on Docker Hub.

And… that’s it, really. The rest was just minor tweaks.

Results

With this setup, I can do proper test-driven development.

Share this page on Twitter