Skip to content
Snippets Groups Projects
Konstantin Seurer's avatar
Konstantin Seurer authored
Improves clang-format(ting) of initializer lists.
ac06b09a
History

Hang Test Suite

This is a collection of Linux programs to trigger GPU hangs in various ways (primarily targeting AMD GPUs) to test the driver stack's ability to recover from such hangs.

Building and installing

glslangValidator is required as a build-time dependency. It is included in the Vulkan SDK.

Use

git clone --recursive https://gitlab.steamos.cloud/holo/HangTestSuite.git

or

git submodule update --init

after cloning. Then, use Meson to compile:

meson setup build
ninja -C build 
# optionally, to directly install to /usr:
cd build
meson install

The install directory can be changed by changing the setup command to meson setup -Dprefix=<install_dir> build.

Usage

The test suite can be run either by installing using meson install, or directly from the build directory.

Launching the runner executable hangtest without any arguments will start a test run that launches each test case once.

The test suite will always run on the first reported Vulkan physical device. To select a different physical device, use the MESA_VK_DEVICE_SELECT environment variable.

For debugging convenience, the executables triggering the different hang conditions can also be run standalone. Each test corresponds to one executable with the same name.

Note: Running the test suite from inside a graphical desktop is dangerous. If resets don't work well, the session might get killed, which in turn also kills the test suite. If possible, run the test suite from a tty or an ssh session.

Options

--stress

To run tests in the stress-test mode, use the --stress option. Stress-test mode will try to hang the GPU in a loop, while also submitting non-hanging GPU work in parallel. Stress-test runs do not terminate on their own unless there is a failure; to end a stress test run, interrupt the process using Ctrl+C on the command line.

--test-filter

To limit the test run to specific tests, use --test-filter. Only tests whose names contain the filter string will be executed.

Example: --test-filter soft_recovery will execute the tests soft_recovery_loop and soft_recovery_pagefault, whereas --test-filter loop will only execute soft_recovery_loop.

--list-tests

Use --list-tests to output the names of all available tests.

--test-dir and --dummy

The test suite assumes that all test executables as well as the dummy executable are located in the same directory as the runner executable (hangtest).

Architecture

The test suite consists of three main components: The runner, the tests, and the dummy.
The runner is responsible for launching tests/dummy processes, communicating with them, and aggregating test results.

Each test simulates one failure case causing a GPU to hang (for example an infinite loop in a shader). After submitting the hanging work, the test checks if subsequent commands correctly return VK_ERROR_DEVICE_LOST to signal the GPU hang.

The dummy process's purpose is checking if the GPU recovery caused device loss in unrelated contexts. The dummy process creates a Vulkan device before the GPU hangs, and after recovery has completed, the dummy process attempts to submit work to that device. It is expected that this succeeds for a successful hang recovery.

Both the test and the dummy process communicate test results to the runner via exit codes. An exit code of 0 means success, anything else means failure.

In order for a test to be considered successful, both the test and the dummy executable need to exit with code 0.

Adding tests

Tests should always be named with a prefix of either soft_recovery_ or hard_reset_, depending on what type of reset they aim to trigger.

To add a new test,

  • write a snippet of Vulkan code triggering the hang and save it as <testname>.cpp in src/tests
  • add 'testname' to the tests array in src/meson.build
  • add "testname" to the TEST_CASES array in src/runner.cpp

After submitting work that triggers the hang, the test should check whether subsequent commands like vkWaitForFences or the next vkQueueSubmit return VK_ERROR_DEVICE_LOST. If device loss is reported properly, the test should exit with return code 0, otherwise with a nonzero code.