Hang Test Suite
This is a collection of Linux programs to trigger GPU hangs in various ways (primarily targeting AMD GPUs) to test the driver stack's ability to recover from such hangs.
Building and installing
glslangValidator
is required as a build-time dependency. It is included in the Vulkan SDK.
Use
git clone --recursive https://gitlab.steamos.cloud/holo/HangTestSuite.git
or
git submodule update --init
after cloning. Then, use Meson to compile:
meson setup build
ninja -C build
# optionally, to directly install to /usr:
cd build
meson install
The install directory can be changed by changing the setup command to meson setup -Dprefix=<install_dir> build
.
Usage
The test suite can be run either by installing using meson install
, or directly from the build directory.
Launching the runner executable hangtest
without any arguments will start a test run that launches each test case
once.
The test suite will always run on the first reported Vulkan physical device. To select a different physical device,
use
the MESA_VK_DEVICE_SELECT
environment variable.
For debugging convenience, the executables triggering the different hang conditions can also be run standalone. Each test corresponds to one executable with the same name.
Note: Running the test suite from inside a graphical desktop is dangerous. If resets don't work well, the session might get killed, which in turn also kills the test suite. If possible, run the test suite from a tty or an ssh session.
Options
--stress
To run tests in the stress-test mode, use the --stress
option. Stress-test mode will try to hang the GPU in a loop,
while also submitting non-hanging GPU work in parallel. Stress-test runs do not terminate on their own unless there is
a failure; to end a stress test run, interrupt the process using Ctrl+C
on the command line.
--test-filter
To limit the test run to specific tests, use --test-filter
. Only tests whose names contain the filter string will be
executed.
Example: --test-filter soft_recovery
will execute the tests soft_recovery_loop
and soft_recovery_pagefault
,
whereas --test-filter loop
will only execute soft_recovery_loop
.
--list-tests
Use --list-tests
to output the names of all available tests.
--test-dir
and --dummy
The test suite assumes that all test executables as well as the dummy executable are located in the same directory
as the runner executable (hangtest
).
Architecture
The test suite consists of three main components: The runner, the tests, and the dummy.
The runner is responsible for launching tests/dummy processes, communicating with them, and aggregating test results.
Each test simulates one failure case causing a GPU to hang (for example an infinite loop in a shader). After submitting
the hanging work, the test checks if subsequent commands correctly return VK_ERROR_DEVICE_LOST
to signal the GPU hang.
The dummy process's purpose is checking if the GPU recovery caused device loss in unrelated contexts. The dummy process creates a Vulkan device before the GPU hangs, and after recovery has completed, the dummy process attempts to submit work to that device. It is expected that this succeeds for a successful hang recovery.
Both the test and the dummy process communicate test results to the runner via exit codes. An exit code of 0 means success, anything else means failure.
In order for a test to be considered successful, both the test and the dummy executable need to exit with code 0.
Adding tests
Tests should always be named with a prefix of either soft_recovery_
or hard_reset_
, depending on what type of reset
they aim to trigger.
To add a new test,
- write a snippet of Vulkan code triggering the hang and save it as
<testname>.cpp
insrc/tests
- add
'testname'
to thetests
array insrc/meson.build
- add
"testname"
to theTEST_CASES
array insrc/runner.cpp
After submitting work that triggers the hang, the test should check whether subsequent commands like vkWaitForFences
or the next vkQueueSubmit
return VK_ERROR_DEVICE_LOST
. If device loss is reported properly, the test should exit
with return code 0
, otherwise with a nonzero code.