Skip to content

pressure-vessel: Let short-term subprocesses inherit non-CLOEXEC fds

Simon McVittie requested to merge wip/a-million-syscalls into master
  • utils: Tolerate G_SPAWN_ERROR_FAILED from pv_capture_output()

    If we don't use a child setup function, then GLib can go into a more optimized code path involving posix_spawn(), where we don't get detailed error reporting.

  • pressure-vessel: Let short-term subprocesses inherit non-CLOEXEC fds

    To work around older versions of GLib having a potential deadlock in their fd-closing code due to use of non-async-signal-safe functions in the forked child process, the Flatpak-derived code we're using here uses a simple, naive implementation of closing fds: it iterates through every possible fd, up to the rlimit.

    Unfortunately, Wine users sometimes set their fd rlimit very high (typically around a million), because the "esync" mechanism in Proton's Wine needs a lot of fds. The time taken by each individual syscall is not significant, but when we do a million of them, it adds up. This is arguably a misconfiguration - the recommendation is now to have a soft rlimit of 1024 and a hard rlimit of about a million, so that only processes that opt-in to dealing with a million fds have to be aware of them - but a significant number of misconfigured systems probably still exist.

    We don't actually need to close all the fds here: there's no security boundary, so giving the child access to unnecessary fds isn't a privilage gain; the parent process outlives the child, so leaking fds won't result in them never being closed; and in any case we're reasonably careful to set CLOEXEC on all our fds. So let's just not close them.

    In particular, this can save literally a million syscalls per helper subprocess invocation (it's not often I get to say that!), reducing pressure-vessel-wrap launch time by 90% on a system with the high fd limit (about 70 seconds down to 7). With the recommended soft limit of 1024, the speedup is less dramatic, but it still takes off nearly 10% (about 6 down to 5.5).

    Resolves: https://github.com/ValveSoftware/steam-runtime/issues/323


/cc @denittis @rbernon

This deliberately only deals with the low-hanging fruit: subprocesses that we launch O(n) times, for which inheriting more fds is obviously not a problem.

Edited by Simon McVittie

Merge request reports