Login
You're viewing the social.mochi.academy public feed.
  • Apr 9, 2026, 10:23 AM

    So, systemd's NOTIFY_SOCKET readiness notification mechanism has a sort of authentication mechanism by pid: it doesn't like it when the notification is sent from another process than the main one, unless you add the NotifyAccess=all directive. Okay, fair.

    Except that even with this directive, it still can fail for some reason:

    if an auxiliary process of the unit sends an sd_notify() message and immediately exits, the service manager might not be able to properly attribute the message to the unit, and thus will ignore it, even if NotifyAccess=all is set for it.

    Huh? Even if the supervisor wants to check that the MAINPID= given in the message is the relevant unit, it shouldn't need the notifier process to still be around, but, let's ignore that for now. So, it has a barrier mechanism to tell the notifier to stick around until the supervisor has acknowledged it. What is that barrier mechanism? The notifier passes an fd to the supervisor via SCM_RIGHTS in the notification message, and when the supervisor closes that fd, it means it has acked the notification, and the notifier can now exit.

    That's right, folks: it's a notification that the notification has been received!

    There are two mistakes here: 1. the supervisor doesn't need the notifier process to be alive when processing the notification, and 2. NOTIFY_SOCKET being a socket means that any process can connect to it so the supervisor needs to authenticate the relevant unit.

    And that is how a bad design decision cascades into a maze of complexity and inefficiency, and a pain to implement. Unless you're willing to use libsystemd to get access to sd_pid_notify_barrier(), good luck getting this to work right.

    (systemd people will say this is conspiratorial, but if they wanted to force users to link against libsystemd, they wouldn't behave differently. But I really don't care if it's intentional or not. If it's not intentional, it's just terrible engineering.)

    What would be a simpler way to handle all this? Have the supervisor run the unit with a pipe from the daemon to the supervisor, not a socket anyone can connect to. Processes that can write to the pipe are the ones that can inherit it, which means they're necessarily part of the same unit. No need for authentication via pid. When the supervisor receives a certain token on the pipe (e.g. a newline), it means that the daemon is ready. In the daemon or any process notifying on its behalf, that spells: write(fd, "\n", 1); and that's it.

    In other words: the s6 readiness notification protocol, which is exactly what is described in the above paragraph, is as efficient as systemd's synchronization protocol for its notification protocol.

    The real tragedy is that when you go through systemd with a looking glass, every part of it is this way. It still works, and nobody notices the inefficiencies because modern machines are blazingly fast, but it is. so. bad.

    (Edit: typos)

    💬 1🔄 4⭐ 4

Replies

  • 💬 1🔄 0⭐ 0
  • Apr 9, 2026, 10:42 AM

    @lanodan I assume OpenRC doesn't authenticate the pid that connects to the notification socket, which isn't a problem in practice. If OpenRC wanted to fully harden against evil services notifying readiness for other services, its supervisor would need to add a whole layer of complexity for that.

    But even then, I don't think it would need a synchronization mechanism, because who cares if the notifier is still around, as long as the main pid is?

    s6's new s6-notify-fd-from-socket binary, doing the systemd-to-s6 protocol conversion, mitigates the problem by only using autobind with abstract sockets, so any foreign would-be notifier would need to guess the abstract socket name.

    💬 0🔄 0⭐ 1
  • Apr 9, 2026, 4:08 PM
    @ska @lanodan

    we don't auth it at all, and we open one socket per service

    one socket per-service means we don't need to use cgroups to locate the service like systemd does, but since we have cgroups support, we could add authentication against the cgroup

    except that would result on the same "race condition" wrt exiting, if the process exits and is reaped before the NOTIFY_SOCKET helper can auth-check it, then we can't auth-check it

    so to implement permission control on our NOTIFY_SOCKET helper we would also need to implement BARRIER, :blep:
    💬 0🔄 0⭐ 0
  • Apr 9, 2026, 4:37 PM

    @navi @lanodan If you have one socket per service with an unpredictable name, I don't think you need to authenticate at all. Only the right service knows the socket name, if the name is in a large enough space, you're good; it's all collaborative, if someone is trying to be malicious you have already lost. (You could verify via SO_PEERCRED that the client is root, though, if you want to be really safe.)

    s6-notify-socket-from-fd uses an abstract socket with autobind. The autobind address space is 2^20; that should be enough for the kernel to make the socket address unpredictable, and if not, 🤷

    💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Apr 9, 2026, 5:16 PM
    @ska > What would be a simpler way to handle all this? Have the supervisor run the unit with a pipe from the daemon to the supervisor, not a socket anyone can connect to. Processes that can write to the pipe are the ones that can inherit it, which means they're necessarily part of the same unit.

    POSIX does guarantee a whole 7 other file descriptors available for such purposes, after all, why not use them?
    💬 0🔄 0⭐ 0
  • Apr 9, 2026, 11:08 PM

    @lispi314 @ska fwiw _POSIX_OPEN_MAX (which is the minimum value for the max fds) is 20

    but honestly i doubt any system now a days has less than at least 1024? if not unlimited (w/o accounting for ulimit)

    💬 0🔄 0⭐ 0
  • Apr 9, 2026, 11:44 PM

    @navi @ska @lispi314 Made me check and OPEN_MAX is 256 on Solaris 10 and 11, didn't except that but well you learn everyday.

    Then UNIX® Conformance Statements wise, seems to be 60000 for HP-UX, 65534 for AIX, RLIM_INFINITY (huh?) for macOS Tahoe (sadly work's macOS testlab machine is currently down and haven't got an AIX or an HP-UX).

    💬 0🔄 0⭐ 0