Login
You're viewing the functional.cafe public feed.

Replies

  • Nov 17, 2026, 1:09 PM

    apachectl comes from ancient, dark times when init systems did basically nothing but run a shell script, service managers were said shell script, and service supervision was unheard of. It was a jungle, and if you were a daemon author, you had to take care of a lot of stuff yourself to ensure that it would work in as many circumstances and hostile environments as possible. Such as closing all your file descriptors (can't rely on the parent to do it, and you don't want to inherit a stealthy ctty) and doubleforking (so the daemon would work in the background, its parent would always be pid 1, etc.)

    That was problematic for a lot of reasons, logging being one of them (in such a configuration, all you can rely on is syslog, which is not great but let's not get into that in this thread), and stopping the daemon being another: if it forked, nobody knows its pid, which you need in order to kill it, so you have to store it in a pid file, which has all the lifetime and race condition problems that we should all be familiar with by now.

    We needed something better, but there was no initiative from the free software OSes to provide it (yet). 1/12

    💬 2🔄 2⭐ 7
  • Nov 17, 2026, 1:10 PM

    So Apache httpd came up with its own management interface, which was a perfectly reasonable decision at the time; lots of other software did the same.

    You start the web server with apachectl start, you stop it with apachectl stop (or graceful-stop or whatever name they give to various stopping semantics), and you can also use the same apachectl command to instruct the daemon to perform other operations, such as reloading its configuration. Sounds good. And from the user's point of view, it is indeed an improvement over starting the thing manually from weird shell scripts and handling pid files.

    From a systems point of view, however, this is not as good. It introduces variety in how software operates, and it ends up in a lot of functionality duplication to start a service, all done by various unrelated pieces of software in slightly different ways. 2/12

    💬 1🔄 2⭐ 4
  • Nov 17, 2026, 1:11 PM

    Enter service supervision: a framework to run a daemon from a process tree distinct from the shell that wants to start it, and to provide a generic interface to send commands to the daemon without having to know its pid. This has multiple advantages: no more pid file race conditions, a clean starting environment (so, reproducible behavior), better logging support, and more - all without the service-specific boilerplate that everyone had to write before.

    The idea of supervising daemons has been around since djb's daemontools in 1998, then expanded upon with various packages such as daemontools-encore, perp, nosh, runit, and s6. However, it unfortunately never got mainstream traction until OS-specific init consolidation. MacOS' launchd implements supervision, so does Solaris' SMF, and so does GNU/Linux's systemd.

    The latter is the one that finally made authors realize that supervision was a good idea and daemons should be able to run under it. I hate how it did it, and systemd is a scourge on the ecosystem and a horror show in how it is designed, but if it did one good thing, it was finally making daemons get rid of auto-backgrounding and auto-syslogging, or at least grow options to bypass them. Credit to systemd for improving the software ecosystem on that one point. 3/12

    (Edit: typo)

    💬 1🔄 2⭐ 10
  • Nov 17, 2026, 1:11 PM

    The problem is that all of the service supervision systems, save nosh and s6, perform supervision as an integral part of being an init system and a service manager. These are different functionalities, but these init systems don't understand that, and mix all of it in one software package.

    I have slides for a detailed talk about this, and when I practiced it, the talk was 4 hours long. (I managed to hack into it enough to make it a 1-hour talk, which I gave, and I found it very superficial and incomplete. There is an even shorter version, 15 minutes long, linked at the end of the thread.)
    I'm not going to go into detail here. But the idea is that supervision is about managing a long-running process (i.e. a daemon), and service management is about managing a set of services, which may or may not involve long-running processes.

    Supervision is "here's a daemon, run it in a controlled and reproducible way without race conditions". Service management is "bring my machine from state A to state B following a series of local state transitions that may include starting daemons, and do so while respecting dependency constraints".

    To everyone but s6 and nosh, these are the same. 🤷 4/12

    💬 1🔄 2⭐ 12
  • Nov 17, 2026, 1:12 PM

    The terminology I use for a state transition that does not involve starting or stopping a daemon is a oneshot, while a transition that does is a longrun.

    Oneshots are typically what you do when you run a process that does a permanent state change to your system then exits. For instance, mounting a filesystem is a oneshot. So is setting the hostname.

    Starting a device manager, such as udevd, is an interesting case: starting the udevd process itself is a longrun, and you'd like it to be supervised, but you then need to follow it by an udevadm trigger command, which is a oneshot, in order to perform the coldplug.

    There's a lot more to say about all this, and by the end of next year I hope to have a full model that takes all cases into account and simplifies writing services for both s6 and OpenRC (with help from the wonderful @navi). 5/12

    💬 1🔄 2⭐ 7
  • Nov 17, 2026, 1:13 PM

    (Don't ask whether bringing up the network is a oneshot or a longrun. The network is a mess. It used to be a oneshot, but with DHCP clients and wifi and mobile and roaming, this is clearly not true anymore, and maintaining a functional network is now a reactive system more than anything. It needs dynamic event management, and I'm still thinking about the right way to integrate that into a service manager. Maybe I'll have a simple and correct model some day, but today is not that day.) 6/12

    💬 2🔄 2⭐ 5
  • Nov 17, 2026, 1:14 PM

    Supervision is great. Service management is great. But with the advent of all this, how do service-specific controllers like apachectl fit in? What do they become?

    Well, they don't fit in well is the problem. There is basically no purpose for them anymore.

    Starting a longrun httpd service would be done via systemctl start httpd, and the httpd service would be properly started under a supervisor, without auto-backgrounding. The httpd process would, ideally, be reloaded with a SIGHUP, gracefully killed via a SIGTERM, not-so-gracefully killed via a SIGQUIT or any equivalent semantically-adjacent signal (not SIGKILL, that's only the very last resort), and so on - conventions that are more than 30 years old, that a lot of software still follows, and when the process' behaviour doesn't match the conventions, the supervisor can be adjusted to send different signals (most supervision systems, including systemd, can do this). httpd would then fully benefit from being supervised; and the whole self-management layer can go.

    The drawback is that if the server is Apache, the apachectl commands then become incorrect, because apachectl does not know of the supervised instance of httpd; apachectl start would start its own instance and do its own thing, completely unaware of the supervision framework. More importantly, apachectl stop would not work, because it would try to stop a nonexistent instance. Also, if it somehow worked, the supervisor would be unaware of the external apachectl stop command, and would only see the httpd process die; unless configured otherwise, it would treat that death as a temporary error and restart the process.

    There is definitely a conflict here. 7/12

    💬 2🔄 2⭐ 4
  • Nov 17, 2026, 1:15 PM

    So what the person who wrote the apache2.service file did instead is ignore the whole supervision infrastructure provided by systemd, and use the old management interface provided by Apache. ExecStart=apachectl start means that systemd will run apachectl which will start a backgrounded httpd and then exit. systemd will only see the apachectl process, and the service will be a oneshot, despite it leaving an httpd running, unbeknownst to systemd. If that httpd dies, systemd won't know to restart it. The apache2.service author is just using the service manager part of systemd to start httpd at boot time, and stop it at shutdown time: there is no integration of the daemon with systemd at all.

    For once, systemd isn't to blame here, unless apache2.service was written by the systemd team. It was written that way because it looked easy: apache2 has a management layer already, so might as well use it, right? apachectl start to start the service, apachectl stop to stop it, and so on. Piece of cake!

    Except it's not that easy. 8/12

    💬 2🔄 2⭐ 5
  • Nov 17, 2026, 1:16 PM

    There are now two management layers to the service: systemd and apachectl. And they have to agree on its state - and they don't. Because from systemd's point of view, it is a oneshot, not a longrun; it stopped as soon as apachectl start exited; so "reloading" makes no sense since the service is "not active". That is what happened to @mcc .

    And this is where systemd is wrong. If it has been given an ExecReload= directive, it should just run it, even if it thinks the service is inactive. If it really is, the reload command will error out and systemd can report the error. But systemd doesn't do that, because it doesn't understand that it's handling a oneshot - and ExecReload= only works for longruns. systemd does not handle the difference properly; if it did, the presence of an ExecReload= directive in a oneshot service file would at least raise a warning.

    Conclusion: systemd is bad, but we already knew that, and more to the point, this unit file is garbage, and simply does not work. Did nobody test it? Does anyone in FOSS test anything anymore? 9/12

    💬 3🔄 2⭐ 5
  • Nov 17, 2026, 1:17 PM

    So what would a working service file look like? Here's what I would do:

    • Study the apachectl layer: how is the httpd process started exactly? what command line is run, and how do subsequent apachectl invocations communicate with it? signals? (how does apachectl find the pid?) Unix domain socket? (how does apachectl know where the socket is?)
    • ExecStart= is a direct command line running httpd in the foreground, with Type=simple (or Type=notify for the daring, that will be for another thread). It is a real longrun, managed by systemd as idiomatically as possible.
    • Other ExecSomething= directives are written depending on the result of the apachectl analysis. With a little luck, some (or all!) of them are just signals, and the directives can be omitted entirely. KillSignal= may need to be adjusted, but that's all fine.
    • And that's it.

    But, but, this means I can't use apachectl directly!

    Well it looks like you already can't, so 🤷. Keep reading though.

    Wait, are you advocating for more integration with systemd? Who are you and what have you done with @ska ?

    Yes I am, because I am advocating for correctness, which is best done by good integration with the service manager, no matter what it is. systemd is trash, but if you're going to write service files for systemd, your service files should actually work with systemd, not be a half-assed shim to your half-assed 20 year old ad-hoc supervision system. Of course, you could also write service files for s6, yknow? Nobody's stopping you. 10/12

    💬 1🔄 3⭐ 8
  • Nov 17, 2026, 1:18 PM

    So what happens to apachectl, and to people who are used to it?

    Well, this kinda sucks. Service-specific control commands are a relic from the past; using supervision or service manager commands to address services is much better - this unifies the interface across services and makes it easy to define distribution policies; and modern supervision systems generally provide better guarantees than service-specific control commands, which are often not much more than three shell scripts in a trenchcoat to hack something that looks like supervision.

    That's... not a very satisfying answer, is it?

    It's possible to get a better one - but it requires effort. Since apachectl and similar control commands are not usable as is in a supervised world, people who provide the service files - whether the distribution, as it used to be and should still be, or upstream, which for some reason took up that burden and is often doing a bad job of it - should also provide replacements for control commands, written on top of service manager commands.

    So, instead of systemd invoking apachectl, it should be apachectl invoking systemctl.

    • apachectl start ➡️ systemctl start httpd
    • apachectl stop ➡️ systemctl stop httpd
    • etc.

    And yes, that requires a lot of writing for someone who supports different service managers - which is what distros should do because monocultures are bad. But the good news is that these can generally be pretty simple shims; this is far from complex or difficult code. 11/12

    (Edit: clarity)

    💬 1🔄 2⭐ 4
  • Nov 17, 2026, 1:20 PM

    I have been interested in supervision systems and service management since daemontools came out, which will soon be 30 years ago. I have written and maintained s6 for 15 years now, and every year I find some new issue, or some rabbit hole of complexity, and it always amazes me that starting daemons, a subject that on paper looks pretty straightforward, can be full of pitfalls and traps and subtle things, and I hadn't clocked it as a fractal subject but looking back, it certainly is. If I prepared a talk on it today, it might end up being more than 6 hours long.

    If the subject interests you, a good starting point is the 15-minute introductory video I gave at FOSDEM 2017: archive.fosdem.org/2017/schedu
    Some stuff is out of date (#s6 is on OFTC, not on Freenode anymore...) and omg did I really have such a French accent? but most of it holds up.

    Thanks for coming to this long-winded, rambling TED talk. This is what happens when I'm waiting for my laundry and scrolling fedi. 12/12

    💬 1🔄 4⭐ 16
  • 💬 0🔄 0⭐ 0
  • Nov 17, 2026, 4:12 PM
    @ska @mcc

    fwiw systemd does have `RemainAfterExit` for oneshots, that makes it then work like openrc works; a started service counts as started until stopped, even if there's no longrun attached; that should make the Reload command work i think, and map better to the notion of it being stateful (even though it's wrong here)

    tho i also wonder why systemd lets you stop a non-remain oneshot but not reload it? either that, or `systemctl stop apache` won't work either

    fwiw² gentoo's systemd unit file for apache does call apache2 -DFOREGROUND directly, not apache2ctl, so someone already did make it work
    💬 2🔄 0⭐ 0
  • 💬 0🔄 0⭐ 3
  • Nov 17, 2026, 9:15 PM

    @navi @mcc @ska

    I don't see any upstream systemd service at all, and the Debian one comes from the debian/ directory.

    By the way, Type=forking would be the real solution. "This daemon is grotty, assume it forks, woe unto you if you don't also specify PIDFile= to tell us how to stop it again".

    Meanwhile in debian/apache2.service land...

    Type=notify
    Environment=APACHE_STARTED_BY_SYSTEMD=true

    *Now* you may start crying.

    💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Nov 17, 2026, 4:22 PM

    @ska I understand why this system must be architected the way it was architected, but that doesn't mean I had a fun day yesterday.

    One problem perhaps bigger than any particular interface decision is that the interface to my applications was changed out from under me without me even knowing it (I still don't know when systemd appeared on my system; it wasn't there when I installed Linux on this server, and one day it was)

    💬 1🔄 0⭐ 0
  • Nov 17, 2026, 7:31 PM

    @mcc Yeah, that branches into the "distros are shit at respecting users' time and habits" rant, which I totally agree with, but it's not my area of expertise (it's more @gro-tsen.bsky.social 's). I stuck to the part of your story that I could provide lighting for. 😅

    💬 1🔄 0⭐ 1
  • Nov 17, 2026, 10:27 PM

    @ska @mcc @gro-tsen.bsky.social I don't think I have any particular expertise to bring (other than thank you for an instructive thread), but as far as communicating user wishes, I believe it ought to be possible to start/stop/restart Apache either with `systemctl` or `apachectl`, and, moreover, they should interact completely harmoniously with each other. I'm not saying I know *how* this can be done, but it *should*.

    💬 2🔄 0⭐ 0
  • Nov 17, 2026, 10:30 PM

    @ska @mcc @gro-tsen.bsky.social … Because this is a problem that has repeatedly plagued the Linux ecosystem at large: you have N different tools that more or less accomplish the same task (here, `systemctl` and `apachectl`), and the user is completely confused as to how they relate or interact (are they synonyms? incompatible? is one obsolete?). Such mazes should always be resolved by harmonious interaction between the tools.

    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 10:34 PM

    @gro_tsen_test @ska @gro-tsen.bsky.social Actually, it is *possible* that newer versions of apachectl do work this way; I am using a somewhat old LTS, on this one system.

    This said there is some design difficulty here because systemd does what it does by *invoking* apachectl, so we'd need something wacky like a --no-really-im-systemd argument to apachectl. It gets awkward.

    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 5:45 PM

    @ska
    > systemd will only see the apachectl process, and the service will be a oneshot, despite it leaving an httpd running, unbeknownst to systemd.

    Are you sure about that? I thought the initial reason for using cgroups in systemd was as a hack to catch those child background processes as well. It still only works in the simple cases (what if httpd already runs and apachectl start becomes a nop), but this case should have been simple enough for it.

    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 7:22 PM

    @mixi That's true, I always forget about cgroups; it could probably work with Type=forking and enough tweaking of directives in the unit file to pretend that apachectl start is the name of the process 😁 it's still a hack though, and even systemd discourages the use of Type=forking.

    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 1:21 PM

    @ska slight tangent, but the design of apachectl is pretty bad. They use signals as their IPC, but since there’s not a sufficient number available for all commands, they reused SIGWINCH. So if you run Apache foregrounded in a terminal and resize it, the Apache process will exit. I sometimes do this just to laugh at how silly it is.

    💬 2🔄 0⭐ 3
  • Nov 17, 2026, 1:23 PM

    @tnorinder Honestly I don't think it's that bad! daemons were never supposed to run in a foreground process group, so I can't blame Apache devs for choosing to use SIGWINCH. It is, shall we say, an unfortunate interaction. 😅

    💬 1🔄 0⭐ 1
  • Nov 17, 2026, 4:44 PM
    @ska @tnorinder Yeah, although can be quite nice for debugging, specially when it's on a pre-supervision system (imagine trying to insert gdbserver/strace into Debian's humongous initscripts or even it's own random perl config-reading wrappers).
    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • Nov 17, 2026, 7:43 PM

    @ska @tnorinder Yeah, although with the addition of SIGRTs that should be pretty rarely needed (8 of them minimum on a POSIX system, usually more like 20~30, see {RTSIG_MAX} in <limits.h>)
    And I guess beyond that one could go for things like sockets.

    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 7:51 PM

    @lanodan @tnorinder Excuse me, only a psychopath would use real-time signals to send commands to daemons. This would be completely arbitrary.

    Imagine, for instance, deciding that the reboot signal for your init process must be SIGRTMAX+3. This is so absurd that everyone would laugh you out of the room.

    ... right?
    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Nov 17, 2026, 3:37 PM

    @ska

    The #systemd people have maintained that "the network" covers a multiplicity of things.

    After a lot of work, I eventually managed to sort it all out into a bunch of genuine long-running services, a bunch of single-shot things, and a bunch of devd/similar rules to auto-start/stop services.

    jdebp.info/Softwares/nosh/guid

    Ironically, as the Guide notes, #Debian's ifup/ifdown were just like apachectl is here, doing Poor Man's service management, badly.

    #nosh #networking

    💬 0🔄 0⭐ 1
  • 💬 1🔄 0⭐ 0
  • Nov 17, 2026, 2:27 PM

    @glitzersachen This is a false dichotomy. systemd and equivalent kitchen-sink-including init systems suck, but so did having nothing.

    I wrote the init system I wanted to see, and I'm not going back - except, of course, for special cases like embedded devices where even s6 is needlessly complex.

    💬 1🔄 0⭐ 1
  • Nov 17, 2026, 3:01 PM

    @ska @glitzersachen

    The middle 1990s, when people were busy re-inventing, for Linux and Minix, systems that Unix had done away with almost a decade earlier in favour of better stuff, was definitely not a golden age. It was an Age Of Reinventing The Wheel Back In Its Original Square Shape. So much of that decade was just reinvention.

    #init

    💬 0🔄 0⭐ 2
  • Nov 17, 2026, 2:03 PM
    @ska Imagine having to do the busywork of splitting (and then joining during reading) your text into tiny chunks due to the insane and obnoxious length limits of your activitypub instance.
    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 2:05 PM

    @L29Ah Or maybe my instance has a high character limit and I chose to split the text into smaller, thematic chunks for an easier read.

    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 2:21 PM
    @ska I double the above comment. there're more appropriate platforms on Fediverse for longreads and solid discussions than Mastodon.
    but I see the upright there and this is a good sign.
    💬 0🔄 0⭐ 0
  • Nov 17, 2026, 2:24 PM

    @iron_bug For longreads? Sure, I could write a blog article, and host it on my web site where it would get a dozen views. Meh.

    For solid discussions? The fediverse has blessed me with more solid discussions in the 4 years I've been on it than all other platforms combined for the past 25 years.

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 1
  • Nov 18, 2026, 12:35 AM

    @ska This thread could have been a blog post :-)

    Seriously, this is a great description of how, and WHY, things are the way they are.

    Please consider making it a blog post and adding a link here - it is valuable information!

    💬 1🔄 0⭐ 0
  • Nov 18, 2026, 3:56 AM

    @rhempel I'm kinda using fedi as a blogging platform tbh, among other things. I trust my instance to keep working and keep availability of my posts as much, or more, as I would trust any website, including my own. And if it stops, they'll warn in advance and I'll migrate my data. So you can consider the thread as perennial as can be, and link to it if you like!

    💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0