For years, I used daemontools (it still runs a bunch of machines) then moved to runit, which follows the same philosophy. Both are excellent tools and have worked extremely well under many different circumstances. Simple tools that do one thing extremely well. Every managed service requires a short, 1 to 3 line, script to actually start the target program, which may, initially, seem off-putting, but actually provides the often missing opportunity to easily customize the program environment.

BTW, runit also comes with svlogd and chpst, both of which have frequently proved invaluable. The latter is useful in its own right.

Most of the tools discussed either lack flexibility, try to do too much, or simply don't offer any benefits over runit and company.

I agree with pretty much all of what you said about how a system like this should work. daemontools was next on my list of solutions to try out. I wish I had discovered runit before doing this rundown...my google-fu is apparently lacking a bit :)

FWIW, from a few months ago:

http://jtimberman.housepub.org/blog/2012/12/29/process-super...

tl;dr use runit

I've always used (and loved) daemontools / runit (http://smarden.org/runit/) - it's a little bit minimal, but it's well-documented, does what it claims to do, and it's rock-solid.

daemontools is high on the list of other options I want to evaluate, I just haven't gotten around to it yet!

Was surprised not to see God (http://godrb.com/) here, has it fallen from grace?

There are several other systems (including God) I still really want to evaluate, unfortunately I just haven't had the time to do so yet. I'll add a note to the post saying this isn't meant to be an exhaustive list yet!

Don't forget Angel and systemd :)

The article is a bit wrong about supervisord. You can load environment variables from a defaults file - just use a wrapper script. Eg:

  #!/bin/sh
  source /etc/myservice/defaults
  exec celeryd $@

And start that from supervisord.

Also, another nice thing, very useful for daemons that spawn their own subprocesses is killasgroup option. This is worth mentioning given the celery examples - celery can and will leave orphan processes around between restarts ...

I swear this is one of the hardest problems in computer science, which is crazy - how hard should it really be to say "run this process, and restart it if it crashes"?

It's not a hard problem at all; it's a rather simple loop:

  while true:
    pid = fork()
    if pid:  # parent
       wait(pid)
    else:    # child
       exec("/path/to/supervised/process")

The key here is not to let the supervised process daemonize itself. If it does, the parent can no longer wait() on the child.

"The convenience of custom commands like /etc/init.d/nginx configtest to check my nginx configuration syntax without affecting the running nginx service is useful enough to keep me from migrating my nginx daemonization to Upstart."

Needing to run "nginx -t" instead of "/etc/init.d/nginx configtest" seems like a minor tradeoff for the benefits Upstart provides. It's even shorter to type!

Timely, since I spent some time over the weekend digging into the internals of Supervisor.

One of the most interesting features, for me, is that Supervisor has an XML-RPC interface[1], allowing for some creative interaction with supervisor and the processes it manages.

[1] http://supervisord.org/api.html

Another recent comparison: http://tech.cueup.com/blog/2013/03/08/running-daemons/

Thanks very much for this! It's great to have someone else's insight into several of these.

You're welcome! Would love to hear your thoughts and insights on the topic. :)

This is a super helpful writeup on a topic I needed an intro to, thanks so much!

You're welcome! I'm glad it was helpful. A lot of these devops and sysadmin related topics can be difficult to break into...especially with the rise of PaaS.

Any experience with 'God', written in ruby? That's the one tool in this class that I've heard of a lot, that wasn't covered. (My guess is you're a pythonista over a rubyist)

systemd is the way to go. Don't waste time debugging someone's half-baked pile of {shell,Perl,Python} scripts