May 8, 2013

Running Supervisor on OS X

I use Supervisor on all of my servers to handle the running of non-daemonized processes, such as Gunicorn processes for Django apps. The truth is, though, that Supervisor is also a damn-awesome tool for local process management in OS X. Here are a few of the things I'm letting Supervisor handle:

Running a Node app on port 80.
Running nginx on port 80.
Running a watched test-suite for a Node app.
Auto-compiling SCSS source files to CSS.
Running a Django app with Gunicorn.
Running a Django app with the built-in dev server.

There are a few things that make Supervisor well-suited for the above tasks. All of the above tasks:

I prefer to have running in the background.
I need quick access to logs, especially streaming via tail -f.
I'd like to have a common interface for starting, stopping, and restarting.

My favorite part of having Supervisor wrap all of these processes is the common interface for accessing logs. Let's say I want to watch the logs for my Django app I'm running via the built-in dev server (wheresniptis the program name I configured -- more on that later):

> sudo supervisorctl tail -f snipt stderr

The output streams to my terminal:

> sudo supervisorctl tail -f snipt stderr ==> Press Ctrl-C to exit <== "GET /static/js/projects.js HTTP/1.1" 304 0 [08/May/2013 18:05:42] "GET /login/?next=/ HTTP/1.1" 200 3811 ...

Or if I just want to dump all of the logs:

> sudo supervisorctl tail snipt stderr

You get the idea. If you want to learn more about what Supervisor can really do, check out the docs.

Installation

Installing Supervisor on OS X is simple:

> sudo pip install supervisor

This assumes you havepip. If you don't:

> curl -O http://python-distribute.org/distribute_setup.py > python distribute_setup.py > sudo easy_install pip > sudo pip install supervisor

Or if you use Homebrew:

> brew install distribute > sudo easy_install pip > sudo pip install supervisor

Or you could install everything from source. Good luck with that.

Regardless of how you've done it, once you've successfully installed you should be able to runsupervisorctland get something like:

Error: No config file found at default paths

That's fine, that means Supervisor is installed and ready to configure. On to bigger and better things (Configuration).

Configuration

The Supervisor documentation provides excellent information on configuration (as well as everything else). For the sake of getting this running on OS X, however, here's what mysupervisord.conflooks like:

[unix_http_server] file=/tmp/supervisor.sock chmod=0700 [supervisord] logfile = /Users/Nick/Sources/dotfiles-private/supervisor/logs/supervisord.log logfile_maxbytes = 50MB logfile_backups=10 loglevel = info pidfile = /tmp/supervisord.pid nodaemon = False minfds = 1024 minprocs = 200 umask = 022 identifier = supervisor directory = /tmp nocleanup = true childlogdir = /tmp [supervisorctl] serverurl = unix:///tmp/supervisor.sock [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [include] files = /usr/local/share/supervisor/conf.d/*.conf

You may want to configure some of these to your liking if you know what you're after, otherwise just change the directories to point to your log file, configuration directory, etc. The configuration directory can be anywhere, just make sure yoursupervisord.confis directed to include configuration files from there.

Supervisor is going to look for thissupervisord.conffile in a few places by default:

/usr/local/share/etc/supervisord.conf /usr/local/share/supervisord.conf ./supervisord.conf ./etc/supervisord.conf /etc/supervisord.conf

You can place it anywhere, though, as we'll be running the Supervisor daemon with a configuration flag (see below).

Running Supervisor

Now that we have a main configuration file somewhere, we can run the daemon. Oh, about that: there are two parts to Supervisor: the daemon (supervisord) and the client (supervisorctl). The daemon runs in the background and does all of the hard work, and the client provides a nice little UI for... doing stuff.

You can run the daemon manually like this:

> supervisord -c /path/to/supervisord.conf

But we're not interested in that. We want Supervisor to run on startup as root so we can have Supervisor do things with root-level priviliges (like running servers on port 80).

So in order to start Supervisor on startup, we need to use OS X'slaunchdsystem, which loads programs asrooton startup. You don't actually use thelaunchdprogram, you write a configuration file and you load it withlaunchctl. Here's the file:

/Library/LaunchDaemons/com.agendaless.supervisord.plist

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>KeepAlive</key> <dict> <key>SuccessfulExit</key> <false/> </dict> <key>Label</key> <string>com.agendaless.supervisord</string> <key>ProgramArguments</key> <array> <string>/usr/local/share/python/supervisord</string> <string>-n</string> <string>-c</string> <string>/usr/local/share/supervisor/supervisord.conf</string> </array> <key>RunAtLoad</key> <true/> </dict> </plist>

Note: The "agendaless" connotation is the organization that created Supervisor. You can use whatever you want there, but "agendaless" makes the most sense.

This plist file tells OS X to run thesupervisordprogram on startup (theRunAtLoadpart), and to run the program as "nodaemon" (the-nflag), meaning we want it in the foreground. Then we pass the configuration file (the-cpart followed by the .conf path). Here's where you'd change the path to your configuration file, if needed.

To register this plist, run this:

> launchctl load /Library/LaunchDaemons/com.agendaless.supervisord.plist

If everything worked well, you should be able to runsudo supervisorctland see something like this:

\t> sudo supervisorctl supervisor>

If you see that, you're in the Supervisor terminal-based UI, the Supervisor daemon is running properly, and thesupervisorctlprogram has found thesupervisord.conffile correctly. This means you're ready to start configuring programs.

If you see anything other thansupervisor>, something is probably wrong. First off: reboot your computer. It's possible that you need to get OS X to loadsupervisordfromlaunchdas root on startup in order to get things to work. If you've rebooted and you're still having problems, drop a note in the comments and we'll see if we can figure it out :)

Otherwise, you're ready to start configuring programs.

Program configurations

In Supervisor, you write configurations like this. It's quite simple, and the configuration options for program definitions are robust. The simplest program configuration looks like this:

[program:foo] command=/bin/cat

That would be the content of the file/usr/local/share/supervisor/conf.d/foo.conf(or wherever you pointed your configuration files at insupervisord.conf.

Let's pretend for a moment that the programcatruns in the foreground and you just want to run it via Supervisor:

Hop into the Supervisor UI: > supervisorctl.
Tell Supervisor to read all of the configuration files: reread.
You should get a notice that foo is available.
Add the foo program: add foo.
foo is now ready to be managed by Supervisor.
Type status to see a list of programs: foo should be in there, with a status of STOPPED.

At this point, thefooprogram is ready to be managed by Supervisor, and is stopped. Here are some common operations:

start foo restart foo stop foo tail foo stderr tail foo stdout tail -f foo tail -f foo stderr

...etc, etc. For a complete list of commands, typehelpin the Supervisor UI, orsudo supervisorctl help. Which brings up a good point: you can run Supervisor commands either inside the UI (viasudo supervisorctl) or directly from the commandline, like this:

sudo supervisorctl start foo

That's about it for creating a basic program to be managed by Supervisor.

Here are some program-specific examples of configuration files:

Django app with built-in dev server

[program:myawesomeprogram] command=/path/to/python /path/to/project/manage.py runserver localhost:4000 directory=/path/to/project autostart=false autorestart=true stopsignal=KILL killasgroup=true stopasgroup=true

Note: thekillasgroupandstopasgroupdeclarations are very important when running the Django dev server via Supervisor.

Django app with gunicorn process

[program:anotherawesomeprogram] command=/path/to/bin/gunicorn -c /path/to/gunicorn.conf.py debug_wsgi:application directory=/path/to/project autostart=false autorestart=true

Node app

[program:yetanotherprogram] command=sudo /usr/local/bin/node app 80 directory=/path/to/node/project autostart=false autorestart=true

Auto-compiling SCSS source files to CSS

[program:css] command=/usr/local/opt/ruby/bin/scss --watch hi.scss:hi.css directory=/path/to/css autostart=false autorestart=true

Nginx

[program:nginx] command=sudo /usr/local/sbin/nginx autostart=false autorestart=true # Note: in your nginx.conf, make sure to set `daemon off;`. # Also note: since Supervisor is run as root, you can configure Nginx to run on port 80 without trouble.

It's important to provide absolute paths to programs, especially if those programs were user-installed, like Ruby gems, Python programs, etc. Remember, thesupervisordprogram does all the work, and it's run as root, which won't source user-level PATHs.