Most likely, you have already read "The Art of Graceful Reloading". Although it does explain a lot, it still leave gaps for the reader to figure out. This can be quite time consuming. Well.. I went through the exercise and this is the result.

I will go through the reload types shortly, but first I need some terminology and theoretical background on how WSGI works.

Pre-fork vs Lazy-app

uWSGI offers two modes how to startup your application. In pre-fork (default), master process will do the WSGI initialization, then he will fork the workers. In lazy-app, master will first fork the workers and they do their own initialization.

pre-fork vs lazy-app

(All diagrams in this post show processes as boxes. Their parent-child relation is denoted by an arrow)

As you can imagine, pre-fork is much less resource intensive (memory consumption, total startup time, time-to-spawn new worker, ..). Lazy-app is more "robust", workers are more isolated from master and each other.

Memory usage example

Pre-forked helloworld app with 10 workers = 46M. The same app with lazy-apps = yes = 225MB. So, even two full copies of pre-forked app (92MB) are way better than lazy-apps (depending on number of workers).

There is also one important difference related to code deploy. Some types of reloads do not reload master, only workers !!

pre-forked workers with mixed code problem

This is a problem for pre-forked apps, because you may end up with old and new python modules loaded in the same process. During code deploy, worker would "inherit" (via fork()) old version of modules already loaded by master. While modules, not part of WSGI initialization, will be loaded from filesystem (new code).

Lazyly-loaded apps are not affected.

Emperor mode

uWSGI can be started as an Emperor. It is a small process responsible for starting/stopping other uWSGI processes, called vassals. Emperor does not have any application logic, think of it as init or systemd. You can create vassal by dropping an .ini file into the directory. Shutdown vassal by deleting his .ini file. Simple.

emperor with three vassals

What is important to remember is that Emperor will re-spawn his vassal when he dies. So, for example, if you gracefully shutdown old instance of your app running v1.0 code, Emperor will happily start it again. There is no distinction (AFAIK) between unexpected death (Emperor should restart vassal) and administrative shutdown (Emperor should not restart vassal).

In addition, if you delete vassal file, Emperor will shut it down ungracefully. Any existing http sessions will be terminated.

Incomplete information

I admit that some information in this blog post may be inaccurate. It is based on my personal empirical experience. In this particular case, there may be obscure option to tweak vassal shutdown but despite intensive research, I haven't found one.

Reload types

Reload can be triggered by sending the corresponding command to master FIFO pipe (see "The Master FIFO")

I will use these terms to estimate downtime/reload time:

  • init - time to initialize an app
  • idle-shutdown - time to gracefully shutdown idle worker (time to run cleanup code)
  • busy-shutdown - time to gracefully shutdown busy worker (waiting for your longest session to finish, plus time to run cleanup code)
  • N - number of workers

re-exec master ("r")

reload type "r"

  1. master will gracefully shutdown all workers at once
  2. wait until all workers are down
  3. re-exec itself - basically, run new code inside the same process
  4. spawn new workers

The problem here is that all idle workers die immediately. Only busy workers remain. So, during shutdown (2.), while you are waiting for your long sessions to finish, there is nobody to handle new requests. Also, nobody will handle requests while initialization (3.) is in progress.

  • downtime: busy-shutdown + init
  • total time: ~ busy-shutdown + init
  • master restart: yes

restart workers ("w")

reload type "w"

  1. master will gracefully shutdown all workers at once
  2. when a worker dies, it is immediately re-spawned
  3. until all workers are re-spawned

This is already better than "r", because master won't wait until all workers are down. Idle worker die/respawn in parallel, while busy workers are left to finish their work and will respawn later. The problem is that this will only restart workers.

  • downtime: idle-shutdown + init
  • total time: ~ busy-shutdown + init
  • master restart: NO (NOT compatible with pre-fork)

chain restart workers ("c")

reload type "c"

  1. master will gracefully shutdown one worker at a time (ordered by id apparently)
  2. when a worker dies, it is immediately re-spawned
  3. until all workers are re-spawned

This is the same as "w", except we shutdown/restart one worker at a time. Workers are restarted sequentially, irrespective to their status (idle/busy). In worst case, you will wait N-times for long session to gracefully finish.

Unlike "r"/"w", in this reload type, you can have a mix of old/new idle workers. So, during the reload period, new requests may be handler by either new or old code.

  • downtime: none
  • total time: ~ N x (busy-shutdown + init)
  • master restart: NO (NOT compatible with pre-fork)
  • mixed (old/new) code

fork master ("f")

fork master "f"

This is technically not a reload. Master will simply fork itself and leave two copies running. It is up to you to shutdown the old one. One possible approach is to use multiple FIFO pipes as suggested by "The Art of Graceful Reloading" doc.

  1. fork new master
  2. when initialization is done, it forks new workers
  3. old master is told to shutdown
  4. old master will wait until last busy worker finishes

You don't have downtime here, but you temporarily use twice as much memory. Forking master is considered "dangerous" (by uWSGI documentation). There is no mix of old/new idle workers either. New master will shutdown old master (and workers) when his workers are ready to accept queries.

There is a problem with shutting down the old master. It does not work in Emperor mode. Emperor would re-spawn his dead vassal.

  • downtime: none
  • total time: ~ init + busy-shutdown
  • master restart: yes
  • NOT compatible with Emperor
  • number of workers: 2x N

Zerg dance

Zerg dance (only the first two steps are depicted in diagram)

  1. start new instance (vassal)
  2. when initialization is done, it forks new workers
  3. old master is told to shutdown
  4. old master will wait until last busy worker finishes

This is similar to fork, except old and new master are not parent-child, but siblings. Temporarily, you would run two full copies of your app, but the mechanism of sharing the socket for requests is different. In fork reload, the descriptor was passed from parent to child by fork(). In this mode, there is a special service (Zerg pool) to pass the socket to new master via another control socket.

Zerg pool process is quite tiny and does not contain any of your application logic. So, you probably won't need to restart/upgrade it often. But it is still a SPOF and quite fragile (it won't survive reload for example).

As you can see, you need to start two top level uWSGI processes and new one for every code update. Emperor is quite handy for this use case (Zerg and App server would both be top-level vassals). You just need to be careful with shutdown/delete of old vassal.

  • downtime: none
  • total time: ~ init + busy-shutdown
  • master restart: yes
  • number of workers: 2x N
  • another single point of failure

Zerg Pool vs Zerg Server

The above information apply to "Zerg Pool". There is another Zerg (Server) which I did not find useful for my graceful code deploy case.

update upstream config

All of the above graceful reloads have one major goal in common. To make sure there is someone listening on the one socket. It would open new possibilities if you could drop that requirement and let each app instance listen on its own socket. Then, you would be able to update upstream (Nginx config) to point to the new socket, while letting existing sessions to gracefully finish on the old one.

  1. start new instance (vassal)
  2. when initialization is done, it forks new workers
  3. update upstream configuration & reload
  4. wait until old instance (all workers) are idle
  5. delete old instance

This approach has many of the same aspects of Zerg dance (minus the Zerg process).

  • downtime: none
  • total time: ~ init + busy-shutdown
  • master restart: yes
  • number of workers: 2x N
  • changing socket name

Examples

I will use my own Helloworld Django app to demonstrate each approach. Examples are minimalist, but fully working. Deploy code is also cut down to bare minimum. Your real deploy would do much, much more. Like update DB schema, collect static files, etc.

I'll be changing the code by git checkout .. "in-place" because it's simpler to demonstrate. In reality, you may want to start each new deploy from a clean copy of your checkout.

re-exec master ("r")

Config:

[uwsgi]
chdir           = /home/foo/helloworld
module          = helloworld.wsgi
processes       = 4
socket          = /home/foo/var/helloworld/app.sock
master-fifo     = /home/foo/var/helloworld/master.fifo

Code deploy:

cd ~/helloworld
git checkout v2.0
echo r > ~/var/helloworld/master.fifo

(chain) restart workers ("c"/"w")

Config:

[uwsgi]
chdir           = /home/foo/helloworld
module          = helloworld.wsgi
processes       = 4
socket          = /home/foo/var/helloworld/app.sock
master-fifo     = /home/foo/var/helloworld/master.fifo
lazy-apps       = true                                      # required for "w"/"c" reloads

Code deploy:

cd ~/helloworld
git checkout v2.0
echo w > ~/var/helloworld/master.fifo
# or
# echo c > ~/var/helloworld/master.fifo

fork master ("f")

You cannot start this from Emperor.

This is minimal configuration, see down below for more fine-tuned one:

[uwsgi]
chdir           = /home/foo/helloworld
module          = helloworld.wsgi
processes       = 4
socket          = /home/foo/var/helloworld/app.sock
vacuum          = false                                     # see below for explanation
master-fifo     = /home/foo/var/helloworld/new_instance.fifo
master-fifo     = /home/foo/var/helloworld/running_instance.fifo

if-exists = /home/foo/var/helloworld/running_instance.fifo
  hook-accepting1-once = writefifo:/home/foo/var/helloworld/running_instance.fifo q
endif =
hook-accepting1-once = writefifo:/home/foo/var/helloworld/new_instance.fifo 1

Code deploy:

cd ~/helloworld
git checkout v2.0
echo f > ~/var/helloworld/running_instance.fifo

Zerg dance

This setup is using Emperor to start/stop vassals.

Emperor startup:

$ uwsgi --emperor ~/vassal

Zerg config (~/vassal/zerg.ini):

[uwsgi]
zergpool = /home/foo/var/helloworld/zerg.sock:/home/foo/var/helloworld/app.sock

Vassal config (~/var/helloworld/vassal.ini):

[uwsgi]
chdir           = /home/foo/helloworld
module          = helloworld.wsgi
processes       = 4
socket          = /home/foo/var/helloworld/app.sock
zerg            = /home/foo/var/helloworld/zerg.sock
stats           = /home/foo/var/helloworld/%n.stats

hook-accepting1-once = write:/home/foo/var/helloworld/%n.ready ok
hook-as-user-atexit = unlink:/home/foo/var/helloworld/%n.ready

Code deploy:

  1. create new vassal vassal/helloworld-<newhash>.ini
  2. wait until new vassal is up (helloworld-<newhash>.ready file is created)
  3. wait until old vassal is idle (all his workers are in idle state)
  4. remove old vassal vassal/helloworld-<oldhash>.ini
VERSION=v2.0
cd ~/helloworld
git checkout "$VERSION"
NEW="helloworld-$(git rev-parse --short HEAD)"
cd ~/var/helloworld/
OLD=$(cat current 2>/dev/null)
cp vassal.ini ~/vassal/"$NEW.ini"
while [ ! -e "$NEW.ready" ]; do sleep 1; done
echo "$NEW" > current
if [ -n "$OLD" ]; then
    ~/bin/uwsgi-wait-for-workers.py "$OLD.stats"
    rm -f ~/vassal/"$OLD.ini"
fi

The script uwsgi-wait-for-workers.py connects to stats socket of a vassal and waits until all his workers are "idle".

Choosing the right approach

This part may be subjective. It all depends on your requirements and use cases, which solution is better for you.

Filtering out candidates

Summary of downsides:

  • with downtime: "r"/"w" - they shutdown before starting ("r" has larger downtime)
  • pre-fork incompatible: "w"/"c" - they don't restart master
  • mixed responses: "c" - it has both old and new idle workers at the same time
  • memory heavy: "f"/Zerg - they spawn full copy side-by-side
  • Emperor incompatible: "f" - cannot allow old master to respawn

"r"/"w" are out due to the downtime. If your app has long sessions (like file upload), then the downtime would be unacceptable.

Zerg is quite frankly very fragile. Without zerg process, nothing works. And you have to ask yourself questions like "when zerg is reloaded, do I have to restart the app as well ?" and "if both zerg/app is running, how do I know that restart is required ?"

So I ended up with:

  • "chain worker restart" - simple, but requires lazy-apps. Mixed old/new responses temporarily.
  • "fork master" - complex, but better memory footprint. Incompatible with Emperor.

memory argument - as I have shown in "Pre-fork vs Lazy-app", two copies of pre-forked app during deploy (fork master) is still better than one lazy-app (chain restart).

mixed responses argument - this might not be such huge deal. When you deploy code on multiple machines you would also get mixed responses. So, why would you put such restriction on multiple processes on the same machine ? It might be better to write app to tolerate it..

Emperor argument - without Emperor, I was forced to start uWSGI directly by systemd. If nothing else, that step made the setup more robust. This is not an argument against fork.

complexity argument - chain restart is much simpler, no doubt

IMHO, the benefit of lower memory usage out-weights the simplicity of chain restart.

Winner: fork master

uWSGI is executed from user's systemd. Here is the unit file (~/.config/systemd/user/helloworld.service):

[Unit]
Description=Helloworld Django app

[Service]
PIDFile=%h/var/helloworld/app.pid
ExecStart=/usr/local/bin/uwsgi --ini %h/.config/helloworld/uwsgi.ini
ExecReload=/home/foo/bin/fifo-write.sh f %h/var/helloworld/running_instance.fifo
KillSignal=SIGQUIT

[Install]
WantedBy=default.target

(fifo-write.sh is a simple bash script that will write to fifo in non-blocking mode)

Config:

[uwsgi]
chdir           = /home/foo/helloworld
module          = helloworld.wsgi
processes       = 4
socket          = /home/foo/var/helloworld/app.sock
pidfile         = /home/foo/var/helloworld/app.pid
vacuum          = false

hook-accepting1-once = write:/home/foo/var/helloworld/helloworld-8f88930.ready ok
hook-as-user-atexit = unlink:/home/foo/var/helloworld/helloworld-8f88930.ready

master-fifo     = /home/foo/var/helloworld/new_instance.fifo
master-fifo     = /home/foo/var/helloworld/running_instance.fifo

if-exists = /home/foo/var/helloworld/running_instance.fifo
  hook-accepting1-once = writefifo:/home/foo/var/helloworld/running_instance.fifo q
endif =
hook-accepting1-once = writefifo:/home/foo/var/helloworld/new_instance.fifo 1P

why vacuum=false ?

During deploy, old and new master share the same socket /home/foo/var/helloworld/app.sock. With vacuum=true, old master would delete it during shutdown.

why pid file ?

Pid file is needed by systemd: a) to figure out if the service is still running, b) where he should be sending TERM/KILL signals. Notice, that pid file is updated by new master via fifo command "P".

what is that helloworld-8f88930.ready file ?

It is a flag to indicate that instance with hash "8f88930" has started and it is ready to accept connections. Each instance has its own ready file. The file is looked up during deploy, but it has no meaning afterwards. Even there, it is not strictly necessary. It's only a safety check.

Code deploy:

  1. update config file ~/.config/helloworld/uwsgi.ini (from template)
  2. initiate forking of master, new master will read the new config
  3. wait until new master is up (helloworld-<newhash>.ready file is created)
  4. wait until old master dies
  5. do cleanup after old code is down
VERSION=v2.0
cd ~/helloworld
git checkout "$VERSION"
NEW="helloworld-$(git rev-parse --short HEAD)"
cd ~/var/helloworld/
OLD_PID=$(cat app.pid 2>/dev/null)

cat > ~/.config/helloworld/uwsgi.ini <<EOF
[uwsgi]
chdir           = /home/foo/helloworld
module          = helloworld.wsgi
processes       = 4
socket          = /home/foo/var/helloworld/app.sock
pidfile         = /home/foo/var/helloworld/app.pid
vacuum          = false

hook-accepting1-once = write:/home/foo/var/helloworld/$NEW.ready ok
hook-as-user-atexit = unlink:/home/foo/var/helloworld/$NEW.ready

master-fifo     = /home/foo/var/helloworld/new_instance.fifo
master-fifo     = /home/foo/var/helloworld/running_instance.fifo

if-exists = /home/foo/var/helloworld/running_instance.fifo
  hook-accepting1-once = writefifo:/home/foo/var/helloworld/running_instance.fifo q
endif =
hook-accepting1-once = writefifo:/home/foo/var/helloworld/new_instance.fifo 1P
EOF

# in case of reload, this will "echo f > ~/var/helloworld/running_instance.fifo"
systemctl --user reload-or-restart helloworld

while [ ! -e "$NEW.ready" ]; do sleep 1; done
if [ -n "$OLD_PID" ]; then
    while [ -e "/proc/$OLD_PID/status" ]; do sleep 1; done
fi
# cleanup here