The daemontools package by Daniel J. Bernstein provides a convenient way to monitor and control UNIX services. Because Zope has to run all day on a production server it is a good candidate to be monitored with daemontools.
At the time of writing this article experience showed that Zope was not
as stable as it was supposed to be. Sometimes
it crashed and then it was essential to have it restarted automatically.
Because I used Bernsteins daemontools anyway for other services it was
natural to look for a way to run Zope supervised by them.
Please beware that this article gives no solution to recover from
a hanging Zope instance. The Zope Process Manager will possibly
adress this problem soon through the builtin heartbeat mechanism.
In the meantime you have to check with an external script if Zope still
works. If not, you may use the daemontools program svc to kill and restart
Zope.
The installation is typically done within a few minutes. Simply download and install daemontools as described in Bernsteins Installation instructions.
If your ps command shows that svscan runs with the parameter /service, all it's fine. How many time did you need for this? It should not be more than 10 minutes inclusive reading the necessary docs.
Now have a look at the svscan documentation: there is an important but somewhat hidden note about setting up a service and it's logging service with a pipe between them. We will use this for Zope too (more in Apache- and Zope logging with Daniel J. Bernsteins multilog), so I will repeat the important things to do this kind of setup:
If these criteria are met, svscan starts the service itself and the logging service with a pipe between them.
Hint: you don't have to type the run scripts because I packaged them
for you to download.
The scripts for Zope
up to version 2.5.1 are also still there.
Starting Zope is done by providing a run script
that is a slightly modified version of the start script that is
generated by the Zope installation.
Here is an example (newer Zope versions use INST_HOME, because PYTHONHOME
creates problems with Python 2):
#! /bin/sh exec 2>&1 echo "------- start by supervise -------" PATH=/command:$PATH:/usr/local/bin reldir=/web/zope umask 077 # running instance INSTANCE_HOME=`cd $reldir; pwd` export INSTANCE_HOME # used Zope-Version INST_HOME=/web/zope export INST_HOME # do event logging through stdout unset ZLOGFILE unset EVENT_LOG_FILE unset STUPID_LOG_FILE HOME=/web/zope export HOME exec setuidgid zope softlimit -m 200000000 python2.1 \ $INST_HOME/z2.py \ -X -a 127.0.0.1 -f '8021' -w '8081' \ -i 280 -Z 0 |
The -Z 0 switch prevents the start of Zope process manager. This way
Zope also doesn't detach from the controlling terminal and thus makes
control by the supervise process possible.
The setuidgid command sets the user to zope, so that Zope doesn't run with root rights.
I got a note from Sascha Gresk that on his FreeBSD 4.5 system Zope
didn't like
to run under setuidgid. He removed setuidgid and used the -u switch of
Zope's start script z2.py. I run Zope under FreeBSD 4.7 myself and there it works as expected.
The script makes it possible to separate the Zope core tree, additional products and the var directory. Just choose different values for INST_HOME, INSTANCE_HOME and HOME. This way you may use one Zope source tree for several zope instances and it's possible to change the Zope version of an instance simply by changing the value of INST_HOME.
The script is designed for a Zope server running behind an Apache and accessed by Apache's mod_proxy (ZServer/mod_proxy solution). If you need a public address, change the script to meet your needs.
You also need the run script for the logging. Because this log contains the messages that
are printed during Zope startup and in the case of errors we will call the logging
directory /web/logs/zopedebug. This is not the only logging that
Zope does - there is still the access logging. A setup for handling this is
in my logging article. Ensure that the logging disk has enough
space to hold your log data - because multilog is designed to never loose any
log data it will stop reading the pipe when the disk fills. Therefor also Zope
may stop working.
In the standard setup given here multilog creates a maximum of 1MB data.
Here is the necessary run script for logging with using the user
zopelog:
#!/bin/sh exec /usr/local/bin/setuidgid zopelog /usr/local/bin/multilog t /web/logs/zopedebug |
Let's say you unpacked your Zope tree into the directory /web/zope. The necessary setup is then done by the following steps (as root):
After Zope ist started the ps axf command under Linux or pstree under FreeBSD gives a tree like the following (parts removed):
207 ? S 0:54 /usr/local/bin/svscan /service 28359 ? S 0:00 \_ supervise zope 28466 ? S 0:07 | \_ /usr/bin/python /web/zope/z2.py -X ... 28360 ? S 0:00 \_ supervise log 28470 ? S 0:00 \_ /usr/local/bin/multilog t ... |
After the first access to ZServer the picture looks like this (only on Linux, pstree on FreeBSD doesn't show threads):
207 ? S 0:54 /usr/local/bin/svscan /service 28359 ? S 0:00 \_ supervise zope 28466 ? S 0:07 | \_ /usr/bin/python /web/zope/z2.py -X -a 127. ... 28476 ? S 0:00 | \_ /usr/bin/python /web/zope/z2.py -m -X ... 28477 ? S 0:00 | \_ /usr/bin/python /web/zope/z2.py -X ... 28478 ? S 0:00 | \_ /usr/bin/python /web/zope/z2.py -X ... 28479 ? S 0:00 | \_ /usr/bin/python /web/zope/z2.py -X ... 28480 ? S 0:00 | \_ /usr/bin/python /web/zope/z2.py -X ... 28360 ? S 0:00 \_ supervise log 28470 ? S 0:00 \_ /usr/local/bin/multilog t /web/logs/zopedebug |
Stopping Zope is done with the svc command. It is necessary to convince supervise not to restart Zope when it is shut down. This is done with the -d switch. It also sends a TERM signal to the supervised application to shut it down:
svc -d /service/zope |
You don't need to stop the logging process in this case. multilog will happily wait for data forever. If Zope is started again multilog will work like before.
Starting Zope again is done with the command svc -u /service/zope.
After changes of the Zope environment it is sometimes required to restart
Zope. One situation is the installation of a new product, another is
restarting Zope to reduce it's memory use.
Note that also in this case there is no need to do anything about
the log service.
When Zope runs under supervise it's easy: simply stop Zope by using the svc command with the -t switch. The supervise program of the daemontools package will notice this and start Zope again quickly. The logging process is not affected by this.
svc -t /service/zope |
Multilog switches logfiles automatically based on their size. While watching
a logfile with tail -f it may be possible that the
logfile is rotated and you don't get output anymore.
For watching the logfile you may use the GNU version of tail that has an option
to recognize a switched file. The conversion of the TAI64N timestamps is
done by Dan Bernsteins utility tai64nlocal.
The solution:
tail --follow=name /web/logs/zopedebug/current | tai64nlocal on FreeBSD: tail -f --follow=name /web/logs/zopedebug/current | tai64nlocal |
If you want to use damontools and multilog for Apache/Zope access logging, see my document Apache- and Zope logging with Daniel J. Bernsteins multilog for more information.