Django on Windows: Run Celery as a Windows Service
In my previous post, I showed how to set up a Django project on a Windows Server to be served behind IIS. After setting up the server, the next thing we want with a Django application is to be able to run background and scheduled tasks, and Celery is the perfect tool for that.
On Windows, background processes are mostly run as Windows Services. Fortunately, Python for Windows Extensions (a.k.a pywin32) provides facilities to create a Windows Service.
I have packaged the related code for this post and the previous one in a project called django-windows-tools available on github and the cheese shop. To make it available for your application, simply install it with the command:
Configuring your project
To run Celery for your project, you need to install Celery and choose a Broker for passing messages between the Django application and the Celery worker processes.
Installation of celery is easy:
Then you add it to your
You can choose among several message brokers. I personnaly use a Windows port of Redis installed as a Windows Service. The advantage of Redis is that it can also be used as an in-memory database. In case you’re interested, you can find here a binay copy of my installation.
The configuration of Redis as Celery’s broker also occurs in the
Finally, you add the
django_windows_tools application to your project:
After the configuration, a
python manage.py syncdb will ensure that the
database of your project is up to date.
Enabling the service
The installed service is going to allow us to run in the backround arbitrary management commands related to our project.
With the application installed, on the root of your project, type the following command:
It will create two files in the root directory of your project .
will help you install, run and remove the Windows Service. It’s much like
manage.py for the service.
service.ini contains the list of management
commands that will be run by the Windows Service.
Configuring the service
A look at the
service.ini file gives us the following:
services section contains :
- The list of background commands to run in the
- The list of files to delete when refreshed or stopped in the
run directive contains only one command:
celeryd. If we look at the
corresponding section of the
ini file, we find:
command specifies the
manage.py command to run and
the parameters to the command.
So here the configurations tells us that the service, when started, will run a python process equivalent to the command line:
And that the
d:\logs\celery.log will be deleted between runs.
log sections defines a log file and logging level for the service process
Installing and removing the service
You need to have administrator privileges to install the service in the Windows Registry so that it’s started each time the machine boots. You do that with the following command:
--startup=auto parameter will allow the service to start automatically
when the server boots. You can check it has been installed:
It can be removed with the following commands:
Please ensure that the Server Manager is not running when you run this command, because in that case a complete removal of the service will need a server restart.
Starting and stopping the service
The service can be manually started and stopped with the following commands:
If everything went fine, the python processes should be there:
Along with the log files :
Running the Beat service
If you deploy your Django project on several servers, you probably want to have
Celery worker processes on each deployed machine but only one unique Beat
process for executing scheduled tasks. You can customize the
service.ini configuration file on that specific machine, but this is
incovenient if you are sharing files between machines, for instance.
The service provides the ability to have several
services sections in the same
configuration file for different host servers. The Windows Service will try to
find the section which name matches the name of the current server and will
fallback to the
services section if it does not find it. This allows you to
have a different behaviour for the service on different machines. In the
preceding configuration, you have one section, named
which adds the
celerybeat command to the
celeryd command. With this
configuration file, the service run on a machine named
BEATSERVER will run the
Celery beat service.
winservice_install facility provides a convenient option for choosing the
current machine as the Beat machine. Let’s try that :
service.py file will contain a section with the name of the current
Now, when run, the service will start a new python process:
And new log files for the beat service will be present:
Changes to the configuration
The Windows Service monitor changes to the
service.ini configuration file. In
case it is modified, the service does the following:
- Stop the background processes.
- Reread the configuration file.
- Start the background processes.
You may have seen in the
service.ini file the
It allows running the runserver command in a separate process. I you edit the
service.ini file and add
runserver to the
As soon as you save the file, you can make your browser point to
http://localhost:8000 and will obtain:
Running arbitrary commands
As shown in the preceding section, virtually any Django management command can
be run by the service at startup or each time the
service.ini file is
modified. You could imagine having a section: