10. Deployment – CherryPy Essentials

Chapter 10. Deployment

Our final chapter will explain in the first section how to configure CherryPy-based applications, and then review different methods to deploy such an application through the use of Apache and lighttpd. Finally, we will review how to make your CherryPy-based application SSL enabled via the built-in CherryPy HTTP server, as well as by using Apache and lighttpd capabilities.

Configuration

While developing an application, you always need to parameterize it, so that it can be tuned as per the requirements of the hosting environment. For instance, the type of database used, PostgreSQL or MySQL, the directory in which the application resides, administrator contacts, etc.

There are different levels of configuration settings required in a web application like our photoblog:

  • Web server: Settings linked to the HTTP server

  • Engine: Settings associated with the engine hosting the application

  • Application: Settings our application will use

CherryPy—Web and Engine Configuration System

Since our application is using CherryPy, we will use the CherryPy configuration capabilities for the web server and the engine. CherryPy uses a configuration based on the syntax of the INI format defined by Microsoft.

The format of a CherryPy configuration file is as follows:

[section]
key = value

The main difference between the original INI format and the format used by CherryPy is the fact that values in the latter case are Python data types. For example:

[global]
server.socket_host = "localhost"
server.socket_port = 8080

With the exception of [global], the sections of configuration files match a requested URI path segment, as illustrated in the following example:

[/css/style.css]
tools.staticfile.on = True
tools.staticfile.file = "app.css"
tools.staticfile.root = "/var/www/photoblog/design/default/css"

When CherryPy tries to match the /css/style.css request, it will inspect the configuration settings for a matching section. If found, it will use the settings defined for that section.

Before we explain how CherryPy differentiates between the web server and the engine settings, let's see how the configuration settings can be defined in a Python dictionary instead. The following code snippet demonstrates the same settings:

{'/css/style.css': {'tools.staticfile.on': True,
'tools.staticfilE.file': "app.css" 'tools.staticfile.root':
"/var/www/photoblog/design/default/css"}}

Functionally, both methods will provide the same capabilities. Using a Python dictionary offers the advantage of residing within the code itself, and thus allows for more complex data types to be provided as values. Eventually, it is usually a matter of taste between the two options.

Now that we have presented how to declare configuration settings, let's see how to pass them to their components. CherryPy API is quite straight forward in that respect:

  • cherrypy.config.update (file or dictionary) is used to configure the CherryPy web server.

  • cherrypy.tree.mount (app, config file, or dictionary) is used to provide the settings for the mounted application.

  • The _cp_config attribute is bound to the page handlers, or to the class containing the page handlers and calls a controller defined as a dictionary (in which case, the settings are propagated by CherryPy to all the page handlers of that controller). It is used to pass the settings directly to where they will be needed.

We will review an example to understand how to use that API in our context:

import cherrypy
class Root:
@cherrypy.expose
def echo(self, some):
repeat = cherrypy.request.config.get('repeat', 1)
return some * repeat
echo._cp_config = {'repeat': 3}
if __name__ == '__main__':
http_conf = {'global': {'environment': 'production',
'server.socket_port': 9090,
'log.screen': True,
'log.error_file': 'error.log',
'log.access_file': 'access.log'}}
cherrypy.config.update(http_conf)
app0_conf = {'/echo': {'tools.response_headers.on': True,
'tools.response_headers.headers':
('Content-Type', 'text/plain')]}}
cherrypy.tree.mount(Root(), script_name='/app0',
config=app0_conf)
app1_conf = {'/echo': {'tools.gzip.on': True,
'repeat': 2}}
cherrypy.tree.mount(Root(), script_name='/app1',
config=app1_conf)
cherrypy.server.quickstart()
cherrypy.engine.start()

Let's see what we have done in our example:

  1. 1. First we declare an application with a page handler named echo. The purpose of this handler is to return the request body and repeat it as many times as defined by the configuration setting key repeat. To do so, we use the _cp_config attribute bound to the page handler. This value can also be passed from the main configuration dictionary. In that case, the value coming from the main dictionary takes precedence over the _cp_config attribute.

  2. 2. Next we declare the web server settings in a dictionary and then we call cherrypy.config.update() with that dictionary. Note that the use of the key, named global, is not compulsory when using a dictionary. CherryPy does interpret it exactly the same way; so the semantic equivalent of the previous example can be written as follows:

    http_conf = {'environment': 'production',
    'server.socket_port': 9090,
    'log.screen': True,
    'log.error_file': 'error.log',
    'log.access_file': 'access.log'}
    cherrypy.config.update(http_conf)
    
  3. 3. Finally we mount two applications on two distinct prefixes with two different configuration settings. It is important to notice that the key we use is the path to the page handler relatively to where the application is mounted. That is why we use /echo, and neither /app0/echo nor /app1/echo. This also means that configuration settings do not leak across mounted applications. CherryPy makes sure that each application receives only the settings it was declared with.

Note

It is a common mistake to pass configuration settings associated with the application to the cherrypy.config.update() method. This will not propagate the settings to the mounted application. You must use the config attribute of cherrypy.tree.mount() to get the expected behavior.

Photoblog Application Configuration System

Configuration settings of an application will not usually be passed through the CherryPy configuration system, which is at a lower level. An application would usually define entities from their domain level, store those values in a back-end storage along with the rest of its data, and ultimately provide a front-end interface to allow the administrator or a user to modify them.

The photoblog application will not go that far but will keep a fairly simple approach to providing configuration settings by using a pure INI file. We make this choice because in the photoblog application case the configuration settings will be simple, defined, and editable by the administrator of the application. We will therefore avoid the burden of developing a more complex solution than an INI file.

However, in order to simplify access to those settings, we will define a specific class that will turn the INI sections, keys, and values into a Python object:

from ConfigParser import ConfigParser
class Config(object):
def from_ini(self, filepath, encoding='ISO-8859-1'):
config = ConfigParser()
config.readfp(file(filepath, 'rb'))
for section in config.sections():
section_prop = Config()
section_prop.keys = []
setattr(self, section, section_prop)
for option in config.options(section):
section_prop.keys.append(option)
value = config.get(section, option).decode(encoding)
setattr(section_prop, option, value)

This class will simply go through the INI file and add attributes to the instance of the Config class on the fly. For instance, imagine you have the following INI file:

[app]
base_url = http://localhost:8080
copyright = Creative Commons Attribution-ShareAlike2.5 License
[storage]
host = localhost
dbname = photoblog
user = test
password = test
type = postgres

Using the above class, we can make the following modifications:

import config
photoblogconfiguringc = config.Config()
c.from_ini('application.conf')
dir(c)
['__class__', '__delattr__', '__dict__', '__doc__',
'__getattribute__', '__hash__', '__init__', '__module__'
'__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__str__', '__weakref__', 'app', 'storage']
c.app.copyright
u'Creative Commons Attribution-ShareAlike2.5 License'

As you can see, we have now modified the INI file into a tree of attributes bound to the instance of the Config class. The photoblog application will have one global instance of that class that will therefore be accessible from everywhere in the application.

In this section, we have briefly reviewed the ways to parameterize a CherryPy application using its built-in configuration system. We have also introduced a simple configuration system using an INI file format allowing application settings. This approach hence provides an easy way to mock up the passing of parameters, before moving towards a system-based database, which can be more demanding.

Deployment

Deploying a CherryPy-based application can be as easy as dropping the application in an environment, where all the required packages (CherryPy, Kid, simplejson, etc.) are available from the Python system path. However, in a shared web-hosted environment, it is quite likely that the CherryPy web server will reside behind a front-end server such as Apache or lighttpd, allowing the host provider to perform some filtering operations if needed, or for instance let that front end serve the static files in a more efficient fashion than CherryPy.

This section will present a few solutions to run a CherryPy application behind the Apache and lighttpd web servers.

Before explaining how to use CherryPy behind Apache or lighttpd, let's define a simple application that we will use throughout the example:

import.cherrypy
def setup_app():
class Root:
@cherrypy.expose
def index(self):
# Will return the hostname used by CherryPy and the remote
# caller IP address
return "Hello there %s from IP: %s " %
(cherrypy.request.base, cherrypy.request.remote.ip)
cherrypy.config.update({'server.socket_port': 9091,
'environment': 'production',
'log.screen': False,
'show_tracebacks': False})
cherrypy.tree.mount(Root())
if __name__ == '__main__':
setup_app()
cherrypy.server.quickstart()
cherrypy.engine.start()

As discussed earlier, there are several ways of deploying CherryPy-based applications. Now, we will discuss the different approaches to deployment.

Apache with mod_rewrite Module

The first solution you can review when running behind the Apache web server is to use the mod_rewrite module. This module allows you to define a set of rules that the module will analyze to transform incoming HTTP requests and re-dispatch them towards the back-end server.

In our example, we will make the following assumptions, which are in fact the requirements:

  • You run Apache 2.2.

  • You have access to the Apache configuration that can usually be found in the file named httpd.conf. You can also stop and restart the apache process. These requirements imply either that you have administrator rights on the machine or that you have a local installation of Apache to play with.

  • You will use the VirtualHost directive that allows encapsulating directives targeting only one particular host. This allows distinct hosts to be handled by one single instance of Apache.

  • We will also assume that you have myapp.com resolvable locally. To do so:

    Under Linux, add the following line to the /etc/hosts file:

    127.0.0.1 myapp.com myapp www.myapp.com

  • Your operating system should now resolve requests to the myapp.com host to your local environment.

Let's now explain how we must configure Apache:

  1. 1. Load the required Apache modules, as follows:

LoadModule rewrite_module modules/mod_rewrite.so
  • Note that you may need to provide the full path to the module itself in some environments.

  1. 2. Next we declare the VirtualHost, as follows:

# Create a virtual host in your apache configuration
# to handle requests for the myapp.com hostname
<VirtualHost 127.0.0.1:80>
ServerName myapp.com
ServerAlias www.myapp.com
# Where our application files reside
DocumentRoot /home/sylvain/photoblog
# What is our directory index by default
DirectoryIndex index.html
# Message to return when our CherryPy server is down and
# apache could not forward the request.
ErrorDocument 502 "Server down"
# mod_proxy magic
# First enable the mod_rewrite engine
RewriteEngine on
# Now we simply rewrite incoming requests URI so that they
# are proxied to our CherryPy web server
# http://myapp.com/archives/2006/10/12/my-article
# would become
# http://127.0.0.1:9091/archives/2006/10/12/my-article
RewriteRule ^(.*) http://127.0.0.1:9091$1 [P]
# Now define the format of the logs to be used by Apache
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\"" combined LogFormat
"%t %a %D %I %O %s %{Content-Type}o %{Host}i
\"%r\" \"%{Referer}i\"" host
CustomLog /home/sylvain/photoblog/access_myapp.log combined
Errorlog /home/sylvain/photoblog/error_myapp.log
</VirtualHost>
  1. 3. The next step is to stop and restart your Apache process so that these modifications are taken into account.

  2. 4. Then start your CherryPy application server.

The mod_rewrite module documentation explains in detail how to build rewriting rules. In the previous example, we defined the most generic one by mapping the request URI path to a new hostname.

When navigating to the URL http://myapp.com, you should now see the following message:

Hello there http://127.0.0.1:9091 from IP: 127.0.0.1

Now that we know how to map a host to our CherryPy application via Apache, we might want to retrieve the actual hostname and remote IP address instead of the local ones. The former is needed when generating links like:

link = "%s/%s" % (cherrypy.request.base, path)

There are two options to achieve this, as they are independent from each other:

  1. 1. Use the mod_proxy module of Apache to forward the host.

    • First you need to load the module like this (consult your documentation):

      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_http_module modules/mod_proxy_http.so
      
    • Add the following directive to VirtualHost:

      ProxyPreserveHost on
      
    • Restart Apache.

  2. 2. Use the CherryPy proxy tool as follows:

    • Add the following entry to your global configuration:

'tools.proxy.on': True
  • Restart your CherryPy application.

In both cases, you will now see the following message in your browser:

Hello there http://myapp.com from IP: 127.0.0.1

The IP address stays the same because the test is being done from the same machine where the server is being hosted, on the local interface.

Let's now explain how the previous recipe works. In the first case, by using the ProxyPreserveHost directive, we tell Apache to keep the HTTP header host field as it is and not to overwrite it with the local IP address. This means that CherryPy will receive the original value of the Host header.

In the second case, we tell CherryPy to look for specific headers set by Apache when doing proxy with the original hostnames. The default header looked up by CherryPy is X-Forwarded-Host.

Lighttpd with mod_proxy Module

Lighttpd is another popular and very efficient HTTP server. The previous section can be translated to lighttpd in a similar fashion using mod_proxy. Here is an example on how you can configure lighttpd to proxy incoming requests to a CherryPy server:

$HTTP["host"] == "myapp.com"
{
proxy.server = ( "" => (("host" => "127.0.0.1",
"port" => 8080)))
}

Add this to the lighttd.conf file and restart the server. When browsing to http://myapp.com, you will see the following message:

Hello there http://myapp.com from IP: 127.0.0.1

Apache with mod_python Module

In the year 2000, Gregory Trubetskoy released the first version of mod_python. It is a module for Apache allowing the Python interpreter to be embedded within the Apache server providing a bridge between the Apache web server and Python applications. One of the strengths of mod_python is that unlike CGI where each request requires a Python process to be launched mod_python does not have any such requirement. Therefore, it gives the opportunity to the developer to benefit from the persistence of the Python process started by Apache when running the module (keeping a pool of database connections for instance).

Before seeing how to configure Apache and mod_python, let's review what are the requirements:

  • Apache 2.2

  • mod_python 3.1.x or superior

We will assume that mod_python is properly installed in your environment.

Now let's explain how to configure mod_python to run a CherryPy-based application:

LoadModule python_module modules/mod_python.so
<Location "/">
PythonPath "sys.path + ['/home/sylvain/app']"
SetHandler python-program
PythonHandler cherrypy._cpmodpy::handler
PythonOption cherrypy.setup my_app::setup_app
PythonDebug On
</Location>

We will take you through the process sequentially:

  1. 1. First we load the mod_python module.

  2. 2. We define a location directive specifying what Apache should do to the request starting with "/".

  3. 3. Then we define several mod_python directives:

    • PythonPath extends the system path and makes sure that our application modules will be found. For instance, here the my_app.py module resides in /home/sylvain/app.

    • SetHandler indicates that all requests starting with the path provided in the location directive will be handled by mod_python.

    • PythonHandler sets the generic handler that will be in charge of generating the output to return to the user agent. We use the built-in mod_python handler provided by CherryPy.

    • PythonOption passes options to the generic handler. Here the option will be named cherrypy.setup and we bind it to the function setup_app that our application provides. We assume the application is saved in a Python module named my_app.py. The setup_app method must be the one mounting the application.

    • PythonDebug is enabled.

  4. 4. Finally, we modify the application as follows:

import cherrypy
def setup_app():
class Root:
@cherrypy.expose
def index(self):
return "Hello there %s from IP: %s " % \
(cherrypy.request.base,cherrypy.request.remote.ip)
cherrypy.tree.mount(Root())
cherrypy.engine.start(blocking=False)
  • The difference is that we start the CherryPy engine in a non-blocking mode so that the Python process started via mod_python does not hang.

Now you can stop and restart the Apache process and navigate to the http://myapp.com URL and you should see the following content:

Hello there http://myapp.com from IP: 127.0.0.1

mod_python with WSGI Application

In the previous approach, we used the built-in mod_python handler that works fine on the applications usually hosted by CherryPy. If your application respects the WSGI interface, you may want to use the ModPythonGateway handler (http://projects.amor.org/misc/wiki/ModPythonGateway) developed by Robert Brewer.

First let's see the CherryPy application in the my_app.py module:

import cherrypy
class Root:
@cherrypy.expose
def index(self):
return "Hello there %s from IP: %s " % (cherrypy.request.base,
cherrypy.request.remote.ip)
# Create an application respecting the WSGI interface
wsgi_app = cherrypy.Application(Root())
# This will be call on the first request
def setup_app(req):
cherrypy.engine.start(blocking=False)

Now, let's review how to configure Apache to use the ModPythonGateway handler:

<Location "/">
PythonPath "sys.path + ['/home/sylvain/app']"
SetHandler python-program
PythonHandler modpython_gateway::handler
PythonOption wsgi.startup my_app::setup_app
PythonOption wsgi.application my_app::wsgi_app
PythonOption wsgi.cleanup cherrypy::engine.stop
</Location>

Thanks to the ModPythonGateway handler, you can use the richness of WSGI-based middlewares within the power of the Apache server.

SSL

SSL (Secure Sockets Layer) can be supported in CherryPy-based applications natively by CherryPy. To enable SSL support, you must meet the following requirements:

  • Have the PyOpenSSL package installed in your environment

  • Have an SSL certificate and private key on the server

In the rest of this chapter, we will assume that you have installed PyOpenSSL properly. Let us explain how to generate a pair of private key and certificate. To achieve this, we will use OpenSSL, a common open-source implementation of the SSL specification.

Creating a Certificate and a Private Key

Let's deal with the certificate and the private key:

  1. 1. First we need a private key:

openssl genrsa -out server.key 2048

  1. 2. This key is not protected by a passphrase and therefore has a fairly weak protection. If you prefer providing a passphrase, you should issue a command like this:

openssl genrsa -des3 -out server.key 2048

  • The program will require a passphrase. If your version of OpenSSL allows you to provide an empty string, do so. Otherwise, enter a default passphrase and then remove it from the generated key as follows:

openssl rsa -in server.key -out server.key

  1. 3. Now we create a certificate as follows:

openssl req -new -key server.key -out server.csr

  1. 4. This process will request you to input some details. The previous step has generated a certificate but it is not yet signed by the private key. To do so, you must issue the following command:

openssl x509 -req -days 60 -in server.csr -signkey
server.key -out server.crt

The newly signed certificate will be valid for 60 days.

Note

Note that, as the certificate is not signed by a recognized authority such as VeriSign, your browser will display a pop up when accessing the application, so that the user can accept or reject the certificate.

Now, we can have a look at the different approaches for creating the certificate and the key.

Using the CherryPy SSL Support

Let's see how we can do it:

import cherrypy
import os, os.path
localDir = os.path.abspath(os.path.dirname(__file__))
CA = os.path.join(localDir, 'server.crt')
KEY = os.path.join(localDir, 'server.key')
def setup_server():
class Root:
@cherrypy.expose
def index(self):
return "Hello there!"
cherrypy.tree.mount(Root())
if __name__ == '__main__':
setup_server()
cherrypy.config.update({'server.socket_port': 8443,
'environment': 'production',
'log.screen': True,
'server.ssl_certificate': CA,
'server.ssl_private_key': KEY})
cherrypy.server.quickstart()
cherrypy.engine.start()

The key is to provide the server.ssl_certificate and server.ssl_private_key values to the global CherryPy configuration. The next step is to start the server; if everything went well, you should see the following message on your screen:

HTTP Serving HTTPS on https://localhost:8443/

By navigating to the application URL, you should see a message such as:

If you accept the certificate, you will be able to continue using the web application via HTTPS.

One caveat of the previous solution is that now your application cannot be reached via non-secured HTTP. Luckily CherryPy provides a fairly easy way to work around this problem by simply starting two HTTP servers at once. You can see how it is done:

import cherrypy
from cherrypy import _cpwsgi
from cherrypy import wsgiserver
import os, os.path
localDir = os.path.abspath(os.path.dirname(__file__))
CA = os.path.join(localDir, 'server.crt')
KEY = os.path.join(localDir, 'server.key')
def setup_app():
class Root:
@cherrypy.expose
def index(self):
return "Hello there!"
cherrypy.tree.mount(Root())
if __name__ == '__main__':
setup_app()
# Create a server which will accept HTTP requests
s1 = _cpwsgi.CPWSGIServer()
# Create a server which will accept HTTPS requests
SSLin CherryPys2 = _cpwsgi.CPWSGIServer()
s2.ssl_certificate = CA
s2.ssl_private_key = KEY
# Our first server uses the default CherryPy settings
# localhost, 8080. We thus provide distinct ones
# for the HTTPS server.
s2.bind_addr = ('localhost', 8443)
# Inform CherryPy which servers to start and use
cherrypy.server.httpservers = {s1: ('localhost', 8080),
s2: ('localhost', 8443)}
cherrypy.server.start()
cherrypy.engine.start()

Upon starting the application, you should now see the following lines on your screen:

HTTP Serving HTTPS on https://localhost:8443/

HTTP Serving HTTP on http://localhost:8080/

Your application will now be reachable via HTTP and HTTPS.

Using the lighttpd SSL Support

Setting SSL support for lighttpd is as simple as adding the following to the global configuration of lighttpd:

ssl.engine = "enable"
ssl.pemfile = "/home/sylvain/application/server.pem"

The server.pem file is the concatenation of the server.key and server.crt files that we have created before. For instance, under a UNIX System we issue the following command:

cat server.key server.crt > server.pem

By using those two lines and the proxy method, we have described in the previous section how to support SSL for the CherryPy application.

Note

Note, however, that the path between lighttpd and CherryPy will be HTTP not secured. SSL support will stop at the lighttpd level.

Using the Apache mod_ssl Support

This approach consists of using the mod_ssl module of Apache based on OpenSSL to handle the SSL exchange before forwarding the request to the CherryPy server, as we did with lighttpd.

To do so, you need to modify your Apache configuration as follows:

LoadModule ssl_module modules/mod_ssl.so
Listen 127.0.0.1:443

The first line loads the mod_ssl module. The second line requests Apache to listen for incoming socket connections on a given IP address on port 443 (which requires administrator rights).

Then, we modify VirtualHost, as follows:

<VirtualHost 127.0.0.1:443>
SSLEngine On
SSLCertificateFile /home/sylvain/application/server.crt
SSLCertificateKeyFile /home/sylvain/application/server.key
</VirtualHost>

Once you have restarted the Apache process, you should be able to navigate to the URL https://myapp.com.

Summary

In this chapter, we have reviewed a few possibilities to configure and deploy the CherryPy-based applications using common products such as Apache and lighttpd. We have also dealt with SSL support. These should give you enough to start with and adapt for your own environment and requirements.

However, deployment goes beyond setting up web servers and this chapter does not cover the discussion of pushing the code into the production environment, neither does it explain how to update the application once in production. This is out of the scope of this chapter and hence not been discussed.

Author's View

If you have read this book, I can only assume that you are interested in the CherryPy library as a candidate for your personal projects. However, my motive behind writing this book was two-fold. Firstly, I wanted to provide a solid reference for CherryPy 3 that could, well hopefully, fill the curiosity of developers using it and this is what I have tried to achieve in the first four chapters of the book.

Secondly, I wished to introduce you, my fellow reader to some of the different aspects of the development of web applications. I did not plan this book as a reference for all the subjects it gets onto, since it would have required ten other tomes. Instead, I have tried to provide you with some of the keys to make you understand that writing a web application is not any different from any other type of application in the process.

With that perspective in mind,Chapter 5 taught us that the persistent mechanism like relational databases could be abstracted, thanks to object-relational mapping like Dejavu, SQLObject, or SQLAlchemy. This is a fundamental concept that allows you to design your application in a relaxed fashion with regards to the manipulated data. Thereafter,Chapter 6 reminded us that a web application could not only serve HTML pages but also expose an API referred to as a web service. This API is precisely what transforms our web application into an actual provider of valuable services. Does it mean we should forget about the actual user experience and be shallow on the designing of the interface of our application? Obviously not, andChapters 7 and8 review the idea behind templating before moving to the additional feature of client-side scripting and Ajax. Eventually,Chapter 9 makes sure that we never forget that an application that has not been tested is a broken one, whileChapter 10 provides a few tips to deploy our application in common environments.

I hope this book will tell you a story of web application development that goes beyond CherryPy itself or any of the products introduced. A story that reminds us that there is no right or wrong but some paths that have already been explored might be good and could be trusted and sometimes they should be pushed even further.

As I have said before, I have not written this book as a reference but as an introduction. It is quite possible that you think there are alternatives or better ways to achieve some of the topics covered. In such a case, I would be pleased to discuss this with you on the CherryPy mailing-lists. If on the other hand you close this book and think about parts of its content, then I will reach my goal.

Note

Founded in 2003 by the original CherryPy creator, WebFaction is a reliable and affordable hosting provider for your CherryPy applications.

You can get an exclusive 20% discount by using the promo code "CHERRYPYBOOK" when you sign up with WebFaction, visit http://www.webfaction.com for more details.