Google App Engine with Python and Pylons

Posted Tuesday, 21 April 2009 at 23:25 by Andrew Liu
Tagged: web development
Read more blogs...

I had tried a few goes at Google App Engine (GAE) a few times in the past, but today tried it with a bit more vigour. Surprisingly, it was worked a treat, so much as I got a simple application going. I expected the installation to fall over, and a few times it did, but that was because I was trying things that were different to the norm as I went along. For this exercise, I wanted to install Pylons as the app server on GAE, and the best source to do that is, of course, Ian Bicking, as he explains well in this post.

The first headache I encountered was that I had Python 2.6.1 installed. The appengine-monkey application only works on Python 2.5, so I had to install that on my server. See my tutorial on how I install Python, and you will get a good idea on how to install dual python installations.

So here's my take on installing GAE on Fedora 10.

1. Get appengine-monkey

I wanted to create a new application called 'webtop', so I installed into a directory called '/apps/webtop'. Of course, all this was done as the 'root' user.

$ cd /apps/webtop
$ svn checkout http://appengine-monkey.googlecode.com/svn/trunk/ appengine-monkey

2. Install Python 2.5

The standard installation, with a defined prefix, does well here.

$ cd /root

$ wget http://www.python.org/ftp/python/2.5.4/Python-2.5.4.tar.bz2

$ tar xvfj Python-2.5.4.tar.bz2

$ mv Python-2.5.4 /usr/local/src

$ cd /usr/local/src/Python-2.5.4

$ ./configure --prefix=/usr/local/Python-2.5.4

$ make

$ make install

$ cd /usr/local/bin

$ ln -s /usr/local/Python-2.5.4/bin/python2.5 python2.5

3. Get the Google Appengine

Download the appengine SDK source code.

$ cd /root
$ wget http://googleappengine.googlecode.com/files/google_appengine_1.2.0.zip
$ unzip google_appengine_1.2.0.zip
$ mv google_appengine /apps/webtop

I think my directory structure ended up different to Ian Bicking's directory structure, but it still worked for me.

4. Create the environment for GAE

Now we see why we grabbed appengine-monkey. This makes installation of GAE for Pylons fairly straight forward. The following creates a virtualenv environment for our new application.

cd /apps/webtop
python2.5 appengine-monkey/appengine-homedir.py --gae /apps/webtop/google_appengine webtop

Our directory structure is now as follows:

apps/
- webtop/ (holds the source directories)
  - appengine-monkey/ (appengine-monkey source)
  - webtop/ (your new application source)
  - google_appengine/ (GAE source)

Immediately after this, I activated the virtualenv. This is a must, otherwise you never know what (and where) you are installing features. This caught me out a few times, and I had to clean up my base installations after tainting them with unwanted third party modules.

$ cd /apps/webtop/webtop
$ ./bin/activate

(webtop)$

5. Install Pylons

Instead of using setuptools, a new installation tool is provided that works with GAE. Pip effectively replaces easy_install here. The homepage for Pip is located at http://pip.openplans.org/.

(webtop)$ cd /apps/webtop/webtop
(webtop)$ ./bin/pip install Pylons

6. Create the Pylons application

Now we physically create our pylons application. Interesting to note that pip is located in '/apps/webtop/webtop/bin', but paster is located in '/apps/webtop/webtop/app/bin'.

(webtop)$ cd /apps/webtop/webtop/app
(webtop)$ ./bin/paster create --template=pylons webtop sqlalchemy=false

The next step required moving directories around, and I started to get a little confused with the number of 'webtop' directories that I was using. So I'll try to be verbose. Now, we have the following directory structure, and highlight the important files.

apps/
- webtop/ (holds the source directories)
  - appengine-monkey/ (appengine-monkey source)
  - webtop/ (your new application source)
    - app/ (where our GAE application will reside)
      - bin/
        
        paster (used to install Pylons)
      - webtop/ (** NOTE 1 ** Lets call this directory 'webtopOne')
        
        webtop/ (** NOTE 2 ** Lets call this directory 'webtopTwo')
    - bin/
      - activate (script to activate the virtualenv)
      - pip (location of the pip installer)
      - python2.5 (our local python binary)
  - google_appengine/ (GAE source)

Now, you can see why 'webtop' is getting a bit confusing! Maybe its just me wanting to use the same name for the application. Anyway, the red directories above need a bit of TLC. Basically, we need to blow away webtopOne (except the child directory webtopTwo), and replace it with webtopTwo. If this is confusing, or if lines of code make more sense, try this out:

(webtop)$ cd /apps/webtop/webtop/app
(webtop)$ mv webtop/webtop apptemp
(webtop)$ rm -rf webtop
(webtop)$ mv apptemp webtop

Now our directory structure is as follows:

apps/
- webtop/ (holds the source directories)
  - appengine-monkey/ (appengine-monkey source)
  - webtop/ (your new application source)
    - app/ (where our GAE application will reside)
      - bin/
        
        paster (used to install Pylons)
      - webtop/ (** Formerly 'webtopTwo' **)
    - bin/
      - activate (script to activate the virtualenv)
      - pip (location of the pip installer)
      - python2.5 (our local python binary)
  - google_appengine/ (GAE source)

7. Edit environment.py

Edit the file in '/apps/webtop/webtop/app/webtop/config/environment.py, and remove the following line

    module_directory=os.path.join(app_conf['cache_dir'], 'templates'),

8. Edit config.py

Edit the file in '/apps/webtop/webtop/app/config.py, and replace the contents with this. The first line, you can replace the word 'webtop' with the name of your application that you have used.

APP_NAME = 'webtop.config.middleware:make_app'
APP_ARGS = ({},)
APP_KWARGS = dict()
APP_KWARGS.update({'beaker.session.type': 'google', 'beaker.session.table_name': 'beaker_session',
                   'beaker.session.key': 'pylonstestapp', 'beaker.session.secret': 'secret',
                   'beaker.cache.type': 'google', 'beaker.cache.table_name': 'beaker_cache'})
# You can overwrite these separately for different dev/live settings:
DEV_APP_ARGS = APP_ARGS
DEV_APP_KWARGS = APP_KWARGS
REMOVE_SYSTEM_LIBRARIES = ['webob']

9. Run the application

Now we can run the application. By default, it will run on localhost on port 8080.

(webtop)$ deactivate

$ cd /apps/webtop/google-appengine

$ python2.5 dev_appserver ../webtop/app

To upload the application to google:

$ python2.5 appcfg.py update ../webtop/app

nb. Quirks

a) Python 2.5 installation didnt work with --enable-shared. I kept getting the following error:

$ bin/paster create --template=pylons webtop sqlalchemy=false
Traceback (most recent call last):
  File "bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/apps/webtop/webtop/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/pkg_resources.py", line 277, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/apps/webtop/webtop/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/pkg_resources.py", line 2180, in load_entry_point
    return ep.load()
  File "/apps/webtop/webtop/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/pkg_resources.py", line 1913, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/apps/webtop/webtop/app/lib/python/paste/script/command.py", line 24, in <module>
    from paste.script.util.logging_config import fileConfig
  File "/apps/webtop/webtop/app/lib/python/paste/script/util/logging_config.py", line 30, in <module>
    import sys, logging, logging.handlers, string, socket, struct, os, traceback, types
  File "/usr/local/Python-2.5.4/lib/python2.5/logging/handlers.py", line 30, in <module>
    import sys, logging, socket, types, os, string, cPickle, struct, time, glob
ImportError: /apps/webtop/webtop/lib/python2.5/lib-dynload/cPickle.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

After much frigging around, I ended up blowing the Python 2.5 installation away, and reinstalling it WITHOUT the --enable-shared option. This fixed it up. The fact that the cPickle library was located in lib-dynload made me suspect the shared libraries were at fault. Of course, this seems strange, because I had Python 2.6 installed and running with --enable-shared, and importing cPickle worked fine there.