October 20, 2008

Zope 3 as a web application framework: an overview

We use Zope related software throughout our day at GateHouse Media. Specifically, we run a hosted solution, Zope4Media provided by Zope Corporation. Our level of involvement with the software on the development end of things is strictly limited to template development in ZPT.

We also build a lot of other generic web applications separate from our Zope4Media installations. Naturally, it makes sense for us to investigate the use of Zope 3 for use as our primary development environment for these other applications.

UPDATE: Aaron Lehmann of Zope Corporation was kind enough to respond to many of the points I make in this post. I've added his responses inline below.

What is Zope 3?

Zope 3 is the latest in the line of Zope-powered web application servers. It was released nearly 4 years ago on November 6th, 2004. It is a standalone server written in the excellent Python programming language that provides a framework for developing applications such as content management systems, intranets, and other dynamic web-based tools. It provides data storage in the form of the ZODB (Zope Object Database) which operates as transparent and persistent storage of Python objects.

Aaron Lehmann: Zope 3 is technically not a web server but a group of libraries that can be used together. While some of the packages handle web publishing, it is possible to write sophisticated applications in Zope 3 that have nothing to do with web publishing.

In fact, the heart and soul of Zope 3 is the Component Architecture, which allows behaviors to be loosely coupled to the data involved.

Zope 3, like other modern development platforms, follows a loose interpretation of the MVC (Model-View-Controller) design pattern. By nature of it being a Python application, it encourages clean and intelligently designed code.

The purpose of this blog post is to journey through the installation, configuration and development of a simple Zope 3 web application.

Installation

First things first, we need to download and install Zope 3. It must be installed and configured on the server before you may begin development.

Since I'm a Subversion junkie, I want to grab a checkout of the Zope 3 application, rather than download.

Aaron Lehmann: The best way to get hold of the packages you need is to make a buildout (analogous to a Makefile) and use zc.buildout to get the necessary dependencies. To ensure the packages you get all work together, you'll want to nail the versions. There is a known good set, which has been consistently inter-compatible. For a very quick start and a sample buildout, you might try the zopeproject egg, which can be gotten from PyPI.

For a less steep learning curve some folks recommend the Grok project, which is built on top of the Zope 3 libraries.

Head over to the Zope 3 website and look for their SVN repository. There's a link in the left sidebar for Zope SVN, so that's probably a good place to start. This page, however, doesn't seem to have any information on where the SVN repo lives, so we hit up the SVN FAQ. The first thing I see on this page is:

svn co svn://svn.zope.org/repos/main/Zope/trunk Zope

That looks to be exactly what we're after, so I go ahead and try to checkout. Unfortunately, this appears to checkout Zope 2, not Zope 3. There's also no indiciation on their FAQ that this is or isn't the case.

I'll save you the headache of trying to figure out just where Zope 3 lives on their repository, since it's not really documented anywhere. The SVN list of Zope 3 releases really lives here:

svn ls svn://svn.zope.org/repos/main/Zope3/tags

From there, you'll see a list of Zope 3 releases. We're interested in the latest release, which from what I can see from here, is 3.3.1. We grab that release by checking out with the following:

svn co svn://svn.zope.org/repos/main/Zope3/tags/3.3.1/ zope

The checkout will take a while. After it's finished, it looks like the entire checkout clocks in at around 111 MB. Quite large.

Now that we've got the latest version of Zope 3 in our hands, we need to install it on our system. Here's where I picked up Benji York's excellent Zope 3 quick start guide.

Following from Benji's tutorial, I run the installation command for Linux, since I'm on OS X:

python setup.py install_data --install-dir .

All went well with installation, so now's time to setup our first instance of Zope 3 (a standalone application):

python zope/bin/mkzopeinstance

The initial instances weighs in at 376k.

Running the server

Our Zope instances makes available to us the Zope 3 application server to host our instance. We do so with the following command from within the instance path:

bin/runzope

Your Zope 3 application should be accessible at http://localhost:8080.

At this point, I'll be completely honest with you. I wrote this tutorial in segments, meaning I wrote down my initial steps first, then iterate on the instructions and comments to increase accuracy. However, the second time around, when trying to run the server for the application I had already built, I got the following error:

ImportError: No module named zope.app.twisted.main

Now, I am completely certain I had a working application when I last touched it. That said, I decided to start fresh with the instructions above. I checked out a fresh copy of Zope 3, reinstalled, and attempted to create a new instance. This is what I get when trying to run 'mkzopeinstance':

ImportError: No module named _zope_proxy_proxy

I honestly have no clue what's going on here. I'm beginning to think the SVN location above is incorrect, but there is absolutely no (good) information on Zope.org to point me in the right direction. Even the official 3.3.1 release page has no reference of the SVN location.

Unfortunately, the list of releases in the Zope 3 SVN project makes it quite hard to figure out what release is correct:

3.2.2/3.2.3/3.3.0/3.3.0-zope.interface/3.3.0b2/3.3.0c1/3.3.1/3.3.2/3.4.0a1/3.4.0b1/3.4.0b2/3.4.0c1/Zope-3.1.0/Zope-3.1.0b1/Zope-3.1.0c1/Zope-3.1.0c2/Zope-3.1.0c3/Zope-3.2.0/Zope-3.2.0b1/Zope-3.2.0b2/Zope-3.2.0b3/Zope-3.2.1/Zope-3.3.0b1/Zope3-29921/

Naturally, '3.3.1/' appears to be (the most) correct.

Anyway, I'm going to continue with the tutorial assuming you finally got it working, somehow.

Zope Management Interface (ZMI)

The Zope Management Interface is the default interface you see when accessing your application for the first time. It goes without saying that this interface is incredibly intimidating to those who are using Zope 3 for the first time. Even after working with the interface a bit, I'm still not sure exactly what everything does, and why everything is there.

Aaron Lehmann: I agree somewhat with you about the Rotterdam Skin (the default ZMI Look and Feel). This is an example where getting a monolithic package works against you, as a custom-assembled application can avoid using it entirely.

That said, I'm sure everything is there for a reason. It'd be nice to have a more formal beginners documentation regarding exactly what everything in the ZMI does.

Aaron Lehmann: The ZMI is intended to be a way to manage all of one's configurations through the web. It's a philosophical holdover from Zope 2. We don't use it much in Z4M, but it does allow quite a lot of options for through the web control.

My biggest pet peeve with ZMI is that there's no way to log out of the application. I can successfully login using the credentials I provided when creating the instance, but there is no logout link anywhere. After a few tries at guessing what the logout URL might be, it turns out that it is:

http://localhost:8080/logout.html

That should come in handy.

Zope Object Database (ZODB)

As mentioned, Zope 3 stores data using the ZODB. From Wikipedia:

The Zope Object Database (ZODB) is an object-oriented database for transparently and persistently storing Python objects. It is included as part of the Zope web application server, but can also be used independently of Zope.

Essentially, this means data is read and written from the database transparently through the use of Python objects, which also means there is no need for SQL. Unfortunately, this also means data portability is somewhat affected, since it's not as easy to import or extract data from ZODB (at least, not apparently).

Aaron Lehmann: Non-trivial relational databases (and the software that interfaces with them) are almost invariably tightly coupled with the peculiarities of a particular RDBMS (DB-specific languages for triggers, custom data types, SQL extensions, etc.), so migrating from one system to another is not easy. The same can be said of ZODB (or any other object database). The end result is that most migrations from any database to another will demand technical sophistication regardless of the database being used.

Similarly, datamarting applications often denormalize their schema so that non-technical people can treat them as spreadsheets. The skill necesary to do that will also allow one to denormalize an ODB. In fact, some Zope 3 shops do just this, pushing their data to an unnormalized RDBMS. One might argue that the RDBMS is still better, because it is what they use for non-technical report needs, but I think it's important to note that no programmer uses an unnormalized database for his code.

There are many factors in choosing the best data storage technology for a particular system. The decision often comes down to programming speed; ZODB lets Python programmers create native Python objects that persist between program restarts. It doesn't get much easier than that.

Serving a static page

Applications have static pages, so the first thing on our list is to serve a static page with our new Zope 3 application. Should be fairly straightforward.

At this point, I'll turn over the mic to Benji, where he can walk you through the quick steps of how to get a static page up and running.

Aaron Lehmann: I think it's worth mentioning that Zope 3 is not optimized with static pages in mind, but rather for applications, which may be published on the web. In that context, the steps involved make code layering in an application much more straightforward. Really, if all you want to display is a static page, your best course is to use a more mainstream web server, such as Apache or Lighttp.

The basic steps are:

Creating the content object (the 'page')
Registering the content type (so Zope 3 knows it exists)
Creating and configuring a view to bind the content object to some method
Configure the security permissions on the view

The steps are quite simple, but it's not entirely apparent how exactly everything works. The most confusing part of the whole thing is the XML-based configuration files. In my opinion, using XML to manage the configuration for these objects rather than native Python configuration objects is not necessary. This is a hot topic in the Zope 3 community.

Aaron Lehmann: It's true that ZCML is opaque to learn and debug. I'm not personally a fan of it, but less because of the fact that it's XML, than because a lot of the directives make too many assumptions and are too elaborate in their implementations. Of course, you don't have to use any of the ZCML if you don't want to. It's all based on Python underneath, and if you want to understand what it is hiding from you, you can do it yourself.

After I created the object using Benji's suggestions, I found that I had to somehow 'enable' the object. This is done by selecting the object from the left menu and then naming the object.

When working with changes to your code, it should be noted that you need to restart the Zope 3 application server to reflect the new code.

Aaron Lehmann: As a clarifying note here, the need for a restart only applies to code that is loaded at server start time, typically Python code. Templates generally get loaded dynamically either out of the database or off of the disk, and so do not suffer from this restriction.

Templating

Templating in Zope 3 is accomplished with Zope Page Templates (ZPT), an XML-based templating system designed specifically for use with Zope. A simple example:

<div tal:attributes="class here/class; style string:border: 1px solid black;" tal:content="string:Hello World!"> Hello World!</div>

I'm very familiar with ZPT, since all of the template development work we do with Zope4Media is done with ZPT. Zope Corporation owns and maintains Zope4Media, which is based on Zope 3.

While ZPT is fairly straightforward and easy to learn, it presents several problems.

ZPT introduces business logic into templating with the allowance of Python code within page templates. Business logic should be separated from presentation logic.

Aaron Lehmann: ZPT allows Python expressions inside of templates. This doesn't set it apart from any other templating system, and it certainly doesn't mix business logic into presentation.

Business logic handles the modeling of real-world objects and managing how those objects are accessed and how they access one another. Since ZPT only allows very restricted expressions, ZPT can only modify things through the API exposed to it. While such an API allowing business logic in templates is possible, it would be a gross misuse of the system, and would fall under Python's "consenting adults" policy with regard to poor practices.

Writing business logic as well as presentation logic in XML is a nightmare. XML was never designed to be a logical scripting language, and thus, should not be used for making logical calculations to manipulate content.

Aaron Lehmann: I agree that writing business logic in ZPT would be extremely challenging and feel your pain about writing presentation logic in it as well.

I suspect that your difficulties stem less from ZPT as a framework than from the APIs you're working with specifically.

From what I understand, one of the major benefits to using ZPT is that it is XML-based, meaning WYSIWYG HTML editors should be able to render and modify the code without messing with the logic. That said, no professional template developer uses a WYSIWYG editor, in any fashion.

Aaron Lehmann: Not everyone who writes templates is a professional templater. Some people might find templating and front-end development to be difficult, and be glad to have a way to save themselves some of the difficulty of writing (X)HTML. ZPT does a fine job allowing them to use tools to generate the markup, while the template programmer works with the logic, at the expense of a slight complexity overhead for both. Unfortunately, when the designer and the templater are the same person, he suffers from the fact that he's got an unnessesary abstraction.

That said, I can agree ZPT can be painful to write. You may have a better way to do things, and the beauty of the Component Architecture is that you can plug it in and use it for your templates, if you develop with a Zope server.

Error pages

At some point in your application, you're going to break something, I promise. It's certainly useful to have a strong debug tool for development, but Zope 3 appears to be lacking in this area. When you break something in Zope 3, you may see an error page something like this:

A system error has occurred.

This is great for production, since you don't necessarily want retail users to see the innards of your code. In development, however, it's not useful at all. I understand there may be ways to display the full traceback to the retail page, but I'm not entirely sure how.

Aaron Lehmann: To show the traceback on the screen, you'll want to register a view on Exception that returns the traceback as HTML. An alternative is to look in the Error Log in the ZMI (Errors link in on the upper right portion of the page). It will allow you to adjust the settings on which errors get logged there, and view the latest ones.

Also useful is including the line 'import pdb; pdb.set_trace()' in your code where you suspect the error will manifest, and running the server in the foreground mode. This will let you step through the call from the point of your choosing, using Python's standard debugger. Caveat: This only works well if you run your servers in single threaded mode. If there are multiple threads, they will all output to the same terminal, which will make debugging more difficult.

Conclusion and pros vs. cons of Zope 3

While Zope 3 has long since been a viable option for developers looking for rapid MVC-style development framework, I don't necessarily agree with some of the approaches the platform takes on templating (ZPT) and data handling (ZODB).

I've come up with a few bullet points highlighting what I see are the pros and cons of Zope 3:

Pros

Zope 3 offers a tried and true development platform used in many high-load production environments.
There is a vast community of very intelligent programmers (though decreasing).

Cons

The vast majority of documentation and tutorials for Zope 3 are at least two years old, with some being over 4 years old.
The Zope 3 book itself is out date.
The Zope Management Interface (ZMI) is far too confusing for beginning developers.
ZPT introduces business logic into presentation logic.
ZODB lacks data portability.

Aaron Lehmann:

The worldwide Zope 3 community is quite robust, as is evidenced if one subscribes to the [Zope3-users](http://mail.zope.org/mailman/listinfo/zope3-users) and [zope-dev](http://mail.zope.org/mailman/listinfo/zope-dev) mailing lists. Zope 3 strives to allow a programmer to mix and match his components, so it's difficult to write a tutorial that uses all, or even most of the available components in a reasonable way without shoehorning them in for no purpose. There is a fine book, ["Web Component Development with Zope 3" by Philipp von Weitershausen](http://www.worldcookery.com), currently in its third edition, with a fourth on the way. The ZMI has its pros and cons: It allows one to manage applications through the web, at the expense of a slower start time, and many dependencies. We use it rarely in Z4M. While it is confusing for a new developer, it is powerful, particularly for more advanced programmers. But in all cases, it is not a requirement. ZPT does not allow statements, meaning there can be no data manipulation except through exposed python code. This renders writing business logic in ZPT impossible. ZPT is for presentation. It is also not required that you use it in your own projects. You may use another templating language, or no templating language. ZODB is just as portable as any other DBMS: data can be migrated from it to another database or schema via scripts which can be written in Python. Zope 3 is a vast a la carte selection of libraries which offers enormous flexibility and capabilities to do anything users can imagine. It is not a 'chef's special', so to speak. What it may not have in terms of simplicity of use, it trumps other frameworks in terms of its power, broad capability, and extensibility.