How to debug Zope/Plone with an IDE

This post gives a quick introduction on how to debug Filesystem Products in Zope/Plone with the following IDEs

1. Wing IDE Professional 2.0.3
2. Boa Constructor 0.4.4
3. Eclipse 3.1 with PyDev 0.98
4. Emacs 21.4.1 with pdb and ipython 0.6.15
5. jEdit 4.2final with JpyDbg 0.9.1

If anyone knows of any other suitable IDEs (or if I've left out features in the ones above), please feel free to let me know in the comment section at the bottom of this post. This post doesn't aim to say which IDE is better than another, just to show how to use them for debugging.
This post is a follow on from Team Development with Plone/ Zope/ ZEO/ Subversion/ ipython. You will need to run these commands to try out the IDE examples in this post.

mkdir ~/plonedev
cd ~/plonedev
wget -c http://homepage.eircom.net/~rachra/setupplonezeo.sh
wget -c http://homepage.eircom.net/~rachra/plonehelper.py
chmod +x setupplonezeo.sh
./setupplonezeo.sh 1 Acme

Running the setupplonezeo.sh script will leave you with the directory structure shown below

~/plonedev/    
    startPlone2.1-rc3.sh        # this script starts the zeo server on port 8100 and plone listening on port 8080
    startIPythonPlone2.1-rc3.sh # this script starts an IPython interpreter on your zeo_client1 (port 9080)
    debugPlone2.1-rc3.py        # a python script to start zeo_client0 (port 8080)
    plonehelper.py              # python helper script for creating a plone instance
                                # installing products and selecting a skin using the plone api
    myproducts/                 # this is where you develop your Zope/Plone products
    build/                      # the directory where you built Zope and Python
    src/                        # the directory where all downloads get stored
    zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/
        Python-2.3.5             # a full python installation including ipython
        Zope-2.8.1-final         # a full zope installation
        thirdpartyproducts/     # all 3rd party products for both zeo clients are stored (symlinked) here,
                                # you shouldn't edit these when developing as you may break your upgrade path
        zeo_server/             # the zeo server that contains the shared ZODB
        zeo_client0/            # the zeo client that will act as our webserver on port 8080
        zeo_client1/            # the zeo client that will act as our interactive debugger (port 9080)

Now we need a filesystem product to debug, we will use the MySite product (mandatory reading for filesystem product development). Among other things the MySite Product subclasses Products.CMFPlone.MembershipTool and modifies the addMember function to send an email to newly registered users (whether they ask for it or not). This causes an exception on a fresh Plone site with no MailHost set up. This is what we will debug below. Install the MySite product by running the following commands.

cd ~/plonedev/src
wget -c http://www.neuroinf.de/LabTools/MySite-0_5.tgz
cd ~/plonedev/myproducts
tar -zxf ~/plonedev/src/MySite-0_5.tgz
cp -R MySite-0_5/* .
rm -rf MySite-0_5
cd ~/plonedev
./startPlone2.1-rc3.sh

then

1. Browse to http://localhost:8080/Acme
2. Log in as the site manager. username:manager, password:manager
3. Click on preferences - Add/Remove Products
4. Select MySite from the "Products available for install" and click the "install" button (you should also add the PloneErrorReporting tool while you're at it)
5. The MySite filesystem product is now installed
6. Logout
7. Click the join button
8. Fill in some registration details and click register
9. You should be presented with this screen

If you've got this far, you're all setup to start testing the various IDE's

Wing IDE Professional 2.0.3

Works really well, out of the box with Zope 2.8.1 and Plone 2.1-rc3 check out the documentation at http://wingware.com/doc/howtos/quickstart
http://wingware.com/doc/howtos/zope
http://wingware.com/doc/howtos/plone
The WingDBG product was installed for you by setupplonezeo.sh above, but you can get it from

wget -c http://wingware.com/pub/wingide/2.0.3/WingDBG-2.0.3-2.tar

1. Download and intall the evaluation version of wingide-2.0.3
2. start wingide
3. Close any existing projects
4. Open a new project
5. Project - add directory tree. Use your zope instance

   ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0
   

   as the directory root (sit back and wait for a long time while wing analyzes all the source files for autocompletion purposes)
6. Project - Properties - Extensions - Enable Zope/Plone support - and add the INSTANCE_HOME of your zope instance (~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0). This will cause your /etc/zope.conf file to be analyzed and will prompt you to add additional directories to your project (very cool).
7. Start Zope/Plone

   ~/plonedev/startPlone2.1-rc3.sh
   

8. open up the ZMI http://localhost:8080/manage
9. add the Wing Debugger product to your root folder
10. Give it a name/id e.g. WingDebugger
11. Click the change button
12. Click the ok button
13. Click start debugger
14. Open the file ~/plonedev/myproducts/MySite/MembershipTool.py in the wingide
15. set a breakpoint in the addMember function
16. Browse to http://localhost:50080/Acme
17. Click join, fill in some random details and then click register
18. The wing ide should pop up and there you go, you're off debugging!
Try click on the stack trace and browse to (~/plonedev//lib/python/ZPublisher/)Publish.py, line 386, put a breakpoint here, you should get into the debugger on every plone request now

BoaConstructor

Provides several ways to debug Zope/Plone as listed in it's documentation, click on help and search for zope Install Boa debugger extensions

1. Download and install Boa-Constructor 0.4.4

   cd ~/plonedev/src
   wget -c http://heanet.dl.sourceforge.net/sourceforge/boa-constructor/boa-constructor-0.4.4.zip
   cd ~/plonedev
   unzip ~/plonedev/src/boa-constructor-0.4.4.zip
   cd boa-constructor-0.4.4
   python Boa.py
   

2. Install the Boa Debug Server extension

   cp boa-constructor-0.4.4/Debugger/BoaDebugServer.py zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0/Extensions/
   

3. Edit BoaDebugServer.py and replace c:/Path/To/Boa with the /home/YOURUSERNAME/plonedev/boa-constructor-0.4.4. You may also optionally replace the username, password passed to the start function for security. e.g.

      1:import sys
      2:
      3:# don't trace this module
      4:__traceable__ = 0
      5:
      6:def StartDebugServer():
      7:    sys.path.append('/home/YOURUSERNAME/plonedev/boa-constructor-0.4.4')
      8:
      9:    from Debugger.RemoteServer import start
     10:    start('', '') # username, password
     11:
     12:    return 'Debug Server started, attach to it from the IDE.'
     13:
     14:def HookDebugServer():
     15:    if hasattr(sys, 'debugger_control'):
     16:        sys.debugger_control.set_traceable()
     17:        sys.debugger_control.set_continue()
     18:    else:
     19:        raise Exception('Not running in the debugger.')
     20:
     21:    return 'Debug Server hooked, breakpoints now active.'
   

4. Start Zope/Plone

   ~/plonedev/startPlone2.1-rc3.sh
   

5. Open the ZMI and add the external methods
   • Id=StartDebugServer, Title=StartDebugServer, Module Name=BoaDebugServer, Function Name=StartDebugServer
   • Id=HookDebugServer, Title=HookDebugServer, Module Name=BoaDebugServer, Function Name=HookDebugServer
6. With a browser, execute the StartDebugServer method.
7. From the IDE open ~/plonedev/myproducts/MySkin/MembershipTool.py choose File->Attach to debugger.
8. Accept the defaults on the dialog box that pops up
9. Set a breakpoint in the addmember function on line 33 by double clicking in the gutter (you should see a red circle appear)
10. With a browser, execute the HookDebugServer method. The debugger is now hooked and ready and soft breakpoints should work.
11. Browse to http://localhost:8080/Acme/join_form and fill in any details, then click register
12. The browser should appear to hang, switch to Boa, you should be ready to step through the code from the breakpoint at line 33

There is also a BoaDebugger product which works well with Zope 2.7, however I could not get this product to work with Zope 2.8.

Emacs with pdb and IPython

This howto is directly lifted from http://docs.neuroinf.de/programming-plone/debug with a few small additions. First of all we need to setup emacs to syntax highlight and code complete python

1. If you haven't already got emacs install it e.g. on gentoo

   emerge emacs
   

2. We want python syntax highlighting and code completion in emacs, so we need to download the appropriate emacs modes, run the following shell commands to get the modes

   mkdir ~/elisp
   cd ~/elisp
   wget -c http://heanet.dl.sourceforge.net/sourceforge/python-mode/python-mode-1.0alpha.tar.gz
   tar -zxvf python-mode-1.0alpha.tar.gz
   wget -c http://scipy.net/cgi-bin/viewcvsx.cgi/*checkout*/ipython/doc/ipython.el?rev=HEAD&content-type=text/plain
   

3. edit the file ~/.emacs and add the following lines to it (if they aren't there already) to ensure your new python modes are installed

   (global-font-lock-mode t)
   (setq font-lock-maximum-decoration t)
   (add-to-list 'load-path "~/elisp")
   (load "~/elisp/python-mode-1.0alpha/python-mode.el")
   (setq auto-mode-alist (cons '("\\.py$" . python-mode) auto-mode-alist))
   (setq interpreter-mode-alist (cons '("python" . python-mode)  interpreter-mode-alist))
   (autoload 'python-mode "python-mode" "Python editing mode." t)
   (setq ipython-command "/usr/bin/ipython")
   (require 'ipython)
   (global-set-key [(f6)] 'ipython-complete)
   

The above commands are a once off emacs setup, you can test syntax highlighting and code completion in emacs as follows

1. open up a console and type

   emacs -nw
   

2. type

   <Alt> x py-shell
   

3. This will bring up the ipython shell, try this

   import os
   os. (press F6) you should see a list of completions (if you added the f6 keybinging command to your ~/.emacs file above)
   os.system?? - should give you the doc string sfor os.system
   

Now on to debugging our Zope filesystem product

1. edit the file ~/plonedev/myproducts/MySite/MembershipTool.py and add the line import pdb; pdb.set_trace() to the addMember function (line 33) e.g.

   20:class MembershipTool(BaseTool):
   21:    
   22:    meta_type = "Membership Tool"
   23:    title="my custom membership tool"
   24:
   25:    security = ClassSecurityInfo()
   26:
   27:    security.declarePrivate('addMember')
   28:    def addMember(self, id, password, roles, domains, properties=None):
   29:        '''
   30:        Adds a new member to the user folder.  Security checks will have
   31:        already been performed.  Called by portal_registration.
   32:        '''
   33:        import pdb; pdb.set_trace()
   34:        BaseTool.addMember(self, id, password, roles, domains, properties)
   35:
   36:        # plus my custom method
   37:        self.notifyAdmin(id,properties)
   38:
   39:    security.declarePrivate('notifyAdmin')
   40:    def notifyAdmin(self, member_id, properties):
   41:        """
   42:        Send email to site admin about the join
   43:        This assumes that the mailhost and email_from_address
   

2. from a console start the zeo server

   ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_server/bin/zeoctl start
   

3. start emacs in a console

   emacs -nw
   

4. open a shell in emacs

   <Alt> x shell
   

5. run zope from the shell in emacs

   cd ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0/bin
   ./runzope
   

6. Open your browser and browse to http://localhost:8080/Acme/join_form fill in some details, click register and ...
7. Switch to emacs, you should be looking at the python debugger in one buffer and some python source code in another buffer.
8. This pdb tutorial shows you what to do from here. In brief, at the pdb prompt:
   type "u" to get to move up the stack to your source code.
   typing "n" steps you through the code.
   typing "l" lists the code around the current line.
   typing "h" will show you the list of available commands.
   typing "h some_command" will show you help for that command.
   You can enter python variables and strings here to test their values.
   
9. emacs keeps the code synchronized with the stepping in pdb which is a very nice feature

PyDev 0.98 with eclipse 3.1

1. Download and install Eclipse 3.1, you will need to use Java 1.5 for PyDev
2. Install PyDev
3. Select Window - Preferences -PyDev - Python Interpreter - New and select ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/Python-2.3.5/bin/python
4. In eclipse, Add a simple project with ~/plonedev as the root directory, call the project plonedev, this will cause pydev to build your workspace for code completion purposes, this will take ages as there is alot of code (Python, Zope etc.)
5. edit the file ~/plonedev/myproducts/MySite/MembershipTool.py and add the line import pdb; pdb.set_trace() to the addMember function (line 33) as shown above
6. open the file debugPlone2.1-rc3.py in your editor, add a breakpoint to last line by double clicking on the margin
7. Right click on debugPlone2.1-rc3 in the Navigator view and select Debug As > Python Run
8. Eclipse will ask you do you want to switch to the Debug perspective (say yes), and you will be brought to a very well laid out debugging environment with the debugger stopped at the breakpoint you set
9. You can step in to your hearts content to see how Zope and Plone start up, but eventually, select Run - Resume to complete the server startup
10. Open your browser and browse to http://localhost:8080/Acme/join_form fill in some details, click register and ...
11. Switch to eclipse and you will see that the console has dropped in to the Python debugger (pdb)
12. Continue debugging as above with emacs, Unfortunately the source code in the editor doesn't follow pdb as emacs does, if anyone knows of a better way to do this, please let me know...

jEdit with JpyDbg

1. Download and install jEdit
2. Install Console Sidekick and JpyDbg, from jEdit, Plugins - Plugin Manager - Install (select all the plugins you need, very extensive list of excellent plugins (especially for xml)), note: it is important not to install the jython interpreter as this conflicts with JpyDbg in the sidekick structure browser.
3. Set up JpyDbg as described here
4. edit the file ~/plonedev/myproducts/MySite/MembershipTool.py (as shown above) and add the line import pdb; pdb.set_trace() to the addMember function (line 33)
5. Open the file ~/plonedev/debugPlone2.1-rc3.py in your editor
6. Utilities - Global Options - Docking, dock the Python Environment and console on the bottom, and dock the structure browser on the left, you should see the structure of your python file in the structure browser now
7. Click the Python Environment button at the bottom of the screen to view the python environment
8. Set a breakpoint on the last line of ~/plonedev/debugPlone2.1-rc3.py by clicking in the left margin
9. Click the green triangle button on the python environment to start debugging
10. Open your browser and browse to http://localhost:8080/Acme/join_form fill in some details, click register and ...
11. It is now possible to use the python debugger (pdb) in the python environment console, however newlines seem to be ignored and the source code does not follow the pdb as emacs does. You should however still be able to step through your code as described in this pdb tutorial

Summary

IDE	Free	Zope Integration
Wing IDE	No	Yes
Boa Constructor	Yes	Yes
Emacs	Yes	No
Eclipse with PyDev	Yes	No
jEdit	Yes	No