Friday, September 02, 2005

How to debug Zope/Plone with an IDE

This post gives a quick introduction on how to debug Filesystem Products in Zope/Plone with the following IDEs
  1. Wing IDE Professional 2.0.3
  2. Boa Constructor 0.4.4
  3. Eclipse 3.1 with PyDev 0.98
  4. Emacs 21.4.1 with pdb and ipython 0.6.15
  5. jEdit 4.2final with JpyDbg 0.9.1
If anyone knows of any other suitable IDEs (or if I've left out features in the ones above), please feel free to let me know in the comment section at the bottom of this post. This post doesn't aim to say which IDE is better than another, just to show how to use them for debugging.
This post is a follow on from Team Development with Plone/ Zope/ ZEO/ Subversion/ ipython. You will need to run these commands to try out the IDE examples in this post.
mkdir ~/plonedev
cd ~/plonedev
wget -c http://homepage.eircom.net/~rachra/setupplonezeo.sh
wget -c http://homepage.eircom.net/~rachra/plonehelper.py
chmod +x setupplonezeo.sh
./setupplonezeo.sh 1 Acme
Running the setupplonezeo.sh script will leave you with the directory structure shown below
~/plonedev/    
    startPlone2.1-rc3.sh        # this script starts the zeo server on port 8100 and plone listening on port 8080
    startIPythonPlone2.1-rc3.sh # this script starts an IPython interpreter on your zeo_client1 (port 9080)
    debugPlone2.1-rc3.py        # a python script to start zeo_client0 (port 8080)
    plonehelper.py              # python helper script for creating a plone instance
                                # installing products and selecting a skin using the plone api
    myproducts/                 # this is where you develop your Zope/Plone products
    build/                      # the directory where you built Zope and Python
    src/                        # the directory where all downloads get stored
    zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/
        Python-2.3.5             # a full python installation including ipython
        Zope-2.8.1-final         # a full zope installation
        thirdpartyproducts/     # all 3rd party products for both zeo clients are stored (symlinked) here,
                                # you shouldn't edit these when developing as you may break your upgrade path
        zeo_server/             # the zeo server that contains the shared ZODB
        zeo_client0/            # the zeo client that will act as our webserver on port 8080
        zeo_client1/            # the zeo client that will act as our interactive debugger (port 9080)
Now we need a filesystem product to debug, we will use the MySite product (mandatory reading for filesystem product development). Among other things the MySite Product subclasses Products.CMFPlone.MembershipTool and modifies the addMember function to send an email to newly registered users (whether they ask for it or not). This causes an exception on a fresh Plone site with no MailHost set up. This is what we will debug below. Install the MySite product by running the following commands.
cd ~/plonedev/src
wget -c http://www.neuroinf.de/LabTools/MySite-0_5.tgz
cd ~/plonedev/myproducts
tar -zxf ~/plonedev/src/MySite-0_5.tgz
cp -R MySite-0_5/* .
rm -rf MySite-0_5
cd ~/plonedev
./startPlone2.1-rc3.sh
then
  1. Browse to http://localhost:8080/Acme
  2. Log in as the site manager. username:manager, password:manager
  3. Click on preferences - Add/Remove Products
  4. Select MySite from the "Products available for install" and click the "install" button (you should also add the PloneErrorReporting tool while you're at it)
  5. The MySite filesystem product is now installed
  6. Logout
  7. Click the join button
  8. Fill in some registration details and click register
  9. You should be presented with this screen
If you've got this far, you're all setup to start testing the various IDE's

Wing IDE Professional 2.0.3

Works really well, out of the box with Zope 2.8.1 and Plone 2.1-rc3 check out the documentation at http://wingware.com/doc/howtos/quickstart
http://wingware.com/doc/howtos/zope
http://wingware.com/doc/howtos/plone
The WingDBG product was installed for you by setupplonezeo.sh above, but you can get it from
wget -c http://wingware.com/pub/wingide/2.0.3/WingDBG-2.0.3-2.tar
  1. Download and intall the evaluation version of wingide-2.0.3
  2. start wingide
  3. Close any existing projects
  4. Open a new project
  5. Project - add directory tree. Use your zope instance
    ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0
    
    as the directory root (sit back and wait for a long time while wing analyzes all the source files for autocompletion purposes)
  6. Project - Properties - Extensions - Enable Zope/Plone support - and add the INSTANCE_HOME of your zope instance (~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0). This will cause your /etc/zope.conf file to be analyzed and will prompt you to add additional directories to your project (very cool).
  7. Start Zope/Plone
    ~/plonedev/startPlone2.1-rc3.sh
    
  8. open up the ZMI http://localhost:8080/manage
  9. add the Wing Debugger product to your root folder
  10. Give it a name/id e.g. WingDebugger
  11. Click the change button
  12. Click the ok button
  13. Click start debugger
  14. Open the file ~/plonedev/myproducts/MySite/MembershipTool.py in the wingide
  15. set a breakpoint in the addMember function
  16. Browse to http://localhost:50080/Acme
  17. Click join, fill in some random details and then click register
  18. The wing ide should pop up and there you go, you're off debugging!
  19. Try click on the stack trace and browse to (~/plonedev//lib/python/ZPublisher/)Publish.py, line 386, put a breakpoint here, you should get into the debugger on every plone request now

BoaConstructor

Provides several ways to debug Zope/Plone as listed in it's documentation, click on help and search for zope Install Boa debugger extensions
  1. Download and install Boa-Constructor 0.4.4
    cd ~/plonedev/src
    wget -c http://heanet.dl.sourceforge.net/sourceforge/boa-constructor/boa-constructor-0.4.4.zip
    cd ~/plonedev
    unzip ~/plonedev/src/boa-constructor-0.4.4.zip
    cd boa-constructor-0.4.4
    python Boa.py
    
  2. Install the Boa Debug Server extension
    cp boa-constructor-0.4.4/Debugger/BoaDebugServer.py zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0/Extensions/
    
  3. Edit BoaDebugServer.py and replace c:/Path/To/Boa with the /home/YOURUSERNAME/plonedev/boa-constructor-0.4.4. You may also optionally replace the username, password passed to the start function for security. e.g.
       1:import sys
       2:
       3:# don't trace this module
       4:__traceable__ = 0
       5:
       6:def StartDebugServer():
       7:    sys.path.append('/home/YOURUSERNAME/plonedev/boa-constructor-0.4.4')
       8:
       9:    from Debugger.RemoteServer import start
      10:    start('', '') # username, password
      11:
      12:    return 'Debug Server started, attach to it from the IDE.'
      13:
      14:def HookDebugServer():
      15:    if hasattr(sys, 'debugger_control'):
      16:        sys.debugger_control.set_traceable()
      17:        sys.debugger_control.set_continue()
      18:    else:
      19:        raise Exception('Not running in the debugger.')
      20:
      21:    return 'Debug Server hooked, breakpoints now active.'
    
  4. Start Zope/Plone
    ~/plonedev/startPlone2.1-rc3.sh
    
  5. Open the ZMI and add the external methods
    • Id=StartDebugServer, Title=StartDebugServer, Module Name=BoaDebugServer, Function Name=StartDebugServer
    • Id=HookDebugServer, Title=HookDebugServer, Module Name=BoaDebugServer, Function Name=HookDebugServer
  6. With a browser, execute the StartDebugServer method.
  7. From the IDE open ~/plonedev/myproducts/MySkin/MembershipTool.py choose File->Attach to debugger.
  8. Accept the defaults on the dialog box that pops up
  9. Set a breakpoint in the addmember function on line 33 by double clicking in the gutter (you should see a red circle appear)
  10. With a browser, execute the HookDebugServer method. The debugger is now hooked and ready and soft breakpoints should work.
  11. Browse to http://localhost:8080/Acme/join_form and fill in any details, then click register
  12. The browser should appear to hang, switch to Boa, you should be ready to step through the code from the breakpoint at line 33
There is also a BoaDebugger product which works well with Zope 2.7, however I could not get this product to work with Zope 2.8.

Emacs with pdb and IPython

This howto is directly lifted from http://docs.neuroinf.de/programming-plone/debug with a few small additions. First of all we need to setup emacs to syntax highlight and code complete python
  1. If you haven't already got emacs install it e.g. on gentoo
    emerge emacs
    
  2. We want python syntax highlighting and code completion in emacs, so we need to download the appropriate emacs modes, run the following shell commands to get the modes
    mkdir ~/elisp
    cd ~/elisp
    wget -c http://heanet.dl.sourceforge.net/sourceforge/python-mode/python-mode-1.0alpha.tar.gz
    tar -zxvf python-mode-1.0alpha.tar.gz
    wget -c http://scipy.net/cgi-bin/viewcvsx.cgi/*checkout*/ipython/doc/ipython.el?rev=HEAD&content-type=text/plain
    
  3. edit the file ~/.emacs and add the following lines to it (if they aren't there already) to ensure your new python modes are installed
    (global-font-lock-mode t)
    (setq font-lock-maximum-decoration t)
    (add-to-list 'load-path "~/elisp")
    (load "~/elisp/python-mode-1.0alpha/python-mode.el")
    (setq auto-mode-alist (cons '("\\.py$" . python-mode) auto-mode-alist))
    (setq interpreter-mode-alist (cons '("python" . python-mode)  interpreter-mode-alist))
    (autoload 'python-mode "python-mode" "Python editing mode." t)
    (setq ipython-command "/usr/bin/ipython")
    (require 'ipython)
    (global-set-key [(f6)] 'ipython-complete)
    
The above commands are a once off emacs setup, you can test syntax highlighting and code completion in emacs as follows
  1. open up a console and type
    emacs -nw
    
  2. type
    <Alt> x py-shell
    
  3. This will bring up the ipython shell, try this
    import os
    os. (press F6) you should see a list of completions (if you added the f6 keybinging command to your ~/.emacs file above)
    os.system?? - should give you the doc string sfor os.system
    
Now on to debugging our Zope filesystem product
  1. edit the file ~/plonedev/myproducts/MySite/MembershipTool.py and add the line import pdb; pdb.set_trace() to the addMember function (line 33) e.g.
    20:class MembershipTool(BaseTool):
    21:    
    22:    meta_type = "Membership Tool"
    23:    title="my custom membership tool"
    24:
    25:    security = ClassSecurityInfo()
    26:
    27:    security.declarePrivate('addMember')
    28:    def addMember(self, id, password, roles, domains, properties=None):
    29:        '''
    30:        Adds a new member to the user folder.  Security checks will have
    31:        already been performed.  Called by portal_registration.
    32:        '''
    33:        import pdb; pdb.set_trace()
    34:        BaseTool.addMember(self, id, password, roles, domains, properties)
    35:
    36:        # plus my custom method
    37:        self.notifyAdmin(id,properties)
    38:
    39:    security.declarePrivate('notifyAdmin')
    40:    def notifyAdmin(self, member_id, properties):
    41:        """
    42:        Send email to site admin about the join
    43:        This assumes that the mailhost and email_from_address
    
  2. from a console start the zeo server
    ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_server/bin/zeoctl start
    
  3. start emacs in a console
    emacs -nw
    
  4. open a shell in emacs
    <Alt> x shell
    
  5. run zope from the shell in emacs
    cd ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/zeo_client0/bin
    ./runzope
    
  6. Open your browser and browse to http://localhost:8080/Acme/join_form fill in some details, click register and ...
  7. Switch to emacs, you should be looking at the python debugger in one buffer and some python source code in another buffer.
  8. This pdb tutorial shows you what to do from here. In brief, at the pdb prompt:
    type "u" to get to move up the stack to your source code.
    typing "n" steps you through the code.
    typing "l" lists the code around the current line.
    typing "h" will show you the list of available commands.
    typing "h some_command" will show you help for that command.
    You can enter python variables and strings here to test their values.
  9. emacs keeps the code synchronized with the stepping in pdb which is a very nice feature

PyDev 0.98 with eclipse 3.1

  1. Download and install Eclipse 3.1, you will need to use Java 1.5 for PyDev
  2. Install PyDev
  3. Select Window - Preferences -PyDev - Python Interpreter - New and select ~/plonedev/zeo_py2.3.5_zo2.8.1-final_pl2.1-rc3/Python-2.3.5/bin/python
  4. In eclipse, Add a simple project with ~/plonedev as the root directory, call the project plonedev, this will cause pydev to build your workspace for code completion purposes, this will take ages as there is alot of code (Python, Zope etc.)
  5. edit the file ~/plonedev/myproducts/MySite/MembershipTool.py and add the line import pdb; pdb.set_trace() to the addMember function (line 33) as shown above
  6. open the file debugPlone2.1-rc3.py in your editor, add a breakpoint to last line by double clicking on the margin
  7. Right click on debugPlone2.1-rc3 in the Navigator view and select Debug As > Python Run
  8. Eclipse will ask you do you want to switch to the Debug perspective (say yes), and you will be brought to a very well laid out debugging environment with the debugger stopped at the breakpoint you set
  9. You can step in to your hearts content to see how Zope and Plone start up, but eventually, select Run - Resume to complete the server startup
  10. Open your browser and browse to http://localhost:8080/Acme/join_form fill in some details, click register and ...
  11. Switch to eclipse and you will see that the console has dropped in to the Python debugger (pdb)
  12. Continue debugging as above with emacs, Unfortunately the source code in the editor doesn't follow pdb as emacs does, if anyone knows of a better way to do this, please let me know...

jEdit with JpyDbg

  1. Download and install jEdit
  2. Install Console Sidekick and JpyDbg, from jEdit, Plugins - Plugin Manager - Install (select all the plugins you need, very extensive list of excellent plugins (especially for xml)), note: it is important not to install the jython interpreter as this conflicts with JpyDbg in the sidekick structure browser.
  3. Set up JpyDbg as described here
  4. edit the file ~/plonedev/myproducts/MySite/MembershipTool.py (as shown above) and add the line import pdb; pdb.set_trace() to the addMember function (line 33)
  5. Open the file ~/plonedev/debugPlone2.1-rc3.py in your editor
  6. Utilities - Global Options - Docking, dock the Python Environment and console on the bottom, and dock the structure browser on the left, you should see the structure of your python file in the structure browser now
  7. Click the Python Environment button at the bottom of the screen to view the python environment
  8. Set a breakpoint on the last line of ~/plonedev/debugPlone2.1-rc3.py by clicking in the left margin
  9. Click the green triangle button on the python environment to start debugging
  10. Open your browser and browse to http://localhost:8080/Acme/join_form fill in some details, click register and ...
  11. It is now possible to use the python debugger (pdb) in the python environment console, however newlines seem to be ignored and the source code does not follow the pdb as emacs does. You should however still be able to step through your code as described in this pdb tutorial

Summary

IDEFreeZope Integration
Wing IDENoYes
Boa ConstructorYesYes
EmacsYesNo
Eclipse with PyDevYesNo
jEditYesNo

4 comments:

Anonymous said...

Just tried out debugging Zope with Eclipse/PyDev and got stuck on step 6. I don't know where to find a file such as debugPlone*.py.
Think this is some startup script for the zope instance. Do I have to write it myself? Is there some template around?

thanks, fRiSi

Michael said...

Hi Frisi, if you run the script at http://homepage.eircom.net/~rachra/setupplonezeo.sh and all goes well you should have a file like debugPlone*.py, if you don't want to run this script you can see the necessary python code at the bottom of the script HTH, Michael

Anonymous said...

Great info, thanks. Do you have a link to your .emacs file and the files it loads? If so, i'd love to see it. Thanks.

Michael said...

Hi Bryan,
Here ya go, http://homepage.eircom.net/~rachra/emacsfiles.tar.gz