Friday, January 7

Python unit test runners

As I do test-driven Python development almost every day, I care a lot about testing infrastructure.

Currently my development environment of choice is Vim + a terminal (for running PyUnit tests). It's rather low-tech, but it works reasonably well. I usually do not use Vim's QuickFix because it's a bit cumbersome to use (I do use quickfix in combination with idutils). I could outline two main inefficiencies of my current approach.

First, to run the tests I have to save the changed files (which I occasionally forget to do), then M-tab to the terminal window and use Up,Enter to run the previous command, which was most likely to run the tests. The latter step, which is now pretty much a reflex as I do it hundreds of times a day, does unexpected things when I forget that the previous command was something different. I have tried to improve upon things by using a shell snippet while true; do ./test -pv foo bar ; read; done, which requires only Enter to be pressed. I find it hard to drop the habit and still press Up,Enter most of the time. Well, at least unexpected things do not happen. Yet this is clearly a poor man's approach. It also makes changing the arguments to the test runner slightly harder.

Second, I am currently doing the jumping to errors manually. QuickFix could do it automatically, but it sometimes does not work as I would like it to, because sometimes I want to jump to a function higher up in the stack rather the one that raised the exception. Typing the line numbers in by hand is not that much trouble, but tedious.

The page that turned my attention into my testing routine inefficiencies is One Button Testing. It contains some interesting ideas. I especially liked the suggestion to use a smart test runner that automatically runs the tests when you save the code. Now I am thinking that, for starters, it might be quite easy to write a small script that would monitor a directory (either by polling or by utilising dnotify/inotify) and would invoke an ordinary text-mode test runner when a file changed.

I have thought up some features (in addition to the usual ones) that my imagined test runner should have:

  • GUI. Command-line interfaces are really powerful and great for scripting, but they require typing, which is exactly what we want to avoid. (I would not mind if the backend was a text-based test runner.)
  • Global keyboard shortcuts. Some examples: C-M-a to run all tests, C-M-l to only run tests that failed the last time, C-M-s to show and hide the test result window. All without having to switch away from the code editor.
  • Notification area support. It would be great to see without switching windows if a test run is in progress, whether the last run had failures or not, and other things.
  • File monitoring. This is the idea from OneButtonTesting. The critical thing here is to run the tests only for the module that has been updated. Otherwise we either have to run all the tests of a project (slow) or type in a specific subsystem to test (error prone and requires typing).
  • Integration with Vim and Emacs. As discussed, clicking on a traceback to jump to corresponding code in the editor would save some keystrokes. It would be great if it also did not require using the mouse (e.g., use Left/Right to switch between failed tests, Up/Down to walk between stack frames in a traceback, Enter to go to the highlighted code)

There are already quite a few test runners out there, and some of them already implement a few of the items in the above list. First, the text-only ones:

  • The Zope 3 test runner is a quite usable, and has some nifty features.
  • The SchoolTool test runner (download) has been written by Marius Gedminas, my colleague and a very bright Python hacker (if you are a Python programmer, make sure to have a look at his blog). I prefer this test runner over the Zope 3 one, because it is faster (demonstration) and neater.

There are also some GUI test runners out there already. In my opinion they are inferior to the text runners available, at least the ones I have tried. Here is a list:

  • The PyUnit test browser has integration with vim, idle and kate, but that's about it. It is hard to figure out, requires a few dependencies and in general looks unmaintained. Besides, it uses Tkinter, therefore it looks ugly.
  • The PyUnit GUI test runner (screenshot) is rather simple and not very useful. It uses Tkinter too.
  • Pycotine is for Macs, I have not tried it.
  • PyUnitOSX seems to be a port of the PyUnit GUI test runner to the Mac. I have not tried it either. Well, at least it looks nicer than the original.
  • The NUnit GUI test runner (screenshot) is not a Python test runner, but I wanted to mention it because it gets some things right. Notably, it sports file monitoring -- whenever you rebuild your project, the runner notices and resets status of all tests. Besides, it is not ugly. However, other things in my wishlist are missing.

As a sidenote, I would like to mention the std library by Armin Rigo and Holger Krekel, two top-notch Python hackers. However, it's buried in obscurity, I do not think it even has a web page. You can check it out from the CodeSpeak Subversion repository (here). The library makes unit tests more Pythonic by doing dissection of expressions (it provides values of context variables with the traceback). It also encourages use of the plain assert statement, which is so much cleaner than the Java-esque self.assertEquals and others. As assertEquals was primarily used so that the value of variables would be shown on failure, there is no need to use it if you use the std library. There are more cool ideas, go see for yourself.

I would implement the stuff I described myself, but I have quite a few unfinished matters already (I will cover them in other posts) and my conscience would not let me drop them and start a yet another project. Therefore, I am hoping that someone with too much free time and good Python skills will have a look at my wishlist, say "These are neat ideas!" and go on and implement them in a couple evenings. Make sure to let me know if you do. By the way, I have seen a pygtk-based GUI test runner by Marius Gedminas . I suspect that it is rather incomplete, but I would not be surprised if Marius, being so ingenious, came up with something really useful, as he has with the SchoolTool test runner.

Darn, another essay-post. Indeed, my original intent was to provide content, but this is too time-consuming to keep up. I will have to try to be more concise.


Tartley said...

Vim quickfix can be configured to show the entire traceback - they you get to choose which entry in the traceback to click on to go to. I don't know exactly how to do it, but I've seen it done. You're looking to set errorformat to something that recognises each whole test failure as a single, multi-line error message, including all the lines of traceback.

caiyan said...

nike shoes
pandora charms
coach factory outlet
birkenstock shoes
coach factory outlet
ralph lauren
cheap nfl jerseys
louis vuitton outlet store
michael kors outlet clearance
ralph lauren outlet online