The future of zope.testing and Python 3
Yesterday I hinted on a post on zope.testing and Python 3, and here it is.
Almost all zope.something modules uses zope.testing for it’s tests. It does this in two ways, the testrunner and zope.testing.doctest. This means that to port zope.something to Python 3, you first need to port zope.testing to Python 3. Except of course, it uses zope.interface, and zope.interface uses zope.testing and you get a Catch 22. I solved this for zope.testing’s dependencies by making it possible for their tests to be run also with Distributes test runner. It’s not as advanced as the Zope test runner, most annoyingly there is no easy way to run just a subset of the tests, but it did the job. This also avoid the use of zc.buildout, which also hasn’t been ported to Python 3, and also depends on zope.testing, so in this way I could start bootstrapping the Python 3 port of the Zope Toolkit.
Extracting the test runner
But zope.testing also depends on zope.testing. Most specifically, it depends on Zopes test runner to run the tests, so much so that running them with Distributes test runner will fail. I did succeed in porting it to Python 3, but it was a pain, and I didn’t feel good about it. When looking at it again in December, I decided the port was such a hacky mess that I started over. One of the things I did that time was to build a custom test command for setup.py, so that I could use Zope’s test runner to run the tests. This was an improvement, but still messy. The final outcome of this was the idea to extract the test runner into it’s own module. I did this, and zope.testrunner was released in a beta yesterday. As you see I messed up the formatting of the documentation, and I also forgot to move zope.testing.exceptions to zope.testrunner, I’ll do that in a beta 2 soon. This means zope.testrunner will not depend on zope.testing at all. Also, this extraction means zope.testing will not depend on zope.testrunner. The tests that are left in zope.testing can be run with Distributes test runner.
The Zope community has been on the forefront of doctests. In fact, The Zope community has tended to overuse doctests, and used them instead of unittests, which is wrong and evil. Doctests should be for testing documentation, not documenting tests. The doctest module wasn’t included in the stdlib until Python 2.4, but Zope 3 was running on earlier versions. To solve this in the days before buildouts, zope.testing simply included it’s own copy of doctest.py. However, it has not been kept in sync with the stdlibs since 2004, and stdlibs doctest has many, many bugfixes that zope.tetsing.doctest doesn’t have. And, funnily, it’s also the other way around, zope.testing.doctest has both fixed some bugs, and even introduced some features that the standard doctest doesn’t have. However, porting zope.testing.doctest, with it’s bugs, to Python 3 would be ridiculous, not to mention difficult, but the two doctests has diverged enough to not be fully compatible.
After some discussion about this, we decided to simply drop zope.testing.doctest. It has been deprecated since Christmas. Since this will be a backwards incompatible version of zope.testing, we decided to also not have any BBB imports of zope.testing.testrunner or zope.testing.exceptions. We’ll release a version with these deprecated as well in the near future. After that I’ll merge the changes to trunk, and sometimes reasonably soon, hopefully in May, a zope.testing 4.0.0 will be released with Python 3 support. We’ll continue to maintain and release zope.testing 3.9 to the end of the year.
Migrating to zope.testing 4.0 and zope.testrunner 4.0
If you use the zope.testrunner, you most likely use it via zc.buildout and the zc.receipe.testrunner module. It will be updated to use zope.testrunner instead, and most likely you’ll not even notice that you are using zope.testrunner instead of zope.testing.testrunner, as initial tests indicate that they are fully compatible.
However, dropping zope.testing.doctest is a different matter, as there are some substantial differences.
First of all zope.testing.doctest has support for using code in footnotes as setup code. Stdlibs doctest doesn’t. If you need this feature you should migrate your doctests to use Manuel.
The doctests using zope.testing.doctest will inherit the TestCases globals. Stdlib doctest doesn’t do this. You’ll have to pass in these variables explicitly instead.
There is a bug in Stdlibs doctest. If one doctest returns a unicode object, all subsequent testresults will be converted to unicode as well. This is not a problem unless you then return a non-ascii string, when you’ll get a unicode decode error. To fix this, don’t return unicode, rather test for equality with the expected unicode string.
DOS linefeeds in Unix
If you have a doctest with DOS/Windows linefeeds in Unix, it will fail, complaining on inconsistent whitespace. To solve this, don’t have DOS linefeeds on Unix. Your version control system will handle this if you don’t mess it up.
REPORT_ONLY_FIRST_FAILURE and REPORT_xDIFF
Stdlibs doctest will not allow you to override reportflags in your test. So if you explicitly in the test set that the test should have a REPORT_NDIFF flag, for example, even if you tell the testrunner to use UDIFF, this will be ignored. That kinda makes sense, I suppose. However, REPORT_ONLY_FIRST_FAILURE is also a report flag and will also be ignored, so if you have a test where you set a diff flag, passing -1 to the test runner will be ignored, which is highly annoying.
These last three bugs have been reported with patches to Pythons bugtracker, but they are likely to be ignored. Only the last one is a problem in Python 3, btw. As a part of the Python 3 port of zope.testing I also converted zope.testing.doctest to a set of monkey patches of stdlibs doctest. Although it was decided to not keep these patches and just drop doctest, I created a product called zope.monkeypatches.doctest. It’s not released yet, and won’t be released unless these bugs turn out to be big real life problems. But they are there and will be preserved there until I’m sure these bugs aren’t big problems.