Skip to content

Why Python 2.6 and 3.0 compatibility would be a Very Good Thing.

March 22, 2008

This was written before Python 3 was released. For information on how things turned out in practice see my book Porting to Python 3 which contains practical information on how to achieve Python 2 and 3 compatibility.

This is a followup to my previous posts on the subject of Python 3 and compatibility, that even ended up on the reddit frontpage, and based on an answer to a question on the python-3000 list. The question basically is why 2to3 code conversion isn’t good enough.

The recommended path for porting Python code to Python 3.0 is to use a tool called 2to3 to convert the code from Python 2.x code to 3.x code. If you need to support both 2.x and 3.0, you should maintain the code as 2.x code, and convert it to 3.x code and test it before you make a new release, and then make two separate releases, one for 2.x and one for 3.0.

That use case works for when you have code that is maintained by a small set of maintainers and regularly released to a large set of users who do not have access to the code repository. That is an ideal situation that isn’t always fulfilled, and is a strategy well optimized for those who develop library type modules, but not for others.

In many other cases, this is not how code is developed. Both within larger organisations and within large communities like Zope and Plone (and I suspect the same is true for communities such as Turbogears and Django), many people are instead working on the same code base at the same time. That codebase is split up into many interconnected modules, and may not always have a very well defined API, and quite often doesn’t get regular releases. What is probably the best (or should I say worst) example of this is the “Plone collective”, a large set of third-party modules for Plone.

So, there are many people working on the same large set of modules. All are using direct svn checkouts, because during development of their product/site they need a module and they discover bugs or add features to this module. They also have wildly varying experience levels and python knowledge.

Either all of these people and modules need to switch to 3.0 at the same time, which is unrealistic, or some will run 2.6 and some will run 3.0, and that means that the modules need to support both 2.6 and 3.0. The 2to3 strategy in that case means that everybody that wants to go over to 3.0 needs to have two complete development environments for every instance of the software that they need. Basically, for every website they develop, they would need one instance that runs under 2.6 and one under 3.0. This even though most of the code developed only would run under 3.0 or 2.6, as it’s custom for that particular website. The developer would then have to check out the 2.6 code in the 2.6 site, change it there, and test the change, and then run 2to3 copying over the code to the 3.0 instance, and test it there. Basically, any sort of code change, no matter how minor, requires a “change -> test -> copy -> 2to3 -> test” dance, instead of the normal “change -> test”. This doubles the effort of making any change. And since testing on the web often means testing with some sort of browser (zope.testbrowser or Selenium or the like) this means restarting a webserver before running the tests. Running tests this was often takes a significant time.

Debugging the code makes things even more complicated. You’ll need to set the debug in the 3.0 copy of the code, then make a change, and copy the code over. This means that the pdb you set is gone, and you have to set it again. In short, using 2to3 means that you have to continually port your code to 3.0. You write it for 2.6, but every change you make you have to port to 3.0. Porting to 3.0 becomes a job that you will have to repeat for every single code-snippet you write, even if that codesnippet turns out to be the wrong solution and you end up reverting it five minutes later.

The end result of this is that people will not move to Python 3 as long as they need any sort of third-party product from the Plone collective or similar set of modules, because it is going to be too much work. So everybody will stay on 2.x. Which means there is no incentive to port the third-party products to 3.0. It becomes a chicken and egg, or a catch 22 problem.

In short, the 2to3 way of supporting both 2.6 and 3.0 works only if you have a well defined API with good testing and a limited access of developers that all can be expected to have a good code discipline. Now, in many cases, that simply will not be the case, and dual support of both 2.6 and 3.0 will not happen is this is the only available path. This leaves you with the choice of going to 3.0 in one big step, or staying with 2.x forever. And when you have hundreds of developers all over the globe, there is no way they are going to move to 3.0 together. End result: They’ll stay on 2.6. Forever.

In the best case this means that Python 3 dies and nobody uses it. Yes, that is the best case. It’s a horrible case, I agree. But the worst case is that the community splits in two, and that will be dangerous for Python as a whole. Python may survive being split into two communities, but it would be negative for the community as a whole, and seriously affect Pythons general acceptance. Zope had this problem with Zope 2 and Zope 3.

So therefore, having so much forwards compatibility *and* backwards compatibility in Python 2.6 that we can run a reasonable set of Python code in both 2.6 and 3.0 would be a Very Good Thing. Because that enables us to either make a clean break by running 2to3 once, and dropping 2,x support, or supporting both by writing compatible code (which is a bit of extra effort and will not perhaps not always work) and supporting both by writing 2.6 code and using 2to3 to support 3.0, which is a lot of extra effort, but will work in almost all cases.

Now, before we all get into doomsday mode, there is good news. First of all, 2.6a1 already contains a lot of forwards compatibility. There is since the 18th of March a “from __future__ import print_function”, which adds more such compatibility, and there is positive noises about “from __future__ import unicode_string_literals”, which would fix the major issue of unicode/string compatibility that I noted in my previous post on this subject. And with these (and an extra module that can be an easy_installable egg, that provides compatibility for the built-in library name changes) there would be no problem in having 95% of all the modules and products out there run under both 2.6 and 3.0. The complete compatibility break that we were so worried about might still never happen, and it looks like it will be possibe for most python implementations to make a gradual upgrade path, by first supporting both 2.5 and 2.6, and then supporting 2.6 and 3.0.

From → python, python 3000

5 Comments
  1. Martin permalink

    Python 3.x is not likely to be a part of the next round of enterprise Linux distros. Red Hat Enterprise Linux, Debian, SLES, and so on is not likely to provide Python 3.x in the upcoming major releases. This will delay the enterprise distro support of Python3 to 2011 or 2012. Ubuntu could theoretically provide python 3.x for their 10.04 LTS, but that is not likely. I guess Ubuntu long term support with Python3 will have to wait until Ubuntu 12.04 in 2012.

    My guess is that Python3 will not be a important part of the Linux platform before 2012 or later.

  2. very useful post , thanks a lot…

Trackbacks & Pingbacks

  1. Top Posts « WordPress.com
  2. New project: python-incompatibility. Is Python 3.0 really incompatible? « Lennart Regebro: Plone consulting

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: