Lets build Bittorent support for Cheese Shop

Edit: The reception was less than overwhelmingly enthusiastic, and for me to push a community project that is not absolutely necessary for me, I need overwhelmingly enthusiastic. So I’m not going to push this. I still think i could be a good solution, though.

There are two problems that can be killed with one piece of software in the world.

One of the problems is that the Python Package Index (aka the Cheese Shop) is a single point of failure when distributing python software. Well, in fact, because some packages doesn’t reside there, but are only indexed there, it’s multipole single points of failure. There is a whole bunch of servers that currently needs tobe running for you to install Plone from a buildout, for example.

A second problem is that bittorent, although being a great idea and useful for all sorts of distribution of data, is currently used pretty much only for file sharing of copyrighted materials. This means that this great protocol often gets filtered by ISPs and similar. We need a legitimate use for bittorrent.

And as you of course quickly realize, bittorent is in fact a solution to the first problem! And not only that, it also provides an answer to some other problems. Like setting up a local egg cache for a company.

So, I will start a project to do this. But before I start, I want to check if more people have had this idea. Good ideas are typically somebody elses ideas, and in this case I know that Matthew Wilkes already started on this earlier. So if you already have some half-finished code for this, or know somebody that has, give us a shout, and lets merge the projects! Also, if you want to help, give us a shout too.

Plone4ArtistsCalendar 2.0a3 released

Plone4ArtistsCalendar 2.0a3 has just been released on the Cheese Shop.
New in this version is various bugfixes, and support for uninstall.

Thanks to all that helped with bugfixes for the various parts of Plone4ArtistsCalendar! It’s been a real community effort with loads of people contributing! Thanks, thanks thanks to all of you!!

So, if this is so bugfree and stable, why is it still an alpha? Well, because we still need migration from Version 1.1, a new iCal implementation that supports recurring events,  and a better UI for recurring events.

What video editing software do you use?

I’m looking for a Windows (or Linux) video editing software or combination of grabbing and editing softwares that

  • doesn’t cost hundreds of dollars/euros and
  • can grab video from a web cam,
  • audio from any audio input,
  • keeps the synchronization well and
  • has at leat two sound tracks (so I can have background musc) and
  • has good frame-by-frame editing and
  • doesn’t require the use of abscure file formats

My PyCon talk now as podcast

Python 2.6 and 3.0 compatibility

I think it doens’t work very well without the slides though.

Plone 3 Developer Training in Sorrento 11th-12th May 2009

I’m holding training in Plone 3 development in Sorrento the 11th-12th May, just before the European Plone symposium. The weather will be great, the people will be great, there sea will be beautiful, the Pizza will surely be almost as amazing as in nearby Naples, and you must see Pompeii before you die

See the link for more info.

Setuptools on Python 3: Work on hold.

I’ve currently suspended the work on porting Setuptools to Python 3. If you need it, there is the version I made earlier on the Python-incompatibility project page. It seems to work. If people want to fix bugs you can still mail me and I’ll give you access to fix it. It’s very unlikely that it ever gets merged.

Why? Well, it basically comes down to that setuptools code is messy and hard to understand and Phillip J. Eby argues against all efforts of improving it. That gives me three options: Fork setuptools, rewrite it from scratch, or give up. I don’t want to fork it, mainly because I don’t think the code is good enough to be worth forking. It’s so hard to maintain, that if we need to fix it without Philip J Ebys support, it’s probably easier to rewrite from scratch. But… I can’t rewrite it from scratch, I simply do not understand the issues well enough to do that. So, at least for the time being, I give up.

This means that there is no official setuptools release. That means it’s going to be hard to port and install any python module using setuptools. This is probably the majority of Python modules on PyPI, because the biggest creators of Python modules is Zope/Plone and these modules all use setuptools. This of course also means that my porting of zope.interface that I started is on hold. In fact, it means all my Python 3 porting work is on indefinite hold. Setuptools is currently the major obstacle for widespread acceptance of Python 3. Sad but true.

So what are the alternatives if you want to port to Python 3? Well, first of all, if you use setuptools, get rid of it. If your module only uses distutils, you can port it to Python 3, and have 2to3 run automatically by using build_py_2to3 in place of build_2to3. It can often be quite a pain to move a project that uses many setuptools features to plain distutils, but currently I can’t recommend using setuptools if you plan to move to Python 3.

If you feel you really need setuptools functionality in your project, join distutils-sig@python.org, and discuss how we can solve the situation. Right now I feel that the most likely way forward is to rewrite setuptools, and possibly also distutils, from scratch.

Extra, extra: Internet outlawed in Sweden!

(Yeah, yeah, sensationalist headlines blah blah blah. I like them. Get used to it.)

Today the court presented the verdict in the Pirate Bay trial. The defendants were found guilty. Basically, they were punished as accomplishes for helping people share files illegally. Hence, from today, under Swedish law, Internet basically became illegal. Linking to illegalities are illegal. Reasonably, by the same logic, Internet providing is illegal if anybody uses your provided Internet to do anything illegal. In short, in Sweden, Internet had become illegal.

I shut down my personal websites hosted in Sweden today (http://www.liberalism.nu/, http://lennart.regebro.nu/) and are looking for a virtual server where I can run low-traffic Zope/Plone sites reliably for a low cost. The server must not be located in Sweden. And from what I’ve seen of French government attitudes, France is probably not good either. My professional websites hosted in Sweden will go away during the weekend. [Update: I really mean go away. I'll shut them down. And no, it's not customer websites, it's still my websites, just not my personal ones, my my company ones. I've now gotten a couple of offers to make some sort of partnership to host my customer websites, so I'll make this clear: I don't do hosting. I do not host any customer websites.I just consult, so there isn't anything to partner on. Sorry... What I'm looking for is a vartual host slice, no more than 50 eur/month. And I'm not going to host at any place I didn't get recommended by somebody I know, so you don't need to spam me about it. I want *recommendations*, not soliciations. Thanks.]

Recommendations, both for countries and providers are most welcome.

Comments on Django’s design decisions

In my talk about what Zope did wrong, I tried to tell the world not to repeat Zope’s mistakes, and I learned at PyCon 2008 that Django have repeated some of them. Mark Ramm also did his keynote on what Django can learn from Zope at last year’s DjangoCon. So I don’t know if it was intentional, but Jacob Kaplan-Moss’ talk about Django’s design decisions at this year’s PyCon felt to me like Django trying to defend itself a bit.

And without a doubt, Django is a good web framework. If it wasn’t, it wouldn’t be so popular. And although I think it needs some big changes to be “future-proof”, I must say that this is basically not very important. One thing I have realized is that when it comes to software choice, choose what works for you, today. Future-proofing is irrelevant, because you don’t know the future. So my comments here are not designed to scare anyone away from Django. Of course, it is partly a question of me sticking the Zope technologies in your face and screaming “Hey Look at This, isn’t this Fantastic!?” and maybe to an even bigger extent just a way to say “Been There, Done That, Not a Good Idea”.

Architecture and Morality

I agree with the basic decisions of Django. They chose Python, open source, of course, having sensible defaults instead of requiring loads of configuration for everything, and also that you shouldn’t have “architecture astronauts”. There is a type of programmer who likes complexity for it’s own sake, and if you let them write a framework, it’s going to be amazing, and amazingly complex to the point of being unusable. But the risk of ignoring architecture astronauts isn’t what Kaplan-Moss says, that you risk reinventing the wheel. The risk is that you paint yourself into a corner. This is not a problem for end user software, because you can always refactor it. But it is a problem for frameworks, because here other people are using the framework and when you refactor it, you break every body else’s software. This is why corners are hard to get out of when building frameworks, because when you walk over the wet paint, it’s not just your work that gets undone, it’s everybody else’s as well. And that’s what the architecture gurus are there for, coming up with a technique of building that means you don’t end up in corners. But yes,you still should be aware of the guys that want to put in flexibility without being able to explain what the use case for that flexibility is. Doing that can often give you complexity because you have flexibility in places where you don’t need it. Zope 3’s publisher has a hook for you to replace the type of object traversal that is done (i.e how you figure out which object is meant with a particular path). And the default traverser has a hook to change how traversal is done on a particular object. And the default object traverser will use __getitem__ to figure out subobjects. Which of course is overridable. That means that you have three places where you can change what the subobjects of your object will be. That is simply not needed. I’ve yet to find a case where the simplest one, __getitem__, isn’t enough.

One common mistake between Zope 2 and Django, is that some functionality, like object security, is handled by special methods or attributes on the content class. That works well, in the beginning, but then you end up wanting to change the way the security works to make it more flexible, and you introduce more attributes, but keep backwards compatibility to not break everybody else code, and you end up with a horrible mess of magic attribute names that change the objects behavior in subtle ways. Zope 2 did this is a big way, and not only with security. I noticed last year that Django did that as well, although not in a big way. Yet.

Full stack or slim waist

I can also not really criticize them for being a full-stack framework. I do disagree with Kaplan-Moss, the dichotomy between a full-stack framework and a “best of breed” framework definitely exists. And even if you, like any good developer, tries to make your full-stack framework consist of separate parts that can be used independently of the whole, unless you make sure it is used independently of the whole, dependencies are going to start to creep in. Zope 2 and Zope 3 where also full-stack frameworks, and for the same reason as Django: There just wasn’t anything else out there. Obviously, when Django started, there was something out there, Zope 2, and Twisted. But the only part that was truly independent of Zope 2 was the ZODB, and Django decided to go for an ORM, and Twisted refused to use thread pools, making it significantly slower, as it has to start a thread for each web request.

But Jacob Kaplan-Moss then goes on to criticize the best-of breed frameworks because you need “glue code” to clue the components together. This is partly true, of course, but it doesn’t distract from the fact that the components in the glued system still works separately and can be learned separately. And that comes into play when Kaplan-Moss talks about the learning curve of Django, and how you have to almost unlearn bits. This is nothing unique to Django. With Zope 2 we have the exact same phenomenon, which is in the Zope community known as “The Z-shaped learning curve”. Jacob claims that this is something that comes with a framework. I do not agree. I think it comes with full stack framework, where there aren’t good separation between different parts, and the internals are messy. But I do not believe this is true for a well designed best of breed framework.

My way or the plugin way

And I have to really protest about Jacob Kaplan-Moss’ criticism over pluggable components. First of all he completely misses the point, and says that it’s not possible to write an application with STORM and then flip a switch and use SQLAlchemy instead. Well, of course it isn’t, unless you actually write a wrapper for them, which seems more work than it’s worth. But the point is that you shouldn’t be stuck. In Zope 2, you can in practice use two templating systems. DTML, which was Zope 2’s first one, and ZPT, which was a much better one introduced some years later. But what if you don’t like them? What if you want to use something else. You might like Zope 2 as a whole, but really detest both DTML and ZPT. Well, in Grok, we have made it very easy to plug in other templating systems. Of course you can’t switch in a project, all your ZPT templates would have to be re-written. But for the next project, you can stick in Genshi or whatever you want. (In fact you can use all of them at the same time, but that’s not true for all types of pluggable components, so it’s besides the point). Pluggability gives you flexibility.

And here Jacob Kaplan-Moss claims that by having choices is bad. That is almost shocking. His main argument is that you force the new user to make choices that he aren’t able to make, which is completely incorrect, because, as he mentioned, Django has sensible defaults. Grok has that same philosophy, and when you start a Grok project ZPT will be there by default. If you want to use Genshi instead, please go ahead and do so. But the sensible default, ZPT, is there from the beginning, and the flexibility adds no extra complexity to the user who doesn’t want to switch, and you force no-one to make choices they can’t make.

Jacob also claims Django can set their own pace, by controlling the whole stack. This is partly correct, but when you make a best-of-breed framework, you will typically have central people involved as contributors to the components you use, and as a major user of that component, you can typically influence the direction and get releases more or less when you need them. And if you aren’t a big user, and won’t have a voice, well then you probably don’t have enough developers to do everything yourself anyway.

Is Django the new Zope 2?

Jacob Kaplan-Moss acknowledges the risk that Django will become Zope. Thanks to that acknowledgement I don’t think Django will become Zope, although they clearly have gone quite a long bit on that path already. Hopefully they have stopped. I don’t know if they need to backtrack, but at least they shouldn’t go much further on the Zope path if they want to continue to be relevant. Because if you go to far, you have a big job in backtracking to become relevant again.

Oh, and Jacob: ZODB is not an ORM. It’s a real OODB. ;-)

Wazzup in San Francisco area end of July?

Before I book tickets and hotels to OSCON, the July 22-24, is there anything else happening in the San Fransisco/San Jose area the days before or after that I shouldn’t miss? It doesn’t have to be Python or even Computer related. If I shouldn’t miss it, whatever it is, tell me!

PyCon 2009: Zope 3 is dead! Long live Zope 3!

This is my subjective account of the discussions about Zope that happened under PyCon 2009. I make no claim that this represents any sort of truth except my own.

On thing that really made med excited about PyCon 2009 was the amount of Zope gurus that was going. Primarily the fact that we got several of the main drivers behind the innovation in Zope web frameworks lately into the same room was cause enough for celebration. The Zope community have a history of “violent agreement”, and I always suspected that it would be fruitful to have these discussions in real life, where misunderstandings are more easily and quickly fixed. My hope was that the result would be a renewed sense of consensus on where Zope was heading, and I think that was pretty much exactly what happened.

No decisions was taken, and I think that is good. When a community decides something it’s important that the people who take the decisions also have the ability to make it happen. For Zope, there is no central group that have the ability to create Zopes future by themselves, and therefore it’s important that no central group takes decisions. The future must grow out organically from what is actually done and created.

The discussions started with an open space on Sunday morning, with a “Hate Zope/Plone” BoF. About ten people showed up and sat around a table complaining. Perhaps not very constructive, but fun and possibly cathartic. At 4pm there was a Repoze Open Space, and at 5pm there was the “Zope: Status + Future” Open Space I announced earlier. This Open Space dragged out and spilled over into the bar. The major revelation for me was that nobody seemed interested in the application server known as “Zope 3″. Of course, opinions on what exactly made up this application server differed. Was the “Application Server” if you included the Zope Management Interface, the ZMI? Or was it zope.app.publisher that was the “Application Server”? It was clear that few, if any, used Zope 3 as a monolithic whole any more. Most Zope 3 users picked and chose the libraries out of Zope 3 that they wanted, and those who didn’t, wanted to. Nobody seemed interested in a big fat Zope 3 Application Server any longer. The wish of the day is to have a lean, mean webserver that doens’t have everything included, but instead uses a dependency tree to include those parts, and only those parts, that your application needs. And this of course was the whole purpose of the eggification of Zope 3, which therefore can be declared to be a resounding sucess.

The discussions continued during the Zope sprint that followed. And one thing that most people seemed to absolutely not want was zope.app.publisher, the central part of Zope 3. It’s too big, too complex and does too much. Chris McDonogh of course wants to use repoze.bfgs minimalistic publisher. Gary Poster from Canonical wanted something more flexible than repoze.bfg, but less complex than z.a.p. Jim Fulton wanted Bobo back, the much simpler publisher that was the center of the early versions of what became Zope 2. Also, nobody was interested in using the ZMI. As these seems to have been the main contenders for being Zope 3 The Application Server, the conclusion is inevitable: The application server known as Zope 3 is dead.

And this is a Good Thing. Not because Zope 3 was crap. It wasn’t. It’s bloody fantastic. But there is a lot of confusion about what Zope is. There is Zope 2, an application server called Zope 3 and a vast library of web application modules also known as Zope 3, so the fact that nobody any longer want the application server help clear this up. We can forget about it. We are then left with a big framework of libraries, which are set to get a release in the not too distant future as the Zope Framework (probably 1.0), and an application server called Zope 2 which includes the framework (and other bits). It’s still not perfect, but it’s way less confusing.