I have definitely never been so ambitious about a conference ever before, and to a large extent this as came out nicely. In fact in my huge list of ambitions I only failed in one thing.
The presentation
I wanted to use impress.js for my presentation. But already in September it was clear to me that I didn’t want to write the HTML directly, as I did when preparing an earlier version of the talk for PyCon PL. I wanted to use ReStructuredText. And I wanted to position the slides somewhat automatically. This ambition lead to the following for goals:
- Create a presentation software that uses ReStructuredText to generate impress.js-style presentations.
- Create a library to handle SVG paths.
- Update Pyroma so tests run on Python 3.3, so I can use it on the software I make.
- Update my talk to use ReStructuredText, and make the slides more useful by themselves.
All of these goals were fulfilled.
- The presentation software is called Hovercraft! and 1.0 was released the 22nd of February. Three main presentations and one lightning talk used Hovercraft! during PyCon, which I declare a success!
- The SVG path library that Hovercraft! uses for it’s most complicated and likely least useful feature: Positioning along SVG Paths, is called, unsurprisingly svg.path.
- Pyroma was updated for Python 3.3. It also got some cleanup and now installs hooks for zest.releaser, so it will run Pyroma on the package before releasing.
- I updated my talk to use ReStructuredText, and it now has better slides. It also available on video.
If you really want to see somebody use Hovercraft! in a useful way, check out Chris Withers talk “Things to make writing tests easier“, where he is using the pan and zoom features to increased pedagogical effect, which is absolutely awesome! Look for example at 5 minutes into the talk.
The PEP
I have also the last few months been writing a Python Enhancement Proposal on getting full time zone support into Python 3.4. It was important to get that PEP reasonably finished before PyCon, as I intended to implement it during the sprints. The idea here was that I would more or less merge pytz into the standard library, with just a bit of refactoring. And this is the ambition that I failed to achieve. Mostly the failure is related to the way pytz works around some limitations in the datetime library. Since we are now enhancing the datetime library, we don’t want to work around those limitations, but we want to instead fix them. Doing that the simple way turned out to be impossible, and the result is instead that timezone support needs to be more or less reimplemented from scratch. The three days I was sprinting ended up with me diving deep into both pytz and also dateutil.tz, comparing them and deciding what needed to be done, so that I now have a plan.
An implementation of the time zone support for Python 3.4 will therefore have to wait a bit, although I still aim to have it done by the first Python 3.4 alpha, which is planned for August.
The book
Now, doing a good presentation and implementing a PEP are reasonable goals for a PyCon, even if it’s a bit over the top to have to write your own presentation tool for it. But I didn’t stop there, oh no! I also decided to update my book on Porting to Python 3. So both leading up to and during the conference I would sit and go through the changes needed to update it for Python 3.3, and on Sunday of the conference I released the second edition of the book! I had hoped to do the release during a lightning talk but competition for those were so tight that it was practically impossible to get them. So I released it during a countdown in the green room instead, with the event captured by David Kua.
The book is now 14 pages longer, includes examples for Unicode-aware sorting and how to deal with the changes in the CSV API and many minor improvements. It is currently only available as HTML, but I hope to get time to make PDF versions in the near future. Perhaps it will even be available in paper again. Only time will tell.
The Plone RV
The last ambition of mine was to also have fun! This was made much easier thanks to the Plone Community having a strong presence on this years PyCon. This was to a large extent thanks to the amazing Spanky and Liz, who not only manned a Plone booth in the expo hall and held a Plone poster session, but also rented an RV, so that us Plonistas could avoid the amazingly boring Santa Clara (the conference center was great, but it is a mile long walk even to the closest restaurant) and instead go into San Francisco every night, and back to PyCon in the morning!
This impeded on my contributing to evening activities such as the Porting to Python 3 clinic, and evening sprinting, but on the other hand, the trips in the RV were a hoot. I’ll probably opt for a hotel room in the future anyway, they are warmer, and it’s less competition for the shower.
Sometimes making a release of a software makes it used, and people come with bug reports and suggestion for improvements and the end result is that you make a lot more releases very quickly.
I wrote Pyroma, a module to check the quality of your Python modules packaging, just before PyCon 2011, and during the sprint I extended it to be more useful. It didn’t get a lot of attention so that was it. I run it from time to time on my own packages, to make sure they are OK.
Then, yesterday, almost two years later, I decided to run it on Hovercraft! to make sure the package data was good. But since I’ve developed Hovercraft! primarily using Python 3.3, and I never even ran Pyroma under Python 3.3, so I decide to first make sure the tests for Pyroma run. They don’t. This turns out to be just testing-errors, but still. I fix the tests and decide to release 1.0. It has after all been stable for two years. Mostly because only I used it.
Well, this time it got re-tweeted a bit, and it immediately turns out it doesn’t work with Twisted. I add some support for that use case and release 1.1 just hours later.
More tweets shows up check-manifest, a new package from Marius Gedminas that checks that the files specified by MANIFEST.in matches the ones checked into the VCS, which is usually what you want. Someone says that the release software zest.releaser should do these checks. Turns out zest.releaser has hooks for that, and Marius adds check-manifest to them, so when you now call fullrelease, check-manifest is called. Well, that’s a great idea, so I now add a hook for pyroma as well! And 1.2 gets released, just the day after 1.0 and 1.1. 
Well, it happens.
Anyway, the end result of this frenzy is quite nice. If you now install zest.releaser, check-manifest and pyroma, you get a triple whammy of:
1. A software that will make releases quick and painless.
2. A check that your MANIFEST.in file is correct before the release is done.
3. A check that the release-tag will have sane metadata before uploading to PyPI.
All by just typing “fullrelease” and pressing enter a couple of times.
Well, I like it.
Pyroma checks your Python packages friendliness to packaging tools and other Python tools. Some of the things it looks at is if you have filled in all the meta data you should fill in, that there are tests and that they pass.
New in Pyroma 1.0 is official support for Python 3.3 (it worked before, but wasn’t tested) and a test that the packages version complies with PEP-386.
Up to probably around a thousand, maybe two thousand years ago, metalworking was special. It was not just a skilled line of work done by a special knowledgeable few, it was magic. There’s a reason Thor wields a big magic hammer. Nobody understood it, not even the metal workers themselves.Through experimentation they came up with rules of how to do things, in a particular sequence, but they did not know why it had to be done exactly like that. And metals generally don’t come in pure forms. The best copper during the copper age was hard. They didn’t know why. Now we know it is because it had high amounts of arsenic in it. Many early metalworkers probably died from strange diseases brought on by being poisoned by metals. What was going on was clearly a dark, dangerous magic.
Today we understand chemistry quite well. And with “we”, I mean humanity, to me it’s still a sort of dark dangerous magic. But when hunter-gatherers see the world as full of small spirits, and there is no difference between religion and daily life, with the start of metal working some people were clearly doing different kinds of magic than others. It may not be a coincidence that everywhere we look around the world, when metal working arrives we see the rise of religions and priesthoods.
And today, we have a new technology that many people do not understand. Just like everyone used metal tools, but very few was able to make them, everyone uses computers today, but most people can not program them. Computers also are incomprehensible, and you often need to do things in a special order even if you don’t know why. Using a word processor or a content management system can become a ritual, where you do a certain list of steps in a particular order, without fully understanding what you are doing. I’ve seen many people have notebooks or word documents full of step-by-step instructions for doing things like posting a new press release on the web page, etc.
For most people, computers are simply magic. But we computer programmers, we seem to make computers and the internet do our bidding. We can handle this magic, and we don’t even die of arsenic poisoning. There is a theory that the legend of drawing a sword from a stone started as a metaphor for metal working. Computer programmers are today’s metal smiths, and we seem to be able to command a kind of magic that enables us to draw information from the internet.
There are no priesthoods yet, partly because society already has it’s own priesthoods to do not embrace this new magic. But where you get stale priesthoods threatened by something external, you also get these priesthoods defending themselves. And that defence comes in two major versions.
The first version is simply a general oppression, reducing freedom and attempting to make the new thing illegal. That is what both SOPA and ACTA is about. They are the establishment reacting to a technology they don’t understand, and hence view like magic. So they are trying to stop it, mainly by simply making it illegal. In Sweden, since the Pirate Bay case, it is illegal to link to material that is infringing on copyright. And since you have no control over what you link points at, any link can be changed at the receiving end, you can suddenly find yourself a criminal. Do I think that will happen? Yes.
Because in some cases the general oppression leaks over into an effort of getting to one particular individual, no matter what. Or in other words, a witch hunt. And in that case, anything that can be used against that person will be used against that person. And since as a part of the general oppression it has become illegal to do what everyone is already doing, anyone can tied to the pole and set on fire.
Aaron Swartz was, in the eyes of those prosecuting him, a magician. They would never admit to this, because they don’t believe in magic ahaha that would be ridiculous. But the fact is they they don’t understand what he did, or how he did it. To them, computers, internet and open source is incomprehensible, and therefore dangerous and therefore it must be stopped at whatever the cost. And as long as these people run the courts and big business and sit in political offices, this will be the case.
And if these people and their witch hunts aren’t stopped soon, anyone with a smart phone will be a criminal.
This weekend there was a yearly global bug fixing sprint for Django. It apparently took place in seven places around the world, stretching from South America to Japan! One of these places was Kraków, organized for the second year in a row by Pykonik.
As an extra incentive this year there was an application written by Aleksandra Sendecka and Tomasz Paczkowski (I think, if I got that wrong, tell me and I’ll change to the correct credits). This app looks at the bug tracker (and I maybe also the github repo, I’m not sure) and gives people achievements according to what they do, such as commenting, adding patches etc. According to the results table there was a total of 43 people involved in the sprint somehow, which is pretty cool.
I personally concentrated on the templating system and threw in a Python 3 fix as well. I added a couple of patches and improved some others. Two of them has been merged to trunk already. It felt like a productive weekend, especially thanks to the achievements app!
Note to all recruiters: I’m not willing to relocate, definitely not ever to London, so the job has to be mostly remote. You can stop reading now.
Yes, I’m coming to the end of my current project. I’m available for consulting or employment for any Python job, although my specialty is web development, especially Plone or Django.
I am a Python expert with 12 years of experience with Python, and 18 years of experience with web. If you need web development done in Python, I’m your man.
Mail me at regebro@gmail.com !
PyPL has gotten a lot of attention lately as index of programming language popularity. It criticizes TIOBE, the most popular popularity index, and it has some valid criticism, mainly that it is a lagging index, it looks, amongst other things, at how many pages about the language that Google finds when you search. PyPL instead uses Google Trends to look at how many searches for a language. That does make sense. And while TIOBE searches for “<language> programming” TIOBE believes this underestimates some languages, as you don’t need to type “programming” as a qualification for PHP.
The difference in results are not just significant, but stunning. In PyPL, Objective-C doesn’t even chart, while in TIOBE it’s the third most popular language. How can this be? One of these indexes must be wildly wrong about Objective-C. And unfortunately, it’s PyPL. Because PyPL falls in the same trap it tries to avoid. Instead of underestimating PHP by looking at “PHP programming” when “PHP” will do, it completely underestimates Objective-C by searching for “Objective-C tutorial” when that’s simply not a search people uses. Objective-C has seen a massive rise in popularity that’s to Apples iOS. Nobody cares that it’s Objective-C. What they search for is “iPhone tutorial” or rather “iPhone programming tutorial” or similar.
But those search terms are too wide, because you can use other languages as well.
The solution? Probably Google hits or searches shouldn’t be used at all, or at least only be a small part of the total score. PyPL also claims that “An analysis of language tag followers on StackOverflow, or of the visits of the Wikipedia language pages result in a similar ranking of objective-c.” That’s complete bogus. If you look at how many visits the Objective-C page on Wikipedia has, or how many new questions per day there are on Stackoverflow, then Objective-C ranks as about half as popular as Python. That’s a big difference from TIOBE who ranks it as twice as popular than Python. But it’s an even bigger difference than PyPL where Python is 7 times as popular as Objective-C.
The only real conclusion to this is of course that ranking languages is hard.
I’ve just released a first beta of Hovercraft!, a tool to generate Prezi-style presentations from reStructuredText. This may need some explanation, but there is a TL;DR version as well.
The basic problem here is that the typical GUI tool for making presentations, from Impress to Prezi, don’t fit the way I make presentations. I do a lot of reorganizing and moving around, and that might mean changing things from bullet lists to headings to text to pictures and back to bullet lists over again. The GUI is only in the way. Some GUI tools have outline modes which are half-way usable, but in the end I want to write my presentations as text, so I can move things around.
And writing them in text and then making slides doesn’t work, since I often realize that I need to change things around once I get to the “making slides” part. So I need to have text as a “source” and “compile” presentations from that text. So, enter reStructuredText.
reStructuredText
reStructuredText, reST or RST, is a so called “lightweight” markup language. Although there is nothing lightweight about it, it is in fact massive and have an incredible amount of features, which is one reason it’s popular. It’s used a lot within the Python community, and is often used to write everything from readme files to books. There are other languages that have their benefits, most notably Markdown and textile, but I don’t know if any of them have the feature set required for Hovercraft, and I’m used to reStructuredText.
There are several ways to make slides from reStructuredText. One is included in the docutils library that implements reStructuredText, adn it can generate S5 slides. Another is Landslide, which i have used as it has a presenter console. But these generate standard left-to-right slide presentations in HTML. Nothing wrong with that, but it’s a bit boring. Enter impress.js.
impress.js
impress.js is a tool to make HTML presentations that zoom and rotate. It’s cool and I used that to make a zooming/panning version of my talk on intellectual properties. And then I made a presentation about Calendaring for PyCon PL 2012 and PloneConf 2012. And by that time I was seriously tired of it, because you write your presentations in HTML, and that sucks in itself, and then you have to position each slide separately. That worked fine in the first case, where I had this huge image to zoom around it. But for the Calendaring talk I ended up having to reposition a lot of slides each time I needed to insert or delete a slide. Not a practical solution. I needed to somehow be able to write reStrcuturedText and get impress.js out. After a false start with a template for Landslide, I hit on the solution: Use docutils to generate XML from reStructuredText and the use XSLT to transform it into an impress.js presentation. That worked, and the solution is Hovercraft!
Positioning
But impress.js is no fun if you have to position each slide separately and you have to reposition all the slides when you insert or delete slides. So for that reason there are several ways in Hovercraft! to position slides:
- Absolute positioning: You simply add X and Y coordinates to a slide, in pixels. Doing only this will not be fun, but someone might need it.
- Relative positioning: By specifying x and/or y with with a starting r, you specify the distance from the previous slide. By using this form of positioning you can insert a slide, and the other slides will just move to the side.
- Automatically: If you don’t specify any position the slide will end up the same distance from the previous slide as the previous slide was from it’s previous slide. This defaults to moving 1600px to the right, which means that if you supply no positions at all anywhere in the presentation, you get the standard boring slide-to-the-left presentation.
- With an SVG path: In this last way of positioning, you can steal an SVG path from an SVG document and stick it in the presentation, and that slide + all slides following that has no explicit positioning, will be positioned on that path. This can be a bit fiddly to use, but can create awesome results, such as positioning the slides as snaking Python or similar.
All in all, the idea here is that you should be able to do the zoom and rotate coolness with as little effort as possible. There may be big changes in how the positioning works in the early versions, to make it more practical. There may also be new ways of positioning. Unknown to me, Gael Pasgrimaud has the same problems as me, and made a tool also called Impress based on Sphinx and impress.js, with the same aim as me (although wildly different implementations). It has one cool feature: you can write functions in Python that does the positioning for you. That could be a cool addition for the future, for example.
impressConsole.js
impress.js also badly needed a presenter console. I took a simple one that David Souther had made, called notes.js, and rewrote it into impressConsole.js. It will show the current slide, the next slide, your notes, the time and a timer. It’s very useful, so Hovercraft! integrates it so you don’t have to think about it.
Installing and using Hovercraft!
Currently Hovercraft! is distributed as a python package on the Cheeseshop. That means that you install it with easy_install or pip. It’s also Python 3 only, so you need Python 3 installed first. I hope future versions will have easier installers for non-Python programmers.
So please test this for your next presentation! Find bugs, come with feedback. What is cool, what is hard, where do I need more docs?
South is the migration framework used with Django. It’s good, use it.
However, it’s a bit awkward to use in the beginning, because the intro documentation doesn’t mention one thing, although it has to be said that it becomes obvious once you learn more about South.
When starting out developing you change the model schemas a lot. So you end up doing a lot of migrations, and having a big stack of migration steps. But the problem is that you often have data in your development database that can’t easily be migrated. But it’s a development database, so the data doesn’t actually need to be migrated. So you have to open up the Django admin, delete the problematic data and then do the migration. In bad cases it becomes easier to just drop the whole database.
Well, it turns out there is an easier way to do that: --update
Instead of making new migration steps for every change, just first make an initial migration step:
python [yourproject]/manage.py schemamigration [yourapp] --initial
Then, each time you need to change the schema, instead of making a new migration, update the first initial one:
python [yourproject]/manage.py schemamigration [yourapp] --initial --update
This well first back out the old initial migration, in practice clearing the app database, and then regenerate the initial migration. When you then run the migration, it will create a new empty database.
python [yourproject]/manage.py migrate [yourapp]
That means you have to recreate the test data. You can do that by making test/demo data once, and dump it.
python [yourproject]/manage.py dumpdata [yourapp] --indent=4 > [yourproject]/[yourapp]/fixtures/test_data.json
After you change the schema, you will have to update the schema in the test_data.json file as well, but then you can load it into the database:
python [yourproject]/manage.py loaddata test_data
As an addition bonus, you now have fixture data for your tests!
You can keep using --initial --update until you start installing a staging server. At that point you need to have proper migrations as people will put data into the staging server. It might not be real production data, but they might get annoyed if it disappears. The you can use --update for all the schema changes you do between two releases, so all the changes in one release only need one upgrade step.
I’m now running django-userena as the plugin for registering users, as it’s most complete and also allows for logging in with either the username or the email.
However, running tests can be a bit tricky, and I got errors like the following:
IntegrityError: insert or update on table “auth_group_permissions” violates foreign key constraint “auth_group_permissions_permission_id_fkey”
or
DoesNotExist: Permission matching query does not exist.
You only get these errors when you do tests that involve the user management and profiles, but when you are doing testing of forms and views, you want to use a webclient of some sort (I use django-webtest) and then you need to be logged in, and then pretty much all tests involve Userena. The trick was a combination of two tricks that took me a lot of digging, so here they are for the benefit of Django-kind: 1. Subclass from Userenas ProfileTestCase. 2. Run UserenaSignup.objects.check_permissions() during the setUp.
from userena.tests.profiles.test import ProfileTestCase
from userena.models import UserenaSignup
class ModelTests(ProfileTestCase):
def setUp(self):
# Setup userena permissions
UserenaSignup.objects.check_permissions()
That’s it (so far at least).
