As a Python programmer, I’m of course biased towards tools that are written in Python or by Pythonistas or at least feel “Pythonic”. I also had a set of requirements:

• The tool chain had to somehow support testing all or almost all the code.
• It had to be able to produce some sort of print-ready format, which in practice means PDF.
• Being able to make HTML would be an added bonus, as I wanted the option to make a website out of the book.

The obvious choice to write this book was therefore to use ReST, and Sphinx to generate PDF/HTML from that. Sphinx goes via LaTeX to generate the PDF and writing it directly in LaTeX would have been an option, it’s used for a lot of technical documentation. But I didn’t feel like learning LaTex, and I didn’t know how good the LaTeX to HTML tools were.

First problem: Testing the code

The “canonical” way to test ReST code is to write it as doctests and then run the ReST source code with a testrunner. As Brandon would accept ReST for Python Magazine, this is who I wrote the articles. This had various problems, most of which Manuel promised to solve, but Manuels syntax for what makes up a doctest is different from Sphinx syntax. I solved this by writing a plugin for Manuel to handle the Sphinx syntax so you could use Manuel to run tests in Sphinx.compatible ReST, but it was relegated by Benji to a branch, where I think it died.

Another problem was that I needed to mix Python 2 and Python 3 code. I had originally intended to have a flag for what versions of Python it should be run with, and let Manuel skip tests is the test wasn’t in the list of Python versions. But I didn’t want to have my own branch of Manuel that only I used, so in the end I decided to drop Manuel and go for a much simpler solution. This solution wouldn’t have worked well for the articles, but it works fine for the book, and that is to use the literalinclude directive. This simply includes an external file, as a literal block.

This way I wrote (nearly) all code examples in the book as separate files, usually doctests, and included them into the book with literalinclude. I then wrote one unit test for each Python version, where I made a DocFileSuite out of all the files that should be tested for that version. The test for Python 3.1 will also run 2to3 on some of these code snippets first. I then have a simple bash file that will run all the unit tests.

Second problem: LaTeX

As mentioned above, Sphinx looked like a promising solution to not have to learn LaTex or other complex publishing softwares. That turned out to be wrong, I still had to learn a lot of LaTeX. The first reason for this is that I wanted to be able to select what fonts I used. So one of the first questions was how to select other fonts in LaTeX. And the answer is: You don’t really. It turns out that LaTeX is just a set of TeX macros to make it easier to use TeX, which is highly involved. LaTex in turn comes in several different distributions, which in turn include different sets of macros, called “packages” and different pdf generators. And each distribution typically includes several different packages that claim to do much the same thing, such as selecting fonts.

Hence, when you Google for how to do this, you get several different answers, one which is actually a good working answer, and some that may be technically correct but may lead you completely astray and only give you more problems.

For Linux, the distribution to use is TeX Live. It’s a sort of mega-distribution that merged all other distributions, if I understand correctly. Although it’s the one included with almost all Linux-distros that may not help you very much, I ended up having to download TeX Live and compile it from source (which takes ages) because of bugs rendering the version included with Ubuntu at that time unusable for me.

And then I had to learn enough LaTeX to change the default templates with Sphinx to do what I wanted. I needed to not only include a “preamble” of LaTeX code in the Sphinx configuration, but also modifying sphinx.sty and sphinxcustom.cls. This was mostly changes in design and margins etc. I did have to fix some bugs in Sphinx usage of LaTeX as well. I still have to feed those bugfixes back to Sphinx, I’m sorry I haven’t done that. Soon, I promise.

Third problem: Fonts

Even after I had figured out that the correct way to select fonts is \usepackage{fontspec} I still had problems. One of my technical reviewers pointed out that the PDF didn’t have any ligatures. That was indeed strange, and after much fiddling around I figured out that the version of the font I was using simply did not have any ligatures. In the end I ended up using TeX Gyre Schola, an open source version of Century Schoolbook for the main text, the also free DejaVu Sans Mono for the code, and the completely commercial Flux Bold for headings. I went with Flux Bold for the headings because it is beautiful, used in the Python logo, and I already owned a copy anyway. DejaVu Sans Mono was selected purely on aesthetic reasons and just happened to be free, and TeX Gyre Schola was selected as a version of Century Schoolbook that worked, with ligatures. That it was free was again pure luck.

The tool chain

So the complete tool chain looks like this:

1. ReStructuredText
2. Python tests using unittest and doctest
3. Sphinx to generate LaTeX output
4. xelatex to generate PDF

Out of this comes a PDF file that makes up the inside of the book. Another PDF makes up the cover, and you upload both to CreateSpace, and they will make a book out of it!

I used Sphinx to generate the book site as well. I could probably have done it faster in Plone, but I decided to use Sphinx to generate a static HTML site because that made it easy for me to put up a sample chapter, and also meant I held the door open to easily put all of the book on the web at a later date (which I indeed also ended up doing, more on that in a future installment).

Next part of this will discuss how to make sure the book is a good book.

If we look at where the conference has been located as an average of the attendance, we get this sort of graph for the the longitude:

We can see that the longitude makes no noticeable  difference. The latitude is a different story, though:

The two outliers are Washington and Bristol, and we can of course ignore them as statistical flukes. And then we get a linear formula for attendance as a function of latitude:

attendance = 400 + latitude in degrees * 1.1

So, the more easterly, the better. Being just on the edge of the international date line would then yield the best results with 598 attendants, but it will be hard to find a good conference location there. Instead I recommend New Zealand. But as east becomes west as an arbitrary location, there is a small risk that the cutoff point is not the international dateline, and that New Zealand is on the “negative” side, and yield only 195 attendants, but I find that unlikely. If we want to be on the safe side, we do need to go gradually east and see. As we like nice weather in the Plone community, good locations would be Cairo (434 attendees), Mogadishu (450 attendees) or if you want to take a bigger risk, The Maldives  with 480 attendees, or possibly 84, depending on where the cutoff is.

There is also a possibility that the function isn’t linear. It could for example be cubic, which would make the projected attendances wrong, but the conclusion should still be to go east, although the east cost of the US would work well too, with Florida in preference for weather. Or for that matter, Cuba.

Or it could be a cyclic function:

In this case the best latitudes would be 148W, 108W, 68W, 28W, 12E, 52E, 92E, 132E and 172E. That gives us the following desirable locations:

• Anchorage, Alaska, US (149W)
• Billings, Montana, US (108W)
• La Paz, Bolivia (68W)
• The Azores (28E-25E)
• Tripoli, Libya (12E)
• Tehran, Iraq (51E)
• Lhasa, Tibet (91E)
• Vladivostok, Russia (131E)
• Christchurch, New Zealand (173E)

What we do see here is that no matter what the function is, New Zealand would be a good bet, and therefore the inevitable conclusion is that there should be no call for proposals, we just you tell Tim Knapp to do the Plone Conference. Remember: Correlation *is* causation.

The book started out of necessity. Not mine, but the communities. There was no good documentation on how to port to Python 3, and I realized that if it was going to be made, I had to do it, and that it would be a crazy amount of work. So I wondered if there was any way I could recuperate some of that in money. The idea I had was to write a series of 3-5 articles in Python Magazine, and then merge those articles with some appendices and forewords and things into a small book. I shipped this idea to Brandon Rhodes, the editor of the magazine, and since Python Magazine had a limited time of exclusivity on their articles, it turned out to work fine. I could write the articles, and after their exclusivity ran out, also publish them as a book.

So I started writing, around December 2009 if I remember correctly. During spring 2010 I had finished and delivered three articles, but by that the reorganizing Python Magazine was doing never got anywhere, and the articles were never published.

Publishing

At this time I started looking for publishers, as I now believed I could actually deliver a book. It’s a short book, but nevertheless, it’s a book. Most publishers, like O’Reilly, will pay out advances, and as such they will only accept your book if they believe in it. So they require you to fill out these long applications full of questions. As it would probably take me days just to fill out these applications and come up with answers to most of their questions (that I felt often didn’t apply) I decided that major publishers probably weren’t the right thing for me.

I also talked to Packt Publishing. I’ve reviewed several of their books here, and they have several times mailed me if I want to write a book, often on weird topics that they think have a market, like “Zope Configuration”. I said that I’ve worked with Zope for 10 years and have no idea what “Zope Configuration” means. Packt first said no to my book, as they didn’t want to publish a book that was likely to be less than 200 pages long. Then they said maybe, can you please write it in Microsoft Word using our templates? I said that I couldn’t. I wanted the examples to be tested, meaning that using Word was out. I needed a format that was basically text.

The option that lay ahead then was self-publishing. This also had the benefit of making it possible to use Creative Commons, something I wanted. After talking to Chris McDonough, who self-published his BFG (now Pyramid) book via CreateSpace, I decided to use CreateSpace as well. CreateSpace has the benefit of being owned by Amazon, and hence you get a much better deal when selling books via Amazon.com from them, then from other self-publishing print houses, and quite a lot of sales will come from Amazon.com.

So I went with Create Space, and I wrote the book in Restructured Text, and used Sphinx and LaTeX to create a PDF output of the book, and I used GIMP to make a PDF cover, using an image of an albino Burmese Python I found on flickr.com by using their creative commons search.  Brett Cannon was nice enough to write a foreword which added a nice touch of authority to the book and makes it sound like I know what I’m talking about.

But although I had the text, there was a long path to having a book.  In Part 2 I’ll write a bit about the toolchain I used and the technical problems I had.

The please export that file, and run it through my anonymizing script: https://github.com/regebro/plone.app.eventindex/blob/master/anonical.py

Send the result to me. Or just send the ical directly to me, I promise to anonymize it and not keep the original around. I will from this put together massive .ics files that can be used for performance testing on calendars with real data. These ics files will be completely anonymous; only the start, end and recurrence if any, will be kept; and the resulting test sets will be made open source so others can benefit from this test data.

It is for the good of mankind! Especially open source!

The book is, as of today, only available directly from the createSpace eStore. It may be available via Amazon for a couple of more days, but I guarantee nothing. By the end of the month, it will be removed from the createSpace sStore as well.

Anything can happen, and it could very well appear in print again, but it looks highly unlikely at the moment. It will however continue to be available for free, online.

I aim to write a post (or possible several) on the experience of self-publishing, which has been very interesting and a lot of fun. The reasons for ending the print-run will be explained there, and has it’s basis in the Polish banking system. Intriguing, eh? Watch this space…

However, to do that will also take time, and I need a Kindle to test on…

So, to make sure I’m just not wasting my time, I would need 10 iPad people and 10 Kindle people promising that they will buy the eBook version. And I need to borrow a Kindle, or have an additional 5-10 Kindle-preorders to cover the cost to buy one. As the printing costs a couple of dollars, I think $25 is a reasonable price for the eBook versions, and you’ll also not have to pay for shipping.

So mail me, or add a comment here if you would buy a PDF version of the book.

There you will find the whole book in HTML format. And you don’t even need 3D-glasses!

Of course, if you want to, you can still buy a hard-copy.

We do both direct consulting and subcontracting, and everything is done remotely (unless you have loads of money).

Contact me at lennart@colliberty.com or regebro@gmail.com.