Skip to content

Python and time zones: Fighting the beast

December 18, 2007

Time zones is not a funny subject. In an effort of finishing the time zone issues in Plone4ArtistsCalendar before a 1.1 release, I’ve now spent more than two days just on this. And I’m not sure I’m done yet. It’s getting rather annoying. If you want to know more about the time zone issue, read on:

The time zone issue is of course made up of several problems.

1. Python has no native timezone implementation.

The datetime library supports timezones, but doesn’t provide implementations. Of course, there are implementations around already, but although I know of two implementations, none of them perfect. One is a part of Gustavo Niemeyers dateutil, and the second is pytz. (There is some sort of basic timezone support in too, but I haven’t found it at all useful). Since I use dateutil otherwise for it’s recurrence module, it was a natural start. However, I found some bugs (which Gustavo says he will fix in the next release). But then I ran into the next problem:

2. Posix has GMT backwards.

Dateutil assumes Posix broken GMT handling. I.e, if you ask for GMT+2, you will get a time zone that is GMT minus two hours (“That’s logic!” said Tweedledee). Only Posix does this afaik, but because of this loads of Unix software thinks it’s the right thing to do, since Posix time strings does this. In fact, most Unix implementations now come with two sets of Etc/XXX zone-definitions, one called posix/Etc and one called right/Etc. However, although dateutil is happy to use posix/Etc/GMT+2, it doesn’t want to use right/Etc/GMT+2 for some reason. And also, I don’t think we can’t count on these files existing on all Unix distributions.

Why is this a problem? Well, mainly because one of the things I need to do is to convert between Zopes DateTime object and pythons datetime objects. And of course, DateTime objects use GMT all the time. Correctly. So DateTime thinks GMT+2 is actually two hours more than GMT, can you imagine. So I need to have a set of timezones, including GMTxx that does the right thing. And dateutil doesn’t give me that.

Is pytz better? Not at all, it also gets GMT+2 backwards, and refuses to use either posix/Etc/GMT+2 or right/Etc/GMT+2. But Zope 2.11 is poised to use pytz as timezone implementation, so code already exists for pytz that sets up a bunch of correct GMT timezones. So I copied that code from Zope trunk and used it. That was that problem solved (or so I thought). But the next problem I encountered was that…

3. Time zone names are not unique.

When you tell your unix in which time zone you are, you do this typically by some UI software where you choose things like “US/Eastern” or “Europe/Paris” that will set things up for you. I’m not sure how most software actually does this, but the general effect is that there is a file called /etc/localtime that contains all the information, and an API to get that information from the OS. However, one very crucial piece of information is missing in that file: The actual location you have configured your computer to use. It contains all other possible information, but not that. This is probably the idiots at Posix that need to be spanked again. The effect is that when you are for example in Australia/Sydney, Python does not know that. All it knows is that you are in a time zone called EST, and the offset. However, there are three time zones this can be with three offsets. The Australian Eastern Standard Time, the US Eastern Standard Time and the Brazilian Eastern Standard Time (although this is usually called BRT, thankfully). Which one is it? Well, we can figure that out by going trough all the timezones, and comparing the name and the offsets. This way we at least will end up in the same time zone. So people in Australia will in fact get a UTC offset of -11 and not -5. Problem solved? Nope.

4. Locations with the same time zone may have different daylight saving.

Figuring out which EST we are in is fine if you want to figure out your current UTC offset. However, that’s not all there is to time zones. No, in calendaring you want to be able to calculate your local time for an event that happens in half a year. Is this a problem? Oh, yeah. For example if you are in Brazil/Eastern this has BRT as a standard time zone and BRST as the daylight saving time zone. But there are MANY locations that has those. Including, for example, America/Araguaina. But while Brazil observes daylight saving, Araguaina does not. So, you will get incorrect offsets for all converted times during Brazilian daylight saving.

This is however only a problem for the local time zone, as you don’t have the name for it. If you want to convert to another time zone, it can be assumed that you know to where you want to convert it. And dateutil has a clever solution there: It can wrap zoneinfo files. This means that to get the local timezone, you just wrap /etc/localtime, and you will be all right. That way you don’t have to know the name of the local zone. So, out goes pytz, and back comes dateutil. It means I have to create a bunch of GMTxx zones and also not use, but my own gettz that first looks in my extra GMT-zones, but that isn’t too much work. But it does gives us a new problem:

5. Some times happen twice.

The night you switch your clock back one hour, a whole hour happens twice. There is no way to know of 1:30 that night is the first 1:30 or the second 1:30. This is written about a bit more in the pytz documentation. The problem comes from that datetime stores it’s data in the local time format. A better solution could have been to store it in UTC, but the problem with that is that there would have to make a conversion each time you use the datetime, which would slow things down. pytz solves this somehow, but it does mean you have to tell ambiguous times whether they are in DST or not. I can’t find any support for this in dateutil, although I might have missed it. And also, it has to be admitted, that this is a rather minor issue. Just don’t put meetings during a daylight saving switch, OK? 😉

Although, OK. Not all problems are solved. One last problem remains:

6. What about windows!?

I have no idea if this works on Windows or not. People are welcome to try.

  1. Daniel Bennett permalink

    Please note that there are some time zones that are not just integers. For example, Newfoundland is on the half hour. I have seen some time zone implementations that typed for integers only.

  2. Right, I need to test what Zope’s DateTime does there, to see if I need to secial case that. Good catch.

  3. “So people in Australia will in fact get a UTC offset of -11 and not -5. ”

    I think you mean +11 there 🙂

  4. yurj permalink

    1) the fix has to be done at os level
    2) guess the timezone from the language setting of the os
    3) 2) should be done by Zope itself, the developer should not “guess and try” about the os

    This in the ideal world…

  5. An OS level fix would indeed be the best, but isn’t feasible. And the fix shouldn’t be done by Zope, but by some python library so others can use it. I think it would be best of and pytz could be merged, as they would together cover all the issues.

  6. Stuart Bishop permalink

    If you want a Unix machine’s timezone with pytz, you can do so using pytz.timezone(open(‘/etc/timezone’).readline().strip()). An API could be added to do this for you, but it would introduce a platform specific call – there is no equivalent under Windows I know of (someone would need to maintain a mapping of Windows timezone names -> Olson timezone names)

    In order for pytz to support dst switchovers better, it had to deviate
    somewhat from the documented specs. dateutil doesn’t do this, so doesn’t
    support this (but has a simpler API because of it). pytz needs t grow a
    ‘dumb’ mode matching the Python API for (most) people who don’t need the
    extra complexity. The particularly annoying thing is the root cause of
    this was because in order to store the flag indicating if dst is in effect
    in a datetime (1 single bit), it would have pushed the pickle size up a
    whole byte. This was very important for the major user at the time, Zope
    2. However, since then the Zope devs have realized that naive datetimes
    such and now just store datetimes with pytz timezone info by default.
    Maybe the Python API needs to be revisited, but it needs someone with the
    interest and C skills to get their hands dirty.

    The pytz module provides a FixedOffset class that could be useful for your
    UTC offset timezones – eg. pytz.FixedOffset(60)

  7. Aha! That you can use the contents of a timeone file like that call should probably be documented better, because I tried to figure out if zoneinfo files was supported and failed.

    Great stuff!

  8. Unfortunately, I finally realized that the above code:

    Did not do at all what I asked for, namely reading the local timezone info file. I mixed /etc/timezone and /etc/localtime up.

    There is no /etc/timezone on Mac OS X. for example, which means that the fix I have been working on in fact does not fix the problem.

  9. Jason R. Coombs permalink

    For win32 time zone support, see win32timezone in the pywin32 package:

Trackbacks & Pingbacks

  1. Please run a test for me (it takes only minutes) « Lennart Regebro: Plone consulting
  2. Python and time zones part 2: The beast returns! « Lennart Regebro: Plone consulting

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: