Skip to content

The compiler doesn’t help you find bugs

November 1, 2007

Via Matt Harrison I saw this blog post by Joe Shaw, where he claims to be too sloppy and undisciplined to use Python.

And lastly, having worked with Trow on a reasonably big desktop Python app, we wanted a strongly typed language. Writing real applications in Python requires a discipline that unfortunately most people (including myself) are unwilling to adhere to, and this easily leads to buggy and hard to maintain programs.

Well, Python is a strongly typed language. He probably means a statically typed language.

You have to be very diligent about unit tests and code coverage for every line of code, because you can’t rely on the compiler to catch errors for you.

Wrong. Yes, the compiler will catch errors. It will in fact catch the type of errors that make the language impossible to compile, syntax errors. In a language without a compiler, you would only see some errors when you reach that line of code and the interpreter tries to interpret that line, this is correct. Joe Shaw thinks that unit tests are there so that you make sure these errors gets caught before delivery. They aren’t. They will do that, sure, but you can do that in much easier ways. In languages like Python, which uses just-in-time byte-code compiling, you’d see that error as soon as the module is loaded. And to make sure you have none of them, you just run a tool like pyflakes regularly. It does what the compiler does: It checks for language errors. That means that if you wrote “tehn” instead of “then” you’ll get to know it. Does this require discipline? Well, no more so than to actually re-run the compiler, before you check in the code you just wrote, so I would say no.

We had been burned by this a bit, and wanted to get back to a strongly typed, but still easy to use language that integrated well with the desktop.

(Again, I assume he means statically typed languages). Well, yes, with statically typed languages it will catch the error if you try to treat a Foo-class object as a Bar-class object. This is not handled in dynamically typed languages. Why? Because it isn’t an error! That’s what the dynamic typing is all about. Does that mean that dynamic typing requires more discipline? Not at all. In fact, it’s the other way around. Static languages want you to be disciplined and set up big object hierarchies with well defined interfaces, and stick to them. But you don’t have to, because there are always ways of making the static language behave dynamically, like using pointers and type casting everywhere, and therefore invalidate the whole argument.

And guess what. Once again, the tests are not there to test this, although they certainly will.

So what are the tests there for? They are there to make sure the program works as intended. And your compiler can’t help you with that. Only testing can. And there is no substitute for testing, and you can’t avoid it. It’s just a matter of who is testing it to find the bugs, the developers or the end users. And if you let the end users do it, I suspect they won’t be happy. And it won’t be cheaper for you either.

From → python

11 Comments
  1. Amen.

    The number of times I have met people that think testing is only done since Python does not have static typing is astounding.

    Testing isn’t because Python doesn’t catch your syntactical errors, it’s to make sure what you write actually does what the user of your application expects it to!

    If you think writing in C (!) or Java will make unit tests redundant, remind me to never spend any money or effort on your software.

    Look no further than Google, who are almost religious about tests, in Java, C++ *and* Python.

  2. Hi Lennart,

    I think you are extending my blog post beyond its intended meaning.

    You’re right, I did misspeak and meant “statically typed” rather than “strongly typed”, but I never implied that a statically typed environment was inherently less buggy or better suited than a dynamic one. Nor did I ever say that I was too sloppy to use Python. In fact, having used it for a large project I am actually better suited to be diligent in my usage of it than most people. I am in fact quite a fan of Python.

    The situation to which I was referring when I said that we had “been burned” was one in which we were unfortunately mixing our use of integers and boolean types when saving configuration data. I think this was Python 2.1 or 2.2, when there wasn’t a first class boolean type. In any case, magically the values were being written out to disk as “true” and “false” rather than “1” and “0”, and when reading those back in as integers, we got errors. In most statically typed languages (at least the ones with a boolean type) this would have been an error, or at least a compiler warning. A unit test would have helped us find this before it was deployed.

    Anyway, again, I never said that unit tests were unnecessary, so please don’t use my omission of that explicit statement to infer it. I feel quite the opposite, and my experience with Python only reinforces that need.

    Joe

  3. Lennart Regebro permalink

    Thanks for the response!

    “but I never implied that a statically typed environment was inherently less buggy or better suited than a dynamic one”

    Well, this is exactly what you *were* saying, because you are saying that you chose a static language because of this exact reason. So don’t try to back out now, you *did* say that. And in fact, you say it here again, in the explanation of the bug, where you say you wouldn’t have gotten this error with a staticly types language. And you are right, you wouldn’t.

    In my opinion, you would have gotten other errors, because the dynamic nature of Python means that some things just get much easier and less error prone. So I would guess it evens out in the end. But that’s really just my opinion and my experience, I don’t have any hard arguments for that.

  4. No, I that wasn’t what I said. I didn’t say that a dynamic language was inherently more buggy overall, just that a certain class of bugs, like the one I mention above, are more likely to happen. Having been burned by some of those bugs (and admittedly lax testing and other supportive measures; it was the first “real” Python project I was involved with), this was one of a few motivating factors for us to choose C# for a later project.

    I don’t doubt that the experience in Python would have been different, although whether it would have been better is unknowable. I loved the freedom that just being able to build tuples and pass them around, for example, without needing to define a throwaway class or radically change function signatures everywhere.

    Again, don’t construe my blog post as an argument against testing. The blog post ultimately wasn’t about Python or development methodology — although multiple people seem to have honed in on that paragraph specifically — so I didn’t feel that I had to do a treatise on that. Maybe I’ll do another blog post.😉

    Thanks,
    Joe

  5. Lennart Regebro permalink

    So you say it is NOT more buggy, and that was a motivating factor for choosing something else?
    Can you explain that logic for me?

    It seems to me that if it should be a motivating factor, is actually has to be more buggy. And also, it seems to me that when you say you have to test it more, you are also saying itä’s more buggy.

    Sorry, Joe, it still seems to me you are claiming you get more bugs with dynamic languages. And I’m not even arguing with you because just one persons experience (mine) isn’t enough to actually say anything for sure.

    What I did argue though was that testing is not necessary because you have a dynamic language, which you also implied. Basically you said that with static languages you need less testing. And I argue that you need just as much testing anyway.

    And now I guess you are going to claim that you didn’t say that either. But the fact is Joe, you did. I don’t know what you *wanted* to say. I can only see what you actually said. And I don’t see why you are arguing about it, if you no longer agree with the things you said.

  6. I’m saying that *class* of bug is what bit us, and I believe it is more prevalent in a dynamic language like Python than a static one like C#. Whether or not that leads to buggier programs, I think we’re both in violent agreement that we have no way of knowing. I also think that comprehensive unit tests are a way to mitigate that class of bugs, but that shouldn’t be construed as a reason not to do unit tests in something like C#. There are certainly classes of bugs which appear more in statically typed languages, and you are definitely sacrificing some “freedom of motion” by using one and that should be a consideration when choosing a development environment.

    It feels to me like you have an ax to grind with regards to static vs. dynamic languages. Myself, I don’t really care. I see uses to both and do use both. C# and Python happen to be my two favorite languages, not that it really matters.

    I honestly do not see how my previous comments are inconsistent with each other or my original post.

    Regards,
    Joe

  7. Lennart Regebro permalink

    No, I don’t have an axe to grind. I just don’t understand why you deny that you wrote the following:

    “Writing real applications in Python requires a discipline that unfortunately most people (including myself) are unwilling to adhere to, and this easily leads to buggy and hard to maintain programs. You have to be very diligent about unit tests and code coverage for every line of code, because you can’t rely on the compiler to catch errors for you.”

    This *is* saying that dynamic languages have more bugs unless you test them. There is no other possible interpretation, sorry, and I don’t see why you want to turn it into a flame war. Flame wars make me irritated and grumpy, so I’m not likely to debate this issue any more.

  8. I don’t deny that I wrote it, and I think it’s true. If you don’t test for these sorts of bugs, your program will have them. Just like how if you have no testing at all on your program, regardless of how or in what it’s written, you will have bugs. And yes, more bugs than you would otherwise have if you didn’t test.

    I don’t think it’s a logical conclusion that in absence of testing a dynamic program will have more bugs than a static one in some sort of absolute measurement. But it will have more of a certain class that is typically caught by the compiler in a static one, as I’ve mentioned.

    I’m speculating, but perhaps you took offense to my characterization that “most people” are unwilling to adhere to such a discipline. This isn’t to say that you are; I have no idea what your code is like. But given how popular Python is these days and the distressing lack of testing much software gets, I don’t think that statement is unfair.

    As an aside, in re-reading my post, I realize that my parenthetical phrase “including myself” makes it seem like this is the case in the present tense, and that was a mistake. It was certainly true for that project, but it also instilled in me this opinion. I can appreciate the discipline now.

    I apologize if I’ve come across as flamey, it’s not been my intention to start a flame war.

    Regards,
    Joe

  9. I think it’s reasonable to say that in a statically typed language, the compiler will catch more errors than in a dynamically typed language. At least if it has a reasonably expressive type system, like Java 5 with generics or C++ with templates. The more primitive static typing in languages like C or the older Java is not really useful. (I don’t know about C#.)

    And I think this is a good thing, because the sooner you discover an error, the cheaper to fix it. Compiler error messages usually points you more accurately to the error than a failed unit test.

    On the other hand, many dynamically typed languages tends to have no “upfront compiler” (i.e. a compiler you run before deployment, as opposed to just-in-time compiler) at all.
    This is quite a disadvantage i many cases.

    BTW, statically typed languages is not only C, C++, C# and Java. Have a look at Haskell which sort of takes static typing to another level.

  10. “I think it’s reasonable to say that in a statically typed language, the compiler will catch more errors than in a dynamically typed language.”

    It will catch misspellings, when you try to use a variable that is not defined. In dynamic languages that isn’t a syntax error, so the compiler doesn’t catch it. But using something like pyflakes will, as it will warn you that the variable you defines isn’t actually being used.

    A compiler will also catch more (but not all) mistakes when you assign to the wrong variable. But do you do that often? I don’t think I do.

    “On the other hand, many dynamically typed languages tends to have no “upfront compiler” (i.e. a compiler you run before deployment, as opposed to just-in-time compiler) at all.
    This is quite a disadvantage i many cases.”

    That’s possible. I’m only used to python when it comes to dynamic languages, where you of course can run the compiling from the command line if you want.
    But you rarely want to, as pyflakes will find syntax errors and also complain about style errors, like “bare excepts” and similar. This is typically at least as fast as compiling, and catches more errors, so thanks to pyflakes at least python people get the minor error catching of compiler and more. But I know Pythonistas are spoiled.😉

    Since I often do test-driven development I don’t bother about pyflakes unless I’m going to do a release, and I probably should use it more often. What I really should do is make a pyflakes macro for my IDE, so I just need to press a button.

    The Delphi IDE had something like that. You just pressed Ctrl-F5 and it would simply take you to the first syntax error you had. I quickly found that I pressed Ctrl-F5 almoust automatically as soon as I wasn’t actually typing code, just to make sure I hadn’t misspelled anything.

  11. Idetrorce permalink

    very interesting, but I don’t agree with you
    Idetrorce

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: