Where modules go to die

When I wrote my “Python doesn’t play well with others” post a couple of weeks ago, I didn’t know that I was treading on ground already well-traveled by Kenneth Reitz. I should have known that the programmer behind the envoy library—which I mentioned in that post and which I’ve adopted as my go-to library for running external processes—would have an overarching interest in making Python libraries simpler and more obvious for common uses.

In addition to envoy, Reitz is the lead developer of

He’s prepared a set of slides for a talk entitled “Python for Humans,” a shortened version of which he delivered at a Django conference earlier this month.

In this abbreviated talk, he spent most of his time on his request library and the problems with urllib2 that he was trying to solve. Reitz’s position is that several of the modules in the standard library violate one or more of the Zen of Python rules, especially

Beautiful is better than ugly.

If the implementation is hard to explain, it’s a bad idea.

and

There should be one—and preferably only one—obvious way to do it.

In the talk, Reitz describes several possibilities a new Python programmer would have to sift through if she were trying to write a program that communicated via HTTP. We saw a similar thing in my subprocess post: the standard library contains at least three ways to call an external program from within Python.

I suspect that some of the standard libraries are convoluted because there’s a sense that they have to cover all cases. The implementation then gets weighted down by the need to handle every edge condition imaginable, and the simple, common uses are buried under layers of objects and method parameter lists.

Reader Carl said, in a comment to another of my Python library complaints

The problem with the “batteries included” philosophy of Python is that a lot of batteries got written back in the Python 2.0 days and haven’t been updated since, but because they already exist, no third party libraries catch on to serve the same niche.

This is exactly right. The standard libraries constitute one of Python’s great features, but they tend to suck up all the oxygen. Programmers are reluctant to write libraries that duplicate their functions, so poor libraries in the standard set persist. Only a few people, like Reitz, are willing to write modules that compete with the standards.

Near the end of his talk, Reitz says, “The standard library is where modules go to die.” An overstatement, certainly, but with more than a germ of truth. Once a library is enshrined in the standard set, it can’t change radically because too many programs rely on it—and its bugs, idiosyncrasies, and complications—remaining stable.

Several Python programmers on Hacker News thought my complaints about the subprocess module were unfounded. I was glad to see that core developer Nick Coghlan agreed with me, as did Reitz:

An awesome article on the frustrations of using the subprocess module: leancrew.com/all-this/2012/…
  — Kenneth Reitz (@kennethreitz) Tue Apr 17 2012

The “awesome” is hyperbole, but it’s nice to see people with real influence in the Python community recognizing the value of making the language and its libraries accessible to lesser programmers.


6 Responses to “Where modules go to die”

  1. Jamie says:

    I write Python daily.

    I really need IPC and similar to come up to the standards Perl has. I’m not in a place to make that happen, but I wish someone was. Perl is becoming the same background noise that so much else is, so something has to replace it. That something should naturally grok how unix command lines work.

    I can imagine disagreeing with that last statement, but you better be Steve Jobs if you want to construct an agument that convinces me (and maybe not then, because iOS is severely limiting).

  2. Steve Holden says:

    It’s not just a matter of making it “more accessible to lesser programmers”—the poor APIs get in the way of facile programmers too. Now we have tools like virtualenvwrapper and pip it is much easier to create tool sets with sensible descriptions of their dependencies. So, we are getting there, but only slowly.

    Nowadays there’s no reason why every Python application shouldn’t be capable of installing all the necessary libraries required to support it, assuming they are not already present in the environment. Except, of course, that large amount of work to engineer it for existing applications. Open source isn’t perfect and it never will be, but if a need is common enough it’s “likely” that it will, eventually, be met.

  3. Marco Lopes says:

    It would be cool to have a PEP20 score on std lib modules, and based on it, create a list of modules that could use a “python for humans” version of that module. This could work as a way to suggest possible projects for developers looking to contribute to python and open source.

  4. R. David Murray says:

    I haven’t read the linked articles, but if I understand what you mean, I’m working on humanizing the email package, and doing it within the stdlib development process. It is possible to do that. We could use more help.

  5. Mathieu Virbel says:

    Compete with standard libraries for is great, and standard module can/could die, ok.

    But where the standard is really defined? What if we made the usage of request, envoy, … as a standard for Python application?

    In order to compete completly, a refined documentation is necessary. A place where people can browse new trending python way to do work. Introduce virtualenv as a standard. Introduce pip to install package. Instead of documenting how to do http request with urllib, go directly with requests…

  6. Clark says:

    Yould think that when they did the big revisions in Python 3.0 that they would have made some serious revisions of the libraries.

    Still Python has gotten better. Auto-installing libraries would be nice but there are a slew of issues with that. (Remember dll hell? Well with all the changes in Python in many ways we are there again.) Over all though I’m pretty impressed with how smart easy_install has gotten. And there are some better third party libraries out there.