Python problems on Lion (mostly self-inflicted)

The upgrade of my MacBook Air to Lion went gone pretty smoothly, but I had problems with Python,1 some of which were my fault, some of which were Apple’s.

Unsurprisingly, the default version of Python on Lion is from the 2.7 branch:

$ python --version
Python 2.7.1

You can, if you wish, run earlier versions instead:

$ python2.6 --version
Python 2.6.6
$ python2.5 --version
Python 2.5.5

Versions earlier than 2.5 aren’t available. The versions available in Snow Leopard were 2.6.1 and 2.5.4.

The first Python program I tried to run was Dr. Twoot. Dr. Twoot itself is written in JavaScript, but since Twitter changed from basic authentication to OAuth, I’ve had Dr. Twoot communicate with the Twitter server via a CGI script (written in Python) running under the Apache server that comes with OS X.

Dr. Twoot failed for what turned out to be a variety of reasons. First, Apache wasn’t running, so I guess the Lion updater didn’t maintain the Web Sharing setting.

Web Sharing setting

Dr. Twoot continued to fail after I turned Web Sharing on, apparently because of some setting in Lion’s /etc/apache/httpd.conf file. I couldn’t access anything under the DocumentRoot, including the necessary CGI script. Fortunately, the Lion updater had preserved my old http.conf under a different name, and when I changed the names and restarted Apache, I was able to access the DocumentRoot again.

But that didn’t mean Dr. Twoot was working. The CGI script was throwing an error, which was a Python problem. The error was a missing third-party library, oauth2. This error made sense to me:

  1. The CGI script is run by the default Python, /usr/bin/python.
  2. The default Python is version 2.7.
  3. Version 2.7 looks for third-party libraries in /Library/Python/2.7/site-packages.
  4. The outh2 library can’t be in the 2.7 site-packages directory because I installed it when Python 2.6 was the default.

The long-term solution would be to install oauth2—and all the other third-party libraries I use—in /Library/Python/2.7/site-packages, but I wanted to get Dr. Twoot up and running quickly, so I changed the shebang line of the CGI script from

#!/usr/bin/python

to

#!/usr/bin/python2.6

because I knew I had oauth2 installed the 2.6 site-packages directory.

When that failed, and the error message still said the oauth2 library couldn’t be found, I navigated to /Library/Python/2.6/site-packages to see what was wrong. Here’s what I found:

$ ls
README

There were no libraries in /Library/Python/2.6/site-packages, just a worthless README. Which means that the Lion updater had overwritten my entire /Library/Python/2.6/site-packages directory.

I’m pretty sure this isn’t kosher. The whole point of a site-packages directory is to keep system updates and third-party libraries separated and unaffected by each other. I checked my office computer, which is still running Snow Leopard. It has libraries in both /Library/Python/2.5/site-packages and /Library/Python/2.6/site-packages. The former were installed under Leopard (when the default Python was version 2.5) and the latter were installed under Snow Leopard; the Snow Leopard update didn’t blow away all libraries installed under Leopard. Lion, unfortunately, wasn’t as well-behaved.

So I set about installing the missing libraries. At first, easy_install failed with every library I tried. Eventually, I realized that I was using a mismatched version of easy_install in /usr/local/bin instead of the Lion-supplied one in /usr/bin. I don’t know why there was an easy_install in /usr/local/bin, but after getting rid of it, the troubles went away.

Until I tried to install appscript. The appscript library, which I use in several scripts to do AppleScript-like things in Python, requires the compilation of some C source. The compilation failed with all sorts of nasty error messages referring to llvm-gcc-4.2. I knew that Apple had switched from the old gcc compiler, and wondered whether that was the problem.

Several Python libraries won’t compile with llvm-gcc-4.2. Is there a way to force easy_install/setuptools to use plain ol’ gcc?

7:44 PM Sun Jul 24, 2011

Almost as soon as I tweeted this, I realized what the problem was: I’d forgotten to update Xcode after updating to Lion. As I began the download of Xcode from the Mac App Store, I got this confirmation from @mr_lightwolf:

@drdrang If you didn’t reinstall Xcode after upgrading Lion, tons of stuff will break.

8:22 PM Sun Jul 24, 2011

Xcode 4.1 took over an hour to download and install, but when it was done easy_install worked perfectly for all my libraries.

To summarize, if you haven’t updated to Lion yet, and you want to avoid the problems I had, be sure to

  1. Make a list of all the third-party libraries you currently have installed under Python 2.6, because you’ll have to reinstall all of them for 2.7 and you won’t have the 2.6 site-packages directory to refer to.
  2. Update to Lion.
  3. Update to Xcode 4.1. It’s free in the App Store.
  4. Make sure the easy_install you use is the one in /usr/bin.

On the non-Python front, if you run a local web server, you might want to put a copy of all your Apache configuration files into your home directory (or Dropbox) before updating to Lion. My configuration files were preserved by the updater, but a belt-and-suspenders approach is safer.

I suppose I’ll go ahead and install Python 3.2 in the next few weeks. None of my workaday scripts use it, but now that it’s 2½ years and 2 point releases old, it’s probably time to get with the program.


  1. Problems with Pythons would be a good title for a Gilderoy Lockhart book. 


10 Responses to “Python problems on Lion (mostly self-inflicted)”

  1. alexandru says:

    A few notes and things that I have learnt, often painfully:

    • Use virtualenv to manage packages as much as possible. It helps keep things clean in general. And since it stores libraries in ~/.virtualenv/ you’d be safe through an upgrade (personally, I installed Lion from scratch)

    • Use pip instead of easy_install. You can get a printout of all the packages installed, save them to a file, and install everything from that file next time you set up a virtualenv.

    • Consider homebrew which will install its stuff in /usr/local/ and shouldn’t be touched by the OS for any reason. I use Python from homebrew because of some local patching I need to do. Homebrew has Python 3.x as well and it should make installing it easier.

    • Submit a bug report to Apple. I agree that this is a pretty big issue, and I reckon I’d be pissed if I lost a bunch of stuff from site-packages the way you did. This has a lot to do with the install path for these packages, which is why I prefer the homebrew approach of /usr/local/ (which, as the project states, is something Apple says they don’t touch, and never put anything into themselves)

  2. speno says:

    Another vote for homebrew. Then you can worry about virtualenv and pip etc. You really don’t want to install Python modules anywhere that your vendor controls and is likely to mess with. Homebrew installs everything in /usr/local.

  3. Dr. Drang says:

    Been meaning to look into pip; inertia’s been keeping me with easy_install.

    How does virtualenv compare with pythonbrew?

    As for Hombrew, I know everyone says it’s great, but people said the same thing about Fink and Mac/DarwinPorts, and both of those turned into dependency nightmares. I’m still leery and prefer to install extras “by hand” in /usr/local.

  4. speno says:

    I saw pythonbrew but wasn’t interested in it so I can’t speak on it.

    Homebrew just automates the way you already want to work. It compiles stuff for you and sticks it in /usr/local. It’s the first package system that does that, or the first one that I actually like. You still get that “by hand” feel except it’s not “by hand” anymore. :-)

    My biggest gripe with easy_install was .egg files. I don’t want .egg files. I know tricks to make it install in non .egg format but pip just does that by default.

  5. Clark says:

    Have you been able to compile the full appscript with Lion? I don’t mean what gets compiled with easy_install but the actual appscript project. I futzed around with it for several hours this week but never made a whole lot of progress - probably due to ignorance on my part of Xcode 4. It was pretty frustrating as I was hoping to test out some obj-C appscript stuff.

  6. Dr. Drang says:

    Clark,
    No, I just did the Python library. No particular interest in ObjC programming at this stage.

  7. Allen MacKenzie says:

    Curses. Things breaking without an Xcode reinstall explains why I never could get any Octave packages to install/compile last week. But I don’t think I have enough hard drive space to reinstall Xcode at the moment. Something I’ve been meaning to do something about, but haven’t had time yet.

    Like I said: Curses.

  8. Shayne says:

    I still have no damn idea where Apple is putting my django files so I can point eclipse at it. Its really really productivity killing and annoying. I just dont get why apple cant leave that stuff alone. Not broken, not for fixing, etc. :(

  9. philotas says:

    If I haven’t upgraded to Lion already, couldn’t I just copy the /Library/Python/2.6/site-packages from SL to /Library/Python/2.7/site-packages on Lion (given that they are compatible with 2.7)?

  10. Dr. Drang says:

    You can do that, philotas, but you should copy the 2.6/site-packages contents to a safe place before upgrading to Lion. Recall that my 2.6/site-packages got wiped out by the upgrade.

    I still think the 4-step process outlined near the end of the post is the best approach. It’ll give you the appropriate modules for 2.7.