Archive for the ‘blogging’ Category
Flickr updates in the sidebar
June 30th, 2008
I just added a section to the sidebar to display my most recent upload to Flickr. The flickrRSS plugin from Eightface makes it very simple. I almost never use my Flickr account, but I hope to change that now that I’ve added the email address for Flickr uploading to my iPhone.
The photo will be 100 pixels wide. Because the width of the sidebar changes with the user’s font size (it’s specified in ems), the photo will take up different amounts of the sidebar for different users. But it should always fit unless you use a really small default font.
I’ve also eliminated the RSS feed links from the sidebar, since browsers now look for feeds on their own and display them in the toolbar.
Automated charts with Gnuplot
June 19th, 2008
Every month since the fall of 2006, I’ve written a post like this one, in which I include a chart of US and coalition fatalities. This is a reduced version of the latest:
I get the data from icasualties.org and generate the chart using Gnuplot, the venerable Unix plotting program. I figured it was about time I showed how I do it.
Gnuplot began its life as a terminal program, and it’s still driven by typed commands rather than by clicks and drags. This makes it a bit difficult to learn, but because it allows sequences of commands to be stored in files, it makes the creation of several similarly-formatted graphs a snap. I started using it when I switched to Linux in the mid ’90s because there were no good Excel-like programs on that platform. I’ve kept using it after switching back to Macintosh because it’s so good at producing the kinds of graphs common in science and engineering. I’m not big on pie charts.
The data for the graphs are stored in a file called “icasualties.txt” that I update every month. The first several lines of that file look like this:
Month US UK CO
2003-03 65 27 0
2003-04 74 6 0
2003-05 37 4 1
2003-06 30 6 0
2003-07 48 1 0
2003-08 35 6 2
2003-09 31 1 1
2003-10 44 1 2
2003-11 82 1 27
2003-12 40 0 8
The columns are separated by spaces; it doesn’t matter that the number of spaces differs from column to column or row to row. I think all columns but the last are self-explanatory. The “CO” column contains the count of military deaths from coalition countries other than the United States and United Kingdom.
The file of Gnuplot commands that generates the graph is called “icasualties.gp.”
1: # input format for dates
2: set timefmt "%Y-%m"
3:
4: # horizontal (time) axis layout
5: set xdata time
6: set format x "%b\n'%y"
7: set xtics "2003-01", 60*60*24*365.2425/4, "2008-12" # quarterly
8: set mxtics 3 # monthly
9:
10: # left vertical axis layout
11: set ylabel "Military Deaths" 2,0
12: set yrange [0:150]
13: set ytics 25
14: set mytics 5
15:
16: # overall layout
17: set title "Iraq War Timeline" 0,-.5
18: set grid
19: set key at "2007-10",20 right width -3 samplen 1.5 box
20:
21: # Make labels for the totals
22: ustot = `perl -e '$s=0;while(<>){($m,$a,$b,$c)=split;$s+=$a}print$s;' icasualties.txt`
23: tot = `perl -e '$s=0;while(<>){($m,$a,$b,$c)=split;$s+=$a+$b+$c}print$s;' icasualties.txt`
24:
25: # Make label for data source
26: set label 3 "Data source: http://icasualties.org" at '2003-05',10 left
27:
28: # choose output and plot it
29: set terminal aqua 0 title "Timeline" size 800 600\
30: fname "Helvetica" fsize 14
31: plot "icasualties.txt" using 1:2 title sprintf("US only (%d)",ustot)\
32: with linespoints pointtype 2 linetype 3 linewidth 3,\
33: "icasualties.txt" using 1:($2+$3+$4) title sprintf("Coalition (%d)",tot)\
34: with points pointtype 3 linetype 1
Like Perl, Python, and the shell, Gnuplot comments start with a hash mark (#). As you can see, most of the commands “set” a Gnuplot parameter that controls either the input or output formatting. Two of the other commands create variables for later use, an the final command creates the plot itself. Here’s the explanation:
Line 2 tells Gnuplot that some of the input data will be time values and that they will be formatted with a 4-digit year (%Y) followed by a hyphen and a 2-digit month (%m). The codes follow the conventions of the well-known strftime C library.
Lines 5-8 cover the formatting of the horizontal (x) axis. Line 5 says that the x-axis will consist of time values. Line 6 sets the format of the axis labels to a 3-letter abbreviation of the month name (%b), new line (\n), an apostrophe, and a 2-digit year (%y). Line 7 sets the major tic marks and the labels to the start of every quarter, which is kind of tricky. The three arguments to set xtics are
- Where we should start counting: 2003-01. This is the January before the war began, and I chose this date to insure that the major tics marks fall on the usual quarterly start dates: January 1, April 1, July 1, and October 1.
- The spacing between the major tic marks, in seconds. Using 356.2425 as the number of days in a year is overly precise, but I’m a big fan of the Gregorian calendar reform.
- Where we stop counting. I’ll have to change this next year.
Line 8 tells Gnuplot to split the space between major tic marks into three parts and put minor tics at the splits.
Lines 11-14 cover the formatting of the vertical (y) axis. The numbers after the axis label in Line 11 nudge the label a little to the right, because I thought the default location was too far from the axis. Since Gnuplot will choose the range and tic locations if they’re not specified, Lines 12, 13, and 14 set the range and spacing to get a consistent vertical axis every time I generate a graph.
Line 17 sets the title at the top of the graph and nudges it down a bit from its default location. Line 18 tells Gnuplot to put faint gridlines that run the full width or height of the graph at every major tic mark.
Line 19 puts the key (or legend) near the bottom of the graph and puts a box around it. The at "2007-10",20 right part positions top right corner of the key at those coordinates. The width -3 part makes the box a bit smaller than its default width. The samplen 1.5 part makes the blue and red point/line examples a bit wider than the default.
Lines 22 and 23 are tricky. I wanted to put the US and coalition casualty totals on the graph, but as far as I can tell, Gnuplot doesn’t have a builtin way to get those figures. But it does have a way of calling another program. So these two lines contain short Perl scripts that scan through the “icasualties.txt” file and sum up the US and full coalition figures. The sums are stored for later use in the Gnuplot variables ustot and tot.
Line 26 puts the acknowledgement text near the bottom left of the graph. The commands for positioning are similar to those in Line 19.
Lines 29-30 is one long Gnuplot command split over two lines. It tells Gnuplot to display the graphs in an 800x600 AquaTerm window with 14-point Helvetica as the base font. The set terminal command is difficult to learn, but makes Gnuplot very flexible in its output. Although I want the graphs in the form of a PNG file, I chose aqua terminal over the png terminal because the precompiled Gnuplot I got with Octave (see this post) doesn’t have support for Macintosh fonts built into its png terminal. And I don’t feel like gathering all the libraries necessary to compile my own version.
Lines 31-34 are what we’ve been leading up to. This is one long Gnuplot command that actually created the plot according to the specifications given in the previous lines. It makes one graph for the US casualties and one for the total coalition casualties. The key labels these data series include the total casualty counts calculated back in Lines 22 and 23. The presentation styles (line type, line width, color, and point type) have numbers rather than names, so there’s usually a bit of trial and error before you hit on a combination you like.
I create the graph by typing gnuplot icasualties.gp into Terminal. AquaTerm launches and shows the graph. I then do a screenshot of the graph and upload it to my server as a PNG file. The whole thing takes less than a minute.
As you can see, Gnuplot is very flexible but very complex. I find that when I’m using it a lot—for example, while writing a report that reduces and presents a lot of data—I get into a rhythm and the command come naturally to my fingertips. But when I’ve been away from Gnuplot for a few months, there’s always some frustration when I start back again.
Good documentation would go a long way toward relieving that frustration. Unfortunately, Gnuplot’s documentation, while quite complete, is terribly difficult to use because it’s organized alphabetically by command. So it’s great if you know the command, but if you knew the command you probably wouldn’t be looking in the manual.
It looks like help is on the way. Gnuplot in Action is a book by Philipp Janert that is scheduled to be published by Manning later this year. Janert and Manning have graciously given me a free review copy of the Early Access Edition of the book—basically a PDF of most of the book in a pre-publication state—and I’m optimistic. The best part of the book can be seen from its table of contents. Janert presents Gnuplot concept by concept rather than command by command. There’s a chapter on axes, a chapter on styles, a chapter on scripting, etc.
He has not, however, simply taken the official Gnuplot manual and rearranged it (although that would be valuable in itself). The chapters are filled with examples showing both the Gnuplot command and the graphical result. The official manual has many examples, but because it’s still text-based, it can only tell you what the commands will do—it can’t show you. I do wish Manning would put the commands and output side by side as is done in The LaTeX Companion, but the sequential layout still gets the job done.
One aspect of the book I don’t expect to like is hinted at in its subtitle, Understanding Data with Graphs. In addition to Gnuplot itself, Janert apparently wants to teach me how to use graphs generally to analyze data. Since that section of the book hasn’t made its way into the Early Access Edition yet, I don’t know what he’s going to say, but I’ve been analyzing data with graphs for 30 years, and I doubt he’s come up with anything new. But that section may be helpful to others.
This isn’t a review. The book isn’t close enough to publication quality for a review to be fair. It is, however, already better at teaching Gnuplot than the official manual is. In some ways, its better than the manual even as a reference because it’s better organized. I’m keeping an eye on it and will write up a review when it’s near its final form.
Snapshot/upload utility with GUI
May 9th, 2008
This post describes a short utility that streamlines my workflow in taking screenshots and uploading them to my server. It has a simple GUI and can create both a full-sized and reduced-sized image in a single step.
Longtime readers of this blog will note that I wrote a very similar utility a couple of years ago and decribed it in this post. In those days, I would invoke the script through Quicksilver, using QS’s Run… action to pass parameters—e.g., the file name—to the script. It worked well, but three things pushed me to write a new version:
- QS’s Run… action stopped working consistently. It would often disappear entirely from my iBook, leaving me no way to use the script.
- I switched from Quicksilver to LaunchBar, in large part because of QS’s quirks. (Here is more complete rundown of my reasons for switching.) But because Launchbar doesn’t allow parameters to be passed to scripts, the older script couldn’t be called from it.
- I wanted a graphical user interface so I didn’t have to remember the order of the options.
I call the new program snapftp, and I invoke it through FastScripts. I’ve given it a keyboard shortcut of Control-Option-Command-4, with is similar to the Apple-standard Command-Shift-4 for snapshots of a portion of the screen. The program launches and presents this dialog box.
The filename for the snapshot is entered without the “.png” extension. Three types of snapshot are possible:
- A single capture file in the original size.
- A single capture file, resized according to the width parameter.
- Two capture files, one original and one resized.
By default, the capture files are saved to my Desktop and uploaded to the “images” directory on my server. The checkbox option at the bottom of the window can preclude the upload.
When I click on the Snap button (or press the Return key) the dialog box goes away, and the computer acts very much like it does when I press Command-Shift-4. The biggest difference is that whereas Command-Shift-4 starts in rectangle capture mode, snapftp starts in window capture mode. I prefer starting in window capture mode because I usually want a snapshot of a window. If I need to, I can change to rectangular capture mode by pressing the Space bar.
After the snapshot is made, the capture file appears on my Desktop and, unless told otherwise, is uploaded to my server via FTP. The URL of the uploaded file is put in the Clipboard for later pasting.
If the “Both” option was chosen, there are two capture files. The full-sized capture file has the name I gave it and the resized file has that name with a “-t” appended to it before the “.png” extension (the “t” is for “thumb” even though the resizing may produce an image much bigger than what is normally considered a thumbnail). For example, if I choose “snap” as the file name, “snap.png” will be the full-sized capture file and “snap-t.png” will be the resized capture file. Only the URL to the uploaded version of “snap.png” is put on the Clipboard.
There are several choices for adding a Mac-native graphical user interface to a script. The methods I considered are:
- The
display dialogcommand in AppleScript. The information available from this is limited to a line of text or a single button click, which is too little for snapftp. - CocoaDialog. This has more options than “display dialog” and easily called from any scripting language, but still collects only one piece of information from the user.
- Pashua. This has text input, radio buttons, checkboxes, popup menus, etc. and can collect several pieces of information from the user in one fell swoop. Its downside is that it requires an explicit textual description in the script to position the dialog elements.
- On My Command. Although the primary use of this utility is to create contextual menu items (which I have done), it can also create applications that make full use of nib files created by XTools’ Interface Builder.
- Platypus. Although this is supposed to provide a simple way to “Mac-ize” command-line scripts, it doesn’t seem to have any user interface hooks. Perhaps it’s meant only for dropping files onto an app.
Because it provided everything I needed for snapftp and didn’t force me to learn Interface Builder (which has always seemed exotic and scary to me), I chose Pashua. There was some trial and error involved in positioning the various parts of the GUI, but it wasn’t too painful.
Here’s the snapftp source code:
1: #!/usr/bin/python
2:
3: import Pashua
4: import sys, os, shutil
5: from subprocess import *
6: from ftplib import *
7:
8: # FTP and local parameters
9: host = "leancrew.com"
10: baseurl = "http://www.leancrew.com/all-this/images"
11: extension = ".png"
12: user = "drdrang"
13: passwd = "itzaseekret"
14: ftpdir = "public_html/all-this/images"
15: localdir = os.environ['HOME'] + "/Desktop"
16:
17: # Dialog box configuration
18: conf = '''
19: # Window properties
20: *.title = Snapshot FTP
21:
22: # File name text field properties
23: fn.type = textfield
24: fn.default = snap
25: fn.width = 264
26: fn.x = 94
27: fn.y = 130
28: fnl.type = text
29: fnl.default = File name:
30: fnl.x = 20
31: fnl.y = 132
32:
33: # Radio button group properties
34: rb.type = radiobutton
35: rb.option = Original
36: rb.option = Resized
37: rb.option = Both
38: rb.default = Original
39: rb.x = 94
40: rb.y = 52
41: rbl.type = text
42: rbl.default = Capture:
43: rbl.x = 30
44: rbl.y = 92
45:
46: # Resized width text field properties
47: rw.type = textfield
48: rw.default = 400
49: rw.height = 22
50: rw.width = 60
51: rw.x = 263
52: rw.y = 71
53: rwl.type = text
54: rwl.default = width:
55: rwl.width = 50
56: rwl.x = 215
57: rwl.y = 73
58:
59: # Local files checkbox properties
60: lf.type = checkbox
61: lf.label = Local files only
62: lf.x = 32
63: lf.y = 5
64:
65: # Default button
66: db.type = defaultbutton
67: db.label = Snap
68:
69: # Cancel button
70: cb.type = cancelbutton
71: '''
72:
73: # Open the dialog box and get the input.
74: dialog = Pashua.run(conf)
75: if dialog['cb'] == '1':
76: sys.exit()
77:
78: # Go to the localdir.
79: os.chdir(localdir)
80:
81: # Set the filenames and url.
82: fn = '%s.png' % dialog['fn']
83: fnt = '%s-t.png' % dialog['fn']
84: url = '%s/%s' % (baseurl, fn)
85:
86: # Capture a portion of the screen and save it to the file.
87: Popen(["screencapture", "-iW", fn], stdout=PIPE).communicate()
88:
89: # Resize the file if asked to
90: if dialog['rb'] == 'Resized':
91: Popen(['sips', '--resampleWidth', dialog['rw'], fn],
92: stdout=PIPE).communicate()
93: elif dialog['rb'] == 'Both':
94: shutil.copy(fn, fnt)
95: Popen(['sips', '--resampleWidth', dialog['rw'], fnt],
96: stdout=PIPE).communicate()
97:
98: # Upload the file unless told not to.
99: if dialog['lf'] != '1':
100: ftp = FTP(host, user, passwd)
101: ftp.cwd(ftpdir)
102: ftp.storbinary("STOR %s" % fn, open(fn, "rb"))
103: if dialog['rb'] == 'Both':
104: ftp.storbinary("STOR %s" % fnt, open(fnt, "rb"))
105: ftp.quit()
106:
107: # Put the URL of the uploaded file onto the clipboard.
108: Popen('pbcopy', stdin=PIPE).communicate('%s/%s' % (baseurl, fn))
My original snapshot utility was written in Perl, but I wrote snapftp in Python. Although Python is my current first choice for a scripting language, I would have stuck with Perl—there would have been much less rewriting—except that I found that Python is curiously faster at launching Pashua than Perl is when invoked from FastScripts. I started with a Perl program that just brought up the dialog box, thinking I would add the logic from the old program later. Although this Perl skeleton was fast on my Intel iMac, it took more than 10 seconds to launch Pashua and bring up the dialog box on my iBook G4, which was far too long. I then switched the skeleton program to Python and found that the dialog box came up almost instantly. I have no idea why.
Lines 9-15 provide the FTP and local parameters needed to ensure that the capture files end up where I want them. If you want your own version of snapftp, you’ll have to change these lines.
Lines 18-71 contain the Pashua configuration for the dialog box. One oddity of Pashua is that it wants the element positions given by their left and bottom coordinates rather than left and top. I got goofy looking results until that sunk in.
After getting the information from the user and putting it into the dialog dictionary (Lines 74-76), snapftp starts the built-in screencapture command-line program. The i option puts it in interactive mode, and the W option starts the interaction in window capture mode. As I said earlier, the user can switch to rectangular capture mode by pressing the Space bar.
If the user asked for a resized version of the capture, Lines 90-96 do the resizing with the built-in sips command-line program. The resampleWidth option sets the width of the resized image but keeps the aspect ratio constant. I’ve set this up to resize on the basis of width because I want to make sure the images fit within the content (white) area of the blog, which is sized according the the reader’s default font size. The default width of 400 pixels should make the image fit even for visitors who use a small default font size.
The last section of the program, Lines 99-108, uploads the image(s) to my server via FTP and puts the URL on the Clipboard using the pbcopy command. Of course, this happens only if the “Local files only” checkbox remains unchecked.
Both sips and screencapture are invoked through Python’s relatively new subprocess module. It’s supposed to replace the popen module, which I had just gotten used to. I still prefer Perl’s backticks. And while I’m complaining (mildly), let me say that I also prefer Perl’s FTP library to Python’s. Perl’s way of handling calls to an outside service is to mimic as much as possible the syntax of the service; Python’s way is to make the call using a Pythonesque syntax. While I understand the Python choice, I think it’s an impediment when—as is usually the case—the programmer knows the syntax of the outside service and knows what he would do if he were using that service directly.
I have snapftp saved in my ~/Library/Scripts folder where FastScripts can get at it. The Pashua application is saved in /Applications, as expected, and the Pashua.py Python module is saved in /Library/Python/2.5/site-packages. The module has one line that I feel is unwarranted and that I have commented out. Down near the bottom of Pashua.py, is a loop that looks like this:
for Line in Result:
print Line
Parm, Value = Line.split('=')
ResultDict[Parm] = Value.rstrip()
I think a print command in a library utility that is supposed to read lines is just plain wrong, so I’ve commented out the print Line.
Overall, I’ve found snapftp to be much nicer and easier to use than my old Quicksilver-based solution (even when it worked). The addition of the GUI has not made it slower, and it is, of course, easier to remember the options when they’re laid out in front of you. This may inspire me to add a GUI gloss to some of my other scripts.
Movable Type to WordPress, Part 4
February 13th, 2008
This will be the last of my posts describing how I moved the blog from Movable Type to WordPress. The first one discussed my rationale for the move; the second covered the export of posts from Movable Type and their import into WordPress, and the special care needed to handle posts written in Markdown; the third one talked about the difference in permalink format between the two blogging engines and how I got MT-style links to point to the right posts after the move. This post will discuss how and why the blog is laid out the way it is.
Let me start with some advice to web designers. Keep your fucking hands off the size of my fonts! I don’t give a flying fuck what your 25-year-old eyes can see—I’m 47, and I can’t read 12 pt Times on a 100dpi screen! Have you noticed how every browser lets the user decide the default font and size? You have? Then where the fuck do you get off overriding that? Do you think I chose 18 pt Georgia as my default font on a whim? I chose it because I’m 47 years old and I need bifocals (sorry, progressive lens) just to read at all! Sure, I used to edit text in 9 pt Monaco back in the 80s, but that was twenty years ago on a 72 dpi monitor—I can’t do it anymore. I hope you all develop early-onset presbyopia. And then die.
So anyway, one of my main concerns in designing the blog was to accommodate the user’s choice in fonts and font sizes, while maintaining a line length conducive to readability. I’ve done that by avoiding pixel-based dimensions in my CSS file. Font sizes are given in percentages, and other dimensions are given in ems, so the layout expands and contracts as the baseline font size changes. You can test this out without changing your browser preferences—just use the “Larger Text” and “Smaller Text” options and watch the blog “breathe.” (In Safari 3, these options are the Make Text Larger and Make Text Smaller items in the View menu. In Firefox, they’re the Increase and Decrease items in the Text Size submenu of the View menu. In Internet Explorer, I don’t know and I don’t care.)
The basic structure of a blog page is simple. There are three parts: the header, the main content, and the sidebar. These are enclosed in a container that’s horizontally centered in the browser window. A skeleton of the HTML looks like this:
<html>
<head>
blah blah
</head>
<body>
<div id="container">
<div id="header">
blah blah
</div> <!-- end of header -->
<div id="sidebar">
blah blah
</div> <!-- end of sidebar -->
<div id="content">
blah blah
</div> <!-- end of content -->
</div> <!-- end of container -->
</body>
</html>
The CSS for the container is
#container {
width: 50em;
margin-left: auto;
margin-right: auto;
background-color: #ded;
}
As I said before, the width is given in ems, not pixels. Setting the left and right margins to “auto” is what centers the container in browser window.
The CSS for the header is
#header {
background: #384338;
padding-left: 3em;
padding-right: 3em;
padding-top: .5em;
padding-bottom: .5em;
margin-bottom: 0em;
color: white;
}
#header h1 {
font-size: 250%;
font-weight: bold;
margin-bottom: .25em;
}
#header h2 {
font-size: 120%;
font-weight: normal;
}
#header a {
color: white;
text-decoration: none;
}
#header a:hover {
color: #ded;
}
Most of this just positions the blog title away from the edges of the header. The text coloring and sizing is pretty much self-explanatory, I think. The last stanza, covering the “hover” psuedo-class, makes the title text—which is a link to the blog’s home page—change color when you hover over it.
I won’t show all the CSS for the sidebar, just the main part:
#sidebar {
font-family: Sans-Serif;
font-size: 75%;
background: #ded;
width: 10em;
float: right;
margin-top: 0em;
padding-top: .5em;
padding-left: 1em;
padding-right: 1em;
padding-bottom: 1em;
}
The important stuff here is how all the text is made smaller (sidebar text is for navigating, not extended reading), and how the entire sidebar <div> is floated to the right. The sizing and styling of headers and links within the sidebar is pretty standard stuff and not worth describing here.
The CSS for the content <div> is
#content {
margin-top: 0em;
margin-right: 9em; /* sidebar: (10 + 1 + 1)*.75 */
padding-top: .5em;
padding-left: 1em;
padding-right: 1em;
padding-bottom: 1em;
background: white;
line-height: 1.25;
}
The most interesting thing here is the right margin. It’s based on the
width and padding of the sidebar, adjusted for the sidebar’s smaller font size. As with the sidebar, the CSS for the elements within the content <div> is pretty standard stuff. The only unusual thing has to do with the styling of computer code. This goes into a <pre> block, which is styled like this:
#content pre, blockquote {
padding-left: 1em;
padding-right: 1em;
background-color: #efe;
}
#content pre {
overflow: auto;
padding-top: .5em;
padding-bottom: .5em;
}
The overflow property puts a horizontal scrollbar at the bottom of the block of code if the lines get longer than the width of the content <div>.
Sometimes I include line numbers with the code. As I described in this post, line numbers get dynamically enclosed in <span> tags and styled like this:
130: #content pre span.ln {
131: color: #888;
132: font-size: 75%;
133: }
As you can see, if the lines are numbered, a button appears at the bottom of the code block to toggle the line numbers on and off.
I think that pretty much covers it. I don’t intend to get into the use of WordPress tags to insert dynamic content into the blog—that’s what the WordPress Codex is for. You can also learn a lot about the proper use of WordPress tags by studying the template files that come with the standard WordPress distribution.
I should mention that the reason I have comments disabled because I haven’t yet figured out a style for comments that I like. But I do intend to get comments going here eventually. In the meantime, you can email or Twitter me at the addresses given near the top of the sidebar.
Movable Type to WordPress, Part 3
February 13th, 2008
The first two parts of this blog’s odyssey from Movable Type to WordPress can be found here and here. The first part discussed the reasoning behind the move, and the second dealt with problems that arise when importing of Markdown-formatted posts into WordPress. Today, I’ll discuss the differences in permalinks between Movable Type and WordPress and how I got my old permalinks working with the new system.
I had my permalinks in Movable Type linked to the monthly archives (I think this is the default, but I don’t remember and can’t go back to find out for sure). So the MT permalink for this post, for example, would be
http://www.leancrew.com/all-this/2008/02/movable-type-to-wordpress-part-3.html
The default permalink format for WordPress is based on the post number, so the WP permalink for this post would be
http://www.leancrew.com/all-this/?p=354
which is certainly more succinct but doesn’t say much about the content of the post.
Fortunately, you can change the format of WordPress permalinks by going to the Permalinks Options page. I changed the format to better match my old Movable Type permalinks by choosing a custom structure of /%year%/%monthnum%/%postname%/.
After this change, the WordPress permalink for this post will be
http://www.leancrew.com/all-this/2008/02/movable-type-to-wordpress-part-3/
The only difference between this and my old Movable Type format is that this has a slash after the title, while the MT link has “.html” after the title.
Getting my new permalinks to match up reasonably well with my old permalinks was not just a matter of aesthetics. Many of my older posts have links to other posts within the blog, and there are links to my older posts at other sites. All of these links will be broken if I don’t take some action. I decided to handle the internal and external links differently:
- For internal links, I went into my posts and edited all the links to reflect the new format.
- For external links, I edited the blog’s
.htaccessfile to rewrite the older-style links into the new format and redirect them.
Editing internal permalinks in over 300 posts sounds like a terrible job, but because I chose the new permalink format to be very close to the older format, the task was really simple. First, I did all the editing in the text file of exported posts from Movable Type before importing them into WordPress (I talked about the MT export file in Part 2 of this series). Editing a single text file that’s sitting on my local computer is much easier than editing hundreds of individual database entries on a remote computer. The necessary change was a simple find-and-replace using regular expressions. The find string was
((www\.)?leancrew\.com/[0-9]{4}/[0-9]{2}/(.+))\.html
and the replace string was
$1/
The “www.” part was an option in the find string because I learned that I had left it out of some internal links.
External permalinks were a bit more difficult. These have to get changed on the fly by my webhost’s Apache server using directives in a special file called “.htaccess” stored at the root directory of the blog. The difficulty came not because the regular expressions for rewriting are more complicated, but because I’m not familiar with the other aspects of Apache’s rewrite rules. After much trial and error, the .htaccess file for the blog now looks like this:
1: <IfModule mod_rewrite.c>
2: RewriteEngine On
3: RewriteBase /all-this
4:
5: # To handle the old MovableType feeds.
6: RewriteRule ^atom\.xml$ feed/atom/ [R,L]
7: RewriteRule ^index\.xml$ feed/ [R,L]
8:
9: # To handle old Movable Type permalinks.
10: RewriteRule ^([0-9]{4}/[0-9]{2}/.*)\.html$ $1/ [R,L]
11:
12: # From WordPress, to allow named permalinks.
13: RewriteCond %{REQUEST_FILENAME} !-f
14: RewriteCond %{REQUEST_FILENAME} !-d
15: RewriteRule . /all-this/index.php [L]
16:
17: </IfModule>
Lines 1-3, 13-15, and 17 were added by WordPress itself when I changed the permalink structure. Lines 5-7 and 9-10 are my insertions. My part and the WordPress part are mixed together because my lines had to be placed after the RewriteEngine and RewriteBase lines and before the RewriteCond lines.
The line for handling permlinks is Line 10, with the search and replace strings very much like what I’ve given above. It was the “[R,L]” option set at the end of the line that took me a while to figure out. And although it works, it still may not be the best way to handle the rewrites.
Shortly after fixing the permalinks, I realized that my RSS feeds had also changed, and that anyone using the old feed URLs would see no updates. Lines 6 and 7 rewrite the Atom and RSS2 feeds from the Movable Type format to the WordPress format.
I believe my next post will cover what I promised last time: the blog’s template and CSS files.
Movable Type to WordPress, Part 2
February 6th, 2008
As the title indicates, this is another post on how I moved this blog from MovableType to WordPress. The first post in the series gave my reasons for making the move. This one will get into the move itself.
Local installation
My first step was to set up a local version of WordPress running on my iBook G4. I wanted the blog to have as little downtime as possible, and I wanted to work out the bugs on a computer I had full control over. Installing WordPress is supposed to be dead simple, and I suppose it would have been if I hadn’t run into problems connecting to the database.
I had MySQL installed and running, using the Hivelogic instructions. So getting WordPress started should have been a simple matter of editing a configuration file and pointing my browser to a particular local URL.
Here’s the configuration file, wp-config-sample.php, that you edit and rename before trying to start up WordPress:
1: <?php
2: // ** MySQL settings ** //
3: define('DB_NAME', 'putyourdbnamehere'); // The name of the database
4: define('DB_USER', 'usernamehere'); // Your MySQL username
5: define('DB_PASSWORD', 'yourpasswordhere'); // ...and password
6: define('DB_HOST', 'localhost'); // 99% chance you won't need to change this value
7: define('DB_CHARSET', 'utf8');
8: define('DB_COLLATE', '');
9:
10: // You can have multiple installations in one database if you give each a unique prefix
11: $table_prefix = 'wp_'; // Only numbers, letters, and underscores please!
12:
13: // Change this to localize WordPress. A corresponding MO file for the
14: // chosen language must be installed to wp-content/languages.
15: // For example, install de.mo to wp-content/languages and set WPLANG to 'de'
16: // to enable German language support.
17: define ('WPLANG', '');
18:
19: /* That's all, stop editing! Happy blogging. */
20:
21: define('ABSPATH', dirname(__FILE__).'/');
22: require_once(ABSPATH.'wp-settings.php');
23: ?>
After setting up a database with the mysql command-line tool, I replaced the DB_NAME, DB_USER, and DB_PASSWORD defaults with the appropriate values, saved it as wp-config.php, and tried to start WordPress. No luck—database connection error. Checked the spelling and tried again—still no luck. Dropped my original database, created a new one, and entered the parameters for that one—still no luck. After a few more rounds, I decided to see if I was in the putative 1% that needed to change line 6. I changed it to
6: define('DB_HOST', '127.0.0.1'); // 99% chance you won't need to change this value
and everything worked fine. I’m still not sure why localhost didn’t work; localhost is set to refer to 127.0.0.1 in my /etc/hosts file. Whatever. Once I got WordPress started, I pushed on.
Importing the old posts
Movable Type has an administration page that lets you export all your blog posts into a simply-formatted text file. WordPress has an administration page for importing posts from a file saved in that format. A match made in heaven—unless your old posts are written in Markdown, which mine are. Here’s what happened.
I figured I had to have the Markdown plugin installed before importing, so I went to Michel Fortin’s site and got both PHP-Markdown-Extra and PHP-SmartyPants. After unzipping the downloaded files, I copied them into the wp-content/plugins directory of my blog and activated them from the WordPress plugins administration page. Then I did the import, which seemed to go quickly and flawlessly; there were no error messages.
When I first looked at the blog, it seemed great. It was styled with the default theme, which I don’t think much of, but I knew I could change that. The important thing was that all the posts were there and their content was intact, and that’s what I thought I had until I looked past the top few posts to an entry that had some programming code. The code was formatted into paragraphs; the line breaks had been lost. (Ironically, the entry with the problem was this one, where I describe how I get nicely-formatted code in my posts.) Further digging showed that all the posts with programming code had the same problem.
Markdown denotes code by indentation; lines that are indented by 4 or more spaces are wrapped in <pre><code> tags. A quick look at the problem entries in WordPress’s editor showed that WordPress had stripped out the leading spaces on all those lines, which is why they were displayed as paragraphs. I posted a plea for help on the Markdown mailing list and was quickly directed (by Sam Angrove) to the culprit in WordPress’s importing script. Line 281 of wp-admin/import/mt.php (in WordPress 2.3.2),
$line = trim($line);
is stripping both the leading and trailing whitespace (including the trailing newline) from each line. Changing that to
$line = rtrim($line, "\n");
fixed the problem. It trims only the trailing newline, which is apparently necessary for the rest of the import logic to work, but leaves the other whitespace intact for Markdown to interpret. Markdown uses leading whitespace for nested lists as well as code, and it uses trailing whitespace for <br> tags.
With the import script changed, I deleted all the posts (using the frightening but useful WordPress Suicide plugin), then re-imported them. Success!
Editing old posts
There’s another gotcha in WordPress for Markdown users. In keeping with this brave new WYSIWYG Web 2.0 world, WordPress’s page for editing posts has, by default, a “visual” text editing interface like a word processor. There’s a tab above the editing field that let’s you toggle between the rich text display and a “code” display, which is meant to allow you to edit the raw HTML of your entry. You would think—or at least, I did think—that using that “code” tab would be the right thing to do for editing in Markdown. But you’d be wrong. Neither the rich text nor the “code” display should be used.
It turns out that when you open a post for editing in WordPress under the default conditions, some of the whitespace gets deleted. This is similar to the importing problem, but has a different solution—one that doesn’t require you to dig into the PHP scripts and make changes. Go to the Users administration page for your blog and click on the Edit action for your user name. When the Edit User page comes up, uncheck the “Use visual editor when writing” checkbox. WordPress will stop stealing your whitespace.
(I didn’t find out about this nasty behavior until the blog was up and running on the new host. And I discovered it not by writing or editing posts—I use TextMate’s Blogging Bundle for that—but by changing the category of several old entries. Even though I made no changes to the content of these posts, the mere act of opening them in the visual editor to change the category caused WordPress to delete some critical whitespace and screw up their formatting. It took quite a while to fix, and I’m still not sure the formatting is back to the way it was.)
Coming up next
I believe my next post in this series will be the last. I’ll describe the templates and CSS that define the blog’s layout.
Update
The new post is here, but covers permalinks.
Movable Type to WordPress, Part 1
January 26th, 2008
A week or so ago, I said I’d describe my blog’s changeover to a new host and, more significantly, to a new blogging engine. Although I can’t say the every detail of the changeover is complete, it does seem substantially finished, so I thought I’d start the description today. Don’t know how many posts this will take, but it’s a safe bet that this won’t be the last. As I add more posts to this series, I’ll put links to them at the bottom.
I suppose I should start with why I wanted to make the moves. The new host offers more disk space, more bandwidth, shell access, Subversion, and a few other goodies for the same price as the previous host. Need I say more? The switch from Movable Type to WordPress wasn’t quite so simple a decision. Here’s my thinking:
First, I really like writing my posts—as well as everything else—in TextMate and uploading them via TM’s blogging bundle. While the blogging bundle works with Movable Type, the interface with MT doesn’t allow you to set a post’s category. I like having categories.
Second, although MT can serve up dynamic pages, it’s really a static page server at heart. This is great if you don’t like fiddling with the layout of your blog and if your traffic is so high that dynamic pages put too much load on the server. As it happens, I fiddle a lot and waiting for MT’s page rebuilds was getting annoying. And the traffic here is so light, it could be handled by a Commodore 64.
Third, because it’s dynamic by default, WordPress can have these nice “Older Entries” and “Newer Entries” on its category archive pages. Apparently, the new Movable Type has something like this with a hybrid category/date archive, which isn’t as straightforward as “Older” and “Newer” and will lead to more pages to generate and longer rebuild times.
Fourth, I’m a little pissed at Movable Type. When I first thought of comparing it with WordPress, I figured the one big advantage of staying with MT was simplicity. I already had a design for MT, right? Wrong. Some history: Back when I put the blog on MT, it was at version 3.2 and had recently undergone a big change in the structure of its default templates. Users were encouraged to learn this new and improved structure—which was a labyrinth of <div>s within <div>s within <div>s—so their customized themes would survive future updates. I followed the advice and jiggered with the Kubrick theme until it looked more or less the way I wanted. Now, of course, MT is at version 4, and if I’m going to install MT on a new host, I’m certainly going to use the latest version. It turns out, though, that MT has changed its structure again, so all my careful, standard-compliant customizations are fucked. Not only does MT lose its advantage, it turns it into a disadvantage fueled by my resentment.
The biggest advantage I could find for Movable Type is that it’s written in Perl, a language I’m much more comfortable with than WordPress’s PHP. This was overwhelmed by the points in favor of WordPress.
The next post in this series will get into the nuts and bolts of exporting and importing over 300 posts.
Update
The second post in the series is here.
That wasn’t so bad
January 13th, 2008
I think the blog is up and running successfully on the new server. It’s working for me, anyway. And after long fiddling with Rewrite Rules, external permalinks and the old RSS feed URLs are transforming themselves to the new system.
Since WordPress seems to have Technorati-style tagging built-in—something I had to handle by hand in Movable Type 3.2—I’ll be adjusting the templates over the next few days to accommodate the new information. [Update WP has tags, but they don’t seem to create Technorati links.] Once I get comfortable with that, I may turn the comments back on.
It’s easy to do too much blogging about blogging, but I think what I learned in going from Movable Type to WordPress would be useful to others, so expect a post or two on that topic once I’ve settled in.
Going into radio silence
January 11th, 2008
Posts have been scarce for the past couple of weeks because of travel and work, and they’ll be scarce for another week or so as I get ready to move the blog to another host and switch from Movable Type to WordPress. The blog will still be at leancrew.com/all-this.
In anticipation of the change, I’ve been testing a WordPress version of the blog on my iBook and have managed to:
- import all the posts from Movable Type;
- fix most, if not all, the internal permalinks; and
- develop a WP theme that I can live with.
Because of the different ways MT and WP handle permalinks, permalinks from other sites and RSS feed links will break until I get some rewrite rules in place. I think I know how to make the rules but can’t test them until the site is up at the new host and the DNS changes have propagated.
No doubt, the site will go through some ups and downs as I debug it. I’m pretty sure old posts will show up in the RSS feeds, so let me apologize for that up front. My next post should be from the new host.
Source code line numbers and JavaScript
December 20th, 2007
A few months ago, I started adding line numbers to some of the programs/scripts I post here. The idea was to make it easier for me to describe the code; I could say things like “look at lines 15-19” instead of “look at the if statement that checks the first line of the file.” I wrote a Python script called “anl” to add line numbers to my code, a JavaScript function called “styleLN” that put the line numbers between <span></span> tags, and some CSS to style those <span>s to make the line numbers easily visible but unobtrusive.
I was happy with the aesthetics of the results. The scripts looked good, and the descriptions of them became both clearer and less verbose. But some usability was lost, because readers could no longer simply copy the code out of my posts and past them into their editors—the line numbers had to be cut out before the scripts would run. My usual text editor, TextMate, has a column selection feature that makes this very easy, but TextMate isn’t universal, and even I don’t use it for AppleScripts.
What I really wanted was a way to toggle the display of the line numbers—they could be on when you’re reading the description of the code and off when you want to select and copy it. I knew I had seen buttons that would do this on other web sites, but I couldn’t remember where they were. So I wrote my own.
The differences between the new version of “styleLN” and the previous version are:
- the whitespace between the line numbers and the code is now inside the
<span></span>tags instead of outside it (lines 13-14); - a toggle button is added below the bottom of the code if the code has line numbers (lines 15-17); and
- There’s a test that aborts the function before any changes are made if the browser is Internet Explorer (see the update at the bottom of the post).
Here it is:
1: function styleLN() {
2: var isIE = navigator.appName.indexOf('Microsoft') != -1;
3: if (isIE) return;
4: var preElems = document.getElementsByTagName('pre');
5: if (0 == preElems.length) { // no pre elements; stop
6: return;
7: }
8: for (var i = 0; i < preElems.length; i++) {
9: var pre = preElems[i];
10: var code = pre.getElementsByTagName('code')[0];
11: if (null == code) { // no code; move on
12: continue;
13: }
14: var oldContent = code.innerHTML;
15: var newContent = oldContent.replace(/^( *)(\d+):( )/mg,
16: '<span class="ln">$1$2$3<' + '/span>');
17: if (oldContent.match(/^( *)(\d+):( )/mg)) {
18: newContent += "\n" + '<button onclick="toggleLN(this.parentNode)">Toggle line numbers</button>';
19: }
20: code.innerHTML = newContent;
21: }
22: }
You can see on line 18 that the onclick handler for the toggle button is a new JavaScript function called “toggleLN.” Here it is:
1: function toggleLN(code) {
2: for (var i=0; i<code.childNodes.length; i++){
3: node = code.childNodes[i];
4: if (node.nodeName == 'SPAN'){
5: if (node.style.display == 'none') node.style.display = '';
6: else node.style.display = 'none';
7: }
8: }
9: }
I don’t have much JavaScript DOM experience, so there may be a nicer way to write this function. It seems to work, but I’m open to any suggestions for improvement.
The purpose of “toggleLN” is to go through the <code> element associated with the clicked button and turn off the display of all the <span> elements within it (the line numbers are inside the <span>s). You should note that “styleLN” put the <button> element between the <code></code> tags. I don’t know if that’s considered good HTML style, but it has the advantage of making the <code> element the parent of the <button> and easy to pass into “toggleLN” via this.parentNode.
All the line-numbered code in this blog should now have a “Toggle line numbers” button at the bottom. It works for me and I hope it works for you.
Update
Late night coding always gets you in trouble. The toggle button doesn’t work in Firefox 2 or Camino 1.5. (Now fixed! It works for both Mac and Windows Firefox.) From Jamie Phelps, I hear that the layout on IE 6 is a disaster, which should be a CSS thing and unchanged from before, so I don’t know what that’s about. I’ll make inquiries into IE7’s behavior today and try to get things fixed.
Later update
My tests with IE7 suggest that it wasn’t doing the replace command correctly; some of the line numbers were being surrounded by <span></span> and some weren’t. This was buggering up the entire block and making the whole thing render as one long line. Since my scripts aren’t particularly useful for Windows users anyway, I’m not going to spend any more time trying to fix this. I just added a test to lines 2 and 3 of “styleLN” that exit the script immediately if the browser is IE.
The upshot is: if you’re using IE, you’ll see the my line-numbered scripts in the raw—the line number, a colon, two spaces, and then the code itself. All monospaced and the same size and color. It’s readable, but not nice looking, which is probably what you’re used to with IE.