Archive for the ‘mac’ Category
Repeating dates in OmniFocus and iCal
June 30th, 2008
In trying to create a repeating OmniFocus task, I learned that its handling of repeating dates is pretty rudimentary. iCal is much better, but is still lacking some niceties that can come in handy.
My company uses an outside bookkeeper who comes to our office on the last Friday of every month to do the accounting that pays our bills, taxes, and salaries. Some paperwork needs to be done by one of the partners before her visit, and if I’m in town, I’m usually the one who does it. Last week I tried to set up an OmniFocus task to remind me of the necessary paperwork, but I was stopped short by some of OmniFocus’s limitations.
I started with what I thought was the obvious approach. I created an Accounting project and added a repeating task to get the needed paperwork ready. Two problems:
- When I’m showing Next Actions—which is my usual working Perspective—all the Next Actions appear. But the paperwork for the bookkeeper shouldn’t be done until a few days before salary checks are cut, so this task will just clutter up my view for most of the month. I may be able to get around this problem by fiddling with start dates or by scheduling an AppleScript to toggle the task status from On Hold to Active near the end of the month.
- OmniFocus’s Repeat Every setting is just too limited. It allows you repeat a task every so many days, weeks, months, or years, but it can’t handle something like “the last Friday of the month.”
This second problem is the killer, and it’s a pretty embarrassing lack of functionality. The algorithms for this kind of repetition are well established and available. Papers and books on the topic by Reingold and Dershowitz can be found here, and you can always look at the source code for the Calendar/Diary module in Emacs, or for the old BSD calendar program, which comes with OS X.
iCal has a much better repeating date implementation. I’ve used it to create an event for the bookkeeper’s visit on the morning of the last Friday of every month.
This is great for keeping track of the visits, but isn’t ideal for keeping track of my paperwork task. David Allen saith that the calendar is only for things that must be done on that day. The paperwork can be done any time in the week before the last Friday of the month and is best done a day or two before the bookkeeper’s visit.1 Since OmniFocus can’t do what I need on its own, the iCal entry will have to suffice until the OmniGroup improves its date repetition code, or until I can figure out a way to force OmniFocus to do my bidding.
I should mention here that iCal’s repeated date handling isn’t perfect. It can’t, for example, handle US election days or the US federal holiday for Christmas. Election day is not the first Tuesday in November, it’s the day after the first Monday in November. And the federal Christmas holiday—the day you get off work—is December 25 only if December 25 falls on a weekday. If it falls on a weekend, the Christmas holiday is the weekday nearest to December 25. Try doing either of these as a repeating event in iCal.
Although I have no solution now, I’m not giving up. Maybe I can use iCal’s ability to schedule AppleScripts to run a few days before an event to create an OmniFocus task telling me to get the paperwork ready. I need to dig into OmniFocus’s AppleScript library.
OmniFocus and OmniOutliner Pro
June 27th, 2008
My free test period for OmniFocus ran out today and I couldn’t open my GTD lists. This bothered me, which I took to be a sign that it’s a program I really use and should buy.
OmniFocus normally sells for $80, but if you’re a licensed owner of OmniOutliner Pro, you can buy OmniFocus for $60. I don’t own OO Pro, but I do have the regular version of OO that was bundled with my iMac. Since that can be upgraded to OO Pro for $30, doing the upgrade and then buying OmniFocus was just $10 more than getting OmniFocus alone. OK, OmniGroup, you win—I’ll get both.
And yes, I’ll get OmniFocus for the iPhone when it comes out. I’m turning into the OmniBitch.
My no-server personal wiki—Part 3
June 26th, 2008
This the last post describing the self-contained wiki-like system I use to keep track of project notes at work. The first post in the series explained my motivation for creating this system. The second post described how I use it. In this post, I’ll show the behind-the-scenes programming that puts it together.
Makefile
Let’s start with the Makefile. I mentioned before that the HTML pages are generated by running the make utility in the notes directory. Here’s the Makefile.
1: # Makefile for project notes.
2:
3: mdfiles := $(wildcard *.md)
4: htmlfiles := $(patsubst %.md, %.html, $(mdfiles))
5:
6: all: notesList.js $(htmlfiles)
7:
8: notesList.js::
9: python buildNotesList.py > notesList.js
10:
11: %.html: %.md project.info header.tmpl footer.tmpl
12: python buildPage.py $* > $@
13:
14: clean:
15: rm $(htmlfiles) notesList.js
Lines 2 and 3 use wildcards and pattern substitutions to create variables that define all the Markdown-formatted (.md) content files and the corresponding HTML pages.
Line 6 defines the all rule; because it’s the first rule in the Makefile, it’s also the default rule, so executing make is the same as running make all. It builds the JavaScript file notesList.js and the HTML pages.
Lines 8 and 9 define the rules for building the notesList.js file. It’s built by running the Python program buildNotesList.py, which we’ll get to in a bit. As we’ll see, notesList.js defines the sidebar links to all the HTML pages in the notes folder. Since new notes can be added at any time, notesList.js must be rebuilt whenever make is run.
Lines 11 and 12 define the rule for building the HTML notes pages. A page gets (re)built whenever
- its corresponding
.mdfile is created; - its corresponding
.mdfile is updated; or - whenever the header or footer template file is updated.
The page is built by running another Python program, buildPage.py, taking the corresponding .md file as input.
Lines 14 and 15 define a cleanup rule that deletes the HTML files and notesList.js. This rule is executed by running make clean from the Terminal. It’s sort of a defensive rule; if things get really screwed up, make clean will take me back to a pristine state. All the deleted files can be regenerated by running make.
notesList.js and buildNotesList.py
As mentioned above, notesList.js is a JavaScript file that’s used to generate the list of links to other notes pages that appears at the top of the sidebar (see the screenshot of a page in Part 2). It defines a JavaScript function, showNotesList, that writes a series of list items with links to the notes pages. In the skeleton version of the notes folder described in Part 2, there is only one HTML notes page so notesList.js is very simple:
function showNotesList(){
document.write('<li><a href="aa-overview.html">Overview</a></li>')
}
The notesList.js file is generated by the Python program, ‘buildNotesList.py`:
1: #!/usr/bin/python
2:
3: import os
4:
5: # Get the titles of all the notes files in the directory. The
6: # title is assumed to be the first line of the file. Truncate
7: # the title at a word boundary if it's longer than maxlength.
8: # Print out a JavaScript function that will write an HTML list
9: # of the notes files.
10:
11: fileLI = []
12: maxlength = 35
13: allFiles = os.listdir('.')
14: baseNames = [ f[:-3] for f in allFiles if f[-3:] == '.md' ]
15: for fn in baseNames:
16: f = file(fn + '.md')
17: top = f.readline()
18: title = top.strip('# \n')
19: if len(title) > maxlength:
20: words = title.split()
21: twords = []
22: count = 0
23: for w in words:
24: if count + len(w) > maxlength:
25: break
26: else:
27: twords.append(w)
28: count += len(w) + 1
29: title = ' '.join(twords) + "…"
30: fileLI.append('<li><a href="%s.html">%s</a></li>' % (fn,title))
31: f.close()
32:
33: print '''function showNotesList(){
34: document.write('%s')
35: }''' % ' '.join(fileLI)
I think the comment at the top of the file describes it pretty well. The trickiest part of the program is getting the title of the link. The title of the page is the first line of the .md file, with any leading or trailing hash marks (#) deleted. But since a page title could be pretty long and the sidebar is rather narrow, I wanted the link titles to be truncated to the nearest word boundary short of 35 characters. That’s what Lines 19-29 do, putting an ellipsis (…, HTML entity …) at the end to indicate the truncation.
buildPage.py
This is the real workhorse of the system.
1: #!/usr/bin/env python
2:
3: import sys
4: import os
5: import os.path
6: import time
7: import string
8: import urllib
9:
10: # The argument is the basename of the Markdown source file.
11: mdFile = sys.argv[1] + '.md'
12:
13: # Open the page files and process the content.
14: header = open('header.tmpl', 'r')
15: footer = open('footer.tmpl', 'r')
16: cmd = 'MultiMarkdown %s | SmartyPants' % mdFile
17: content = os.popen(cmd, 'r')
18:
19: # Make the template.
20: templateParts = [header.read(), content.read(), footer.read()]
21: template = string.Template(''.join(templateParts))
22:
23: # Close the page files.
24: header.close()
25: footer.close()
26: content.close()
27:
28: # Initialize the dictionary of dynamic information.
29: info = {}
30:
31: # Dictionary entry with long modification date of the Markdown file.
32: mdModTime = time.localtime(os.path.getmtime(mdFile))
33: info['modldate'] = time.strftime('%B %e, %Y', mdModTime)
34: info['modldate'] = info['modldate'].replace(' ', ' ')
35:
36: # Dictionary entry with short modification date of the Markdown file.
37: info['modsdate'] = time.strftime('%m/%e/%y', mdModTime)
38: info['modsdate'] = info['modsdate'].replace(' ', '')
39:
40: # Dictionary entry with modification time of the Markdown file.
41: info['modtime'] = time.strftime('%l:%M %p', mdModTime)
42: if info['modtime'][0] == ' ':
43: info['modtime'] = info['modtime'][1:]
44:
45: # Dictionary entry with absolute path to the Markdown file (for editing).
46: info['mdpath'] = os.path.abspath(mdFile)
47:
48: # Add project info to the dictionary.
49: projInfo = open('project.info', 'r')
50: for line in projInfo:
51: if line[0] == '#' or line.strip() == '':
52: continue
53: name, value = [s.strip() for s in line.split('=', 1)]
54: if name in info:
55: info[name] += '\n' + value
56: else:
57: info[name] = value
58:
59: projInfo.close()
60:
61: # Dictionary entry with absolute path to project info file (for editing).
62: info['infopath'] = os.path.abspath('project.info')
63:
64: # Convert the contacts into a series of HTML list items.
65: if 'contact' in info:
66: contactLI = []
67: cl = [s.split(':',1) for s in info['contact'].split('\n')]
68: for c in cl:
69: if len(c) == 1:
70: contactLI.append('<li>%s</li>' % c[0])
71: else:
72: contactLI.append('<li><a href="addressbook://%s">%s</a></li>'\
73: % tuple(reversed(c)))
74: info['contactlist'] = '\n'.join(contactLI)
75: else:
76: info['contactlist'] = ''
77:
78: # Output the template with the dynamic information substituted in.
79: print template.safe_substitute(info)
It does basically five things:
- It processes the given Markdown file through MultiMarkdown (Fletcher Penney’s extended version of Markdown that includes support for tables and other niceties) and SmartyPants (John Gruber’s typographical conversion program that substitutes curly quotes for straight quotes and m- and n- dashes for multiple hyphen sequences). See Lines 16 and 17.
- It concatenates the header template, the just-generated main content, and the footer template into a new string template that will later be turned into the HTML page file. See Lines 20 and 21.
- It queries the file system for info about the source Markdown (
.md) file and creates a set of dictionary entries with that information. See Lines 31-46. - It goes through the
project.infofile, and creates another set of dictionary entries with the data it reads from that file. See Lines 48-76. - It creates the HTML page by substituting the dictionary entries from Steps 3 and 4 into the string template from Step 2.
header.tmpl and footer.tmpl
The header template file looks like this:
1: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
2: "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3: <html>
4: <head>
5: <title>$projname ($projnumber)</title>
6: <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
7: <link rel="stylesheet" type="text/css" media="all" href="notes.css" />
8: <link rel="stylesheet" type="text/css" media="print" href="notes-print.css" />
9: <!-- <script type="text/javascript" src="file:///Users/drang/Library/JavaScript/jsMath/easy/load.js"></script> -->
10: <script type="text/javascript" src="styleLineNumbers.js"></script>
11: <script type="text/javascript" src="notesList.js"></script
12: </head>
13: <body onload="styleLN()">
14: <div id="container">
15: <div id="title">
16: <h1 class="left">$projname</h1>
17: <h1 class="right">$projnumber</h1>
18: </div> <!-- title -->
19: <div id="sidebar">
20: <h1>Project notes:</h1>
21: <ul>
22: <script type="text/javascript">showNotesList()</script>
23: </ul>
24: <hr />
25: <h1>Contacts:</h1>
26: <ul>
27: $contactlist
28: </ul>
29: <hr />
30: <h1>Source:</h1>
31: <ul>
32: <li><a href="txmt://open?url=file://$mdpath">Edit in TextMate</a></li>
33: <li>Last modified<br />
34: $modldate<br />
35: at $modtime</li>
36: </ul>
37: <hr />
38: <ul>
39: <li><a href="txmt://open?url=file://$infopath">Edit project info</a></li>
40: </ul>
41: </div> <!-- sidebar -->
42:
43: <div id="note">
Although it’s called header.tmpl, you’ll see that it really contains both the header and the sidebar.
Line 9 in the <head> section contains a call to a JavaScript file that isn’t in the notes folder. This is one of the files that comes with the jsMath library, a set of JavaScript and PNG files created by Davide Cervone that allow equations to be embedded in the pages without the need for MathML support. Since most people don’t need equations in their notes, I’ve commented this line out. My project notes often do need equations, so I usually have this line uncommented and it brings in jsMath library from its spot in my ~/Library/JavaScript folder.
The footer template looks like this:
1: <hr />
2: <p class="info">
3: Source: <a href="txmt://open?url=file://$mdpath">$mdpath</a><br />
4: Last modified: $modldate at $modtime<br />
5: <!-- This page built: $buildtime -->
6: </p>
7: </div> <!-- note -->
8: </div> <!-- container -->
9: </body>
10: </html>
It adds a little notation at the bottom of the page, telling where the source file is and when it was last updated.
The structure of the resulting HTML page is pretty simple:
<div id="container">
<div id="title">
blahblahblah
</div> <!-- title -->
<div id="sidebar">
blahblahblah
</div> <!-- sidebar -->
<div id="note">
blahblahblah
</div> <!-- note -->
</div> <!-- container -->
CSS
I’m not going to go through the CSS files because they’re long and not that interesting. suffice it to say that notes.css floats the sidebar to the right and defines a set of colors, type sizes, and spacing that I find pleasing. Not surprisingly, it’s quite similar to the layout of this blog. The CSS file for printing, notes-print.css, hides the sidebar because navigation links don’t work on paper and turns all the colors to black and white because that’s the kind of printer I use.
styleLineNumbers.js
If a notes file contains source code with line numbers, the JavaScript functions in this file will style it nicely and allow me to toggle the line numbers on and off. It’s the same set of functions I use on this blog and which I’ve described in an earlier post.
All together now
If you’re interested in playing around with this system, I’ve made a zip file of my skeleton notes folder available to download. Have fun, and let me know of any improvements you make.
My no-server personal wiki—Part 2
June 25th, 2008
In my last post, I gave my reasons for needing a wiki for project notes, what I wanted it to do, and why I decided to roll my own instead of going with any of the readily available free or commercial offerings. In this post, I’ll describe how my system works. If some of the choices I’ve made seem odd to you, the earlier post should explain the whys and wherefores.
The files for the wiki—both the content and the build files—sit in a folder called “notes.” This makes the system portable. I keep a skeleton version of this folder on my Desktop, and whenever I need a notes wiki for a new project, I drag a copy from the Desktop into the project folder. Here’s a view of the skeleton.
The content is kept in the Markdown-formatted files with an .md extension. The skeleton provides just one of these, aa-overview.md, which is filled with mostly nonsense. Any file added to the folder with an .md extension will be taken as the content for a new page. The wiki pages themselves are the .html files with the same basenames as the .md files. Everything else in the folder is either a programming, template, or configuration file.
The HTML version of aa-overview is intended to be the home page of the wiki. For the skeleton file, it looks like this:
The skeleton aa-overview.md, as with all the Markdown files, contains just the content of the main section of the page:
# Overview #
## Background ##
This is where I describe the project.
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
The header, the sidebar, and the footer are all built from the template and configuration files. The page is built statically, so aa-overview.html is a full HTML file, not a fragment.
The top section of the sidebar contains links to all the notes files in the folder. The links are created automatically by the build script, which looks through the folder for all the .md files. These links will be in the sidebar of every page, so I can go directly from any page in the wiki to any other. This would make for a very cluttered sidebar if my projects had hundreds of notes pages, but I’ve never needed more than a dozen or so.
Each page’s name is taken from top line of its .md file. The page links are sorted according to the file name, which is why the overview page is given the “aa-” prefix—I want it at the top of the list. I use this method of sorting to group similar pages in the list of links; inspection notes have an “ii-” prefix, lab notes have an “ll-” prefix, etc.
The project name and number in the header and the contact link in the sidebar come from the project.info file:
projname = Project Name
projnumber = 9999
contact = Apple:A1A2AA41-FA30-40AE-9925-FD6DB270B0A5%3AABPerson
This file defines variables that are used by the build script to create the pages. The projname and projnumber are obvious; the contact variable contains the name of the contact (“Apple” for the skeleton file) and its Address Book ID, separated by a colon. I described a script for extracting the ID earlier this month—this is where I use it. When you click on a contact link in the sidebar, the Address Book opens to the card for that entry.
This is convenient when I’m reviewing a project and want call or email my client.
Although the skeleton version of the project.info file has just one contact, a page can have any number of contact links. I just add more contact = lines to the file.
I can, of course, edit a page by going to the Finder, working my way to the appropriate notes folder, and opening the .md file in my text editor. But I’ve made it easier than that. The editor I use, TextMate, is the handler for special URLs of the form
txmt://open?url=file:///Users/drang/Desktop/notes/aa-overview.md
By including a link like that in the sidebar, I can just click on it and TextMate will open the .md file associated with the page. The bottom of the sidebar has a similar link to the project.info file.
I mentioned before that the HTML files in the notes folder are built statically, which means that, unlike a regular wiki, I have to do something to get them built. And what I need to do is run the make utility on the Makefile in the notes directory. This could be done through the Terminal, but I’ve made a simple TextMate command that runs make whenever I type Command-Option-Control-M.
There’s probably a clever way to use Folder Actions to run the Makefile whenever an .md file is added or changed, but I haven’t looked into that yet.
In summary, when I want to create a new wiki for project notes, I
- Drag the skeleton
notesfolder from my Desktop to the project folder. - Edit the
project.infoandaa-overview.mdfiles to include the appropriate information. - Add new
.mdfiles as my work on the project grows.
In my next post in this series, I’ll describe the template files and build scripts and will provide a zipfile with my skeleton notes folder.
Update
Part 3 is now up.
My no-server personal wiki—Part 1
June 24th, 2008
This is the first of what I expect to be a two- or three-part post on a wiki-like note taking system I’ve developed for keeping track of the work I do on my professional projects. I’ll start by explaining what I want out of such a system and how other solutions didn’t fit my needs.
At any given time, I’m working on several projects, each with its own client and its own set of facts, figures, and research results. My habit has always been to keep all computer files related to a particular project in its own folder (usually with subfolders) on my hard disk, just as I keep all physical files related to the project in a set of manila folders that are labeled and kept together in my file cabinets. Keeping everything related to a project together is important for three reasons:
- It’s just easier to keep track of things this way. Despite the improvements in Spotlight, dragging a file into a folder is less time-consuming than tagging it with a project name.
- Sometimes I need to copy all my work on a project and send it to the client. My paper files go off to a copy shop for photocopying, and my computer files get burned to DVD. If all the computer files are in one folder, burning a DVD is a one-step operation.
- When the project is done, I archive the computer files to DVD, put them with the paper files, and send them off to storage. Again, if all the computer files are in one folder, burning a DVD is a one-step operation.
Until recently, most of my project notes were on paper rather than in computer files. This was primarily because most of my notes are taken in the field, away from my computer, and there’s been no organizational advantage to rewriting those notes on the computer. But I did want the notes on the computer, so I began looking for ways to organize my notes that way.
At first, a wiki seemed like a natural fit. It’s easy to create new pages, and the pages are easy to navigate and edit. If I choose the right wiki software, I can use Markdown formatting, which I use to write almost everything these days. Unfortunately, the usual server-based wikis keep all the information in a central database, which means that my project notes would not be kept in the project folder. This makes it too difficult to archive the notes with the other project files, so server-based wikis were out.
The notion of a file-based wiki led me to TiddlyWiki and its various offshoots. Because it’s run with JavaScript, TiddlyWiki would let me keep my notes for a particular project in a single big HTML file in the project folder, just as I want. But there were two problems with TiddlyWiki: First, I didn’t like the default style and found its CSS structure very difficult to delve into and modify. Second, TiddlyWiki doesn’t work well with Safari because Safari’s JavaScript doesn’t like the idea of saving changes to the HTML file, which is essential to the idea of a wiki. (I tried using the Java applet that gets around this problem, but it didn’t work for me. I suspect I could have gotten it to work if I’d kept at it, but since I didn’t like the look of TiddlyWiki it didn’t seem worth the effort.)
So then I moved on to VoodooPad, Gus Mueller’s personal wiki application for the Mac. This seemed like the perfect solution. Its pages can be styled however I like, its data are saved to a file that can be put anywhere on my hard disk and can be exported to various open formats—this is important because I don’t want to get stuck in a proprietary system—and it’s backed by a developer known to be responsive to his customers. I tried it, I bought a license for it, but I just couldn’t get used to using it because it doesn’t use Markdown. I was a bit surprised at this reaction, but after a decade of using text-only systems (SGML, LaTeX, and now Markdown) I just couldn’t stand using something that works like a word processor. Eventually, I gave up and gave my copy of VoodooPad to my daughter to help her organize her college notes. Since it came from me, I suspect she’s ignored it—most parents of teenagers will understand that—but I still think VoodooPad would be a great solution for someone who likes writing in a word processor.
At this point, I decided to create my own system. The rules I set for myself were:
- It has to use text files that can be stored anywhere and are easily moved and archived.
- The notes are to be written in Markdown. If I later decide that Textile or reStructuredText or something else is better, changing to the new markup system should be easy.
- The notes are to be written in a text editor rather than in an HTML text box so I don’t feel like I’m writing in a straightjacket.
- The notes should be styled according to my taste. Since my taste can change, the style should be easy to change.
- The creation of new pages and new links to those pages should be simple and/or automatic.
I’ll start describing the system I came up with in the next post.
Update
Part 2 and Part 3 are now up.
iPhone ringtones flowchart
June 20th, 2008
I don’t convert songs to iPhone ringtones very often, so I made this flowchart to help me remember the process. And by posting it here, I’ll always know where it is. Oddly, I find that Googling “create iphone ringtone” or something like that often brings up out-of-date information. This flowchart works with iTunes 7.6.2 and iPhone 1.1.4 system software, I’ll try to keep it current as the versions change.
Here it is:
You can see a larger version by clicking on it.
This is a Mac-centric workflow, and it takes advantage of the software I own. I perform the actions in the yellow boxes using Rogue Amoeba’s Fission audio editor, but you could substitute any sound file editor, like Freeverse’s SoundStudio or the free Audacity or whatever’s popular on Windows.
Converting to AAC is done by choosing the Convert Selection to AAC command from iTunes’ Advanced menu. If you don’t see a Convert Selection to AAC item in that menu, it’s because your iTunes Importing Preference is set to use a different encoder. Change the encoder to AAC and the menu item will change, too.
A clip about 10-20 seconds long seems to work well. Once the clip is in the iTunes Ringtone Library, it will transfer to your iPhone the next time you sync.
Automated charts with Gnuplot
June 19th, 2008
Every month since the fall of 2006, I’ve written a post like this one, in which I include a chart of US and coalition fatalities. This is a reduced version of the latest:
I get the data from icasualties.org and generate the chart using Gnuplot, the venerable Unix plotting program. I figured it was about time I showed how I do it.
Gnuplot began its life as a terminal program, and it’s still driven by typed commands rather than by clicks and drags. This makes it a bit difficult to learn, but because it allows sequences of commands to be stored in files, it makes the creation of several similarly-formatted graphs a snap. I started using it when I switched to Linux in the mid ’90s because there were no good Excel-like programs on that platform. I’ve kept using it after switching back to Macintosh because it’s so good at producing the kinds of graphs common in science and engineering. I’m not big on pie charts.
The data for the graphs are stored in a file called “icasualties.txt” that I update every month. The first several lines of that file look like this:
Month US UK CO
2003-03 65 27 0
2003-04 74 6 0
2003-05 37 4 1
2003-06 30 6 0
2003-07 48 1 0
2003-08 35 6 2
2003-09 31 1 1
2003-10 44 1 2
2003-11 82 1 27
2003-12 40 0 8
The columns are separated by spaces; it doesn’t matter that the number of spaces differs from column to column or row to row. I think all columns but the last are self-explanatory. The “CO” column contains the count of military deaths from coalition countries other than the United States and United Kingdom.
The file of Gnuplot commands that generates the graph is called “icasualties.gp.”
1: # input format for dates
2: set timefmt "%Y-%m"
3:
4: # horizontal (time) axis layout
5: set xdata time
6: set format x "%b\n'%y"
7: set xtics "2003-01", 60*60*24*365.2425/4, "2008-12" # quarterly
8: set mxtics 3 # monthly
9:
10: # left vertical axis layout
11: set ylabel "Military Deaths" 2,0
12: set yrange [0:150]
13: set ytics 25
14: set mytics 5
15:
16: # overall layout
17: set title "Iraq War Timeline" 0,-.5
18: set grid
19: set key at "2007-10",20 right width -3 samplen 1.5 box
20:
21: # Make labels for the totals
22: ustot = `perl -e '$s=0;while(<>){($m,$a,$b,$c)=split;$s+=$a}print$s;' icasualties.txt`
23: tot = `perl -e '$s=0;while(<>){($m,$a,$b,$c)=split;$s+=$a+$b+$c}print$s;' icasualties.txt`
24:
25: # Make label for data source
26: set label 3 "Data source: http://icasualties.org" at '2003-05',10 left
27:
28: # choose output and plot it
29: set terminal aqua 0 title "Timeline" size 800 600\
30: fname "Helvetica" fsize 14
31: plot "icasualties.txt" using 1:2 title sprintf("US only (%d)",ustot)\
32: with linespoints pointtype 2 linetype 3 linewidth 3,\
33: "icasualties.txt" using 1:($2+$3+$4) title sprintf("Coalition (%d)",tot)\
34: with points pointtype 3 linetype 1
Like Perl, Python, and the shell, Gnuplot comments start with a hash mark (#). As you can see, most of the commands “set” a Gnuplot parameter that controls either the input or output formatting. Two of the other commands create variables for later use, an the final command creates the plot itself. Here’s the explanation:
Line 2 tells Gnuplot that some of the input data will be time values and that they will be formatted with a 4-digit year (%Y) followed by a hyphen and a 2-digit month (%m). The codes follow the conventions of the well-known strftime C library.
Lines 5-8 cover the formatting of the horizontal (x) axis. Line 5 says that the x-axis will consist of time values. Line 6 sets the format of the axis labels to a 3-letter abbreviation of the month name (%b), new line (\n), an apostrophe, and a 2-digit year (%y). Line 7 sets the major tic marks and the labels to the start of every quarter, which is kind of tricky. The three arguments to set xtics are
- Where we should start counting: 2003-01. This is the January before the war began, and I chose this date to insure that the major tics marks fall on the usual quarterly start dates: January 1, April 1, July 1, and October 1.
- The spacing between the major tic marks, in seconds. Using 356.2425 as the number of days in a year is overly precise, but I’m a big fan of the Gregorian calendar reform.
- Where we stop counting. I’ll have to change this next year.
Line 8 tells Gnuplot to split the space between major tic marks into three parts and put minor tics at the splits.
Lines 11-14 cover the formatting of the vertical (y) axis. The numbers after the axis label in Line 11 nudge the label a little to the right, because I thought the default location was too far from the axis. Since Gnuplot will choose the range and tic locations if they’re not specified, Lines 12, 13, and 14 set the range and spacing to get a consistent vertical axis every time I generate a graph.
Line 17 sets the title at the top of the graph and nudges it down a bit from its default location. Line 18 tells Gnuplot to put faint gridlines that run the full width or height of the graph at every major tic mark.
Line 19 puts the key (or legend) near the bottom of the graph and puts a box around it. The at "2007-10",20 right part positions top right corner of the key at those coordinates. The width -3 part makes the box a bit smaller than its default width. The samplen 1.5 part makes the blue and red point/line examples a bit wider than the default.
Lines 22 and 23 are tricky. I wanted to put the US and coalition casualty totals on the graph, but as far as I can tell, Gnuplot doesn’t have a builtin way to get those figures. But it does have a way of calling another program. So these two lines contain short Perl scripts that scan through the “icasualties.txt” file and sum up the US and full coalition figures. The sums are stored for later use in the Gnuplot variables ustot and tot.
Line 26 puts the acknowledgement text near the bottom left of the graph. The commands for positioning are similar to those in Line 19.
Lines 29-30 is one long Gnuplot command split over two lines. It tells Gnuplot to display the graphs in an 800x600 AquaTerm window with 14-point Helvetica as the base font. The set terminal command is difficult to learn, but makes Gnuplot very flexible in its output. Although I want the graphs in the form of a PNG file, I chose aqua terminal over the png terminal because the precompiled Gnuplot I got with Octave (see this post) doesn’t have support for Macintosh fonts built into its png terminal. And I don’t feel like gathering all the libraries necessary to compile my own version.
Lines 31-34 are what we’ve been leading up to. This is one long Gnuplot command that actually created the plot according to the specifications given in the previous lines. It makes one graph for the US casualties and one for the total coalition casualties. The key labels these data series include the total casualty counts calculated back in Lines 22 and 23. The presentation styles (line type, line width, color, and point type) have numbers rather than names, so there’s usually a bit of trial and error before you hit on a combination you like.
I create the graph by typing gnuplot icasualties.gp into Terminal. AquaTerm launches and shows the graph. I then do a screenshot of the graph and upload it to my server as a PNG file. The whole thing takes less than a minute.
As you can see, Gnuplot is very flexible but very complex. I find that when I’m using it a lot—for example, while writing a report that reduces and presents a lot of data—I get into a rhythm and the command come naturally to my fingertips. But when I’ve been away from Gnuplot for a few months, there’s always some frustration when I start back again.
Good documentation would go a long way toward relieving that frustration. Unfortunately, Gnuplot’s documentation, while quite complete, is terribly difficult to use because it’s organized alphabetically by command. So it’s great if you know the command, but if you knew the command you probably wouldn’t be looking in the manual.
It looks like help is on the way. Gnuplot in Action is a book by Philipp Janert that is scheduled to be published by Manning later this year. Janert and Manning have graciously given me a free review copy of the Early Access Edition of the book—basically a PDF of most of the book in a pre-publication state—and I’m optimistic. The best part of the book can be seen from its table of contents. Janert presents Gnuplot concept by concept rather than command by command. There’s a chapter on axes, a chapter on styles, a chapter on scripting, etc.
He has not, however, simply taken the official Gnuplot manual and rearranged it (although that would be valuable in itself). The chapters are filled with examples showing both the Gnuplot command and the graphical result. The official manual has many examples, but because it’s still text-based, it can only tell you what the commands will do—it can’t show you. I do wish Manning would put the commands and output side by side as is done in The LaTeX Companion, but the sequential layout still gets the job done.
One aspect of the book I don’t expect to like is hinted at in its subtitle, Understanding Data with Graphs. In addition to Gnuplot itself, Janert apparently wants to teach me how to use graphs generally to analyze data. Since that section of the book hasn’t made its way into the Early Access Edition yet, I don’t know what he’s going to say, but I’ve been analyzing data with graphs for 30 years, and I doubt he’s come up with anything new. But that section may be helpful to others.
This isn’t a review. The book isn’t close enough to publication quality for a review to be fair. It is, however, already better at teaching Gnuplot than the official manual is. In some ways, its better than the manual even as a reference because it’s better organized. I’m keeping an eye on it and will write up a review when it’s near its final form.
TypeIt4Me problem with AppleScript snippets
June 13th, 2008
Two weeks ago, I posted the following bug report to the the TypeIt4Me technical support forum:
I am having a problem with TypeIt4Me’s AppleScript clipping feature on my iBook G4. I have a clipping with the following AppleScript: tell application “Safari” to get URL of front document When I run this clipping from within the text editor TextMate, it comes out looking like thisThe “spaces” before each character are not really spaces, they’re unprintable characters. I’ve saved this in a file called tm-g4.txt and run the xxd utility on it to see what the characters are. Here are the results:h t t p : / / m y . y a h o o . c o m /The unprintables are nulls. At first I thought this was some weird interaction between TypeIt4Me and TextMate, but… When I run the same clipping from within TextEdit, the spaces don’t appear, but saving it to a file called te-g4.txt and running xxd on it gives me the same thing:% xxd tm-g4.txt 0000000: 0068 0074 0074 0070 003a 002f 002f 006d .h.t.t.p.:././.m 0000010: 0079 002e 0079 0061 0068 006f 006f 002e .y...y.a.h.o.o.. 0000020: 0063 006f 006d 002f .c.o.m./So the nulls are there, but TextEdit just isn’t showing them. I don’t get this behavior with the same clipping on my Intel iMac. On this machine, there are no blanks before the characters when the clipping is inserted in a TextMate document. Saving the document to a file called tm-intel.txt and running xxd on it yields% xxd te-g4.txt 0000000: 0068 0074 0074 0070 003a 002f 002f 006d .h.t.t.p.:././.m 0000010: 0079 002e 0079 0061 0068 006f 006f 002e .y...y.a.h.o.o.. 0000020: 0063 006f 006d 002f .c.o.m./No nulls, which is how it should be. I got the same result from using the clipping in TextEdit on the Intel iMac:% xxd tm-intel.txt 0000000: 6874 7470 3a2f 2f6d 792e 7961 686f 6f2e http://my.yahoo. 0000010: 636f 6d2f com/Running the wc utility on these files also shows that the G4 versions have two bytes for every byte in the Intel versions:% xxd te-intel.txt 0000000: 6874 7470 3a2f 2f6d 792e 7961 686f 6f2e http://my.yahoo. 0000010: 636f 6d2f com/Is this some sort of big-endian/little-endian thing? Can it be fixed?% wc tm-g4.txt 0 1 40 tm-g4.txt % wc tm-intel.txt 0 1 20 tm-intel.txt % wc te-g4.txt 0 1 40 te-g4.txt % wc tm-intel.txt 0 1 20 tm-intel.txt
The problem arose when I tried to use the snippets I discussed in this post on my iBook G4. I really like the idea of having programmable snippets available system-wide, not just in TextMate. But I can’t use the feature if it keeps spitting out these spurious nulls.
So far, there’s been no response from Ettore Software, which is a bit annoying. I’m pretty sure the forum is where Ettore wants bugs reported.
While waiting for a response, I downloaded the trial version of TextExpander to see if it has the same problem. It doesn’t; AppleScript snippets in TextExpander expand perfectly on both my Intel and G4 machines.
Were it not for the AppleScript snippet bug, I’d prefer to use TypeIt4Me, because it has the AutoCue feature that allows me to enter variable text at placeholder locations within the snippet—similar to the tab stops in TextMate’s snippets. With the bug, however, the scale tips to TextExpander. If I don’t get some encouraging response from Ettore, I’ll have to go ahead and buy it when the trial period is over.
Address Book URLs, revisited
June 5th, 2008
In an earlier post I discussed the Mac’s addressbook URI scheme and how you can open a particular contact in your Address Book with a command like
open addressbook://A1A2AA41-FA30-40AE-9925-FD6DB270B0A5:ABPerson
from the Terminal. Everything after the double slash is the ID of the contact, accessible via the id property in AppleScript. In a similar way, you can create links in HTML documents which, when clicked, will open Address Book to the contact with that ID. (On my work computer, the above ID is for Apple, Inc.; the ID for Apple on your computer will be different.) I’ve been using such links a lot recently—for a serverless wiki-like system I’ll be writing about soon—and needed a utility for quickly extracting the ID of an Address Book contact.
The Address Book opener described in that earlier post used a pair of scripts—one AppleScript and one bash script. I never liked that setup, because the AppleScript was somewhat convoluted and because it used two script to do basically one thing. So even though the ID extraction script would be almost identical to the Address Book opener, I didn’t want to reuse my old code.
The script works like this: entering
abid john smith
at the Terminal prints the ID of the first contact with the names “john” and “smith.” Any number of arguments can be given to the abid command; the name of the contact must have each of the arguments as a substring. The search through the Address Book is case-insensitive. If no match is found, abid prints “No match!” instead of the ID.
Here’s the code. It’s written in Python using the appscript module.
1: #!/usr/bin/python
2:
3: import sys
4: from appscript import *
5:
6: # Find the contacts that have all the names in the given list.
7: def searchName(nameList):
8: names = [ x.lower() for x in nameList ]
9: # Get everyone whose last name matches.
10: matches = [ x for x in app('Address Book').people.get() if names[-1] in x.name.get().lower() ]
11: # Look for matches with the other names if there are any.
12: while len(names) > 1:
13: del names[-1]
14: matches = [ x for x in matches if names[-1] in x.name.get().lower() ]
15: # Return the list of matches.
16: return matches
17:
18: # Print the ID of the top match. Or print an error message.
19: try:
20: print searchName(sys.argv[1:])[0].id.get().replace(':', '%3A')
21: except IndexError:
22: print "No matches!"
The searchName function does almost the same thing as the identically-named function in my old AppleScript, but is much shorter and the logic is cleaner. Which is why I wanted to get away from AppleScript. I suppose the really cool, Lispy way to write searchName would be to make it recursive instead of iterative, but the while loop works fine.
The name of a contact is the full name—prefix, first, middle, last, suffix—as a string. If the contact is a company, the name is the company name. The list comprehensions in Lines 10 and 14 are basically substring filters.
You can see in Line 20 that I’ve URL-encoded the colon near the end of the ID, replacing it with %3A. The encoding isn’t necessary to make the URL legal, but eliminating that colon from the string makes the ID easier to use in the serverless wiki I mentioned at the top of this post.
The program works through the argument list from back to front instead of front to back. I expected to give it the names in “first last” order, and I thought that filtering by last name first would be faster because identical last names are rarer than identical first names and the list comprehension in the while loop would therefore be working with smaller lists. This notion could be completely off base; I haven’t done any benchmarking.
There’s no need to pass abid more than one name if one of the contact’s names is unique. I have only one Adolf in my Address Book, so
abid adolf
is all I need to get his ID. Nicknames will work if the nickname is a subset of the name given in Address Book. For example, if someone is listed as Robert Johnson, then
abid rob johnson
will find him, but
abid bob johnson
will not.
This script would likely be unnecessary if Address Book would show us the ID and let us copy it into the clipboard. But I haven’t found any way to get Address Book to show us the ID; if you know of a way, I’d like to hear about it.
Simple AppleScript tuneup for iTunes
June 2nd, 2008
My iTunes library has many songs that I ripped back in my Linux days, when I got track information from freeDB, a service that was started up when Gracenote changed the licence on the previously-free CDDB database. As with Gracenote—but to a greater extent than with Gracenote—some of freeDB’s entries are a little funky.
One set of entries that’s been bothering me for years is Creedence Clearwater Revival’s Chronicle album of greatest hits. The name field for every track on that album is prefixed with “CCR - ”. Thus,
CCR - Susie Q
CCR - I Put A Spell On You
CCR - Proud Mary
and so on. Eventually I got tired of seeing this and wrote the following AppleScript to delete the six-character prefix.
1: tell application "iTunes"
2: set sel to selection
3: repeat with theTrack in sel
4: set theName to name of theTrack
5: set len to length of theName
6: set name of theTrack to (characters 7 thru len of theName as text)
7: end repeat
8: end tell
It probably took a bit longer to write the script than it would have to change the 20 or so tracks by hand, but it was more interesting. As you can see from Line 2, the script assumes that all the tracks that need to be changed are selected before the script is run.
The first time through, I thought Lines 5 and 6 could be the single line
set name of theTrack to (characters 7 thru last of theName as text)
but apparently the “last” specifier can’t be used that way. Hence the “len” variable in Line 5 to get the index of the last character. (Had I written this in Python and appscript, I could have done that with a single line because Python has a simple way of indexing the end of the string without knowing the string length. But by the time I realized “last” wouldn’t work the way I wanted, I didn’t feel like rewriting in Python.)
The “as text” stuck on the end of Line 6 is needed because without it “characters 7 thru len of theName” will return a list of characters instead of a string.
This is very much in the Unix tradition of one-shot, throwaway scripts. It solves a specific problem by automating a tedious process and will never be used again.