Installing PVM under MacPorts on Leopard

I've been having trouble getting PVM 3.4.5 to install under MacPorts on Leopard. The conf/DARWIN.def configuration file defines the FAKEXDRFLOAT preprocessor symbol; it may be needed for OS X 10.4, but it causes a compiler error w. GCC 4.0.1 under 10.5:

../../src/pmsg.c:785: error: static declaration of 'xdr_float' follows non-static declaration.


The workaround is simple: install the Darwin8 variant.

$ sudo port install pvm +darwin_8

Likin' bzr

I've been using bzr for the past couple of months, and am liking it.

I'm not using it on any projects involving multiple people. All of my employer's code still lives in subversion repositories.

Rather, I'm using bzr for local, lightweight revision control. It sure is nice to be able to go into a directory tree where I'm whanging together a prototype, or where I'm working up some documentation; and to say

$ bzr init
$ bzr add .
$ bzr commit -m "New repository"'


And bingo, I can make changes and back out of them at will, with little fear of trashing anything. And of course, for technical writing it's nice to be able to refer to old revisions.

Lately I've started fiddling around w. bzr branch, building prototype web servers in one directory tree and occasionally pulling them out to a "deployment" branch. This feels a little more like working with subversion, but bzr lets me make local "checkpoint" commits, where some things are broken, without fear of breaking the build.

At this point my only problem with bzr is that I don't quite grok what's going on under the hood. Distinctions between branches and checkouts seem clear enough. But I don't understand why, when I commit from a "checkout" workspace, a subsequent bzr status in the source workspace will tell me that my workspace is out of date; whereas a similar sequence of actions from a "branch" workspace -- a commit followed by a push, followed by a status in the target workspace -- offers no warning that the code has been updated.

Given some time and experience these confusing behaviors should make more sense. Meanwhile, it sure is nice having cheap, local revision control. Kinda like XCode 3's Project Snapshots, but without the XCode :)




Technorati Tags:
, ,


Using supervisor on OS X

Last week's sojourn with Leopard gave me a chance to learn about some new ways of performing common system administration tasks. I've already written about using iCal in place of cron. I also had a chance to get acquainted with launchctl.

I have some custom web app servers which need to launch at system startup, and which need to be restarted if they fail. (I guess I could launch them from within Apache...)

The last time I needed to launch anything at system startup on OS X, /Library/StartupItems was in vogue. It's still usable, but now launchctl and /Library/LaunchDaemons et al are recommended in its place. As the launchd man page states:

In Darwin it is preferable to have your daemon launch via launchd instead of modifying rc or creating a SystemStarter Startup Item.


At some point my web apps may be deployed on Linux. So instead of going all the way with launchctl I decided to introduce supervisor into the mix: use launchctl to keep supervisor running, and use supervisor to keep the custom services running.

Of course this is a stupid thing to do. Both launchctl and supervisor are intended for the same types of problems. But supervisor will be available on Linux whereas, presumably, launchd will not. This is as good a place as any to learn how to use supervisor.

Besides, I have a long history of solving problems in ways which are almost, but not quite, entirely incorrect.

After much bumbling about, this became the plist script which would get supervisor up and running at system boot:
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN
http://www.apple.com/DTDs/PropertyList-1.0.dtd >
<plist version='1.0'>
<dict>
<key>Label</key><string>com.agendaless.supervisor</string>
<key>ProgramArguments</key>
<array>
<string>
/Library/Frameworks/Python.framework/Versions/Current/bin/supervisord
</string>
</array>
<key>Debug</key><true/>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
</dict>
This script is installed as /Library/LaunchDaemons/com.agendaless.supervisord.plist. As the launchctl.plist man page says, that's the place for "System wide daemons provided by the administrator."

Both the name of the plist file and the value of the Label are based on the domain of the copyright owner for supervisor: http://www.agendaless.com/

The ProgramArguments consist solely of the path to the supervisord executable.

RunAtLoad ensures that supervisor is started as soon as it is loaded.

Setting Debug to true increases the odds that, if some problem arises in running supervisor, the problem will be noted in the system log.

Finally, KeepAlive ensures that, if supervisor dies, it will be restarted. (KeepAlive is not documented in the OS X 10.4 man pages; it may make sense only for OS X 10.5. I need to re-read the blog where I saw it mentioned to better understand its intended use.)

I'm still such a novice with supervisor (not to mention launchctl) that I'm not sure supervisor meets the expectations of launchctl. But in any case, the above seems to work.




Technorati Tags:
,


Intel iMac hard drive failure

Just after returning from Christmas vacation my iMac (Intel Core Duo 1.83GHz, 17") started showing spinning beachballs. And the system log started accumulating error messages similar to those described on this blog. The hard drive was dying.

Luckily I'd run a SuperDuper backup the night before starting vacation.

It was also lucky that I had an early-model Intel iMac. Supposedly newer models are much harder for amateurs to service.

Thank goodness for John Wood's iMac_Disassembly web page. If it weren't for him I'd either be running from an external hard drive right now or shipping the iMac off for a very expensive (not under AppleCare) repair.

Using the photos and instructions on the disassembly page I was able to swap out the failed 160GB Seagate Barracuda 7200.7 for a 320GB Western Digital WD3200AAJS, purchased from NewEgg. Required tools:

  • A #0 Phillips screwdriver for the memory slot cover
  • A T-8 Torx driver for the top case
  • A small Allen wrench (not sure about the size) for the screws securing the LCD panel to the back case
  • A sharp knife and a spot of rubber cement for moving the temperature sensor to the new drive
The whole job took about an hour and a half, not because it was difficult, but because I am a coward :)

There were two minor differences between my iMac and the one Mr. Woods took apart. Mine had two "hooks" at the top of the front plate, and I bent one of them slightly before discovering it was there. And in addition to the black tape overlying the sides and bottom of the LCD panel, I had perforated foil tape running along the top edge.

Everything's up and running again, and now I have lots of free disk space :) Time to do another backup. Of course in the meantime I've been doing development under Leopard on the MacBook; so it's tempting to follow that backup with an archive-and-install of 10.5 onto the iMac...

Leopard, 'ls -l' and extended attributes

Under OS X v10.5 the permissions column from ls -l contains extra characters which I haven't seen before.

$ ls -l
total 16
-rwxr--r--@ 1 me me 165 Jan 2 06:38 run_webalizer
-rw-r--r--@ 1 me me 0 Jan 3 10:04 run_webalizer.scpt
The man pages didn't say anything about these @ characters. Google was no help since it ignores punctuation characters like "@".

Eventually I realized that, since I had done an "upgrade" installation of Leopard, my man pages might be out of date. Indeed they are: Apple's online man pages explain @ and much else that's new in the ls command:
If the file or directory has extended attributes, the permissions field printed by the -l option is followed by a '@' character. Otherwise, if the file or directory has extended security information, the permissions field printed by the -l option is followed by a '+' character.


And
The following options are available:
-@ Display extended attribute keys and sizes in long (-l) output.


Ah, Bach. Look at some of the interesting tidbits revealed by -@:
$ ls -l@
total 16
-rwxr--r--@ 1 me me 165 Jan 2 06:38 run_webalizer
com.macromates.caret 33
-rw-r--r--@ 1 me me 0 Jan 3 10:04 run_webalizer.scpt
com.apple.FinderInfo 32
com.apple.ResourceFork 754
So that's how TextMate knows where to put the cursor when I re-open a file.

Now I just need to figure out how to get my man pages up to date.

Replacing cron with iCal

cron is deprecated in OS X. iCal makes a straightforward, if tedious, replacement, despite its limited support for scripted alarms.

In iCal you can easily create a repeating event with an associated alarm that runs a script.

ical_repeating_event.png


Put all of your periodic system management events into a hidden "Automation" calendar (so they don't clutter up the calendar view) and you're all set.

automation_calendar.png


Of course, there's a problem. I prefer to write system management scripts using Python. iCal can run only AppleScripts.

It's easy to write an AppleScript "wrapper" which just launches a shell script, but the process involves lots of button-clicking:
  • Launch Script Editor
  • Enter do shell script "/path/to/shell/script"
  • Save the script
  • Quit Script Editor
Luckily, the osacompile command lets you do the same thing from the command line. Together with Python's pathname-manipulation facilities, osacompile makes it simple to create AppleScript wrappers for shell scripts.

Given the path to a shell script, the following Python program creates a corresponding AppleScript wrapper in the same directory:
#!/usr/bin/env python
import sys, os, subprocess, logging
def main():
    for filename in sys.argv[1:]:
        pathname = os.path.abspath(filename)
        scriptname = os.path.splitext(pathname)[0] + ".scpt"
        if os.path.exists(scriptname):
            os.remove(scriptname)
        status = subprocess.call([
            "osacompile", "-e", 'do shell script "%s"' % pathname,
            "-o", scriptname])
        if status:
            logging.error("Got status %d creating %s" % 
                          (status, scriptname))

if name == "main":
    main()

Custom keybindings in OS X

Since I keep forgetting how to customize keyboard navigation, here's a pointer to the documentation:
http://developer.apple.com/documentation/Cocoa/Conceptual/EventOverview/TextDefaultsBindings/chapter_9_section_2.html

And here are the bindings I'm currently using in my ~/Library/KeyBindings/DefaultKeyBinding.dict file. Basically they cause the "Home" and "End" keys on the Apple keyboard to move the cursor to the beginning and end of the current text line. Hitting these keys while holding "Shift" will also select all text covered by the move. And the page up/down keys do what you'd expect.

{
"$\Uf729" = "moveToBeginningOfLineAndModifySelection:";
"$\Uf72b" = "moveToEndOfLineAndModifySelection:";
"\Uf729" = "moveToBeginningOfLine:";
"\Uf72b" = "moveToEndOfLine:";
"\Uf72c" = "pageUp:";
"\Uf72d" = "pageDown:";
}

CDK, Jython and chemical structure format conversion

These days I'm learning how to work with some new (to me) Java cheminformatics toolkits. Google, Jython 2.2.1, and VoodooPad are helping me get up to speed:


  • Google to find the documentation for the Java APIs

  • Jython to learn API nuances without an edit/compile/test cycle

  • VoodooPad as a scratchpad for sample scripts -- it's easy to save them in pages that sit alongside my worklog

During initial exploration I just bang up the script in VoodooPad, then repeatedly select all (Cmd-A), copy (Cmd-C), switch to a Terminal session running Jython (Cmd-Tab), and paste (Cmd-V).

The "learnings" are going into proper Jython scripts maintained with TextMate.

It's nice to be able to deploy using Jython instead of pure Java. Since almost all of the heavy lifting is being done inside the cheminformatics jars, I don't need to worry about the overhead of running a Python interpreter inside a JVM...

Here's a simple example using CDK, to transform an SD string into a SMILES string.

#! /usr/bin/env jython
# You must have set your CLASSPATH to pick up the CDK jar.
# encoding: utf-8

import java.io
from org.openscience import cdk

def getCDKMol(sdf):
    reader = cdk.io.MDLReader(java.io.StringReader(sdf))
    result = cdk.Molecule()
    result = reader.read(result)
    return result

def getSmiles(cdkMol):
    result = java.io.StringWriter()
    writer = cdk.io.SMILESWriter(result)
    writer.write(cdkMol)
    return str(result)

def sdfToSmiles(sdf):
    return getSmiles(getCDKMol(sdf))

# Example:
print sdfToSmiles("""1-7


42 44 0 0 0 0 999 V2000
-0.8349 0.9908 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0
-0.0502 1.2457 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-0.0502 2.0707 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-0.8349 2.3257 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-1.3198 1.6582 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
0.6172 0.7608 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-1.0898 0.2062 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
0.5310 -0.0597 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
1.1984 -0.5446 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
1.9521 -0.2090 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
2.0383 0.6114 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
1.3709 1.0964 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-1.8968 0.0346 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-2.1517 -0.7500 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-1.5997 -1.3631 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-0.7927 -1.1915 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-0.5378 -0.4069 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-2.1448 1.6582 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
2.6195 -0.6940 0.0000 S 0 0 0 0 0 0 0 0 0 0 0 0
3.1044 -0.0265 0.0000 O 0 0 0 0 0 0 0 0 0 0 0 0
2.1346 -1.3614 0.0000 O 0 0 0 0 0 0 0 0 0 0 0 0
3.2869 -1.1789 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
-1.8546 -2.1477 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 0
0.6172 2.5557 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-1.0898 3.1103 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-0.2227 -0.3952 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
1.1122 -1.3651 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
2.7920 0.9470 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
1.4571 1.9168 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.4488 0.6477 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.9587 -0.9215 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-0.2407 -1.8046 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-0.1632 -1.1420 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.1448 0.8332 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.1448 2.4832 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.9698 1.6582 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
3.7719 -0.5114 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
2.8020 -1.8463 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
3.9544 -1.6638 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-1.0700 -2.4026 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.6392 -1.8928 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
-2.1096 -2.9323 0.0000 H 0 0 0 0 0 0 0 0 0 0 0 0
1 2 1 0 0 0 0
1 5 1 0 0 0 0
1 7 1 0 0 0 0
2 3 2 0 0 0 0
2 6 1 0 0 0 0
3 4 1 0 0 0 0
3 24 1 0 0 0 0
4 5 2 0 0 0 0
4 25 1 0 0 0 0
5 18 1 0 0 0 0
6 8 2 0 0 0 0
6 12 1 0 0 0 0
7 13 2 0 0 0 0
7 17 1 0 0 0 0
8 9 1 0 0 0 0
8 26 1 0 0 0 0
9 10 2 0 0 0 0
9 27 1 0 0 0 0
10 11 1 0 0 0 0
10 19 1 0 0 0 0
11 12 2 0 0 0 0
11 28 1 0 0 0 0
12 29 1 0 0 0 0
13 14 1 0 0 0 0
13 30 1 0 0 0 0
14 15 2 0 0 0 0
14 31 1 0 0 0 0
15 16 1 0 0 0 0
15 23 1 0 0 0 0
16 17 2 0 0 0 0
16 32 1 0 0 0 0
17 33 1 0 0 0 0
18 34 1 0 0 0 0
18 35 1 0 0 0 0
18 36 1 0 0 0 0
19 20 2 0 0 0 0
19 21 2 0 0 0 0
19 22 1 0 0 0 0
22 37 1 0 0 0 0
22 38 1 0 0 0 0
22 39 1 0 0 0 0
23 40 1 0 0 0 0
23 41 1 0 0 0 0
23 42 1 0 0 0 0
M END
$$$$
""")

SQLElixir/SQLAlchemy/SQLite3 and ISO-8859 content

See here for the full discussion. The upshot is that, if you are storing string data into an SQLite3 database via Python, and if you suspect that the data may contain non-ASCII characters, you need to ensure that it's properly converted to (Python) Unicode before inserting.

#!/usr/bin/env python
# encoding: utf-8
"""Explores a problem importing a chemical structure file into an
SQLElixir/SQLAlchemy/SQLite3 database.

The file was in SD format. One of the tags contained ISO-8859 characters.
Python read the file contents without trouble; SQLite3 stored the contents.
But on retrieval SQLite3 raised an OperationalError:
Could not decode to UTF-8 column '[...]' with text '[...]'

This script explores the problem, and demonstrates two successful
Unicode encodings: utf-8/replaced and utf-8/ignored.

NB: Only the latter actually worked with SQLElixir:
s = unicode(s, encoding='utf-8', errors='ignore')
"""
import unittest, sqlite3

class TestCase(unittest.TestCase):
    def setUp(self):
        self.conn = sqlite3.connect(":memory:")
        self.conn.cursor().execute('CREATE TABLE demo (value TEXT)')

    def insertAndDump(self, encoding, errors, expectFailure=False):
        caseName = repr([encoding, errors])
        data = 'K\xf6 1366'
        try:
            if encoding is not None:
                data = unicode(data, encoding=encoding, errors=errors)
            cursor = self.conn.cursor()
            cursor.execute("INSERT INTO demo (value) VALUES (?)", [data])
            cursor.execute('SELECT * FROM demo')
            self.failUnless(len(cursor.fetchall()) == 1)
            self.failIf(expectFailure, "Unexpected success for %s" % caseName)
        except Exception, info:
            self.failUnless(expectFailure, "Failed %s: %s" % (caseName, info))

    def testCannotRetrieveUnencoded(self):
        self.insertAndDump(None, None, True)

    def testCannotEncodeStrict(self):
        self.insertAndDump("utf-8", "strict", True)

    def testCanRetrieve8859Strict(self):
        self.insertAndDump("8859", "strict")

    def testCanRetrieveUTF8Ignored(self):
        self.insertAndDump("utf-8", "ignore")

    def testCanRetrieveUTF8Replaced(self):
        self.insertAndDump("utf-8", "replace")

unittest.main()