Good on you, Wolf Rentzch

rentzsch.tumblr.com: [C4 release]:
"With resistance to Section 3.3.1 so scattershot and meek, it’s become clear that I haven’t made the impact I wanted with C4. It’s also clear my interests and the Apple programming community’s interests are farther apart than I had hoped."
I signed up for an iPhone developer account this spring, just before the change to the developer agreement which included the revised 3.3.1. The change was galling. Never mind Flash; Apple was showing what it thinks of its developer community. "We can further limit you any time we want. Not a big deal. Just rewrite your code." I bought back into Apple in 2002, drawn by OS X. OS X developers are still free from the strictures imposed on iP* developers. Let's hope it stays that way. If not, then it's back to Linux for me. Meanwhile, here's hoping for good things from HP/Palm, Notion Ink, and the Google Android ecosystem.

Upgrading a VMware VM to Ubuntu 10.04

Yesterday I tried to update an Ubuntu 9.10 VM, running VMware Fusion 3.1 beta, to Ubuntu 10.04. Two significant problems arose:

  1. At the login screen, the keyboard didn't work
  2. the boot process paused because hgfs could not be mounted

Here's a log of the problems and worries encountered, together with their solutions.

Grub Configuration

Late in the Ubuntu upgrade process a warning appeared, saying Grub could not be configured (or words to that effect — I wasn't paying close attention) and asking for confirmation that the installation should continue without any changes to grub's configuration. I confirmed. After all, this was a VM instance with only a single operating system installed.

Can't Mount hgfs

On reboot the console reported that a problem had occurred while mounting /mnt/hgfs. If offered to continue without mounting, and I accepted.

Unresponsive Keyboard

When the Ubuntu login screen appeared, the keyboard was unresponsive. This made it difficult to enter my password.

Thanks to this thread in the VMware Communities forums, I was able to login after enabling the onscreen keyboard. (Click on the accessibility icon — the "little man in the circle" — check the onscreen keyboard toggle, and restart the VM.) Once logged in, the keyboard worked just fine.

VMware Tools Out of Date

VMware detected that VMware Tools were out of date for this VM. (It may have reported that they weren't installed. I can't recall.)

No xorg.conf?

I started the VMware Tools installation process and encountered a warning I hadn't seen before:

Detected X.org version 7.6.6.

The configuration file /etc/X11/xorg.conf can not be found.  Do you want to create a new one? (yes/no) [yes]

Not sure what to do, I accepted the default response.

Keyboard Fixed

After the tools finished installing I rebooted the VM. This time, at the login screen the keyboard worked. A wild guess is that the VMware Tools re-installation fixed this problem by reconfiguring the console.

Fixing hgfs Ownership

/mnt/hgfs had been auto-mounted, but as usual the ownership and permissions were wrong. I patched this as documented in a previous post.

Fixing hgfs automount Failure

Ubuntu continued to complain at startup that it had failed to mount hgfs. It continued to request user permission to skip mounting hgfs. But once the boot completed /mnt/hgfs was mounted with correct permissions.

To prevent Ubuntu complaining of failure to mount hgfs, I added a "nobootwait" option to its entry to tell mountall(8) not to hold up the boot for this filesystem. The entry now reads:

.host:/ /mnt/hgfs vmhgfs rw,ttl=1,uid=my_uid,gid=my_gid,nobootwait 0 0

Now Ubuntu boots without complaint, the keyboard works, and /mnt/hgfs is mounted with correct ownership and permissions.

VMware Fusion 3.0.1, Ubuntu 9.10 and shared folders

In VMware Fusion I share a folder with many of my VMs. In a new Ubuntu 9.10 VM, that shared folder is always mounted with the wrong owner and group IDs. Here's a fix.

After booting the Ubuntu VM for the first time, I downloaded and installed the latest vmware-tools distribution, using the VMware "Virtual Machine -> Install VMware Tools" menu item. Then I added an entry to /etc/fstab, specifying the correct owner and group ID for the /mnt/hgfs mount point.

Unfortunately, the owner and group ID specifications were ignored. As it turns out, /etc/init.d/vmware-tools does not examine /etc/fstab when deciding whether or how to mount the shared filesystem.

Here's how I changed /etc/init.d/vmware-tools to address this problem.

Old

# Mount all hgfs filesystems
vmware_mount_vmhgfs() {
  if [ "`is_vmhgfs_mounted`" = "no" ]; then
    vmware_exec_selinux "mount -t vmhgfs .host:/ $vmhgfs_mnt"
  fi
}

New

is_vmhgfs_in_fstab() {
    if grep -q " $vmhgfs_mnt " /etc/fstab; then
        echo "yes"
    else
        echo "no"
    fi
}
# Mount all hgfs filesystems
vmware_mount_vmhgfs() {
  if [ "`is_vmhgfs_mounted`" = "no" ]; then
    if [ "`is_vmhgfs_in_fstab`" = "yes" ]; then
        vmware_exec_selinux "mount $vmhgfs_mnt"
    else
        vmware_exec_selinux "mount -t vmhgfs .host:/ $vmhgfs_mnt"
    fi
  fi
}

PyCon 2010 Poster Session

I finally presented something at a Python conference. Many thanks to Vern Ceder for organizing the PyCon 2010 poster session! Here's a copy of the poster, which describes a way to run QUnit-based JavaScript unit tests using Python's unittest facilities:

The code described by the poster is available from bitbucket.org, under the BSD license.

Mercurial, Migration Assistant, and dotfiles

I recently upgraded my iMac. Migration Assistant moved all of my files to the new machine without issue -- or so it seemed. I had created Mercurial repositories in a couple of virtualenv environments, to track changes locally.[1] I didn't notice that Mercurial had put each virtual environment's .Python file under revision control. Shortly after completing the migration I made changes in one of these virtual environments. A quick 'hg status' before committing, and...

$ hg status
abort: data/.Python.i@13b27e856c38: no match found!
WTF? After much investigation it appears that the following has happened.
  1. Mercurial represented the .Python link in its .hg/store/data directory as ._Python.i
    1. The leading underscore appears to be Mercurial's way of noting that the 'P' should be capitalized.
  2. I think Migration Assistant uses ditto to copy files.
    1. ditto saw the leading '._' and concluded this was an orphaned resource file.[2] So it didn't copy the file.
  3. Mercurial knew that it was supposed to have a .hg/store/data/._Python.i file; when it couldn't find it, it decided the repository was corrupted.
Luckily the problem cropped up before I traded in the old machine, so I was able to copy across the missing files manually. In my experiments, the problem manifested only when the capitalized dotfile was a symbolic link, not when it was a regular data file. Well... the above write-up contains several unproven assertions, e.g. about the conditions under which Mercurial will create a '._' filename. I'm not really sure whether this is a bug or merely a caveat regarding an obscure corner condition. For now, the easiest workaround is: Don't Track virtualenv .Python Files With Mercurial. [Update 2010/02/22: Someone has already filed this as a Mercurial bug.]
[1] (Mercurial makes a great filesystem "undo" facility, useful even for directory trees which you never intend to share with anyone else.) [2] OS X still supports something ike resource forks. In tarballs and other non-OS Extended filesystems, resource forks are represented as dot-files with a leading underscore. (See Norman Walsh's blog for more info.) For example, the resource fork for a file named 'foo.txt' might be '._foo.txt'.

Python, webbrowser, OS X, and "execution error"

Once in awhile I write Python scripts which produce static HTML pages and then use the webbrowser module to open the results in a browser. I do this just rarely enough that, when the scripts fail on OS X with the following error, I have to waste time diagnosing the problem: execution error: An error of type -2110 has occurred. (-2110) Despite the obscure phrasing, the cause of the error is simple. I'm passing a filesystem pathname to webbrowser.open, when I should be passing a file: URL. This produces the error:
webbrowser.open("/path/to/file.html") This works:
webbrowser.open("file:///path/to/file.html")

TextMate, Emacs and META indent-region

I haven't used GNU Emacs very much since switching to TextMate in 2005. One Emacs feature which I really miss in TextMate is indent-region. It lets you take an entire region of code, whatever its language, whatever its mix of tabs and spaces and indentation widths, and re-format it using your preferred indentation style.

But wait! Emacs has a batch mode, and you can drive it from TextMate. Many thanks to Gragusa's Things for showing the way.

The post on Gragusa's Things is specific to R code, but I'm more interested in re-formatting C and C++ code. Here's my first cut at a general TextMate Bundle to re-format code regardless of the source language:

#!/usr/local/bin/python2.6

"""
Use Emacs to re-indent regions of the current buffer.
Inspired by 
http://gragusa.wordpress.com/2007/11/11/textmate-emacs-like-indentation-for-r-files/
"""
import tempfile
import os
import sys
import subprocess

# Use the same filename extension so Emacs will know which
# mode to use. 
ext = os.path.splitext(os.environ["TM_FILEPATH"])[-1]
outf = tempfile.NamedTemporaryFile(suffix=ext, delete=False)
pathname = outf.name

outf.write(os.environ["TM_SELECTED_TEXT"])
outf.close()

args = [
    "emacs", "-batch", pathname, 
    # Assume no emacs-startup.el
    "--eval", "(setq indent-tabs-mode nil)",
    "--eval", '(c-set-style "java")',
    "--eval", "(setq c-basic-offset 4)",
    "--eval", "(indent-region (point-min) (point-max) nil)", 
    "-f", "save-buffer"]
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode:
    print(err)

inf = open(pathname, "r")
sys.stdout.write(inf.read())
inf.close()

os.remove(pathname)

NB:

  1. Due to the use of the delete=False keyword argument to tempfile.NamedTemporaryFile, this command bundle requires Python 2.6+.
  2. TextMate on OS X 10.5 won't, by default, have /usr/local/bin in its path; hence the pathetic shebang.

Anyway, install this as a new TextMate command bundle, assign a Key Equivalent such as ⌘-Shift-R, and enjoy.

Creating an 'hg ignore' extension

I often wish Mercurial had an 'hg ignore' command similar to 'bzr ignore'. Turns out it's pretty easy to add one:

#!/usr/bin/env python
"""Ignore pathnames and patterns"""

import os

def ignore(ui, repo, *pathnames):
    """Ignore the given pathnames and patterns."""
    outf = open(os.path.join(repo.root, ".hgignore"), "a")
    for p in pathnames:
        outf.write(p + "\n")
    outf.close()
    return

cmdtable = {
    'ignore': (ignore, [], "hg ignore pathname [pathname]"),
}

To use this, save it to a file such as ${HOME}/platform/independent/lib/hg/ignore.py. Then add the extension to your ${HOME}/.hgrc:

[extensions]
~/platform/independent/lib/hg/ignore.py

Running TileCache within a Django Application

[Cross-posted from Bottled Text, as usual.]

Punchline

Here is how to serve TileCache tile images from within a Django application.

from TileCache.Service import Service
_service = Service(...)
def get_tile(request):
    global _service
    format, image = _service.dispatchRequest(
        request.GET, request.path, request.method, 
        request.get_host())
    result = HttpResponse(str(image), mimetype=format)
    return result

Scenario

You're building a low-traffic Django-based GIS application, and you need to serve your own map layers. You're using TileCache to improve your application's performance. But installation and configuration are hassles.

  • All of your servers must run with the right user and group IDs, so the Django app can expire the tile cache when necessary.
  • Your Django app needs to understand the structure of the tile cache, so it can remove the correct tile images when the underlying data changes.
  • Etc.
standalone_tilecache.png

This would all be much easier if you could serve TileCache requests from within your Django application. They're both Python-based; why not?

django_plus_tilecache.png

The TileCache code base includes sample code that shows how to run TileCache as a CGI or a FastCGI service. I couldn't find any sample code for running TileCache within a Django application, but it was easy to convert the cgiHandler code for use with Django's HttpRequest objects.

Installation Prerequisites

In order for TileCache to generate its own tiles, instead of delegating to a separate mapserver instance, you must already have compiled and installed mapserver's Python mapscript bindings. For instructions on compiling the bindings see the mapscript/python/README file in the mapserver source distribution.

Configuring TileCache

import os
thisdir = os.path.abspath(os.path.dirname(__file__))
def relpath(p):
    return os.path.abspath(os.path.join(thisdir, p))

from TileCache.Service import Service
import TileCache.Layers.MapServer as MS

# Create the service 'singleton'.
_mapfile = relpath("../mapserv/data/mapfile.map")

_service = Service(
  _cache,  # See "Cache Invalidation", below
  { 
    "basic": MS.MapServer(
        "basic", _mapfile, layers="basic", debug=False),
  }
) 

Handling Tile Requests

This is the sweet part. It's derived from the cgiHandler() example in the TileCache source code, but Django's HttpRequest class makes the implementation very simple:

def get_tile(request):
    global _service
format, image = _service.dispatchRequest( request.GET, request.path, request.method, request.get_host()) result = HttpResponse(str(image), mimetype=format) return result

What About Feature Info Requests?

I don't know much about the required web API of a WMS server, but it appears as if the same URL must serve both tiles and feature info requests; the type of request is determined by the Request querystring parameter.

Django's dispatch system is based on URL pathnames; I'm not aware of any way to dispatch based on query string parameters. So you'll need to either configure your web server (e.g. Apache) to rewrite WMS requests to distinct URLs provided by your Django app, or you'll need to do some dispatch within your Django app.

Suppose you opt for the latter. Then your urls.py might look something like this:

    ...
    url(r'^wms/$', 'world.views.wms', name='wms'),
    ...

and in world/views.py you might have this:

def wms(request):
    if request.GET.get("request") == "GetFeatureInfo":
        return get_feature_info(request)
    return get_tile(request)

Cache Invalidation

For my web app, several of the tile layers are derived from a Django model which is updated via the admin interface. Whenever the model changes, the tile cache for the corresponding layer(s) needs to be invalidated, so the images can be regenerated.

The TileCache Cache interface doesn't provide for invalidation. Since I'm using a filesystem-based cache, I subclassed TileCache.Caches.Disk to create a Disk cache which does support invalidation.

import shutil
from TileCache.Caches.Disk import Disk

class InvalidatingDisk(Disk):
    """A Disk cache which can invalidate its contents, 
       layer by layer."""
    def invalidate(self, layerName=None):
        if self.basedir:
            pathname = self.basedir
            if layerName is not None:
                pathname = os.path.join(self.basedir, 
                                        layerName)
            shutil.rmtree(pathname, ignore_errors=True)

Ted Leung on web app design patterns

DjangoCon 2009:
"Avi Bryant’s keynote took its root in his experiences building Trendly. As one might expect, Avi started building Trendly using Seaside. But by the time he finished, he noticed that very little of Seaside was actually being used. He attributed this to the fact that Trendly’s architecture involves loading a single HTML, with a ton of Javascript. That Javascript then manages all of the interaction with the server, which consists of snippets of JSON data. This range true to me because we used a similar architecture for Chandler Hub, the web based version of Chandler (our interaction with the server was based on atom and atompub, not JSON), and it’s the kind of architecture that GMail is based on."
The same has been true for us (Mesa Analytics) in most of our web applications. Sometimes we do have "multi-page" web apps, but each of the individual pages typically follows the pattern above: load, then let JavaScript drive a bunch of task-specific server interactions. Between Ted's summary of DjangoCon 2009 and the chatter about Tornado, it has been an interesting week for web application developers.