Distributed Binary Data Sorter

One machine (the coordinator) breaks a big file of numbers into chunks and sends them out to worker machines with Dispy. Each worker sorts its chunk, and the coordinator merges finished chunks two at a time while the rest are still running.

In the final version everything stays in binary — numbers get packed into 4-byte ints before they're sent out and stay that way through every merge, until the very end when it's written out as text.

The coordinator reads the input as it goes instead of loading it all at once, packs each chunk with Python's struct module, and sends it off right away — so workers start sorting before the whole file has even been read. Every time a job finishes, a callback adds its output to a queue and merges pairs of files, with a lock so callbacks don't step on each other.

I built this in three steps: get a distributed sort working, figure out where it was doing extra work, test a fix on its own, then put it back in.