Nick's blog - tech

Getting O2 Ireland's "Mobile Broadband" working in OSX 10.5

nospam@example.com (Arachnid) — Tue, 29 Jul 2008 16:11:42 +0100

O2 Ireland proudly advertise that their mobile broadband offering works for both Windows and Mac. Then they proceed to offer only windows instructions, with no hint of how to get it working on a mac. This is, obviously, less than helpful. To complicate things, the generic instructions for getting Huawei modems working in OSX aren't entirely sufficient: The PIN on the sim card has to be disabled first, and the Huawei app for setting the APN doesn't seem to work in 10.5. So here are the instructions, in a nutshell:

1. Find a windows computer with administrator priveliges. Plug in the modem and follow the installation instructions. When prompted, enter your pin, then select Tools -> Pin Options -> Disable Pin. Enter your pin again. If you're trying these directions for a network other than O2 Ireland, this would be a good time to check the settings for the APN name, too.

Yes, I know it sucks to have to use a PC to set it up. It might be possible to do this by putting the 3G sim into a cellphone and disabling the PIN using that - I haven't tried, but it seems like it should work.

2. Follow the rest of the instructions here, with the following addenda:
- You may not need to add the device as described. In my case, it automatically added itself to the network preferences panel, and just needed configuring.
- The E220 drivers work fine if you have an E270, too.
- If you're using O2 Ireland, the APN you want to enter is "open.internet".

Nearly all DHT implementations vulnerable to 'merge' bug.

nospam@example.com (Arachnid) — Sun, 15 Jun 2008 13:34:09 +0100

As DHT implementations proliferate and harmonise, the prospect of multiple widely-deployed applications using the same or compatiable DHT implementations is increasingly becoming a reality. There are a large and increasing number of DHT libraries out there, such as Entangled , and FreePastry, being used by an increasing number of applications.

However, most of these implementations are vulnerable to a simple but subtle bug: They have no way of distinguishing one DHT network from another. Although each application's DHT network or networks start off separate, if, by chance or deliberate action a node from a different but compatiable DHT is introduced, the 'self healing' property of DHTs will ensure that, sooner or later, the two networks become merged into a single DHT.

This is not a problem for single-purpose DHT implementations such as those used by BitTorrent or Overnet, since they generally establish a single DHT with all compatiable clients participating in any case. Nor is this a problem for networks designed with heterogenous applications in mind, such as CSpace. However, this still leaves a number of DHT libraries that don't fall into either of these categories.

If DHTs from two distinct applications using compatiable implementations become merged, the outcome depends to a large extent on how those applications make use of the DHT. If they use compatiable parameters and only make use of the basic DHT store/retrieve/find-node functionality, the impact may be minimal, though differences in persistence time and maximum size for stored elements can still lead to issues. If the DHTs have different parameters, such as the number of nodes to replicate inserted data to, the result may be subtly broken - a DHT that no longer fulfils the guarantees the software expects from it.

If the two applications expect very different things from their DHTs, though, such as a varying set of RPCs supported, the result could be a complete breakdown in the usefulness of the DHT. If your application expects that the vast majority of participating nodes will support its custom doFoo RPC (not an unreasonable expectation when you control the application), and suddenly half the nodes don't, the probability is that the application's behaviour will be severely degraded, at best.** This amounts to an entirely accidental Denial of Service attack on both applications involved.

The solution is relatively simple, of course - simply add an application identifier or 'magic number' to the DHT protocol, and require each application to specify an identifier unique to it. Then, the DHT library can simply discard packets with IDs it doesn't recognise.

In a quick survey of current DHT implementations, at least the following seem to be vulnerable:

Entangled

KHashmir

Plan-X

SharkyPy

Open Chord

Chimera

Freepastry is notable in that it does provide for a 'magic number' facility. However, this is buried deep in the protocol stack, and is not required in order to create a Freepastry DHT instance.

I'm not aware of this bug occurring yet, but with increasing use of DHTs, it seems to me to be all but inevitable that it will occur at some point. Further, restoring normal behaviour after it has occurred is likely to be extremely difficult, likely requiring starting an entirely new DHT instance and moving all existing clients over to it.

** Please forgive my overuse of generalities. As I'm speaking of a general bug shared across many libraries and implementations, it's hard to be specific about the impact.

SMTP to HTTP gateway for your App Engine (and other) apps!

nospam@example.com (Arachnid) — Sat, 14 Jun 2008 22:23:13 +0100

In response to a comment in the freenode.net/#appengine channel by someone wishing their App Engine app could receive email, I put together smtp2web, a simple service that accepts mail for an address (or your entire domain), and sends it via HTTP POST to a URL you specify. If you're running in a restricted environment such as App Engine, this means you can now receive email. Even if you're not, this is a lot simpler to use than writing your own SMTP server (or adding custom handlers to most existing servers).

Someone's already blogged about it, too.

Update on Anagram Trees

nospam@example.com (Arachnid) — Sat, 20 Oct 2007 04:01:31 +0100

Original Post

One nice thing about working at Google is that you are surrounded by very smart people. I told one of my coworkers about the anagram tree idea, and he immediately pointed out that reordering the alphabet so that the least frequently used letters come first would reduce the branching factor early in the tree, which has the effect of reducing the overall size of the tree substantially. While this seems obvious in retrospect, it's kind of unintuitive - usually we try to increase the branching factor of n-ary trees to make them shallower and require fewer operations, rather than trying to reduce it.

Trying it out with an ordering determined by looking at the branching factor for each letter produces results that bear this out: Memory is reduced by about a third, and the number of internal nodes is reduced to 858,858 from 1,874,748, a reduction of more than 50! Though I haven't benchmarked it, difficult lookups are substantially faster, too.

The next logical development to try is to re-evaluate the order of the alphabet on a branch-by-branch basis. While I doubt this will have a substantial impact, it seems worth a try, so I'll give it a go and update with results.

Edit: Re-evaluating the symbol to choose on a branch-by-branch basis had a bigger impact than I anticipated: The tree created with my sample dictionary now has a mere 661,659 internal nodes. Here's the procedure for creating a tree using this method:

Assuming you have:

A dictionary
A set of symbols that have not yet been used (initially set to the alphabet)

If the symbol set is empty, this is a leaf node - store the dictionary in the node and return.
Find the symbol from the set that, if used, will result in the smallest number of branches (that is, the symbol that has the least variation in number of occurrences).
Mark the current node with the chosen symbol
Partition the dictionary into sub-dictionaries based on how many occurrences of the chosen symbol they have
For each sub-dictionary, recurse with the sub-dictionary and the set less the symbol you selected.

Implemented in Python, this is actually substantially larger in memory and on disk than the previous approach, likely due to overhead with using classes instead of tuples as the nodes. In statically-typed languages, however, the overhead should be substantially outweighed by the benefit of the reduction in node count.

Note that the result of this alternate method is that while the letter to branch on is different for every node, following nodes from any leaf to the root of the tree always results in a valid permutation of the alphabet used.

Edit 2: The code for a Python implementation incorporating these ideas can be found here.

Damn Cool Algorithms, Part 3: Anagram Trees

nospam@example.com (Arachnid) — Fri, 19 Oct 2007 04:00:50 +0100

I hesitate to call this algorithm "damn cool", since it's something I invented* it myself, but I think it is rather cool, and it fits the theme of my algorithms posts, so here it is anyway.

When it comes to finding anagrams of words, a frequent approach is to use an anagram dictionary - simply put, sort the letters in your word to provide a unique key that all anagrams of a word have in common. Another approach is to generate a letter-frequency histogram for each letter in your word. (Both these approaches are more or less equivalent, in fact.) These approaches make the problem of finding exact single-word anagrams for strings very efficient - O(1) if you use a hashtable.

However, the problem of finding subset anagrams - a word that contains a subset of the letters in a string - is still rather inefficient, requiring either a brute force O(n) search through the dictionary, or looking up every substring of the sorted input string, which is O(2^l) with the number of letters in the input string. Finding subset anagrams is significantly more interesting, too, as it has applications in finding multi-word anagrams, as well as being applicable to problem domains such as scrabble.

However, with a little more effort, and the above observation that we can generate a histogram that uniquely represents a given set of letters, we can generate a tree structure that makes looking up subset anagrams much more efficient. To build the tree, we follow this simple procedure:

Assume we have the following information:

A lexicon or dictionary of words to populate the tree with
An alphabet for words in the lexicon
The tree we are building
A current node

For each term in the lexicon:

Generate a letter-frequency histogram for the term.
Set the current node to the root of the tree.
For each symbol in the alphabet:
1. Get the frequency of the current symbol in the current term. Call it f
2. Set the current node to the f^th child node of the current node, creating it if it doesn't exist
Append the current term to the list of words on the current (leaf) node

The result of following this simple procedure is a fixed-height tree, 27 nodes deep, with all the words in the leaf nodes, and each internal tier of the tree corresponding to a symbol from the alphabet. Here's an (abbreviated) example:

Once the tree is built, we find subset anagrams for an input string as follows:

Assume we have the following information:

The tree we built using the above procedure.
The alphabet we used above.
A frontier set, initially empty.

Initialize the frontier set to contain the root of the tree.
Generate a letter-frequency histogram for the input string.
For each symbol in the alphabet:
1. Get the frequency of the current symbol in the input string. Call it f.
2. For each node in the current frontier set, add the subnodes numbered 0 through f to the new frontier set.
The frontier set now consists of leaf nodes, containing all the subset anagrams of the input string.

Runtime analysis of this algorithm is rather difficult, for me, at least. Intuitively and in practice, it's a lot faster than either of the brute-force approaches, but quantifying that in big-O notation is something that's escaped me. As an upper bound, it cannot be less efficient than O(n) - only a constant factor worse than the brute-force approach. As a lower bound, a lookup in which the frontier set always has one node, lookup time is proportional to the length of the alphabet, or O(1). The average case depends on how large a subset of the dictionary the input string selects. Quantifying by the size of the output, approximately O(m) operations are required. If anyone knows how to determine more solid bounds for runtime, please do let me know in the comments.

One disadvantage of this approach is that there is substantial memory overhead. Using my Python implementation of the algorithm, and importing /usr/share/dict/words, which is approximately 2MB on this machine results in over 300MB of memory consumed. Using the Pickle module to serialize to disk, the output file is over 30MB, and compresses with gzip down to about 7MB. I suspect part of the large memory overhead is due to the minimum size of Python's dictionaries; I will modify the implementation to use lists and update this post if I can make it more efficient.

Here's a few stats on the tree generated that may be of interest:
Total words: 234,936
Leaf nodes: 215,366
Internal nodes: 1,874,748

From this we can see that the average cardinality of internal nodes is very low - not much more than 1. A breakdown of the number of nodes in each tier helps clarify this:

Tier	Number of nodes
0	1
1	7
2	25
3	85
4	203
5	707
6	1145
7	1886
8	3479
9	8156
10	8853
11	10835
12	19632
13	28470
14	47635
15	73424
16	92618
17	94770
18	125018
19	156406
20	182305
21	195484
22	200031
23	203923
24	205649
25	214001

The cardinality of nodes towards the top of the tree is fairly high, but the tree quickly flattens out, and the last four tiers of the tree account for almost half of the total nodes. This suggests one possible space optimisation: Removing the last few tiers of the tree and concatenating their leaf nodes together. When performing lookups, check the selected nodes to ensure they are actually subset anagrams of the input string.

* It's possible I'm simply rediscovering something that's well known in the computer science community, or perhaps mentioned in a computer science paper 30 years ago. Significant searching hasn't turned up anyone using an algorithm like this, or anything else more efficient than the brute-force approaches outlined.

Edit: The source to my initial implementation is here.

Edit: Converting my Python implementation to use lists reduced memory consumption by roughly half. I'll post figures for the pickled tree and the source code when I have the opportunity.

Edit: More updates can be found here.

Damn Cool Algorithms, Part 2: Secure permutations with block ciphers

nospam@example.com (Arachnid) — Sun, 30 Sep 2007 01:34:26 +0100

It's been too long since I blogged about anything much, and way too long since I posted the first Damn Cool Algorithms post, which I promised would be a series. So here's part 2.

To start, I'm assuming you know what a permutation is - basically a shuffling of a sequence of items in a particular order. A permutation of the range 1-10, for example, is {5,2,1,6,8,4,3,9,7,10}. A secure permutation is one in which an attacker, given any subset of the permutation, cannot determine the order of any other elements. A simple example of this would be to take a cryptographically secure pseudo-random number generator, seed it with a secret key, and use it to shuffle your sequence.

What if you want to generate a really, really big permutation - one so big precomputing and storing it isn't practical or desirable? Furthermore, what if you want it to be a secure permutation? There's a really neat trick we can pull with block ciphers that allows us to generate a secure permutation over any range of numbers without first having to precompute it.

A block cipher, for anyone that isn't familiar with them, is a common cryptographic primitive. It takes blocks of ciphertext of some fixed lengths - 64 or 128 bits is common - and encrypts it. Given the same key and the same block of plaintext, it will always generate the same block of ciphertext. Messages larger than a single block are encrypted using one of a number of modes of operation, allowing messages much larger than a single block to be encrypted and decrypted securely. When using a block cipher for encryption, choice of the mode of operation is critical. For the purposes of generating a secure permutation, however, we're only going to be encrypting a single block at a time, so we don't have to worry about modes of operation.

If you look at how a block cipher operates - taking any block of a given length (think of blocks as very large numbers here) and converting it uniquely to another block, such that it can be converted back again, a block cipher is already a secure permutation. If we progressively encrypt larger numbers (1, 2, 3, and so on), we get out a random seeming sequence of output numbers that is guaranteed not to repeat as long as we don't repeat our input. It's easy to prove this to yourself: If it repeated, then you would have two input numbers with a single output number, and it would thus be impossible to decrypt uniquely. So the same properties that a block cipher requires are the properties that make it useful to us.

All very well, you say, but what if I want a permutation over a range that isn't a power of two? This is where the clever trick comes in. Take a block cipher that's got a block length slightly larger than you want. Use it as described above, encrypting progressively higher numbers in a sequence to generate elements in the permutation. Whenever the output of the encryption is outside the range you want for your permutation, just encrypt it again. Repeat until you get a number within the range you want. Again, we're guaranteed uniqueness by the block cipher, and we're also guaranteed (by exhaustion) that we will eventually get a number within the desired range.

Obviously, there are some factors that need to be taken into consideration before pursuing this route. You want to select a block cipher that is not much larger than the range you wish to generate a permutation over - preferably the next power of two. The ratio of the cipher's range to the permutation's range defines the average amount of work you will have to perform, so if the cipher has a range four times that of your permutation, you'll need to do an average of four encryptions for each value. Since most block ciphers are 64, 128, or more bits, this can be problematic. For this purpose, I've found the TEA cipher to be particularly adaptable. It is easy to create variants that are 32, 64, 128 or more bits long, and from there, the bitshifts in the main loop are easily adjusted to produce a cipher with a length that's any power of 4, without needing to shorten the key to the point where it's easily brute-forced.

It's also worth noting that although this technique is aimed at generating very large secure permutations, it is equally useful for a permutation that doesn't need to be secure or secret - your secret key simply becomes your random seed for the permutation. There are many situations in which this can be useful - what you essentially have is a mapping function from index to permutation value, so you can calculate the value of any subset of the permutation that you wish.

Finally, bear in mind that due to the factorial explosion of the number of possible permutations, the keyspace of your cipher is almost certainly going to be much smaller than the number of possible permutations. For most purposes this probably does not matter, since the number of possible permutations is too large to enumerate anyway, but if your key is sufficiently short, it allows the possibility of an attacker doing an exhaustive search of your keyspace to find the permutation that generates the subsequence of the permutation he has access to.

Update: Yossi Oren points to this excellent paper in the comments. It covers exactly what I describe here (only much more comprehensively, of course).

I am teh famous, LOL!

nospam@example.com (Arachnid) — Tue, 28 Aug 2007 07:58:33 +0100

LOLCode.NET was featured at TechED07, in a presentation by Nick Hodge, a "Professional Geek" for Microsoft. He's posted a video of the presentation on his blog.

Now I just need to convince them to integrate it into the next edition of VS...

LOLCode.net - Now your LOLCats can use the CLR!

nospam@example.com (Arachnid) — Sun, 03 Jun 2007 04:04:24 +0100

The past few days, I've been working on a .NET compiler for the LOLCode language.

LOLCode is an emerging esoteric (and hilarious) language based on the dialect used in LOLCats images. It's been siezed upon by a group of people (myself included, now), and is being expanded into a real, workable, turing complete esoteric language (though nobody has proven its turing completeness yet!).

The LOLCode.NET compiler is now working, and as a nearly-free bonus for using the .NET platform, you can even debug it in Visual Studio:

I still can't believe I'm looking at LOLCode and its x86 disassembly in Visual Studio.

Dynamic code generation in .NET

nospam@example.com (Arachnid) — Tue, 29 May 2007 02:21:21 +0100

I've written up a rather lengthy document on different methods of dynamically generating code in .NET - two current methods, and one hypothetical one based on a C extension called "`c" (tickc). I think it does a fairly good job of illustrating just how awkward dynamic code generation is in current mainstream languages, and how much simpler and more understandable (and in many cases, more efficient) it could be.

Damn Cool Algorithms, Part 1: BK-Trees

nospam@example.com (Arachnid) — Mon, 02 Apr 2007 03:52:30 +0100

This is the first post in (hopefully) a series of posts on Damn Cool Algorithms - essentially, any algorithm I think is really Damn Cool, particularly if it's simple but non-obvious.

BK-Trees, or Burkhard-Keller Trees are a tree-based data structure engineered for quickly finding near-matches to a string, for example, as used by a spelling checker, or when doing a 'fuzzy' search for a term. The aim is to return, for example, "seek" and "peek" if I search for "aeek". What makes BK-Trees so cool is that they take a problem which has no obvious solution besides brute-force search, and present a simple and elegant method for speeding up searches substantially.

BK-Trees were first proposed by Burkhard and Keller in 1973, in their paper "Some approaches to best match file searching". The only copy of this online seems to be in the ACM archive, which is subscription only. Further details, however, are provided in the excellent paper "Fast Approximate String Matching in a Dictionary".

Before we can define BK-Trees, we need to define a couple of preliminaries. In order to index and search our dictionary, we need a way to compare strings. The canonical method for this is the Levenshtein Distance, which takes two strings, and returns a number representing the minimum number of insertions, deletions and replacements required to translate one string into the other. Other string functions are also acceptable (for example, one incorportating the concept of transpositions as an atomic operation could be used), as long as they meet the criteria defined below.

Now we can make a particularly useful observation about the Levenshtein Distance: It forms a Metric Space. Put simply, a metric space is any relationship that adheres to three basic criteria:

d(x,y) = 0 <-> x = y (If the distance between x and y is 0, then x = y)

d(x,y) = d(y,x) (The distance from x to y is the same as the distance from y to x)

d(x,y) + d(y,z) >= d(x,z)

The last of these criteria is called the Triangle Inequality. The Triangle Inequality states that the path from x to z must be no longer than any path that goes through another intermediate point (the path from x to y to z). Look at a triangle, for example: it's not possible to draw a triangle such that it's quicker to get from one point to another by going along two sides than it is by going along the other side.

These three criteria, basic as they are, are all that's required for something such as the Levenshtein Distance to qualify as a Metric Space. Note that this is far more general than, for example, a Euclidian Space - a Euclidian Space is metric, but many Metric Spaces (such as the Levenshtein Distance) are not Euclidian. Now that we know that the Levenshtein Distance (and other similar string distance functions) embodies a Metric Space, we come to the key observation of Burkhard and Keller.

Assume for a moment we have two parameters, query, the string we are using in our search, and n the maximum distance a string can be from query and still be returned. Say we take an arbitary string, test and compare it to query. Call the resultant distance d. Because we know the triangle inequality holds, all our results must have at most distance d+n and at least distance d-n from test.

From here, the construction of a BK-Tree is simple: Each node has a arbitrary number of children, and each edge has a number corresponding to a Levenshtein distance. All the subnodes on the edge numbered n have a Levenshtein distance of exactly n to the parent node. So, for example, if we have a tree with parent node "book" and two child nodes "rook" and "nooks", the edge from "book" to "rook" is numbered 1, and the edge from "book" to "nooks" is numbered 2.

To build the tree from a dictionary, take an arbitrary word and make it the root of your tree. Whenever you want to insert a word, take the Levenshtein distance between your word and the root of the tree, and find the edge with number d(newword,root). Recurse, comparing your query with the child node on that edge, and so on, until there is no child node, at which point you create a new child node and store your new word there. For example, to insert "boon" into the example tree above, we would examine the root, find that d("book", "boon") = 1, and so examine the child on the edge numbered 1, which is the word "rook". We would then calculate the distance d("rook", "boon"), which is 2, and so insert the new word under "rook", with an edge numbered 2.

To query the tree, take the Levenshtein distance from your term to the root, and recursively query every child node numbered between d-n and d+n (inclusive). If the node you are examining is within d of your search term, return it and continue your query.

The tree is N-ary and irregular (but generally well-balanced). Tests show that searching with a distance of 1 queries no more than 5-8% of the tree, and searching with two errors queries no more than 17-25% of the tree - a substantial improvement over checking every node! Note that exact searching can also be performed fairly efficiently by simply setting n to 0.

Looking back on this, the post is rather longer and seems more involved than I had anticipated. Hopefully, you will agree after reading it that the insight behind BK-Trees is indeed elegant and remarkably simple.

Serializing JavaScript objects with circular references

nospam@example.com (Arachnid) — Thu, 21 Sep 2006 09:51:05 +0100

One problem we've been dealing with at work for a while is serializing JavaScript objects to a string representation, when objects can contain circular references. The simplest example of this is an object a, which contains one property (call it 'foo') which is a reference to a.

Mozilla provides a toSource() function on the Object class (and hence every object) which returns a serialized representation of that object, allowing it to be reconstituted with eval(). To handle circular references, any object or array can be tagged with a numeric identifier by prefixing it with "#x=" (where x is an integer), and a tagged object can be referenced with the syntax "&x;". Using this, any object graph can be serialized, and we can even make use of this system to compress the serialized text down further than Mozilla does on its own by using this syntax for all backreferences, instead of just circular ones. The example earlier, of an object referencing itself would serialize as "#1={foo:&1;}".

All fine and dandy, but unfortunately, IE doesn't implement the toSource() function. I don't know about other browsers either - toSource() is part of Netscape's original Javascript documentation, but not part of the ECMAScript standard. So where does that leave us? We could use this syntax or something similar and simply do both serialization and deserialization manually in IE, but this leads to some fairly serious performance issues. We need something that's far less manual, but works in any JS-supporting browser.

Our first approach was simply to replace the #x syntax with standard variables, in the form of a special context array. If we name this array "_", and initialize it before deserializing, our serialized object now looks like this: "_[1]={foo:_[1]}". Unfortunately, JavaScript doesn't assign to _[1] until it's finished deserializing the value that's being assigned to it (naturally), and so any uses of _[1] inside that value return 'undefined'. Backreferences that aren't circular are just fine.

Eventually, we concluded that an entirely automatic deserialization that works in all JS-supporting browsers simply isn't possible. One that requires a minimum of manual intervention is, however: We continue using the array syntax above for assignments, but we replace all backrefs with placeholder objects. After the eval() step, we recurse through the graph finding all the placeholders and replacing them with the appropriate backrefs from the array. The best part of this is that our array of backrefs has been conveniently populated for us by the JavaScript that we used to identify the destinations of the backrefs in the first place. Thus, our example from above becomes: "_[1]={foo:{_ref:1}}". After initializing "_" and calling eval() on this, we recurse through the tree, find the object "{_ref:1}" and replace it with the value of _[1]. Done! We don't even have to keep track of where we've been to avoid loops, because we just stop recursing whenever we replace a backref.

Indexing directory structures

nospam@example.com (Arachnid) — Wed, 30 Aug 2006 01:00:32 +0100

After much messing around with SQL DBs and table structures, I eventually resorted to writing my own indexing scheme and storing the index in memory for my FS-indexing needs.
Here's how I went about it:

1) Construct a tree out of the FS you're indexing. Each node in the tree needs a reference to its parent, but if you're just using it for this index, it doesn't actually need references to its children.
2) Construct a dictionary of lists of nodes to act as the index. Dictionary> in generics/templates speak.
3) Iterate through each item, and extract terms for that component. Terms are only extracted for the current component, not its parents - "c:\foo\bar" would only have 'bar' as a term.
4) Find the relevant item in the index for each term (or add it, if need be), and add the current node to the list of nodes against that item.

Once you've constructed the index, searching is as follows:
1) Extract the terms for your search query in the same manner as used for path components above
2) Obtain the list from the index for each of these terms. If any of the lists are empty, return immediately with an empty set - no results are possible.
3) For each item in the returned lists, follow the chain of parents, checking which terms from the original search are present in the item and all its ancestors. If you match all terms, add it to the result set. If you reach the root of the tree without doing so, discard the node.

I can't help thinking it could be more efficient - there's got to be a way to intersect the sets returned for each term, or to make use of the property that the end results for a search can consist only of items in each term's list, or their descendents, but I can't think how to do it right now.

Performance tests: Indexing over 600,000 items, searches for 4 terms return in less than 1/100 of a second. Searches with more terms show only linear increase in time taken, of course.

Obsession

nospam@example.com (Arachnid) — Wed, 23 Aug 2006 05:58:58 +0100

For some time now, I've been working, on and off, on a P2P filesharing system, which I've dubbed 'Attercop' (named after the term used for 'spider' used in Old English and LOTR). Simply put, it's a replacement for DC++, designed mostly for LAN environments. It uses multicast, so no central hub is required, and it improves on a number of the most glaring weaknesses in DC++. I'm writing it in C#.NET.

Like most projects I embark on, I started off hugely enthusiastic, spending as much time as I could spare on it. Normally, I'd continue like that until either I get the project completed, or I burn-out on it. Quite frequently, the latter has happened - I've burned out and lost interest, with the project unfinished.

However, since I've been spending all the time I've not been at work with Hayley, I've had very little time for it lately - a lack that's hard to mourn, since I consider time with Hayley much better spent - and I'd begun to lose enthusiasm for the project. I concluded that I was (unfortunately) suffering from the same sort of mid-project disinterest I often get. However, we spent the last weekend at a Lan, during which I coded nearly the whole time, and I find my enthusiasm suddenly renewed. I'm really enthused about it again, and hugely enjoy working on it.

My conclusion, obvious as it seems: Limiting how much time you spend on a project - no matter how enthused you are by it - can help prevent burning-out on it. It just takes something really good to pull me away from something I'm this engrossed in.

Transforming ATOM 1.0 into HTML using XSLT

nospam@example.com (Arachnid) — Fri, 21 Apr 2006 04:52:17 +0100

In response to Myke's request for a way to generate an HTML page containing the newest items from a bunch of RSS feeds, I made a suggestion: Use Google Reader to aggregate the feeds for you, export them using the 'sharing' functionality, and then simply transform the resulting ATOM feed into an HTML page using some XSLT on serverside.

I thought there'd be a ready-made XSLT stylesheet for transforming ATOM out there, but I had absolutely no luck finding one. So, I hacked together a basic one for the purpose. Here it is:

CODE:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:atom="http://www.w3.org/2005/Atom">
  <xsl:template match="/">
    <html xmlns="http://www.w3.org/1999/xhtml">
      <head>
        <title><xsl:value-of select="atom:feed/atom:title"/></title>
      </head>
      <body>
        <h1><xsl:value-of select="atom:feed/atom:title"/></h1>
        <ul>

          <xsl:for-each select="atom:feed/atom:entry">
            <li>
              <h2><xsl:element name="a">
                <xsl:attribute name="href">
                  <xsl:value-of select="atom:link[@rel='alternate']/@href"/>
                </xsl:attribute>
                <xsl:value-of select="atom:title"/>
              </xsl:element></h2>
              <xsl:if test="atom:published">
                <div class="published">
                  Published: <xsl:value-of select="atom:published"/>
                </div>
              </xsl:if>
              <xsl:if test="atom:updated">
                <div class="updated">
                  Updated: <xsl:value-of select="atom:updated"/>
                </div>
              </xsl:if>
              <div class="entrydesc">
                <xsl:if test="atom:summary[@type='html']">
                  <xsl:value-of select="atom:summary" disable-output-escaping="yes" />
                </xsl:if>

                <xsl:if test="atom:summary[@type='text']">
                  <xsl:value-of select="atom:summary" disable-output-escaping="no" />
                </xsl:if>
              </div>
            </li>
          </xsl:for-each>
        </ul>
      </body>
    </html>

  </xsl:template>
</xsl:stylesheet>

I've only tested it with google's atom, but it ought to work with any valid atom feed. I also haven't attempted to format the dates into something more user-friendly - I'm sure XSLT has the facility, but I'll be damned if I can find it right now. Finally, only the title, url, dates and summary are exposed, but it's pretty trivial to add more.

Optimum phone keypad layouts, or, what to do with a boring weekend.

nospam@example.com (Arachnid) — Tue, 18 Apr 2006 19:32:08 +0100

So, I was a little bored, and had an idea. The commonly used (alphabetic) phone keypad layout is not particularly efficient when it comes to T9 (one digit per letter) style entry - there are a lot of collisions. What would an ideal keypad layout look like?

So I hunted up a computer-readable english dictionary with frequency scores (a 0-16 scale), and devised a way to score the efficiency of a layout: group words by the digits they translate into with that layout, and multiply together their frequency scores. So, for example, two colliding words with frequency 8 would score 64 points, while a collision between a word with frequency 1 and one with frequency 15 would score only 15 points. The idea is that collisions between words with greatly different frequencies are much easier to resolve - it'll almost always be the more common word - whilst collisions between similar words are generally bad. Then, sum up all the products to get the final score.

I then seeded a Genetic Algorithm with the default keypad layout, and ran generations of randomly mutating layouts (by reassigning letters at random) then scoring them, then selecting the few layouts with the lowest scores, and repeating.

The default keypad layout scores a whopping 8,853,348 points.
The most optimal layout I could find scores only 71,063 points. The layout was:

{gr',aky,cej,lw-,fimxz,hn,ps,qtu,bdov}

Various variations on the placing of the apostrophe and hyphen also had the same score.
Each group represents letters to go on one key - order doesn't matter for the purpose of this.

This was a 40th generation result of the original (generation 0) default layout. I have a complete 'geneaology', of course. This was the best run of several I did totally from scratch - the best results seem to occur when I select a reasonably large 'breeding population', not just the few very fittest ones - the geneaology for the winner has several steps where the parent scored better than the child.

I just found it interesting how incredibly inefficient the default layout is when it comes to collisions. A pity nobody will ever use anything else.

The next step, of course, would be to construct another GA based on the optimum keypad layout for this key mapping, so as to minimise finger movement between letters when typing a large corpus.