Here, you can find the robots.txt parser and matcher open sourced by google. It is fun to see the code that read your robots.txt file and acted on it.