I'm not sure they quite got the idea of the purpose of robots.txt. Might it not be easier to block the whole site, or at least vast swathes of it rather than umpteen individual files?

Don't ask how I found this out ;-)
<--break-->