I'm not sure they quite got the idea of the purpose of robots.txt. Might it not be easier to block the whole site, or at least vast swathes of it rather than umpteen individual files?
Don't ask how I found this out ;-)
<--break-->
I'm not sure they quite got the idea of the purpose of robots.txt. Might it not be easier to block the whole site, or at least vast swathes of it rather than umpteen individual files?
Don't ask how I found this out ;-)
<--break-->
Comments
IBM not having a clue about what they're doing? Surely not!