"It's less like flypaper and more an infinite maze holding a minotaur, except the crawler is the minotaur that cannot get out. The typical web crawler doesn't appear to have a lot of logic.

It forms the web equivalent of an infinite “maze of twisty little passages, all alike”. While the project has been a success, confirmed by the amount of time various web crawlers have spent ...