Harry

Almost a Blog

Category: Spiders

Miragorobot

I have seen plenty of robots spider my site but by far the most polite as far as time goes has got to be the miragorobot. It waited a full minute between each page request which is almost unheard of.

500,000 Pages

The spiders have now collected over 500,000 pages. Considering I have been taking it easy it was surprisingly fast. I now need another 500,000 to get my goal of 1 Million for testing.