Yahoo Slurp should stop slurping the fucking bandwidth. (3)

1 Name: 2005-06-14 11:24 ID:Heaven

"Slurp" is the name of Yahoo's webspider.
"Slurp" is also what it does to websites bandwidth, sometimes nailing sites 12 times a day for no apparent reason.
For instance, here at 4-ch its completely banned. Why? Last month it made more requests than Opera, or by Googlebot & MSNbot combined.

So that fixes the problem? No. At the time of posting, Slurp has made over 1400 pointless requests to robots.txt, the only file its now allowed to touch. That's not huge (way less than 2Mb), but 1400 requests? No wonder my requests figures have been a bit high.

http://www.robotstxt.org/
http://www.ysearchblog.com/archives/000078.html

2 Name: !WAHa.06x36 2005-06-14 13:46 ID:A8Ut9TlP

Bandwidth saver:

Make a link to a script on every page. Make the link display:none. Forbid access to the script in robots.txt. Make the script ban everybody who touches it.

There are a LOT of badly behaved spiders out there, running on zombie machines. This will ban them. It will also ban badly-behaved website downloaders, which you may or may not want to do.

3 Name: 404 - Name Not Found 2005-06-15 17:44 ID:PibcYD8j

Bandwidth-saver:
http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html

Return Entire thread Last 50 posts
This thread has been closed. You cannot post in this thread any longer.