Windows Vista Tips

Windows Vista Tips > Newsgroups > Windows Server > Limit web crawlers impact on IIS site

Reply
Thread Tools Display Modes

Limit web crawlers impact on IIS site

 
 
Miha
Guest
Posts: n/a

 
      05-07-2010

Hi

Is it possible and how to limit web crawlers (Google) impact to web site
running on IIS 6/7 ?
Dealing with Google's webmaster tools - site config - custom crawl rate
doesn't give us less impacts, so I'm wondering if there is any way to limit
bandwidth available to all crawlers (identified by bot in User-Agent header)
using bandwidth limiting on IIS.
I know that it is possible on Apache, but cannot find this how to set on MS
IIS.
Any help or ideas would be appriciated.
Regards,
Miha


 
Reply With Quote
 
 
 
 
.._..
Guest
Posts: n/a

 
      05-11-2010
Use robots.txt for this. Either just ban the search crawlers that don't
work for you "Baide" or whatever that chinese one is useless for most sites,
and it's aggressive.

Ther are also commands you can use in robots.txt to tell them to slow down.

http://www.kavoir.com/2010/02/how-to...your-site.html

http://searchengineland.com/a-deeper...obotstxt-17573

Google honors the robots.txt.

For the crawlers that don't, I'd just band them at the firewall based on IP
address. There's about 5 search engines that most people use and the rest
of them are garbage. Just block the garbage ones.

You may also want to pay attention to what might be causing the issue, it
could be revealing bad code or files that should be compressed better.


"Miha" <> wrote in message
news:...
>
> Hi
>
> Is it possible and how to limit web crawlers (Google) impact to web site
> running on IIS 6/7 ?
> Dealing with Google's webmaster tools - site config - custom crawl rate
> doesn't give us less impacts, so I'm wondering if there is any way to
> limit bandwidth available to all crawlers (identified by bot in User-Agent
> header) using bandwidth limiting on IIS.
> I know that it is possible on Apache, but cannot find this how to set on
> MS IIS.
> Any help or ideas would be appriciated.
> Regards,
> Miha
>
>



 
Reply With Quote
 
 
 
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
What is Impact on Web-Application when IE6 should be changed to IE masakiy Internet Explorer 0 12-12-2007 02:12 AM
Halloween creepy crawlers.... Tie Various Windows Vista General Discussion 2 11-01-2007 03:16 PM
Web console / Remote control of Media Center 2005 (IIS web site) David Windows Media Center 3 12-01-2005 07:08 PM
Impact of moving FSMO's to different site? Active Directory 2 10-29-2005 05:07 AM
Default Web Site vs Company Web Site JerryF Windows Small Business Server 4 09-24-2004 07:05 PM