Limit web crawlers impact on IIS site

Discussion in 'Windows Server' started by Miha, May 7, 2010.

  1. Miha

    Miha Guest

    Hi

    Is it possible and how to limit web crawlers (Google) impact to web site
    running on IIS 6/7 ?
    Dealing with Google's webmaster tools - site config - custom crawl rate
    doesn't give us less impacts, so I'm wondering if there is any way to limit
    bandwidth available to all crawlers (identified by bot in User-Agent header)
    using bandwidth limiting on IIS.
    I know that it is possible on Apache, but cannot find this how to set on MS
    IIS.
    Any help or ideas would be appriciated.
    Regards,
    Miha
     
    Miha, May 7, 2010
    #1
    1. Advertising

  2. Miha

    .._.. Guest

    Use robots.txt for this. Either just ban the search crawlers that don't
    work for you "Baide" or whatever that chinese one is useless for most sites,
    and it's aggressive.

    Ther are also commands you can use in robots.txt to tell them to slow down.

    http://www.kavoir.com/2010/02/how-t...t-search-engine-crawler-visits-your-site.html

    http://searchengineland.com/a-deeper-look-at-robotstxt-17573

    Google honors the robots.txt.

    For the crawlers that don't, I'd just band them at the firewall based on IP
    address. There's about 5 search engines that most people use and the rest
    of them are garbage. Just block the garbage ones.

    You may also want to pay attention to what might be causing the issue, it
    could be revealing bad code or files that should be compressed better.


    "Miha" <> wrote in message
    news:...
    >
    > Hi
    >
    > Is it possible and how to limit web crawlers (Google) impact to web site
    > running on IIS 6/7 ?
    > Dealing with Google's webmaster tools - site config - custom crawl rate
    > doesn't give us less impacts, so I'm wondering if there is any way to
    > limit bandwidth available to all crawlers (identified by bot in User-Agent
    > header) using bandwidth limiting on IIS.
    > I know that it is possible on Apache, but cannot find this how to set on
    > MS IIS.
    > Any help or ideas would be appriciated.
    > Regards,
    > Miha
    >
    >
     
    .._.., May 11, 2010
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tie Various

    Halloween creepy crawlers....

    Tie Various, Nov 1, 2007, in forum: Windows Vista General Discussion
    Replies:
    2
    Views:
    159
  2. masakiy
    Replies:
    0
    Views:
    145
    masakiy
    Dec 12, 2007
  3. David
    Replies:
    3
    Views:
    237
    Doug Knox MS-MVP
    Dec 1, 2005
  4. JerryF

    Default Web Site vs Company Web Site

    JerryF, Sep 23, 2004, in forum: Windows Small Business Server
    Replies:
    4
    Views:
    423
    Michelle Dixon [MSFT]
    Sep 24, 2004
  5. Guest

    Impact of moving FSMO's to different site?

    Guest, Oct 27, 2005, in forum: Active Directory
    Replies:
    2
    Views:
    225
    Guest
    Oct 29, 2005
Loading...

Share This Page