User-agent: * Crawl-delay: 60 Disallow: /awstats/ Disallow: /awstats-icon/ Disallow: /cgi-bin/ Disallow: /cms/ Disallow: /htdig/ Disallow: /webalizer/ Disallow: /Library/ Disallow: /phpmyadmin/ Disallow: /plaatjes/ Disallow: /Templates/ Disallow: /handleiding/ Disallow: /handleiding-kaal/ Disallow: /sendto_form User-agent: Slurp Crawl-delay: 20 Disallow: /*sendto_form$ Disallow: /*folder_factories$ Disallow: /awstats/ Disallow: /awstats-icon/ Disallow: /cgi-bin/ Disallow: /cms/ Disallow: /htdig/ Disallow: /webalizer/ Disallow: /Library/ Disallow: /phpmyadmin/ Disallow: /plaatjes/ Disallow: /Templates/ Disallow: /handleiding/ Disallow: /handleiding-kaal/ Disallow: /sendto_form # Add Googlebot-specific syntax extension to exclude forms # that are repeated for each piece of content in the site # the wildcard is only supported by Googlebot # http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling User-Agent: Googlebot Disallow: /*sendto_form$ Disallow: /*folder_factories$ Disallow: /awstats/ Disallow: /awstats-icon/ Disallow: /cgi-bin/ Disallow: /cms/ Disallow: /htdig/ Disallow: /webalizer/ Disallow: /Library/ Disallow: /phpmyadmin/ Disallow: /plaatjes/ Disallow: /Templates/ Disallow: /handleiding/ Disallow: /handleiding-kaal/ Disallow: /sendto_form # Block URLs that include ? except those that end with ? Allow: /*?$ Disallow: /*? User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: SkreemRBot Disallow: / User-agent: Yandex Disallow: /