User-agent: * Crawl-delay: 60 Disallow: /test/ Disallow: /search/results/ User-agent: GnowitNewsbot Disallow: / User-agent: Bingbot Crawl-delay: 3 Sitemap: http://www.noozhawk.com/sitemap.xml # Spam crawlers blocked in htaccess as well #Baiduspider and Yandex User-agent: Baiduspider User-agent: TweetmemeBot User-agent: Yandex User-agent: Yandexbot User-agent: YandexImages User-agent: SemrushBot User-agent: MJ12Bot Disallow: /