User-agent: * Disallow: /WebService.asmx/ Disallow: /version.html User-agent: MJ12bot Disallow: / # BLEXBot blocked due intensive crawling of non-existing URLs in Jan '17 # http://webmeup-crawler.com/ User-agent: BLEXBot Disallow: / # SemrushBot blocked due too intensive crawling in May '20 User-agent: SemrushBot Disallow: /