User-agent: * Disallow: /WebService.asmx/ Disallow: /version.html User-agent: MJ12bot Disallow: / # BLEXBot blocked due intensive crawling of non-existing URLs in Jan '17 # http://webmeup-crawler.com/ User-agent: BLEXBot Disallow: / # SemrushBot blocked due too intensive crawling in May '20 User-agent: SemrushBot Disallow: / # Blocked due to crawling once per minute of this non-exsiting url in April 2023 # https://www.studiobookr.com/schoenmacherei-aesthetik-institut-58479/Willkommen/Produkte/Beauty-Blog/index.php/;session=9433dd7f4adb1a2671ce4f6096dd89a2&focus=STRATP_com_cm4all_wdn_Flatpress_10917523&path=?x=y:19%3Bm:07 User-agent: PetalBot Disallow: / # 2024-07-11, RA+CK User-agent: DataForSeoBot Disallow: / # Temporarily blocked due too intensive crawling in Nov '24 User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot Disallow: /searchresults/*