[mail-web] Additional crawler exclusions: too aggressive

[Andrea Veri <averi src gnome org>]

commit dec6bd6ef5014ce9a955453a9e136d1d7b87a116
Author: Andrea Veri <averi redhat com>
Date:   Wed Nov 21 18:29:09 2018 +0100

    Additional crawler exclusions: too aggressive

 robots.txt | 9 +++++++++
 1 file changed, 9 insertions(+)
---
diff --git a/robots.txt b/robots.txt
index 1e756e2..670f772 100644
--- a/robots.txt
+++ b/robots.txt
@@ -1,5 +1,14 @@
 User-agent: BUbiNG
 Disallow: /
 
+User-agent: SemrushBot
+Disallow: /
+
+User-agent: SemrushBot-SA
+Disallow: /
+
+User-agent: Exabot
+Disallow: /
+
 User-agent: *
 Disallow: /pipermail/