add robots.txt

disallow crawlers on exports (since some of them can be slow to generate)
sfu-dhil · Aug 26, 2024 · 2091183 · 2091183
1 parent af22369
commit 2091183
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/public/robots.txt b/public/robots.txt
@@ -0,0 +1,3 @@
+User-agent: *
+Disallow: */export
+Disallow: */search/export/*