From 2091183668360ecc8bcaa6de4b98d48adab705c4 Mon Sep 17 00:00:00 2001
From: Andrew Gardener <andrew.e.gardener@gmail.com>
Date: Mon, 26 Aug 2024 15:07:02 -0700
Subject: [PATCH] add robots.txt

disallow crawlers on exports (since some of them can be slow to generate)
---
 public/robots.txt | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 public/robots.txt

diff --git a/public/robots.txt b/public/robots.txt
new file mode 100644
index 00000000..6c9465d4
--- /dev/null
+++ b/public/robots.txt
@@ -0,0 +1,3 @@
+User-agent: *
+Disallow: */export
+Disallow: */search/export/*
\ No newline at end of file