Disable all bots and remove sitemap (#402)

* Disable all bots and remove sitemap

* allow bots to access only the landing page

public/robots.txt

@@ -4,14 +4,7 @@
 
 # See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
 
-# Disallow every other robot
+# Disallow every robot except for the landing page
 User-agent: *
+Allow: /$
 Disallow: /
-
-# Google can only index individual works pages
-# Allow syntax is only really listened to by google and a few others
-User-agent: Googlebot
-Allow: /doi.org/
-
-# Sitemap specific location
-Sitemap: https://commons.datacite.org/sitemaps/sitemap.xml.gz