Add more entries to robots.txt (#2044)

* Add more entries to `robots.txt` * remove some entries as discussed --------- Co-authored-by: Masterjun3 <[email protected]>
TASVideos · Nov 26, 2024 · ee2a5a8 · ee2a5a8
1 parent 92796ec
commit ee2a5a8
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/TASVideos/Middleware/RobotHandlingMiddleware.cs b/TASVideos/Middleware/RobotHandlingMiddleware.cs
@@ -17,8 +17,12 @@ public async Task Invoke(HttpContext context)
 						Disallow: /MovieMaintenanceLog
 						Disallow: /UserMaintenanceLog
 						Disallow: /InternalSystem/
+						Disallow: /Search
 						Disallow: /*?revision=*
 						Disallow: /Wiki/PageHistory
+						Disallow: /Wiki/PageNotFound
+						Disallow: /Wiki/Referrers
+						Disallow: /Wiki/ViewSource
 
 						User-agent: Fasterfox
 						Disallow: /