robots.txt policy

[agnlez]

From #116

Based on the article Álvaro shared, shall we change our robots.txt policy? Right now, any production application will deploy with robots disabled which may impact SEO results if the file is not changed consciously.

I think like the article: staging/preview sites are development exceptions and should be treated as if. There are better ways to stop robots from crawling and avoid modifying robots.txt per environment.

Barrenechea added this link to the conversation: https://github.com/vercel/examples/tree/main/edge-functions/basic-auth-password