SEO: Configure robots.txt file.

[jimtyhurst]

Expected Behavior

We want the company website to score as high as possible in the Lighthouse SEO scoring.

Actual Behavior

One of the Gatsby Cloud build reports from Lighthouse dinged our code a few points on SEO, saying:

Search engines are unable to include your pages in search results if they don't have permission to crawl them.
...
Blocking Directive Source
x-robots-tag: none

Analysis

• I looked at the meta tags in the build. I do not see that x-robots-tag anywhere. Maybe it is none by default? Maybe we need to explicitly set that tag.
• See gatsby-plugin-robots-txt as a way to create robots.txt from metadata in gatsby-config.js

[craigfisk]

If I understand correctly (from the first link below), Lighthouse runs on the results of indexing, and you wouldn't want the Gatsby cloud version to be indexed, so what you are seeing is ok, and Lighthouse will work with the https://marmalade.ai version only. Is that the way you see it, too? 1) https://answers.netlify.com/t/disabling-x-robots-tag-header-on-hosted-site/8857 2) https://www.google.com/search?q=%22x-robots-tag%22+gatsby+lighthouse&oq=%22x-robots-tag%22+gatsby+lighthouse&aqs=chrome..69i57.34455j0j7&client=ms-android-google&sourceid=chrome-mobile&ie=UTF-8#scso=_jdJFYeq7C8G2tAa3y7boBQ14:0 https://www.linkedin.com/in/craigfisk <https://about.me/craigfisk> (650) 336-8770

…

On Fri, Sep 17, 2021 at 8:23 PM Jim Tyhurst ***@***.***> wrote: Expected Behavior We want the company website to score as high as possible in the Lighthouse SEO scoring. Actual Behavior One of the Gatsby Cloud build reports <https://build-1af2efe3-37c6-4b51-b75f-495fbac4d79f.gtsb.io/reports/lighthouse/index.html> from Lighthouse dinged our code a few points on SEO, saying: Search engines are unable to include your pages in search results if they don't have permission to crawl them. ... Blocking Directive Source x-robots-tag: none Analysis - I looked at the meta tags in the build. I do not see that x-robots-tag anywhere. Maybe it is none by default? Maybe we need to explicitly set that tag. - See gatsby-plugin-robots-txt <https://www.gatsbyjs.com/plugins/gatsby-plugin-robots-txt/> as a way to create robots.txt from metadata in gatsby-config.js <http://./gatsby-config.js> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#130>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAZH6WWBH6TMLZOF67ZEYDUCPSXVANCNFSM5EITZ2PQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jimtyhurst]

Yes, I was assuming that whatever was in the branch build would be deployed, but it makes sense that it would be different in this case. I will leave this issue open until we deploy the new company website to production, so we can verify that the new site is indexed correctly.

See also: gatsby-plugin-next-seo.