[BUG] Investigate 503

[dcastro]

Description

In some of our gitlab projects, we have links such as https://gitlab.com/morley-framework/morley-infra/-/issues/new.

For some reason, when xrefcheck attempts to verify these links, it gets a 503 Service Temporarily Unavailable, which causes xrefcheck to fail. As a result, our gitlab projects ignore those links with a xrefcheck: ignore link annotation.

However, if you manually verify it using curl or using a browser, you get a 302 redirection (which would make xrefcheck pass).

• Let's investigate the reason for this discrepancy.
• If there's anything that can be done in xrefcheck to fix this, let's do it.
• If we manage to fix it, let's remove the xrefcheck: ignore annotation from those gitlab projects. These are the ones I'm aware of:
  • https://gitlab.com/morley-framework/morley-debugger
  • https://gitlab.com/morley-framework/morley-infra
  • https://gitlab.com/morley-framework/indigo/-/issues

To Reproduce

Steps to reproduce the behavior:

$ echo '[a](https://gitlab.com/morley-framework/morley-infra/-/issues/new)' > a.md
$ git add a.md
$ xrefcheck

Configuration file not found, using default config for GitHub repositories

=== Invalid references found ===                                                                                                                    

  ➥  In file a.md
     bad reference (external) at src:7:1-66:
       - text: "a"
       - link: https://gitlab.com/morley-framework/morley-infra/-/issues/new
       - anchor: -
     
     ⛂  Resource unavailable (503 Service Temporarily Unavailable)
     
Invalid references dumped, 1 in total.

However, if we check the same link via curl, it replies with a 302 redirect.

$ curl -I https://gitlab.com/morley-framework/morley-infra/-/issues/new
HTTP/2 302 
...
location: https://gitlab.com/users/sign_in
...

Similarly, opening the link in the browser (while not authenticated into gitlab.com) gets us redirected to https://gitlab.com/users/sign_in.

Expected behavior

Expected the link to reply with a 302 and for xrefcheck to pass.

Environment

• xrefcheck v0.2.2

[Martoon-00]

AFAIR if I visit that page from the browser without being authenticated into any account, I will get 503 too. So this is an issue around authentication.

[dcastro]

Just double-checked to be sure, it replies with a 302:

I suspect the discrepancy between xrefcheck vs curl/browser might be due to some header.

The most obvious suspect would be the User-Agent header, so I tried using curl without this header to see if I would get a 503, but nope, didn't work :/

$ curl -I https://gitlab.com/morley-framework/morley-infra/-/issues/new -H 'User-Agent:' -v                                                                                                               
...
HTTP/2 302 

[aeqz]

We can reproduce the xrefcheck behaviour by telling curl to follow redirects (-L):

$ curl -IL https://gitlab.com/morley-framework/morley-infra/-/issues/new                                                                                                              
...
HTTP/2 503

This is because the default HTTP configuration that is being used in the project follows automatically up to 10 redirect responses. I have tried to set the corresponding value to 0 and then xrefcheck shows a 302 response code.

Now, even with this 302 response, xrefcheck keeps reporting it as an error. Perhaps we should also change that to not consider a 302 response as an error?

[Martoon-00]

Oh, then that's exactly #25.

We have discussed this for a while, but didn't manage to make a final decision yet. Could you read that discussion and leave your thoughts, and I will reiterate over it and drop some thoughts too.