Linkcheck failing because of user agent used by Documenter

[giordano]

As discovered in JuliaParallel/MPI.jl#856 (comment), MPI.jl has a linkcheck failing because of the useragent hardcoded in Documenter

Documenter.jl/src/docchecks.jl

Lines 216 to 217 in c2ed0b7

	"--user-agent",
	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",

$ curl -I 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html'
HTTP/2 200
content-type: text/html;charset=utf-8
server: Apache
x-dispatcher: dispatcher2uswest1-b80
x-vhost: publish
last-modified: Wed, 07 Aug 2024 13:09:27 GMT
etag: W/"30f33-61f179eb85c9b-gzip"
expires: Wed, 07 Aug 2024 21:40:56 GMT
cache-control: max-age=0, no-cache
pragma: no-cache
date: Wed, 07 Aug 2024 21:40:56 GMT
set-cookie: detected_bandwidth=LOW; path=/; domain=.intel.com; secure; HttpOnly
set-cookie: src_countrycode=gb; path=/; domain=.intel.com; secure; HttpOnly
alt-svc: h3=":443"; ma=93600
content-security-policy: frame-ancestors 'self' https://premiersupport.intel.com https://c0.avaamo.com *.intel.com; object-src 'self';
x-grn: 0.8d3e1202.1723066856.1047ccba
x-frame-options: SAMEORIGIN
access-control-allow-origin: *
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000 ; preload
x-content-type-options: nosniff
$ curl --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" -sI 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html'
HTTP/2 403
server: AkamaiGHost
mime-version: 1.0
content-type: text/html
content-length: 460
expires: Wed, 07 Aug 2024 21:46:56 GMT
cache-control: max-age=0, no-cache
pragma: no-cache
date: Wed, 07 Aug 2024 21:46:56 GMT
set-cookie: detected_bandwidth=LOW; path=/; domain=.intel.com; secure; HttpOnly
set-cookie: src_countrycode=gb; path=/; domain=.intel.com; secure; HttpOnly
alt-svc: h3=":443"; ma=93600
content-security-policy: frame-ancestors 'self' https://premiersupport.intel.com https://c0.avaamo.com *.intel.com; object-src 'self';
x-grn: 0.8d3e1202.1723067216.1050300f
x-frame-options: SAMEORIGIN
access-control-allow-origin: *
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000 ; preload
x-content-type-options: nosniff

I'm not sure of what would be a good resolution of this issue, a somewhat customisable user agent?

[giordano]

Bump. Would it be possible to have a way to set a custom user agent, and don't pass --user-agent at all if empty?

% curl -sI --proto '=http,https,ftp,ftps' -H 'accept-encoding: gzip, deflate, br' 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html' --max-time 10 -o /dev/null --write-out '%{http_code} %{url_effective} %{redirect_url}'
200 https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html 

That'd work for my use case.

[giordano]

I'm a bit confused: with Documenter.jl v1.7.0 (which includes #2569) I'm still seeing the check for this link fail, but locally I get

% curl -sI --proto '=http,https,ftp,ftps' -H 'accept-encoding: gzip, deflate, br' --user-agent 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36' 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html' --max-time 10 -o /dev/null --write-out '%{http_code} %{url_effective} %{redirect_url}'
200 https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html 

Any clue of what could be the issue now?

[giordano]

Ah, setting linkcheck_useragent = nothing, like in #2571, seems to have done the trick. I'm a bit confused by why this is necessary, since it works locally with the set user agent (do they also check IP address or something?!?), but at least I got it working. Thanks!