muffet generates 403 on pixabay.com

[c33s]

my site links to https://pixabay.com/ and if i check it with muffet it leads to a 403. tried to set a custom header but i still get a 403

--header="User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"

i don't want to scrape pixabay but i would like to check all external links on my site. currently my workaround is to exclude the site.

same problem for pexels.com.

looks like it is not really a problem of muffet as wget also produces a 403 but maybe muffet can do something about it

[ruzickap]

Seems like it is by default.
If you use curl - you got it as well:

curl -v https://pixabay.com/
...
> GET / HTTP/2
> Host: pixabay.com
> user-agent: curl/7.54.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403
< date: Sat, 26 Feb 2022 07:12:19 GMT
...

Same results with

curl -v --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0' https://pixabay.com/

I'm using: --exclude=(linkedin.com|pixabay.com|html5up.net)

[maxmeyer]

curl https://www.cyberciti.biz/

has the same problem. Both sites - cyberciti and pixabay - secure their sites with cloudflare.

If you check the output, Cloudflare tries to check “browser”.

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>

            <p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
          </div>

This post about curl and Cloudflare might shed some more light on the issue: https://community.cloudflare.com/t/curl-command-getting-403-in-my-subdomain/299578/10.