Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ERROR] No RateLimit present in response #228

Closed
thoys opened this issue Mar 20, 2020 · 49 comments
Closed

[ERROR] No RateLimit present in response #228

thoys opened this issue Mar 20, 2020 · 49 comments

Comments

@thoys
Copy link

thoys commented Mar 20, 2020

I get the following error when trying to build my docker file which worked fine a couple hours ago.

Step 51/64 : RUN phive --no-progress install --global --trust-gpg-keys 31C7E470E2138192 squizlabs/php_codesniffer
 ---> Running in d4b64f83fa46
Phive 0.13.2 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors

[ERROR]   No RateLimit present in response

In a previous run it failed one package earlier:

Step 50/64 : RUN phive --no-progress install --global --trust-gpg-keys E82B2FB314E9906E FriendsOfPHP/PHP-CS-Fixer
 ---> Running in daaa9daa3027
Phive 0.13.2 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]   No RateLimit present in response

The 6 packages I downloaded before this worked fine. What could be going on? Am I downloading too fast?

@theseer
Copy link
Member

theseer commented Mar 20, 2020

No, probably not "too fast" but github's http response headers should contain a rate limit information that we parse to avoid just that. But for whatever reason, that header seems to be missing for your requests.

I cannot reproduce this though:

theseer@nyda /tmp/xx $ phive install FriendsOfPHP/PHP-CS-Fixer
Phive 0.13.2-12-gbf2bb2e-dirty - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/repos/friendsofphp/php-cs-fixer/releases
Downloading https://github.com/FriendsOfPHP/PHP-CS-Fixer/releases/download/v2.16.1/php-cs-fixer.phar
Downloading https://github.com/FriendsOfPHP/PHP-CS-Fixer/releases/download/v2.16.1/php-cs-fixer.phar.asc
Linking /home/theseer/.phive/phars/friendsofphp/php-cs-fixer-2.16.1.phar to /tmp/xx/tools/php-cs-fixer

Are you having any type of (transparent?) proxy in between which might filter our these headers?

I guess we could change our download code to not barf on missing headers and just assume everything is fine, but it feels weird...

@theseer
Copy link
Member

theseer commented Mar 20, 2020

May also have been a temporary issue on github's end ...

@thoys
Copy link
Author

thoys commented Mar 21, 2020

I probably had exceeded the request limit or something before and got myself blocked. It worked fine again today. Do you have a check for that?

@thoys
Copy link
Author

thoys commented Mar 21, 2020

Next time it happens I could try an make a wireshark capture if thats helpful.

@thoys
Copy link
Author

thoys commented Mar 21, 2020

I read For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests. on the page https://developer.github.com/v3/#rate-limiting . If I ran the docker build (which contains about 8 phive calls) multiple times in one hour it might have exceeded the limit?

@theseer
Copy link
Member

theseer commented Mar 21, 2020

You certainly could exceed your limits. I ran into that while developing phive a lot before we implemented authentication token support.

But: Github usually supplies a set of X-RateLimit-XXXX headers, regardless whether the request was successful or (already) blocked. That's why it is confusing to me that - at least according to our error message - those headers seem to be missing in the response.

If you run into this problem again, you could try to do a curl/wget request to the URL shown before it happend in verbose mode so you get the HTTP response headers along. (If no URL is shown in the output, please make sure you get STDOUT as well as STDERR output ;)

Phive supports using access tokens - simply set the GITHUB_AUTH_TOKEN environment variable. We do have pending tickets to document and enhance its usage, though (see #178 and phar-io/phar.io#25).

@thoys
Copy link
Author

thoys commented Mar 21, 2020

Great thanks, for now I don't think this ticket needs action, if someone finds it through a search engine when having the same issue it could be re-opened. I will set a GITHUB_AUTH_TOKEN and don't expect it to happen again in the near future.

@thoys thoys closed this as completed Mar 21, 2020
@Surf-N-Code
Copy link

I just ran into this issue as well.

export GITHUB_AUTH_TOKEN=gittoken fixed it.
Thanks for the tip!

@theseer
Copy link
Member

theseer commented Mar 28, 2020

Looks like the code is having issues with either not seeing or not receiving the RateLimit headers from github. I'll reopen this and have a look.

@theseer theseer reopened this Mar 28, 2020
@mjnbock
Copy link

mjnbock commented Mar 30, 2020

I am having the problem now.
wget -S 'https://github.com/phar-io/phive/releases/download/0.13.2/phive-0.13.2.phar shows

--2020-03-30 14:46:14--  https://github.com/phar-io/phive/releases/download/0.13.2/phive-0.13.2.phar
Resolving github.com (github.com)... 140.82.118.3
Connecting to github.com (github.com)|140.82.118.3|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  date: Mon, 30 Mar 2020 14:45:05 GMT
  content-type: text/html; charset=utf-8
  server: GitHub.com
  status: 302 Found
  vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
  location: https://github-production-release-asset-2e65be.s3.amazonaws.com/31674534/04998680-faac-11e9-87d1-3a90ceae0228?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T144505Z&X-Amz-Expires=300&X-Amz-Signature=d695b84022e0458a46bf048568c3b445e29c80d85be530c60be3fc05240729cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dphive-0.13.2.phar&response-content-type=application%2Foctet-stream
  cache-control: no-cache
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  x-frame-options: deny
  x-content-type-options: nosniff
  x-xss-protection: 1; mode=block
  expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
  Set-Cookie: _gh_sess=4bQzi%2FKOHKW88fgcQlTSn5UYiwD7htihXvbp1gJN1xjkGATQLMAYFQCfaGhwjkkQyda%2BxifEY%2FOMioar4u1WXGwuRhg877dQ8qVJjfzEe3RaLYH2et8QAQhI4d9Kfc%2Bkm2wCrvhgwyO18Uuzzuaa53mPCRuVkxEE4OVn1PeqY9FP2rRhgwD2qKs0fzpwVIbxZ%2BafS5VFLJ76cU9t1N4vo%2BqcJCeT989RUsEgODvqkQnuvkYdTR6c8PVG%2Fvzr5b%2FTyMtT%2FAHHIGHRXjW14pyVZQ%3D%3D--O0lwOhxdwgWIxDDS--dgTNJg7j7F49X8TBYtQDFQ%3D%3D; Path=/; HttpOnly; Secure
  Set-Cookie: _octo=GH1.1.576589217.1585579575; Path=/; Domain=github.com; Expires=Tue, 30 Mar 2021 14:46:15 GMT; Secure
  Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Tue, 30 Mar 2021 14:46:15 GMT; HttpOnly; Secure
  Content-Length: 607
  X-GitHub-Request-Id: BA81:EEC4:3FF0BE8:5D4A8F7:5E820636
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/31674534/04998680-faac-11e9-87d1-3a90ceae0228?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T144505Z&X-Amz-Expires=300&X-Amz-Signature=d695b84022e0458a46bf048568c3b445e29c80d85be530c60be3fc05240729cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dphive-0.13.2.phar&response-content-type=application%2Foctet-stream [following]
--2020-03-30 14:46:15--  https://github-production-release-asset-2e65be.s3.amazonaws.com/31674534/04998680-faac-11e9-87d1-3a90ceae0228?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T144505Z&X-Amz-Expires=300&X-Amz-Signature=d695b84022e0458a46bf048568c3b445e29c80d85be530c60be3fc05240729cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dphive-0.13.2.phar&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.143.140
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.143.140|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  x-amz-id-2: K4ZSELOa7GSDEhp6/vpPwBw5IyQi6/GqYH3lkyh+gz1a/G57lxwidIYvd1TNjTrT7VX6CQz1Aks=
  x-amz-request-id: 23A7CB96BB664D4F
  Date: Mon, 30 Mar 2020 14:46:16 GMT
  Last-Modified: Tue, 29 Oct 2019 23:27:15 GMT
  ETag: "1e44235059d526839dd1fbece1e1d5b8"
  Content-Disposition: attachment; filename=phive-0.13.2.phar
  Accept-Ranges: bytes
  Content-Type: application/octet-stream
  Content-Length: 163055
  Server: AmazonS3
Length: 163055 (159K) [application/octet-stream]
Saving to: 'phive-0.13.2.phar.1'

     0K .......... .......... .......... .......... .......... 31%  263K 0s
    50K .......... .......... .......... .......... .......... 62%  526K 0s
   100K .......... .......... .......... .......... .......... 94% 16.0M 0s
   150K .........                                             100% 17.8M=0.3s

2020-03-30 14:46:15 (551 KB/s) - 'phive-0.13.2.phar.1' saved [163055/163055]

@dnaber-de
Copy link
Contributor

I face the same issue when running phive on Codeship.com CI. On a local docker container it works though.

@theseer
Copy link
Member

theseer commented Apr 8, 2020

Looks like Github changed the download of assets to be a redirect to s3 (of all places, why not azure? ;-p ) rather than having it count as an API hit.

While that's a good thing, phive seems to be confused about it. Not sure why it would work in some cases then and not fail all the time.

This needs a bit more investigation but i'll probably relax the code to simply be happy if no RateLimit header is there.

@theseer
Copy link
Member

theseer commented Apr 8, 2020

The code already is relatively flexible in that regard.

But: As it seems, Github no longer allows HEAD requests against their rate limit check api endpoint.

Using HEAD for other endpoints still counts against the API limit, so we'll use GET now and ignore the response data.

@theseer
Copy link
Member

theseer commented Apr 8, 2020

It would be awesome if someone could try this before I make a new release to see if that actually fixes this issue... :)

@mjnbock
Copy link

mjnbock commented Apr 8, 2020

On Debian the problem persists:

$ git clone https://github.com/phar-io/phive
$ cd phive && composer install
$ php phive selfupdate
Phive 0.13.2-14-g37a22dc - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/rate_limit
[ERROR]    No RateLimit present in response 

@theseer
Copy link
Member

theseer commented Apr 8, 2020

sigh

I wish I could reproduce this ;)

theseer@nyda /tmp/xxx $ git clone https://github.com/phar-io/phive
Cloning into 'phive'...
remote: Enumerating objects: 85, done.
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 9045 (delta 32), reused 61 (delta 28), pack-reused 8960
Receiving objects: 100% (9045/9045), 2.26 MiB | 1.77 MiB/s, done.
Resolving deltas: 100% (6329/6329), done.
theseer@nyda /tmp/xxx $ cd phive && composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 6 installs, 0 updates, 0 removals
  - Installing phar-io/filesystem (dev-master 4fc0abe): Cloning 4fc0abed6a from cache
  - Installing phar-io/executor (dev-master 73ee06f): Cloning 73ee06f202 from cache
  - Installing phar-io/gnupg (dev-master 788b591): Cloning 788b591d04 from cache
  - Installing phar-io/version (2.0.1): Loading from cache
  - Installing phar-io/manifest (1.0.3): Loading from cache
  - Installing mikey179/vfsstream (v1.6.7): Loading from cache
Generating autoload files
theseer@nyda /tmp/xxx/phive master $ php phive selfupdate
Phive 0.13.2-14-g37a22dc - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/rate_limit
Downloading https://api.github.com/repos/phar-io/phive/releases
[ERROR]    An error occurred while processing your request:

          Version string '0.13.2-14-g37a22dc' does not follow SemVer semantics

The rate limit check works just fine?
Anyone got any idea how to force the error this issue is about?

@theseer
Copy link
Member

theseer commented Apr 8, 2020

Looks like I managed:

[theseer@chase phive]$ php phive selfupdate
Phive 0.13.2-14-g37a22dc - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/rate_limit
[ERROR]   No RateLimit present in response

I have no idea what's different on that box though ;)

@mjnbock
Copy link

mjnbock commented Apr 8, 2020

GithubAliasResolver.php:

private function initRateLimit(): void {
    if ($this->rateLimit !== null) {
        return;
    }

    $response        = $this->httpClient->get(new Url('https://api.github.com/rate_limit'));
    $this->rateLimit = $response->getRateLimit(); // XXX throws
}

HttpResponse.php:

public function __construct(int $httpCode, string $responseBody, ETag $etag = null, RateLimit $rateLimit = null) {
    $this->responseBody = $responseBody;
    $this->httpCode     = $httpCode;
    $this->etag         = $etag;
    $this->rateLimit    = $rateLimit; // defaults to null
}
// ...
public function hasRateLimit(): bool {
    return $this->rateLimit !== null;
}
// ...
public function getRateLimit(): RateLimit {
    if (!$this->hasRateLimit()) {
        throw new HttpResponseException('No RateLimit present in response');
    }

    return $this->rateLimit;
}

CurlHttpClient.php

private function execRequest(): HttpResponse {
    $this->rateLimitHeaders = [];

    $result = $this->curl->exec();

    $httpCode = $this->curl->getHttpCode();

    if ($httpCode >= 400 || \in_array($httpCode, [200, 304], true)) {
        return new HttpResponse($httpCode, $result ?: '', $this->etag, $this->parseRateLimitHeaders());
    }
// ...
private function parseRateLimitHeaders(): ?RateLimit {
    $required  = ['Limit', 'Remaining', 'Reset'];
    $exisiting = \array_keys($this->rateLimitHeaders); // empty []

    if (\count(\array_intersect($required, $exisiting)) < 3) {
        return null; // XXX so HttpResponse hasRateLimit() returns false
    }

@theseer
Copy link
Member

theseer commented Apr 8, 2020

Found it!

Technically, it's a bug on Github's side as their response headers are inconsistent. Some servers' response header use X-RateLimit-xxxxx (as is "correct") while others respond with X-Ratelimit-xxxx (which, according to their own documentation, is incorrect).

Phive only looks for the variant with capital L. I'll change that to be case independent...

@thoys
Copy link
Author

thoys commented Apr 8, 2020 via email

@theseer
Copy link
Member

theseer commented Apr 8, 2020

@mjnbock I know where the exception came from but I wanted a scenario to reproduce the issue. But thanks :)

@theseer
Copy link
Member

theseer commented Apr 8, 2020

Okay, fix implemented and tested. Will be in the next release.

@theseer theseer closed this as completed Apr 8, 2020
@ravage84
Copy link

Okay, fix implemented and tested. Will be in the next release.

@theseer I just ran into this on a Windows 7 box. Is there an ETA for the next release?

This was referenced Apr 24, 2020
@theseer
Copy link
Member

theseer commented Apr 26, 2020

@ravage84 Working on it, with a bit of luck (read: my daughter sleeping a bit earlier today ;) ) I'll release 0.14.0 today.

@dereuromark
Copy link
Contributor

dereuromark commented Apr 27, 2020

I also just ran into it today for "selfupdate". So the release is very welcome indeed :)

@ostrolucky
Copy link

To encourage fixing this, I am posting obligatory "me too"

@michaelKaefer
Copy link

Me too on phive selfupdate.

@amenk
Copy link

amenk commented May 27, 2020

There is no backport for PHP7.1, right? We still build docker images for PHP7.1 and currently phive does not seem to work here anymore :(

@theseer
Copy link
Member

theseer commented May 27, 2020

I created a 0.13.3 Release for you.

This will mark the definite end for PHP 7.1 compatible releases though.

@amenk
Copy link

amenk commented May 27, 2020

wow, awesome, thanks :)

@ramsey
Copy link

ramsey commented Jul 20, 2020

I ran into this problem just now, and adding a GITHUB_AUTH_TOKEN environment variable fixed it for me.

@theseer
Copy link
Member

theseer commented Jul 20, 2020

@ramsey What version where you having this issue with? While you of course might have run into a Rate Limit issue in general, this particular bug should no longer exist.

@ramsey
Copy link

ramsey commented Jul 20, 2020

@theseer I was contributing to CaptainHook when I ran into this issue, so it's possible they're using an older version of Phive. Their CONTRIBUTING.md instructions show this command, which is where I saw the error:

tools/phive --home ./.phive install --trust-gpg-keys 4AA394086372C20A,31C7E470E2138192,8E730BA25823D8B5 --force-accept-unsigned

@theseer
Copy link
Member

theseer commented Jul 20, 2020

Thanks!

The command call looks sane, but indeed they are on 0.13.2. The problem is fixed in 0.13.3 and up.

/ping @sebastianfeldmann

@sebastianfeldmann
Copy link
Contributor

Thanks for the ping, I updated phive to the latest version

@MacDada
Copy link

MacDada commented Sep 15, 2020

I'm getting the error on version 0.13.3:

$ bin/phive.phar install
Phive 0.13.3 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]    No RateLimit present in response 

$ bin/phive.phar version
Phive 0.13.3 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors

$ cat phive.xml 
<?xml version="1.0" encoding="UTF-8"?>
<phive xmlns="https://phar.io/phive">
  <phar name="wapmorgan/phpcodefixer" version="^2.0" installed="2.0.15" location="./tools/phpcf" copy="false"/>
</phive>

@theseer
Copy link
Member

theseer commented Sep 17, 2020

I don't see how this would be possible. And, worse, I cannot reproduce it. Is that the only output you get?

@theseer
Copy link
Member

theseer commented Sep 17, 2020

Okay, managed. Interesting.

@theseer
Copy link
Member

theseer commented Sep 17, 2020

Looks like I didn't backport all of the Fix in 0.13.3. Try 0.13.4.

@MacDada
Copy link

MacDada commented Sep 21, 2020

$ bin/phive.phar selfupdate
Phive 0.13.3 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]    No RateLimit present in response 

xD

@MacDada
Copy link

MacDada commented Sep 21, 2020

I downloaded latest 0.14.4 manually. When I look into phive.phar/.phar/stub.php I can see that it is 0.14.4 –– OK.

It no longer complains about RateLimit.

It does complain that I use PHP 7.1, but that's a different issue…

@theseer
Copy link
Member

theseer commented Sep 21, 2020 via email

@MacDada
Copy link

MacDada commented Sep 21, 2020

I said 0.13.4 for PHP <7.2 ;-)

Oh, my bad. It works! Thx :)

@MacDada
Copy link

MacDada commented Sep 21, 2020

Well, it worked for a few times…

$ bin/phive.phar install
Phive 0.13.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Copying phpcs-3.5.2.phar to ./tools/phpcs
Linking /home/vagrant/.phive/phars/php-cs-fixer-2.14.0.phar to ./tools/php-cs-fixer

$ bin/phive.phar install
Phive 0.13.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Copying phpcs-3.5.2.phar to ./tools/phpcs
[ERROR]    No RateLimit present in response 

$ bin/phive.phar --version
Phive 0.13.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors

@theseer
Copy link
Member

theseer commented Sep 21, 2020

I admit I'm confused.

The exception is thrown when phive tries to retrieve rate limiting information from the response. The only point in time where that is not wrapped into a check (by calling $response->hasRateLimit()) is when initializing the RateLimit in the github alias resolver.

So that must be the culprit. But for that to fail, Github must not be responding with X-RateLimit-*-Headers on the API call for Rate Limits. Which I consider unlikely - and fail to trigger.

So I'm lost as to why this might be happening.
Would you be interesting i trying a debug-build I'd provide to you?

@theseer theseer reopened this Sep 21, 2020
@theseer
Copy link
Member

theseer commented Sep 21, 2020

Ah! Github started to block HEAD requests to their rate limit API endpoint.

@theseer theseer closed this as completed Sep 21, 2020
@MacDada
Copy link

MacDada commented Sep 22, 2020

Ah! Github started to block HEAD requests to their rate limit API endpoint.

Okeeeeey… So i guess you know how to fix that?

Would you be interesting i trying a debug-build I'd provide to you?

Sure, bring it on :)

@theseer
Copy link
Member

theseer commented Sep 24, 2020

Try 0.13.5 :)

@JSabelli-FDX
Copy link

It's happening again on 0.15.2

$ phive install phpmetrics/PhpMetrics
Phive 0.15.2 - Copyright (C) 2015-2024 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]   No RateLimit present in response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests