Build downloads failed to complete for a period of time

[zachallaun]

Original report: elixir-lang/elixir-lang.github.com#1781

For a period of time (at least ~20 minutes), attempting to download https://builds.hex.pm/builds/otp/amd64/ubuntu-22.04/master.tar.gz was failing with the following errors:

$ curl -fsSLo master.tar.gz https://builds.hex.pm/builds/otp/amd64/ubuntu-22.04/master.tar.gz
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

$ curl --http1.1 -fsSLo master.tar.gz https://builds.hex.pm/builds/otp/amd64/ubuntu-22.04/master.tar.gz
curl: (18) transfer closed with 24841216 bytes remaining to read

At the time, it was specifically master.tar.gz that was failing; I was able to successfully download both Elixir archives as well as OTP 27.1.2. Unfortunately, I can no longer reproduce the issue, but @wojtekmach suggested reporting here to create a paper trail. 🙂

[wojtekmach]

I just got the same error:

~% curl -vvvfO "https://builds.hex.pm/builds/otp/amd64/ubuntu-24.04/master.tar.gz"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host builds.hex.pm:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.37.91
*   Trying 151.101.37.91:443...
* Connected to builds.hex.pm (151.101.37.91) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [318 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2571 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=builds.hex.pm
*  start date: Oct 20 06:47:39 2024 GMT
*  expire date: Jan 18 06:47:38 2025 GMT
*  subjectAltName: host "builds.hex.pm" matched cert's "builds.hex.pm"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://builds.hex.pm/builds/otp/amd64/ubuntu-24.04/master.tar.gz
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: builds.hex.pm]
* [HTTP/2] [1] [:path: /builds/otp/amd64/ubuntu-24.04/master.tar.gz]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /builds/otp/amd64/ubuntu-24.04/master.tar.gz HTTP/2
> Host: builds.hex.pm
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< x-amz-id-2: s1dmLoKWoC7Pd4VQ6k7HpSodrrjLeCFisUFSbqyIvMTGg9JKvN8dCv2YftTj7i89am5w6jLcJo0=
< x-amz-request-id: R7HV480DRDGX4Q9G
< last-modified: Mon, 04 Nov 2024 15:24:18 GMT
< etag: "5315f4ca3f4e7ab26949d9b71676d3c8-9"
< x-amz-server-side-encryption: AES256
< cache-control: public,max-age=3600
< x-amz-meta-surrogate-control: public,max-age=604800
< x-amz-meta-surrogate-key: builds builds/otp builds/otp/amd64 builds/otp/amd64/ubuntu-24.04 builds/otp/amd64/ubuntu-24.04/master
< x-amz-version-id: ZX5cg_sWIYeuQwFPBw2A0fIL8m.u3sm7
< content-type: application/x-tar
< server: AmazonS3
< accept-ranges: bytes
< date: Tue, 05 Nov 2024 09:06:38 GMT
< via: 1.1 varnish
< age: 16
< x-served-by: cache-iad-kiad7000127-IAD, cache-ams21040-AMS
< x-cache: HIT, HIT
< x-cache-hits: 3, 1
< x-timer: S1730797599.571087,VS0,VE1
< content-length: 72648215
<
{ [16375 bytes data]
  8 69.2M    8 5682k    0     0  1594k      0  0:00:44  0:00:03  0:00:41 1594k* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
 11 69.2M   11 8192k    0     0  1831k      0  0:00:38  0:00:04  0:00:34 1831k
* Connection #0 to host builds.hex.pm left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

changing the URL slightly to force cache miss worked:

~% curl -vvvfO "https://builds.hex.pm/builds/otp/amd64/ubuntu-24.04/master.tar.gz?v2"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host builds.hex.pm:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.37.91
*   Trying 151.101.37.91:443...
* Connected to builds.hex.pm (151.101.37.91) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [318 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2571 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=builds.hex.pm
*  start date: Oct 20 06:47:39 2024 GMT
*  expire date: Jan 18 06:47:38 2025 GMT
*  subjectAltName: host "builds.hex.pm" matched cert's "builds.hex.pm"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://builds.hex.pm/builds/otp/amd64/ubuntu-24.04/master.tar.gz?v2
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: builds.hex.pm]
* [HTTP/2] [1] [:path: /builds/otp/amd64/ubuntu-24.04/master.tar.gz?v2]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /builds/otp/amd64/ubuntu-24.04/master.tar.gz?v2 HTTP/2
> Host: builds.hex.pm
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< x-amz-id-2: UvIG1hDs37Sotqhm1iDnPrEp5R2BSm3dHaNgEeMqHLV35LPd5Obq+tfKo0XmqUOiJn43p+M+T4k=
< x-amz-request-id: AQE218NQZEN82K54
< last-modified: Mon, 04 Nov 2024 18:29:33 GMT
< etag: "47e064e699faf0828bd4dadbd32816ec-9"
< x-amz-server-side-encryption: AES256
< cache-control: public,max-age=3600
< x-amz-meta-surrogate-control: public,max-age=604800
< x-amz-meta-surrogate-key: builds builds/otp builds/otp/amd64 builds/otp/amd64/ubuntu-24.04 builds/otp/amd64/ubuntu-24.04/master
< x-amz-version-id: KXdtS3CUMFyNmw_JqNnUvbNuu7wdPbDe
< content-type: application/x-tar
< server: AmazonS3
< accept-ranges: bytes
< age: 0
< date: Tue, 05 Nov 2024 09:07:43 GMT
< via: 1.1 varnish
< x-served-by: cache-iad-kjyo7100093-IAD, cache-ams2100127-AMS
< x-cache: MISS, MISS
< x-cache-hits: 0, 0
< x-timer: S1730797663.809332,VS0,VE375
< content-length: 72647951
<
{ [1369 bytes data]
100 69.2M  100 69.2M    0     0  4320k      0  0:00:16  0:00:16 --:--:-- 4841k
* Connection #0 to host builds.hex.pm left intact

I noticed this difference between requests:

- < x-served-by: cache-iad-kiad7000127-IAD, cache-ams21040-AMS
+ < x-served-by: cache-iad-kjyo7100093-IAD, cache-ams2100127-AMS