Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal worker error if github is down #168

Open
vtjnash opened this issue Jul 6, 2023 · 0 comments
Open

fatal worker error if github is down #168

vtjnash opened this issue Jul 6, 2023 · 0 comments

Comments

@vtjnash
Copy link
Member

vtjnash commented Jul 6, 2023

[ Info: [Node 3 | 2023-06-29T13:32:24.262]: running on node 3: BenchmarkJob JuliaLang/julia@5db930e [daily]                                                                                             
      From worker 3:    fatal: unable to access 'https://github.com/JuliaCI/BaseBenchmarks.jl/': Failed to connect to github.com port 443 after 294689 ms: Connection timed out                         
      From worker 3:    error: Could not fetch origin                                                                                                                                                   
┌ Error: [Node 3 | 2023-06-29T13:59:00.488]: Encountered error: encountered job loop error: HTTP.Exceptions.ConnectError("https://api.github.com/repos/JuliaLang/julia/commits/5db930e67a30c4ef75de43c72
b1e9d617d55673c/comments", Base.IOError("connect: connection timed out (ETIMEDOUT)", -110))                                                                                                             
│   exception =                                                                                                                                                                                         
│    HTTP.Exceptions.ConnectError("https://api.github.com/repos/JuliaLang/julia/commits/5db930e67a30c4ef75de43c72b1e9d617d55673c/comments", Base.IOError("connect: connection timed out (ETIMEDOUT)", -1
10))                                                                                                                                                                                                    
│    Stacktrace:                                                                                                                                                                                        
│      [1] (::HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.va
r"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Type, readtimeout::Int64, kw::Base.Pairs{Symbol, Union{Nothing, 
Integer}, NTuple{5, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :status_exception, :idle_timeout), Tuple{Nothing, Nothing, Int64, Bool, Int64}}})                                          
│        @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/ConnectionRequest.jl:80                                                                                                 
│      [2] (::HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequ
est.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; canonicalize_headers::Bool, kw::Base.Pairs
{Symbol, Union{Nothing, Integer}, NTuple{5, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :status_exception, :idle_timeout), Tuple{Nothing, Nothing, Int64, Bool, Int64}}})                  
│        @ HTTP.CanonicalizeRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/CanonicalizeRequest.jl:17                                                                                             
│      [3] (::Base.var"#88#90"{Base.var"#88#89#91"{ExponentialBackOff, HTTP.RetryRequest.var"#3#6"{Int64, HTTP.Messages.Request, Base.RefValue{Int64}}, HTTP.CanonicalizeRequest.var"#1#3"{HTTP.Canonica
lizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.Exception
Request.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(args::HTTP.Messages.Request; kwargs::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{5, Symbol}, NamedTuple{(:iofunction, :deco
mpress, :verbose, :status_exception, :idle_timeout), Tuple{Nothing, Nothing, Int64, Bool, Int64}}})                                                                                                     
│        @ Base ./error.jl:296                                                                                                                                                                          
│      [4] (::HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.
var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.
Messages.Request; retry::Bool, retries::Int64, retry_non_idempotent::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{5, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :status_ex
ception, :idle_timeout), Tuple{Nothing, Nothing, Int64, Bool, Int64}}})                                                                                                                                 
│        @ HTTP.RetryRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/RetryRequest.jl:62                                                                                                           
│      [5] (::HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2
#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{
typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{5, Symbol}, Nam
edTuple{(:iofunction, :decompress, :verbose, :status_exception, :idle_timeout), Tuple{Nothing, Nothing, Int64, Bool, Int64}}})                                                                          
│        @ HTTP.CookieRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/CookieRequest.jl:42                                                                                                         
│      [6] (::HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1
#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"
#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; detect_content_type::Bool, kw::Base.Pairs{Sy
mbol, Union{Nothing, Integer}, NTuple{5, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :status_exception, :idle_timeout), Tuple{Nothing, Nothing, Int64, Bool, Int64}}})                     
│        @ HTTP.ContentTypeDetection ~/.julia/packages/HTTP/kWE2h/src/clientlayers/ContentTypeRequest.jl:23                                                                                             
│      [7] (::HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieReques
t.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.
var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(req:
:HTTP.Messages.Request; basicauth::Bool, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{5, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :status_exception, :idle_timeout), Tuple{Not
hing, Nothing, Int64, Bool, Int64}}})                                                                                                                                                                   
│      [8] (::HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.Con
tentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.Canonicalize
Request.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequ
est.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, kw::Base.Pairs{Symbol, Integer, Tuple{Symbol, Symbol, Sy
mbol}, NamedTuple{(:verbose, :status_exception, :idle_timeout), Tuple{Int64, Bool, Int64}}})                                                                                                            
│        @ HTTP.DefaultHeadersRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/DefaultHeadersRequest.jl:47                                                                                         
│      [9] (::HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuth
Request.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.
var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutReques
t.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::
Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::Base.Pairs{Symbol, Integer, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:verbose, :status_exception, :idle_timeout),
 Tuple{Int64, Bool, Int64}}})                                                                                                                                                                           
│        @ HTTP.RedirectRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/RedirectRequest.jl:25                                                                                                     
│     [10] #1#2                                                                                                                                                                                         
│        @ ~/.julia/packages/HTTP/kWE2h/src/clientlayers/DebugRequest.jl:23 [inlined]                                                                                                                   
│     [11] (::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.var"#1#2#5"{HT
TP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.ContentTypeDetectio
n.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequest.var"#1#2
#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.var"#1#2#4"{
typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; response_stream::Nothing, http
_version::VersionNumber, kw::Base.Pairs{Symbol, Integer, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:redirect, :status_exception, :idle_timeout), Tuple{Bool, Bool, Int64}}})                           
│        @ HTTP.MessageRequest ~/.julia/packages/HTTP/kWE2h/src/clientlayers/MessageRequest.jl:17                                                                                                       
│     [12] request(stack::HTTP.MessageRequest.var"#1#3"{HTTP.MessageRequest.var"#1#2#4"{HTTP.DebugRequest.var"#1#4"{HTTP.DebugRequest.var"#1#2#5"{HTTP.RedirectRequest.var"#1#4"{HTTP.RedirectRequest.va
r"#1#2#5"{HTTP.DefaultHeadersRequest.var"#1#3"{HTTP.DefaultHeadersRequest.var"#1#2#4"{HTTP.BasicAuthRequest.var"#1#3"{HTTP.BasicAuthRequest.var"#1#2#4"{HTTP.ContentTypeDetection.var"#1#3"{HTTP.Content
TypeDetection.var"#1#2#4"{HTTP.CookieRequest.var"#1#5"{HTTP.CookieRequest.var"#1#2#6"{HTTP.RetryRequest.var"#1#4"{HTTP.RetryRequest.var"#1#2#5"{HTTP.CanonicalizeRequest.var"#1#3"{HTTP.CanonicalizeRequ
est.var"#1#2#4"{HTTP.ConnectionRequest.var"#1#7"{HTTP.ConnectionRequest.var"#1#2#8"{HTTP.TimeoutRequest.var"#1#6"{HTTP.TimeoutRequest.var"#1#2#7"{HTTP.ExceptionRequest.var"#1#3"{HTTP.ExceptionRequest.
var"#1#2#4"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}}}}}}}}}, method::String, url::String, h::Dict{String, String}, b::String, q::Nothing; headers::Dict{String, String}, body::String, qu
ery::Nothing, kw::Base.Pairs{Symbol, Integer, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:redirect, :status_exception, :idle_timeout), Tuple{Bool, Bool, Int64}}})                                      
│        @ HTTP ~/.julia/packages/HTTP/kWE2h/src/HTTP.jl:417                                                                                                                                            
│     [13] #request#16                                                                                                                                                                                  
│        @ ~/.julia/packages/HTTP/kWE2h/src/HTTP.jl:277 [inlined]                                                                                                                                       
│     [14] #post#36                                                                                                                                                                                     
│        @ ~/.julia/packages/HTTP/kWE2h/src/HTTP.jl:486 [inlined]                                                                                                                                       
│     [15] github_request(api::GitHub.GitHubWebAPI, request_method::typeof(HTTP.post), endpoint::String; auth::GitHub.OAuth2, handle_error::Bool, headers::Dict{Any, Any}, params::Dict{String, String},
 allowredirects::Bool)                                                                                                                                                                                  
│        @ GitHub ~/.julia/packages/GitHub/O5XRk/src/utils/requests.jl:66                                                                                                                               
│     [16] #gh_post#5                                                                                                                                                                                   
│        @ ~/.julia/packages/GitHub/O5XRk/src/utils/requests.jl:73 [inlined]                                                                                                                            
│     [17] gh_post_json(api::GitHub.GitHubWebAPI, endpoint::String; options::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:auth, :params), Tuple{GitHub.OAuth2, Dict{String, String}}}})  
│        @ GitHub ~/.julia/packages/GitHub/O5XRk/src/utils/requests.jl:79                                                                                                                               
│     [18] create_comment(api::GitHub.GitHubWebAPI, repo::String, item::String, kind::Symbol; options::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:auth, :params), Tuple{GitHub.OAuth2, 
Dict{String, String}}}})                                                                                                                                                                                
│        @ GitHub ~/.julia/packages/GitHub/O5XRk/src/issues/comments.jl:71                                                                                                                              
│     [19] #create_comment#188                                                                                                                                                                          
│        @ ./none:0 [inlined]                                                                                                                                                                           
│     [20] reply_comment(sub::Nanosoldier.JobSubmission, message::String)                                                                                                                               
│        @ Nanosoldier /nanosoldier/Nanosoldier.jl/src/submission.jl:133                                                                                                                                
│     [21] #reply_comment#11                                                                                                                                                                            
│        @ /nanosoldier/Nanosoldier.jl/src/jobs/jobs.jl:12 [inlined]                                                                                                                                    
│     [22] reply_comment                                                                                                                                                                                
│        @ /nanosoldier/Nanosoldier.jl/src/jobs/jobs.jl:12 [inlined]                                                                                                                                    
│     [23] delegate_job(server::Nanosoldier.Server, job::Nanosoldier.BenchmarkJob, node::Int64)                                                                                                         
│        @ Nanosoldier /nanosoldier/Nanosoldier.jl/src/server.jl:114                                                                                                                                    
│     [24] macro expansion                                                                                                                                                                              
│        @ /nanosoldier/Nanosoldier.jl/src/server.jl:72 [inlined]                                                                                                                                       
│     [25] (::Nanosoldier.var"#63#64"{Nanosoldier.Server, DataType, Int64, Vector{Int64}})()                                                                                                            
│        @ Nanosoldier ./task.jl:484                                                                                                                                                                    
└ @ Nanosoldier /nanosoldier/Nanosoldier.jl/src/config.jl:48                                                                                                                                            

(this is the last we heard from worker 3)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant