Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connections times out when Opencast takes long to inspect files #8

Open
tibroc opened this issue Apr 9, 2021 · 5 comments
Open

Connections times out when Opencast takes long to inspect files #8

tibroc opened this issue Apr 9, 2021 · 5 comments

Comments

@tibroc
Copy link
Collaborator

tibroc commented Apr 9, 2021

In long recordings Opencast seems to take long to inspect all the files. During this the connection to the BBB-instance can time out causing an error (see log output below). I do not know how grave this is, because the whole workflow still seems to work, but it certainly is confusing in the logs and there might be an easy fix for that.

From /var/log/bigbluebutton/post_archive.log:

E, [2021-04-08T17:50:52.600932 #5969] ERROR -- :  A problem occured for request: /ingest/ingest/bbb-upload
I, [2021-04-08T17:50:52.601036 #5969]  INFO -- : 504 Gateway Timeout (RestClient::GatewayTimeout)
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
/usr/lib/ruby/2.5.0/net/http.rb:910:in `start'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
/var/lib/gems/2.5.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
/usr/local/bigbluebutton/core/scripts/post_archive/oc_modules/oc_util.rb:104:in `requestIngestAPI'
/usr/local/bigbluebutton/core/scripts/post_archive/post_archive.rb:1194:in `<main>'
I, [2021-04-08T17:50:52.601082 #5969]  INFO -- : <html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

I, [2021-04-08T17:50:52.601093 #5969]  INFO -- : LOG ERROR Aborting ingest with BBB id 9993503bcee5edce408f7e34ef38b5a7af2b3c94-1617879754745and OC idbd3c9497-f52e-407e-a32f-3b702199c495
@tibroc tibroc mentioned this issue Apr 22, 2021
Open
@tibroc
Copy link
Collaborator Author

tibroc commented Apr 22, 2021

As discussed, the issue seems to be that the ingest takes very long and that is why the timeout happens. See also #13

@Arnei
Copy link
Member

Arnei commented Apr 26, 2021

To clarify this a bit, the timeout is likely not triggered by Opencast itself or the post_archive.rb script, but the reverse proxy in front of Opencast.

@Rillke
Copy link
Contributor

Rillke commented May 9, 2021

It's clearly not triggered by Opencast. nginx reports 504 Gateway Time-out meaning that proxy_read_timeout is too small in your nginx config.

@metzenseifner
Copy link

It seems to me that inspecting the files after the ingest step would be a more scalable solution. The proxy timeout setting has an upper limit that depends on the SYN retransmit count limit and timeouts of the TCP protocol implemented by the operating system. Changing these global settings on the operating system would affect the entire operating system, including any other applications. That is often undesirable.

@metzenseifner
Copy link

metzenseifner commented Nov 4, 2021 •
edited
Loading

I have tested increasing the proxy_read_timeout on our (ironically) production machine from 60s (default) to 300s, and again to 600s. In all cases, the problem persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@metzenseifner @Rillke @Arnei @tibroc