Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Close threads at server shutdown #4029

Merged
merged 6 commits into from
Oct 1, 2023

Conversation

petersv5
Copy link
Contributor

@petersv5 petersv5 commented Aug 3, 2023

To exit normally all non-daemon threads of the java process are supposed to have stopped. There are several cases of EventLoopGroup-created thread pools that are not shut down as a part of the server shutdown.

These threads are spawned on the first Bedrock player login. The identified cases are:

  • TcpClientSocket created by GeyserSession
  • The listener threads in GeyserServerInitializer
  • The player threads in GeyserServerInitializer
  • The EXECUTOR_SERVICE in the SkinProvider

This patch depends on the PR
GeyserMC/MCProtocolLib#744 submitted to MCProtocolLib.

This is a very quickly thrown together proof of concept to verify that the execution thread pools left running were what blocked a clean exit with Geyser active.

@Camotoy
Copy link
Member

Camotoy commented Aug 3, 2023

Can we make sure this works with /geyser reload?

@petersv5
Copy link
Contributor Author

petersv5 commented Aug 3, 2023

Can we make sure this works with /geyser reload?

Will that call GeyserServer.shutdown()? If so things will break for the cases where the EventLoopGroup is held in a static class variable and some other approach will be needed.

@Konicai
Copy link
Member

Konicai commented Aug 3, 2023

Spigot's reload should also be tested (despite it being terrible) to make sure things don't get any worse than they already are

@petersv5
Copy link
Contributor Author

petersv5 commented Aug 3, 2023

Like I wrote in the MCProtocolLib PR, it is probably best for someone that knows Geyser to reimplement these changes properly. This patch was literally created with grep, jstack and jdb.

@rtm516 rtm516 linked an issue Aug 3, 2023 that may be closed by this pull request
@rtm516 rtm516 changed the title close threads at server shutdown Close threads at server shutdown Aug 3, 2023
@rtm516 rtm516 added PR: Needs Testing When a PR needs testing but is currently not under review PR: Bugfix When a PR contains a bugfix labels Aug 3, 2023
@petersv5
Copy link
Contributor Author

petersv5 commented Aug 5, 2023

I can confirm /geyser reload is completely broken, as expected.
I'll make a version that actually unsets the variables tracking the event loop groups which should allow them to be recreated.

However, I wonder if the better solution would be to make the event loop groups instance variables instead of class members. I think we only create one instance of any of the classes holding event loop groups so there is no real point in making them static class variables. We never reuse them anyway. I need to go over all of them, but I think we never create many of them. At least not on a regular server. Are there cases with multiple instances in one process and where it is desireable to share worker pools?

@petersv5
Copy link
Contributor Author

petersv5 commented Aug 5, 2023

I pushed an update that allows /geyser reload to work.
The patch now nullifies EventLoopGroup holders when shut down.
EventLoopGroup users that are class variables can automatically restart the EventLoopGroup on first use. This affected TcpClientSession (see the updated patch to MCProtocolLib) and SkinProvider. The remaining EventLoopGroups were instance variables of objects that were recreated after a /geyser reload.
I also changed the name of player_group to playerGroup as requested.

I am not able to test spigot reload as I do not have such a system set up anymore.

@petersv5
Copy link
Contributor Author

petersv5 commented Aug 7, 2023

Rebased to latest master.

@petersv5
Copy link
Contributor Author

petersv5 commented Aug 9, 2023

Rebased to latest master

@petersv5
Copy link
Contributor Author

Switched to using the alterantive PR for ProtocolLib (GeyserMC/MCProtocolLib#748) after comments from Camotoy. This one no longer has an explicit shutdown of the TcpClientSession EventLoopGroup. That is now reaped on process exit.

The remaining EventLoopGroup threads are still shutdown on Geyser shutdown.

I have tested /geyser reload and that seems to work.

@petersv5
Copy link
Contributor Author

Rebased to latest master.

@petersv5
Copy link
Contributor Author

petersv5 commented Sep 3, 2023

Rebased

@petersv5
Copy link
Contributor Author

Rebased and tested

@Konicai
Copy link
Member

Konicai commented Sep 21, 2023

thank you

To exit normally all non-daemon threads of the java process
are supposed to have stopped. There are several cases of
EventLoopGroup-created thread pools that are not shut down
as a part of the server shutdown.

These threads are spawned on the first Bedrock player login. The
identified cases are:
- The listener threads in GeyserServerInitializer
- The player threads in GeyserServerInitializer
- The EXECUTOR_SERVICE in the SkinProvider

This patch depends on the PR
GeyserMC/MCProtocolLib#748
submitted to MCProtocolLib to allow the TcpCLientSession to
shut down on process exit.

This is a very quickly thrown together proof of concept to verify that
the execution thread pools left running were what blocked a clean exit
with Geyser active.

Nullify EventLoopGroup variables after shutdown to allow them to be
restarted.
Copy link
Member

@onebeastchris onebeastchris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on all platforms (compiled locally with the mcpl pr), works perfectly! Thanks for this fix & for keeping the PR updated :)

@Konicai Konicai added PR: On hold When a PR is on hold like if it requires a dependency to be updated first and removed PR: Needs Testing When a PR needs testing but is currently not under review labels Sep 23, 2023
@Konicai
Copy link
Member

Konicai commented Sep 23, 2023

Until 1.20.2 when we can target a newer MCPL

@Konicai Konicai removed the PR: On hold When a PR is on hold like if it requires a dependency to be updated first label Oct 1, 2023
@Konicai Konicai merged commit 7d489c7 into GeyserMC:master Oct 1, 2023
1 check passed
@Konicai
Copy link
Member

Konicai commented Oct 1, 2023

thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: Bugfix When a PR contains a bugfix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fabric server does not shutdown on stop command
5 participants