Add EngineFragment #3120
Add EngineFragment #3120
Conversation
…ager # Conflicts: # python/sglang/srt/openai_api/adapter.py
# Conflicts: # python/sglang/srt/managers/detokenizer_manager.py
# Conflicts: # python/sglang/srt/entrypoints/engine.py # python/sglang/srt/managers/generation_manager.py # python/sglang/srt/managers/tokenizer_manager.py
# Conflicts: # python/sglang/srt/entrypoints/engine.py # python/sglang/srt/entrypoints/http_server.py # python/sglang/srt/managers/scheduler.py
…into feat/separate_comm
| traceback.print_exc() | ||
| raise | ||
|
|
||
| fragment.shutdown() |
There was a problem hiding this comment.
This PR modifies shutdown(). Should we assert something to check shutdown() is appropriately implemented? For example, we may check allocated cuda memory.
There was a problem hiding this comment.
IIRC last month I added it because, otherwise without shutdown, the process would run forever, possibly because there is a while-true running if the shutdown is not triggered, then Verl training scripts can never end.
The shutdown in std mode kills the whole process tree, thus fully release memory. On the other hand, this SPMD one does not seem to release memory. However, it seems that the primary use case of SPMD, e.g. Verl training, does not care about this, because shutdown is only called (if any) at the end of whole training.
Motivation
With refactor PRs before this one, adding SPMD logic is quite short:
EngineFragmentclass (for SPMD). Both EngineFragment and Engine extends EngineBase to ensure consistent API.Modifications
Checklist