[Feature]: Multi-Node Serving

[ppatel-eng]

🚀 The feature, motivation and pitch

We are attempting to serve Llama 3.1 405B and cannot fit the model into a single node with 8 HPUs so would like to use multiple nodes to serve the model. I see this MR attempted to address this but was never merged: #218

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[michalkuligowski]

Hi @ppatel-eng, multinode solution is not planned for now.