Use absolute paths instead of relying on HF cache

[rishic3]

Relying on Huggingface cache to load LLMs is finicky on DBFS and when an access token is needed. Switch to downloading the model and using the abs path, which is better practice anyway.
Added barrier to pytriton server startup to ensure all servers are shut down if one fails.

[eordentlich]

Added barrier to pytriton server startup to ensure all servers are shut down if one fails.
What is the benefit of this since they are independent?

[rishic3]

Added barrier to pytriton server startup to ensure all servers are shut down if one fails. What is the benefit of this since they are independent?

If only one server fails and thus raises an error, the barrier stage will fail and the other servers will be left dangling without the manager having stored their PIDs. The subsequent tasks rely on all executors having a live server.

[eordentlich]

Ok. Good point. Does this actually solve the problem then or are the dangling servers still running after the barrier fails. Only the python workers are killed. Does this take out the triton servers too?

[rishic3]

Ok. Good point. Does this actually solve the problem then or are the dangling servers still running after the barrier fails. Only the python workers are killed. Does this take out the triton servers too?

Yes killing the parent python worker, which is running the pytriton event loop, will kill all associated triton server processes.

[eordentlich]

👍