Autoscaling inference endpoints

[clefourrier]

This PR:

• makes a lot of parameters completely optional to allow creating an inference endpoint from only the model name if needed
• tries to auto-suggest relevant endpoint size
• if not possible, tries to spin up the endpoint with the lowest hardware needed, then tries rescaling automatically for an hour until an appropriate hardware size is reached (or until 1h has elapsed)
• catches a bunch of possible exceptions (endpoint already present, connection issues, compute not available, etc)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[clefourrier]

cc @albertvillanova this could interest you btw :)

[albertvillanova]

Thanks for the ping.

Just a preliminary comment while trying to understand the logic:

• The PR does not allow the user to create an endpoint with an arbitrary name. The user can only:
  • Either create an endpoint without being able to choose the endpoint name
  • Or reuse and existing (arbitrarily named) endpoint

[clefourrier]

Yep! That's the idea, since I wanted to streamline process as much as possible - either the user already has an endpoint, or they want one created and don't care about the name

[clefourrier]

Would you want do be able to select a name in all cases?

[albertvillanova]

I do not have a strong opinion. I just wanted to be sure to understand all possible use cases and if we should allow all of them:

• Use an existing endpoint
  • endpoint_name is required, model not passed, instance not passed, endpoint not deleted
• Create a new endpoint
  • model is required
  • Custom endpoint_name? Either pass it or not
  • Delete endpoint at the end? Yes or not
  • Custom instance? Yes or not (autoscale)

The simplest cases:

• User only passes endpoint_name: use an existing endpoint
• User only passes model: create endpoint (we decide its name), autoscale, and delete

[clefourrier]

At the moment we only cover the simple cases with this PR: user passes endpoint name or model name, we see if it's an already existing endpoint or if we need to spin it up, if we need to spin it up we delete it afterwards

[albertvillanova]

My general suggestion would be:

• make it as easy as possible for simple use cases, convenient for most users
• also make it highly customizable, for advanced users

But some aspects if considered useful could be addressed in other PRs.

The only inconvenient I see is:

• this PR changes current YAML format: endpoint_name and model are replaced by model_or_endpoint_name
  • this is a breaking change and has an impact in UX
• future advanced use cases may again require differentiating between endpoint_name and model

[clefourrier]

Very fair point, I'll revert this specific change

[NathanHB]

Good ! Only small nit on the config file