Dont set scale to zero as default when creating an Endpoint

[tomaarsen]

Hello!

Pull Request overview

• Dont set scale to zero as default when creating an Endpoint

Details

As discussed internally, this PR sets the scale_to_zero_timeout to None by default. It's not entirely clear to me if the min_replica/max_replica is also intended to be set to None from the discussion though - we can make those changes still.

Also, when someone specifies the scale_to_zero_timeout, we can help them out by overriding the min_replica to 0, but we can also do nothing as users will get an error anyways that the minReplica should not be 1.

I've also added an example of how to create an endpoint that scales to zero, and I quickly tested that running that example with the new scale_to_zero_timeout & min_replica defaults indeed creates an endpoint that doesn't scale to zero.

cc @Wauplin @Vaibhavs10 @ErikKaum

• Tom Aarsen

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hanouticelina]

Thanks @tomaarsen, looks good! I left two comments about the default values of min_replica/max_replica and another one about the typing syntax

[hanouticelina]

Looks good, let's merge!

Also, when someone specifies the scale_to_zero_timeout, we can help them out by overriding the min_replica to 0, but we can also do nothing as users will get an error anyways that the minReplica should not be 1.

I’d prefer not to add any pre-processing logic here or override min_replica based on the value of scale_to_zero_timeout. The error message is already quite clear as you mentioned:

Bad Request: 'scaleToZeroTimeout' can only be set when 'minReplica' is 0

I think it's better to let the user handle it explicitly.