Slow model loading performance after version 3.0.1

[ogo-adp]

System Information

• OS & Version: Windows 10, Windows 2016
• ML.NET Version: ML.NET > 3.01 (including 5.0 preview)
• .NET Version: .NET 8.0

Describe the bug
We use ML.NET on a large application with approx 20 models using only ML.NET FastTree regression.
For all ML.NET versions after 3.0.1 the time to load a model from a MLContext has increased dramatically

To Reproduce
Create a regression model using FastTree
Load the model from a MLContext with version 3.0.1. Note the loading time
Load the same model with version 4.0.2. The loading time should be much higher
We observed between 7 and 50 times increased in loading times

Expected behavior
The loading time should not be different

Screenshots, Code, Sample Projects
var mlContext = new MLContext(seed: 0);
var loadedModel = mlContext.Model.Load("path_of_model.zip", out _);
-> loading time 111ms with 3.0.1, 1100ms with 4.0.2

Additional contextA
Loading a model produced with version 4.0.2 using ML.NET 3.0.1 is fast
Loading a model produced with version 4.0.2 using ML.NET 4.0.2 is slow
Our model zip file range from 1000ko to 3000ko

[ogo-adp]

I've found the cause of the problem. Starting with version 4.0.0 ML.NET create a temporary directory in %appdata%\local\temp named "ml_dotnet" to load the model file. It does never delete this directory, and they pile up ...

On one of my servers there were near 200 000 "ml_dotnet" directories hence the problem. Once deleted the problme disappear (but will reappear after some time).

Version 3.0.1 did not create any temporary directory.

Culprit commit seems to be this

Especially this awkward code :
int dirNumber = 0;
string mlNetTempDir = null!;
while (Directory.Exists(mlNetTempDir = Path.Combine(Path.GetFullPath(tempPath), $"ml_dotnet{dirNumber++}"))) ;
var path = Path.Combine(mlNetTempDir, Path.GetRandomFileName());

In my opinion this is a huge bug with serious consequences on production servers using ML.NET. Would it be possible to release a 4.0.3 version that include a fix ?

Best regards,
Olivier

[ssanders82]

I am seeing this as well. Incredible. I've got 39,710 folders called ml_dotnetxxxxx in my local appdata