Hyperparameter tuning

[ablaom]

Looking at bit into MLJ integration. For better or worse, hyper-parameter optimization (eg, grid search) in MLJ generally works by mutating the field values of the model struct. I wonder if TableTransforms.jl would consider changing their transformer types to mutable structs? I think in ML applications, at least, any loss in performance would be pretty minimal, but perhaps there are wider use-cases to consider?

The alternative for our use case is for the MLJ model wrapper to be mutable (for now a wrapper is necessary anyway) and that a user wanting to do a search does something like

values = [Scale(low=0, high=x) for x in 1.0:0.1:10]     <---- extra step 
values = range(wrapped_transformer, :model, values=values)

However, while this might be fine for Grid search, it doesn't really work for other optimization strategies.

Thoughts?

[ablaom]

I suppose with some work we could create a wrapper that essentially "mutafies" the original model (overload setproperty! to re-generate the atomic model each time it's called) but that would be kinda complicated.

[juliohm]

I wonder if hyperparameter tuning could leverage Setfield.jl or its successor Acessors.jl instead of enforcing mutability of the structs? My understanding is that mutable structs in Julia are only justified when we have hot loops. In other words, mutable structs feel like an anti-pattern in the language unless you have a "array-like" struct that you need to setindex! many times.

My main concern with enforcing mutability is that it may compromise parallelization later on. I am not an expert in compiler technology but I know that non-mutable structs help the compiler do more aggressive optimizations. And sending copies of these tiny transforms to different threads and processes should be more performant than dealing with chunks of memory in the heap? Again, I am not an expert in compilers, but my intuition is pointing to the non-mutable design as the more conservative and optimal design.

@eliascarv do you have any comments to add?

[eliascarv]

I have an idea. We can make a wrapper type to make Transforms mutable.
Something like this:

mutable struct Mut{T<:Transform}
  transform::T
end

Base.getproperty(transf::Mut, prop::Symbol) = getproperty(transf.transform, prop)

function Base.setproperty!(transf::Mut, prop::Symbol)
  # code
end

apply(transf::Mut, table) = apply(transf.transform, table)

# code

statictransf = Scale(low=0, high=1)
mutabletransf = Mut(statictransf)

[juliohm]

Yes, it is always possible to wrap the structs as @ablaom mentioned.

Let's postpone this decision to a future major release when hyperparameter tuning can be considered.

[ablaom]

I wonder if hyperparameter tuning could leverage Setfield.jl

We already do (in-house) recursive setproperty!. I'm not understanding how that helps us, though. Except to make implementation of the proposed wrapper easier. Perhaps that's what you meant?

[juliohm]

I mean that there is nothing intrinsic to hyperparameter tuning that requires types to be mutable. We can certainly provide an API that extracts parameters of a pipeline into a "flat vector" of parameters, performs an optimization step, and then instantiates a new pipeline with improved parameters. The amount of computing we save by forcing the structs to be mutable is not very clear, specially considering that we are talking about a couple of dozen parameters in general?

[CameronBieganek]

This proposal for MLJTuning would allow MLJ to handle immutable pipelines.

[ParadaCarleton]

@juliohm has there been any work on hyperparameter tuning with TableTransforms.jl? I'd like to add asinh (pseudolog) transformations to the package, but that would require the ability to tune 1 or 2 hyperparameters.

[juliohm]

We are starting to brainstorm an API for this feature. It would be nice to have an additional hand :) Em qua., 25 de out. de 2023 00:20, Carlos Parada ***@***.***> escreveu:

…

@juliohm <https://github.com/juliohm> has there been any work on hyperparameter tuning with TableTransforms.jl? I'd like to add asinh (pseudolog) transformations to the package, but that would require the ability to tune 1 or 2 hyperparameters. — Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZQW3PXLLCXRRLRRMPIOQ3YBCAOZAVCNFSM5VD22KQKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXHA2DIMJVGU4A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ParadaCarleton]

We are starting to brainstorm an API for this feature. It would be nice to have an additional hand :)

Sounds good! Should we just stick to just the MLJ model interface for now?