Experience with Gin for configuring hyperparameters

[marcvanzee]

Gin provides a lightweight configuration framework for Python, based on dependency injection. Functions or classes can be decorated with @gin.configurable, allowing default parameter values to be supplied from a config file (or passed via the command line) using a simple but powerful syntax. This removes the need to define and maintain configuration objects (e.g. protos), or write boilerplate parameter plumbing and factory code, while often dramatically expanding a project's flexibility and configurability.

[marcvanzee]

Below is a summary of opinions collected from a discussion between various experienced users (@lienz, @adarob, @markomernick, @dulacarnold, @olivierteboul, @johnpjf, @alextp):

• We use gin for everything from data processing, flax models, loss functions, training loop, etc. We started with configdict but then switched to gin as (in my opinion) it's the most practical way of handling hyperparameters, in particular when your hyperparameters can be classes (like the type of model to train).

• The main benefits of gin are:

  • ability to use functions/classes as values
  • ability to inherit from other configs (rather than starting from scratch each time)

  However, you have to be very careful not to overuse it. Create a barrier where you agree not to make things below it gin
  configurable or all hell can break loose.

• Treat Gin like flags, and only use it in the top level of your trainer - never in a library. You really want to confine Gin configurables to the highest level of your binary.

• Our experience has been that gin can get unwieldly, we have a mixture of dataclass-based configdicts with gin-like auto-generation of flags. Like this all the config happens in the top-level, experiments are defined by a single .py file, and everything is flag over-writeable.

• I feel like as long as the code can be used with or without gin, gin is very convenient and elegant. As soon as you fully rely on gin to inject parameters, I agree it becomes challenging to reason about.

• We've been using gin, but have a rule that you can only call / create configgurable things from the top level main. So your model can be marked as gin.configurable in some source file, but you can only create it inside the top level main.

• Essentially, using gin to configure your binary makes it super easy to control it, but allowing library functions to be gin.configurable leads to very-hard-to-read code sometimes as you can think you're calling a function with some arguments when reading the code but you're actually calling another function with different arguments. And this can also prevent more advanced use cases (say, if you make the # of transformer layers gin configurable in the library you can end up in a situation where you cannot have a big transformer and a small transformer in memory at the same time because both have to have the same # of layers)