I think this can work well with Python's implicit namespace packages, https://peps.python.org/pep-0420/
In a nutshell, we can leave the fast_llm namespace open, so that third-party packages can add to it. As long as these packages are in the python path, python will find them and create the full fast_llm namespace dynamically. All that's needed is removing the init.py files in modules that should be open and their parents.

Not sure how much this helps though, since we still need to add a way to auto-import. We could import them all, but it would add to the startup time and the config/implementation separation complicates things a bit. I was thinking about somethings that makes the imports explicit, some options:

1. In the fast-llm command .Maybe not ideal since it's too separate from the config.
   fast-llm --plugins my_package.my_module other_package.other_module train ....

2 Somewhere near the config root. All in one place but distant from the classes that use them

training: ...
plugins: 
  - my_package.my_module
  - other_package.other_module
...

3. Next to the class name. Fully modular, but a bit scattered over the place.

data:
  datasets:
    Training:
      type: my_dataset
      module: my_package.my_module
...

Probably 2. is best.

Do we really need this though, especially now?
I'd rather focus on making the codebase more friendly to work with for people who need or want to directly contribute to Fast-LLM's development.

I'm not planning to implement plugins now, I think it should come once we have 2-3 places with dynamic classes and/or other teams working on this that would benefit from it.

Short-term use case: components for testing, ex GPTTestSlowDatasetConfig from #129