Plugin system

[DenuxPlays]

Feature Request

Is your feature request related to a problem? Please describe.

I wrote a few "plugins" for loco for some projects and I've noticed that there is no real documentation or features on how to do that.

Describe the solution you'd like

Disclaimer: I use packages and plugins interchangeably. What name gets used does not really matter as long as it is consistent.
My Solution would be the following (inspired by symfony and their packages/bundles):

Configuration

Document a config folder where custom config should be located.
I thought about something like config/packages (which I would prefer but it doesn't really matter) or config/plugins
Example: config/packages/my-plugin.yaml

Also I would expose a config that loads a config into a struct just like locos configuration with env resolving, etc.
Example:

pub struct Config(u8)

let config = load_package_config::<Config>("my-plugin.yaml");

Which would automatically resolves the config/packages folder structure.

Also I have considered that plugins should could extend the normal config.yaml but this could lead to name conflicts.
The other method is ofcourse not immune to it but it would reduce the chance.
Also having more then one config increases the readability if correctly used.

Plugin registration hook?

Maybe a hook that just exposes the normal AppHooks to the plugin so that the user would not need to write for example every initializer themselfs into the hook.
So for example:

pub async fn load_packages() -> Result<Vec<Box<dyn PackageHooks>>> {
    vec![my_plugin::PackageHook.into()]
}

or something like this what would reduce the acutal lines of code to one.
And the plugin maintainer has more control which would result in less user error.
Disclaimer: This should just point out the structure I guess that the rest is complete non-sense.

Documentation

Ofcourse all this needs to be documented and maybe an example plugin or a template repository would be great.

Describe alternatives you've considered

Doing it myself for every package but thats kinda annoying.
I just don't like copying code.

Misc

This is more of a concept/discussion.
I definitely forgot something so I will probably add more to it over time.
If you like this idea and this concept is more thought through I would contribute it.

[kaplanelad]

what is the difference between your suggestion and initializers?

[kaplanelad]

Adding another hook to run code, plugins, or modules before appContext seems overly complicated.
If you need to execute code before appContext, use the boot hook. If you need to run it after appContext, use initializers.

If something is missing in initializers, we can improve it, but introducing another hook feels unnecessary.

[DenuxPlays]

what is the difference between your suggestion and initializers?

initializers are always run after AppContext is created which some plugins might want to modify or do something before the Database connection was established.
With the new Data subsystem ( #1267 ) some might want to change/update/create data files.

For example a plugin for correctly/better loading env variables.

Also a plugin might want to use both.
Add one or more initializers and do something before AppContext.

Now the user would need to modify their main method and add one or more initializers to their local hooks which is prone to user error and adding/removing an initializer would always be a breaking change

Things like registering tasks and worker or tables that need to be truncated is not even possible with initializers.

This "Hook" does not need to be complicated just expose the normal Hooks to plugin so that these things are possible for the maintainer to do not the user.

[DenuxPlays]

Adding another hook to run code, plugins, or modules before appContext seems overly complicated. If you need to execute code before appContext, use the boot hook. If you need to run it after appContext, use initializers.

If something is missing in initializers, we can improve it, but introducing another hook feels unnecessary.

Will boot work with env loading?
The config is already parsed at this time or?

Also adding another hook would not be that complicated.
It just needs to expose the normal Hooks to the plugin.
Maybe there is a better solution that I am not thinking of.

My goal is that the user just needs to say "load that plugin" and the plugin then does the rest.
Being able to hook in various states during Start-up or able to register worker without leaving the responsibility to the user.

[kaplanelad]

The boot function gets environment. So, in your case, if you need to execute code before the app context is created, use boot. On the other hand, if you want to register plugins after the AppContext is initialized, use initializers.

Let's look at a real example of a plugin you want to implement and see how we can integrate it using the existing hooks. do you want to give one?

[DenuxPlays]

Currently I cannot show my plugins I've already made.
But integrating them look like this:

1. Find all the hooks I need
2. Add all the specific functions from the plugin

So for example in my Env Plugin I've done something like this:

async fn load_config(env: &Environment) -> Result<Config> {
        // Other stuff..
        EnvLoading::load_from_env(env);

        env.load()
    }

fn register_tasks(tasks: &mut Tasks) {
        EnvLoading::register_commands(&mut tasks);
        // tasks-inject (do not remove)
    }

And this is a simple plugin.

I currently making plans to propose an enhance event/message system which I want to make as a plugin first.
And currently the User MUST do all this:

    async fn load_config(env: &Environment) -> Result<Config> {
        EventSystem::before_boot(env);
        MessageSystem::before_boot(env);

        env.load()
    }

    async fn boot(mode: StartMode, environment: &Environment, mut config: Config) -> Result<BootResult> {
       config = EventSystem::update_config(config);
       config = MessageSystem::update_config(config);

        create_app::<Self, Migrator>(mode, environment, config).await
    }

    async fn initializers(_ctx: &AppContext) -> Result<Vec<Box<dyn Initializer>>> {
        Ok(vec![
            Box::new(EventSystem::initializer()),
            Box::new(MessageSystem::initializer())
        ])
    }


    async fn connect_workers(ctx: &AppContext, queue: &Queue) -> Result<()> {
        // Yeah I think you get it        

        Ok(())
    }

// Also some things need to be done in: register_tasks, truncate, after_context

Which is absolutely not feasible for the user.
Too error prone and changing anything would be breaking.

And this are just my basic Plan for the 0.1.0 I've not even started developing or extended features.
I have some other things on my todo list but I wanted to open this issue to maybe resolve/discuss things before I start.

[kaplanelad]

Following your example:

1. Yes, if you want to load a custom environment variable for login, load_config is the right place to do it. I don’t see a strong reason to create a separate hook (or plugin) for this, as it only needs to be done once.

2. Regarding the boot process—why should the eventSystem return a config? What is the logic behind that? If your goal is to define configuration for the MessageSystem, you can use the settings section in the configuration. This section is unstructured, allowing for dynamic configuration. That way, you can remove the initialization of MessageSystem from the boot process and manage it solely within the initializers. you can also see example here.

If you can provide a GitHub repository with your setup, I’d be happy to review it and suggest improvements to enhance your implementation and better understand the problem.

[DenuxPlays]

1. Yes with simple plugins there is no problem and it does not really matter

2. Oh didn't noticed the settings but this would only resolve this problem.
   If the plugins defines, worker, tasks or other intialiazers then the USER would need to list/add ALL of the manually.
   So one of my commands will be to list all events/messages with a specific status, one will be to delete, one will be add etc.
   The user is not responsible to manage this but is currently forced to.
   And this is just for one plugin.

When I take a look at ever symfony, spring, ruby etc. projects they all use a lot of third-party extensions but no-one is required to manage that much of custom code for EVERY extension because they all provide some kind of enhanced Plugin system.

Also i would replace the unstructured settings with a place for custom .yaml files like mentioned above.
Would reduce the chance of naming collisions and clean-up the config.

I don't currently have a Repo atleast none with code just a bunch of markdown files with concepts.

But maybe a better description would be a Plugin system.
Initializers is good for basic things but useless when introducing custom workers etc. and even more useless when you want to do something during boot etc.

[DenuxPlays]

So with the example.
Instead of having the config in settings there will be a confi/packages/multi_db.yaml config file with the structure that the multi_db plugin describes.

And the initializer works fine but try adding a truncate specific db command.
Every command must now be listed by the user instead of by the plugin.
A plugin system would allow the maintainer of the plugin to take care of that and decide which command are exposed to the user or what will be executed when.

[kaplanelad]

What prevents you from reading the config in the initializer?

[DenuxPlays]

Nothing
Its just not readable when a Lot of plugins are using it.

And this still doesnt solve the other problems

[kaplanelad]

Nothing Its just not readable when a Lot of plugins are using it.

And this still doesnt solve the other problems

I think it's opinionated, and I don't currently see enough value

[DenuxPlays]

Yes ofcourse.
But seeing that other frameworks are doing exactly this is a point for it.

For example when I take a look at my symfony config files.
If I would put everything into one (which is still possible even in symfony) I would be 1387 lines long...

But yes thats why I am open for discussions but this isn't my big point