[Task] explore the issue of changing parameters value during the simulation

[SarahAlidoost]

Currently, the parameters of diffwofost are static meaning they are fixed during the entire simulation period. In some applications, we might need to change this condition.

An example is implemented in DMC-MTL. Earlier I commented on this in #13 as:

The process of “updating parameters before each integration step” is similar to BMI functionality (they implemneted set/get_parameters instead of the typical set/get_variable see the code here). This seems needed when using RNN to estimate the crop parameters. But I couldn't find any tests or examples showing that this works in practice. This also needs future discussion.

Another example is implemented in PCSE-Gym. I have commented on this in #6 as:

According to the paper (Nitrogen management with reinforcement learning and crop growth models), PCSE_gym supports interaction between a Reinforcement Learning agent and a crop model (i.e. WOFOST80 or LINTUL3). For example, the use case from the paper shows how to use LINTUL3 for nitrogen management in rain-fed winter wheat. I found that the PCSE version for crop model (i.e. WOFOST80 or LINTUL3) used in crop_gym is the one in the branch develop_lintul3. In this branch, the module signals has apply_npk in addition to irrigate. These two signals are used in pcse_gym to apply an action to the crop model. The workflow: the pcse crop model (i.e., WOFOST80 or LINTUL3) gets initialized and an action is calculated. Then, in a single step of the pcse_gym, the action is applied to the model. Model runs one more step in time (t+1) and based on the model outputs, observations and reward (i.e., increase in yield) are calculated. It is nice to see that the pcse.engine.Engine and _model._send_signal are used to interact with the crop model. Details can be found at Advanced_Machine_Learning_CropGym_Tutorial.ipynb

In this context, we want to explore the challenges and answer:

• what does it mean to change the parameters during the simulation? see point (i) in my comment.
  • which parameter and how?
• what would be the effect of changing a parameter value on gradients during the optimization?
  • this is similar to "model-based RL" or "differentiable decision making" using "Backpropagation Through Time"
• how to change engine to pause and resume a simulation or update the crop models internally see running a model with different parameter sets #21 and [Tasks]: Implement Data array organization inside Engine #25
  • engine should not do initialize, calculate_rate, calculate_state at once
• how can be this integrated in hybrid modelling see [Backlog Item]: Build hybrid modeling framework #11

[SarahAlidoost]

@ronvree in the update meeting, we discussed the issue of changing a parameter during the simulation. I listed two examples above (I'm not sure how PCSE-Gym is relevant though). Is there any other use case/examples? can you please share it here, thanks!

[ronvree]

One thing I can imagine happening is that you would like to update your beliefs of either the model state or parameters based on new observations you obtain during a simulation. PCSE-Gym, for example, is basically an interface between WOFOST/PCSE and a reinforcement learning setting, where in each day in a simulation you would like to reconsider your situation and update your policy/management actions accordingly. The ability to "pause" or adjust (and copy) a simulation mid-way is essential here. Would including this functionality in the engine not also imply the possibility of changing parameters mid-simulation? Would it significantly complicate the implementation?

[ionathan]

I am not entirely sure what would be the main scenario for updating parameters during simulation. Maybe in the context of data asssimilation with joint state-parameter estimation methods (EnKF). Otherwise, I would think that the parameters are estimated during calibration/training and not updated during simulation/testing.

[SarahAlidoost]

We decided to skip this case and consider the parameters static. No further exploration needed.