Merge pull request #182 from darrylmelander/lifecycle-doc

Lifecycle doc

Author: darrylmelander

Diff for: doc/OnlineDocs/source/_static/image/PrescientSimulationCycle.png

@@ -2,7 +2,7 @@ Time Series Data Streams
 ========================
 
 Prescient uses time series data from two data streams, the real-time stream
-(i.e., actuals) and the forecast stream. As their names imply, the real-time 
+(i.e., actuals) and the forecast stream. As their names imply, the real-time
 stream includes data that the simulation should treat as actual values that
 occur at specific times in the simulation, and the forecast stream includes
 forecasts for time periods that have not yet occured in the simulation.
@@ -13,25 +13,25 @@ generation data.
 Real-Time Data (Actuals)
 ------------------------
 
-The real-time data stream provides data that the simulation should treat as 
+The real-time data stream provides data that the simulation should treat as
 actual values. Real-time values are typically used only when the simulation reaches
 the corresponding simulation time.
 
 Real-time data can be provided at any time interval. The real-time data interval
-generally matches the SCED interval 
-(see :ref:`sced-frequency-minutes<config_sced-frequency-minutes>`), but this is 
-not a requirement. If the SCED interval does not match the real-time interval 
+generally matches the SCED interval
+(see :ref:`sced-frequency-minutes<config_sced-frequency-minutes>`), but this is
+not a requirement. If the SCED interval does not match the real-time interval
 then real-time data will be interpolated or discarded as needed to match the SCED
 interval.
 
 Forecasts
 ---------
 
-Forecast data are provided by the forecast data stream. The frequency of data 
+Forecast data are provided by the forecast data stream. The frequency of data
 provided through the forecast stream must be hourly.
 
-New forecasts are retrieved each time a new RUC plan is generated. The 
-forecasts retrieved in a given batch are those required to satisfy the RUC horizon 
+New forecasts are retrieved each time a new RUC plan is generated. The
+forecasts retrieved in a given batch are those required to satisfy the RUC horizon
 (see :ref:`ruc-horizon<config_ruc-horizon>`), starting with the RUC activation time.
 
 
@@ -40,17 +40,17 @@ forecasts retrieved in a given batch are those required to satisfy the RUC horiz
 Forecast Smoothing
 ~~~~~~~~~~~~~~~~~~
 
-As forecasts are retrieved from the forecast data stream, they may be adjusted so that 
+As forecasts are retrieved from the forecast data stream, they may be adjusted so that
 near-term forecasts are more accurate than forecasts further into the future. This serves
 two purposes: first, to avoid large jumps in timeseries values due to inaccurate forecasts;
 and second, to model how forecasts become more accurate as their time approaches.
 
-The number of forecasts to be smoothed is determined by the 
+The number of forecasts to be smoothed is determined by the
 :ref:`ruc-prescience-hour<config_ruc-prescience-hour>` configuration option. Values for
-the current simulation time are set equal to their actual value, ignoring data read from 
+the current simulation time are set equal to their actual value, ignoring data read from
 the forecast stream. Values for ``ruc-prescience-hour`` hours after the current simulation
 time are set equal to data read from the forecast stream. Between these two times,
-values are a weighted average of the values provided by the actuals and forecast data 
+values are a weighted average of the values provided by the actuals and forecast data
 streams. The weights vary linearly with where the time falls between the current time
 and the ruc prescience hour. For example, if ``ruc-prescience-hour`` is 8, then the adjusted
 forecast for 2 hours after the current simulation time will be ``0.25*forecast + 0.75*actual``.
@@ -62,8 +62,8 @@ Real-Time Forecast Adjustments
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Forecasts are adjusted further each time a SCED is run. This is done by comparing the forecast
-for the current time with the actual value for the current time. The ratio of these two 
-values is calculated, then used as a scaling factor for forecast values. For example, if the 
+for the current time with the actual value for the current time. The ratio of these two
+values is calculated, then used as a scaling factor for forecast values. For example, if the
 forecast for a value was 10% too high, all future forecasts for the same value are reduced by 10%.
 
 .. note::

Diff for: doc/OnlineDocs/source/concepts/data_streams.rst

@@ -1,10 +1,10 @@
-Modeling Concepts
------------------
-
-.. toctree::
-   :maxdepth: 2
-
-   ruc_sced_cycle.rst
-   data_streams.rst
-   reserves.rst
-   markets.rst
+Modeling Concepts
+-----------------
+
+.. toctree::
+   :maxdepth: 2
+
+   ruc_sced_cycle.rst
+   data_streams.rst
+   reserves.rst
+   markets.rst

Diff for: doc/OnlineDocs/source/concepts/index.rst

@@ -1,3 +1,3 @@
-Energy Markets and Pricing
-==========================
-
+Energy Markets and Pricing
+==========================
+

Diff for: doc/OnlineDocs/source/concepts/markets.rst

@@ -1,2 +1,2 @@
-Reserves and Ancillary Services
+Reserves and Ancillary Services
 ===============================

Diff for: doc/OnlineDocs/source/concepts/reserves.rst

@@ -4,7 +4,7 @@ The Prescient Simulation Cycle
 .. image:: ../_static/image/RucScedCycle.png
 	:class: align-right
 
-Prescient simulates the operation of a power generation network throughout a 
+Prescient simulates the operation of a power generation network throughout a
 study horizon, finding the set of operational choices that satisfy demand at
 the lowest possible cost.
 
@@ -19,7 +19,7 @@ The RUC Cycle
 -------------
 
 The RUC cycle periodically generates a RUC plan. A RUC plan consists of two types of
-data: a unit commitment schedule and optionally a pricing schedule (when
+data: a unit commitment schedule and, optionally, a pricing schedule (when
 :ref:`compute-market-settlements<config_compute-market-settlements>` is True).
 The unit commitment schedule
 indicates which dispatchable generators should be activated or deactivated during
@@ -36,38 +36,48 @@ will go into effect at midnight, one at 8:00 a.m., and one at 4:00 p.m. Each
 RUC plan covers the time period that starts when it goes into effect and ends
 just as the next RUC plan becomes active.
 
-A RUC plan is based on the current state of the system at the time the plan is 
-generated (particularly the current dispatch and up- or down-time for dispatchable generators), and on
-forecasts for a number of upcoming time periods. The forecasts considered when
-forming a RUC plan must extend at least to the end of the RUC's planning period,
-but typically extend further into the future in order to avoid poor choices at
-the end of the plan ("end effects"). The amount of time to consider when
-generating a RUC plan is known as the RUC horizon. A commonly used RUC horizon
-is 48 hours.
+A RUC plan is based on the current state of the system at the time the plan is
+generated (particularly the current dispatch and up- or down-time for dispatchable
+generators), and on forecasts for a number of upcoming time periods. The forecasts
+considered when forming a RUC typically extend beyond the end of the RUC's planning
+period, to avoid poor choices at the end of the plan ("end effects").
 
 The simulation can be configured to generate RUC plans some number of hours before
-they take effect. This is done by specifying a time of day for one of the plans to
-be generated. The gap between the specified generation time and the next time a RUC
-plan is scheduled to take effect is called the RUC gap. Each RUC plan still covers
+they take effect. Each RUC plan still covers
 the expected time period, from the time the plan takes effect until the next RUC plan
-takes effect, but its decisions will be based on what is known at the time the RUC 
+takes effect, but its decisions will be based on what is known at the time the RUC
 plan is generated.
 
+More information about RUCs is found in :doc:`../reference/ruc_details`.
+
+
 The SCED Cycle
 --------------
 
-The SCED process selects dispatch levels for all active dispatchable generators 
+The SCED process selects dispatch levels for all active dispatchable generators
 in the current simulation time period. Dispatch levels are determined using a process
 that is very similar to that used to build a RUC plan. The current state of the
 system, together with forecasts for a number of future time periods, are examined to
 select dispatch levels that satisfy current loads and forecasted future loads at the
 lowest possible cost.
 
-The SCED cycle is more frequent than the RUC cycle, with new dispatch levels selected 
+The SCED cycle is more frequent than the RUC cycle, with new dispatch levels selected
 at least once an hour. The SCED honors unit commitment decisions made in the RUC plan;
-whether each generator is committed or not is dictatated by the RUC schedule 
+whether each generator is committed or not is dictatated by the RUC schedule
 currently in effect.
 
-Costs are also determined with each SCED, based on dispatchable generation selected by 
-the SCED process, the commitment and start-up costs as selected by the associated RUC
-process, as well as current actual demands and non-dispatchable generation levels.
+Costs are determined with each SCED, based on dispatchable generation levels selected by
+the SCED process, commitment and start-up decisions selected by the active RUC
+plan, and actual demands and non-dispatchable generation levels for the current simulation
+time. If market settlement is enabled, market-based generator revenue is also calculated.
+
+More information about SCEDs is found in :doc:`../reference/sced_details`.
+
+
+.. seealso::
+
+   A more detailed description of the Prescient simulation process can be found
+   in the :doc:`../reference/detailed_lifecycle` documentation.
+
+   More information about RUCs and SCEDs is available from
+   :doc:`../reference/ruc_details` and :doc:`../reference/sced_details`.

Diff for: doc/OnlineDocs/source/concepts/ruc_sced_cycle.rst

@@ -1,2 +1,2 @@
-Examples and Tutorials
+Examples and Tutorials
 ----------------------

Diff for: doc/OnlineDocs/source/examples/index.rst

@@ -1,2 +1,2 @@
-Python Classes and Functions
-----------------------------
+Python Classes and Functions
+----------------------------

Diff for: doc/OnlineDocs/source/reference/code.rst

@@ -0,0 +1,181 @@
+Detailed Prescient Simulation Lifecycle
+=======================================
+
+As Prescient simulates the operation of a power generation network, the simulation
+follows a repeating cycle of Reliability Unit Commitment (RUC) plans and Security
+Constrained Economic Dispatch (SCED) plans. This cycle is described at a high level
+in the concepts section (see :doc:`../concepts/ruc_sced_cycle`). This page
+provides a more detailed description of the simulation process, including plugin
+points that provide opportunities for custom code to observe or modify the
+simulation.
+
+.. image:: ../_static/image/PrescientSimulationCycle.png
+	:class: align-right
+
+A Prescient simulation consists of three phases: startup, the main simulation
+loop, and finalization. Each phase includes one or more tasks that are executed
+in a specific order. In the case of the main simulation loop, these tasks are
+carried out multiple times, once per SCED during the simulation period.
+
+The Prescient simulation lifecycle is executed when you run the Prescient
+command-line application, or in code when the `simulate()` method is called on
+a `prescient.simulator.Prescient` object.
+
+Startup
+------------------
+
+The startup phase consists of one-time activities that occur before the main
+simulation loop begins.
+
+Pre-Simulation Startup
+......................
+
+During the pre-simulation startup task, Prescient carries out activities such
+as parsing options, initializing plugins, and setting up data structures.
+
+First, any plugins specified in the simulation configuration are given an
+opportunity to register their callbacks. See :ref:`Identifying Plugins` and 
+:ref:`Plugin Module Initialization`.
+
+After plugins have been initialized, two plugin callbacks are called: 
+
+  * :ref:`plugin-options_preview`
+  * :ref:`plugin-initialization`
+
+After callbacks have been called, the current simulation time is set to midnight of
+the :ref:`simulation start date<config_start-date>`.
+
+
+The Main Simulation Loop
+------------------------
+
+The main simulation loop is executed once for every simulation time step, where
+the simulation time step duration is the 
+:ref:`SCED frequency<config_sced-frequency-minutes>`. The first simulation time
+step occurs at midnight of the first day (midnight is the beginning of the day,
+not the end). The last simulation time step occurs at the end of the final day
+of the simulation, just before midnight of the next day.
+
+A SCED is solved every time through the loop. Some times through the loop, a RUC may
+also be generated and/or activated.
+
+.. _Generate RUC:
+
+Generate RUC
+............
+
+If the current simulation time is a RUC generation hour, a new RUC is generated.
+This is either the same timestep the RUC will be activated, or an earlier
+timestep if a RUC delay has been specified. See :doc:`ruc_details` for information on the
+timing and frequency of RUC generation and its relationship to RUC activation.
+
+Note that the initial RUC is always generated on the first timestep of the
+simulation, even if Prescient has been configured to generate other RUCs earlier
+than they are activated.
+
+If a RUC is generated before its activation time, the first step of the RUC
+generation process is to solve a SCED-like model to estimate what the state of the
+system will be at the RUC activation time. Solving this model causes a single
+callback to be called:
+
+  * :ref:`plugin-after_get_initial_model_for_sced`
+
+This callback is only called if the RUC is generated in a different timestep than
+the RUC will be activated. The initial RUC never triggers this callback.
+
+As part of the RUC generation process, forecasts and actual values for upcoming
+periods are retrieved from the data source and loaded into Egret model. The
+callbacks listed below are called as a new batch of values is about to be loaded,
+giving plugins an opportunity to load any custom data they may need:
+
+  * :ref:`plugin-after_get_initial_model_for_simulation_actuals`
+  * :ref:`plugin-after_get_initial_model_for_ruc`
+
+Finally, the RUC itself is generated and solved. The following callbacks will be
+called:
+
+  * :ref:`plugin-before_ruc_solve`
+  * :ref:`plugin-after_ruc_generation`
+
+.. _Activate RUC:
+
+Activate RUC
+............
+
+If the current simulation time is a RUC activation time, the most recently
+generated RUC will be activated. Activating a RUC simply marks the point in the
+simulation when the RUC's decisions first begin to be followed. RUC activation
+hours occur at regular intervals starting at midnight of the first day and repeating
+at the :ref:`RUC frequency<config_ruc-every-hours>` for the rest of the simulation.
+See :doc:`ruc_details` for information on the timing and frequency of RUC
+activation.
+
+The following callback is called each time a RUC is activated:
+
+  * :ref:`plugin-after_ruc_activation`
+
+.. _Deploy SCED:
+
+Deploy SCED
+...........
+
+A SCED is generated, solved, and applied every simulation timestep.
+When a SCED is applied, generator setpoints are set for the current simulation time.
+See :doc:`sced_details`.
+
+The following callbacks are called each time a SCED is deployed:
+
+  * :ref:`plugin-after_get_initial_model_for_sced`
+  * :ref:`plugin-before_operations_solve`
+  * :ref:`plugin-after_operations`
+  * :ref:`plugin-update_operations_stats`
+
+.. _Finalize Timestep:
+
+Finalize Timestep
+.................
+
+After SCED deployment is complete, statistics for the timestep are published
+and the simulation clock advances to the time of the next SCED, as determined
+by the :ref:`SCED frequency<config_sced-frequency-minutes>`.
+
+Several callbacks related to statistics may be called at this time. Calling a
+callback related to statistics is referred to as "publishing" statistics.
+
+Operations statistics (statistics about SCED results) are published every timestep
+by calling the following callback:
+
+  * :ref:`plugin-operations_stats`
+
+If the timestep is the final timestep in a given hour, hourly statistics are
+published:
+
+  * :ref:`plugin-hourly_stats`
+
+If the timestep is the final timestep in a given day, daily statistics are
+published:
+
+  * :ref:`plugin-daily_stats`
+
+The simulation clock is advanced after all relevant statistics have been published.
+If the new time is later than the simulation end date, the main simulation loop
+ends and Prescient moves to the Finalization stage. Otherwise Prescient repeats the
+main simulation loop for the new timestep.
+
+Finalization
+------------
+
+The finalization phase consists of tasks that occur once at the end of the
+simulation.
+
+Finalize Simulation
+...................
+
+Statistics for the simulation as a whole are published during finalization:
+
+  * :ref:`plugin-overall_stats`
+
+Another callback is called to notify callbacks that the simulation is complete,
+giving plugins a chance to cleanly shut down:
+
+  * :ref:`plugin-finalization`

Diff for: doc/OnlineDocs/source/reference/detailed_lifecycle.rst

@@ -16,7 +16,7 @@ to acquire initial data, and to request updates to data for specific time
 periods.
 
 For an example or a custom data provider, see the
-`example <https://github.com/grid-parity-exchange/Prescient/blob/main/prescient/simulator/tests/custom_data_provider.py>`_ 
+`example <https://github.com/grid-parity-exchange/Prescient/blob/main/prescient/simulator/tests/custom_data_provider.py>`_
 in the source code, or examine one of the `standard data providers
 <https://github.com/grid-parity-exchange/Prescient/tree/main/prescient/data/providers>`_.