From 4eb8837b789ca36591a367911d9bcf436e201d9e Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Mon, 12 Aug 2024 13:00:07 -0400
Subject: [PATCH 001/136] Update 2_Analysis_and_Visualization.md
update docs (in progress)
---
docs/wiki/2_Analysis_and_Visualization.md | 29 +++++++++++++++++------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/docs/wiki/2_Analysis_and_Visualization.md b/docs/wiki/2_Analysis_and_Visualization.md
index 5ef53b9..67bb167 100644
--- a/docs/wiki/2_Analysis_and_Visualization.md
+++ b/docs/wiki/2_Analysis_and_Visualization.md
@@ -22,7 +22,7 @@ To simplify the description, assuming the preferred direction of any response va
If there are multiple experimental outcomes ($J \geq 1$), the $j^{th}$ outcome in vector $Y_i$ is denoted as $Y_i^{j}$.
-### Evaluating the Measured Experimental Conditions
+#### Evaluating the Measured Experimental Conditions
A direct assessment of input features is assigning a binary indicator (True/False) to identify the best performer $X_{opt}$ or an opimal solution set $\chi_{opt}$.
@@ -58,7 +58,7 @@ For multi-objective optimization, the evaluation is subjective to user preferenc
-### The Overall Optimization Performance Metrics
+#### The Overall Optimization Performance Metrics
To monitor the progress of SMBO workflow, we need to define a scalar evaluation metric to summarize performance over all the $N$ data points.
@@ -75,14 +75,29 @@ To monitor the progress of SMBO workflow, we need to define a scalar evaluation
---
## Surrogate Model Interpretation
+In this session, we assume the input feature $X$ is a $K$-dimensional vector $(X_1, X_2, ..., X_K)$, and all the model explanation techniques are applied to each surrogate model outcome individually.
-### SHAP (SHapley Additive exPlanations)
+#### SHAP (SHapley Additive exPlanations)
-### Partial Dependence Plot
+We use the Kernel SHAP algorithm to estimate the Shapley values, which is a feature attribution method that quantifies the contribution of each feature towards the surrogate model's prediction for any input data, providing insights into variable importance and model explanation.
-### Individual Conditional Expectation
+The Shapley value is a concept from game theory that aims to fairly allocate the total gains among the players in a coalitional game. In the original definition of the Shapley value, the contribution of each player is the difference in gains when including or excluding this player, averaged over all possible permutations of players. Let $v(S)$ be the gain of any player subset $S$, the Shapley value $\varphi_k(v)$ for the $k^{th}$ player is defined as:
-### Sensitivity Analysis
+\begin{equation*}
+\varphi_k(v) = \frac{1}{K!} \sum_{S \subseteq K \setminus {k}} |S|! \times (K-|S|-1)! \times \big(v(S \cup \{k\}) - v(S)\big)
+\end{equation*}
+
+It can be used to explain the outputs of a machine learning model, where the input features are considered as the players and model prediction is interpreted as the total gains achieved through the collaborative effort of these features.
+
+Calculating the exact Shapley values is not feasible due to the large number of $2^K$ possible subsets and the need to train a new prediction model for each possible subset of features for obtaining $v(S)$.
+The Kernel SHAP algorithm implemented in [SHAP](https://github.com/shap/shap) package provides a model-agnostic and computationally efficient approach to estimate Shapley values.
+
+
+#### Partial Dependence Plot
+
+#### Individual Conditional Expectation
+
+#### Sensitivity Analysis
@@ -93,7 +108,7 @@ To monitor the progress of SMBO workflow, we need to define a scalar evaluation
-### Prediction Uncertainty
+#### Prediction Uncertainty
(TBA...)
From a35ba74fb6950ebd1fcc626e1048798fbaade784 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Tue, 13 Aug 2024 01:13:15 -0400
Subject: [PATCH 002/136] Update 2_Analysis_and_Visualization.md
First draft in multiple sub-sessions
---
docs/wiki/2_Analysis_and_Visualization.md | 33 +++++++++++++++++++++--
1 file changed, 31 insertions(+), 2 deletions(-)
diff --git a/docs/wiki/2_Analysis_and_Visualization.md b/docs/wiki/2_Analysis_and_Visualization.md
index 67bb167..453bc33 100644
--- a/docs/wiki/2_Analysis_and_Visualization.md
+++ b/docs/wiki/2_Analysis_and_Visualization.md
@@ -95,11 +95,35 @@ The Kernel SHAP algorithm implemented in [SHAP](https://github.com/shap/shap) pa
#### Partial Dependence Plot
+Partial Dependence Plot (PDP) is a powerful visualization tool used in the interpretation and explanation of complex machine learning models.
+It helps to visualize the relationship between a specific feature and the target variable while holding all other features constant.
+PDPs are particularly valuable when working with sophisticated models like deep neural networks, random forests, and gradient boosting machines, which are often considered "black boxes" due to their complexity.
+By isolating the effect of a single feature, PDPs can reveal whether the relationship with the target variable is linear, monotonic, or more intricate.
+They also have the ability to uncover interactions between features, providing deeper insights into the model's behavior.
+One of the key advantages of PDP is their relative ease of computation and interpretation, making them an effective means of communicating model insights to both technical and non-technical audiences.
+This versatility has made PDPs an essential technique in the field of explainable AI (XAI), allowing stakeholders to gain trust in and understanding of complex predictive models across various domains.
+
#### Individual Conditional Expectation
-#### Sensitivity Analysis
+Individual Conditional Expectation (ICE) plots serve as a powerful complement to Partial Dependence Plots (PDPs), offering a granular, instance-level perspective that reveals how predictions change for individual data points as a feature value varies, thereby uncovering heterogeneous effects and non-linear relationships that might be obscured in the aggregate view provided by PDPs.
+While Partial Dependence Plots (PDPs) provide a global view of a feature's impact on model predictions, ICE plots offer a more granular, instance-level perspective.
+These plots illustrate how the prediction for a specific data point changes as the value of a particular feature is varied, while keeping all other features constant.
+This approach allows us to observe the model's behavior at a local level, providing crucial insights into how the model makes predictions for individual instances.
+ICE plots are particularly valuable when dealing with complex, non-linear relationships or when there are significant interactions between features that might be obscured in aggregate visualizations.
+By displaying a separate line for each instance in the dataset, ICE plots can reveal heterogeneity in feature effects that might be averaged out in PDPs.
+This makes them especially useful for identifying subgroups within the data that may be affected differently by changes in a feature.
+#### Sensitivity Analysis
+
+Sensitivity analysis around the optimal solution, particularly after the suggestions have stabilized over several iterations, serves as a critical step in validating and understanding the robustness of the identified solution.
+As the optimization process converges, it's essential to examine how small perturbations in the input variables affect the output, ensuring that the algorithm hasn't fallen into a local optimum or prematurely converged.
+This analysis helps quantify the trade-off between exploration and exploitation, a key consideration in Bayesian optimization.
+By systematically varying the parameters around the suggested optimal point, we can gauge the stability of the solution and identify any regions of high sensitivity.
+This process not only provides insights into the model's behavior but also helps in assessing the reliability of the optimization results.
+Moreover, sensitivity analysis can reveal potential areas for further refinement or highlight the need for additional iterations if the optimal point proves to be unstable.
+In cases where the analysis indicates a robust optimal solution, it strengthens confidence in the APO outcome and provides valuable information about the parameter space surrounding the optimum.
+This understanding is particularly crucial in complex, high-dimensional problems where visualizing the entire optimization landscape may not be feasible.
@@ -109,8 +133,13 @@ The Kernel SHAP algorithm implemented in [SHAP](https://github.com/shap/shap) pa
#### Prediction Uncertainty
-(TBA...)
+Adding prediction intervals as an uncertainty metric to suggested candidates is crucial for enhancing both the performance and interpretability of the optimization process.
+These intervals provide a quantifiable measure of uncertainty around predicted values, enabling a balanced approach between exploration of uncertain areas and exploitation of promising regions.
+This balance is key to avoiding premature convergence to local optima and making more informed decisions about where to sample next.
+Furthermore, they greatly enhance the explainability of the process by visually and numerically representing the model's confidence across the parameter space.
+This additional context allows for clearer communication of potential risks and rewards associated with different candidate points.
+By incorporating prediction intervals, APO becomes a more transparent and interpretable tool, which is crucial for its effective real-world applications where understanding the rationale behind suggestions is as important as the suggestions themselves.
From 4d9541a221b8fa13b45bef37faa77b48b3716bf7 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Tue, 13 Aug 2024 15:31:20 -0400
Subject: [PATCH 003/136] update wiki in progress...
---
.gitignore | 3 -
docs/wiki/1_APO_Workflow_and_CodeStructure.md | 4 +-
docs/wiki/2_Analysis_and_Visualization.md | 6 +-
docs/wiki/3_Data.md | 115 +++++++++++++++++-
4 files changed, 119 insertions(+), 9 deletions(-)
diff --git a/.gitignore b/.gitignore
index c2e89ce..a5fae96 100644
--- a/.gitignore
+++ b/.gitignore
@@ -85,9 +85,6 @@ instance/
# Scrapy stuff:
.scrapy
-# Sphinx documentation
-docs/_build/
-
# PyBuilder
.pybuilder/
target/
diff --git a/docs/wiki/1_APO_Workflow_and_CodeStructure.md b/docs/wiki/1_APO_Workflow_and_CodeStructure.md
index 47f39c2..2b8c3d7 100644
--- a/docs/wiki/1_APO_Workflow_and_CodeStructure.md
+++ b/docs/wiki/1_APO_Workflow_and_CodeStructure.md
@@ -48,7 +48,9 @@ The data structure in a typical APO work flow includes:
See section
[Data](3_Data.md)
and submodule
-[`parameters`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/parameters)
+[`parameters`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/parameters),
+and submodule
+[`objectives`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/objectives)
for more details.
diff --git a/docs/wiki/2_Analysis_and_Visualization.md b/docs/wiki/2_Analysis_and_Visualization.md
index 453bc33..f157d03 100644
--- a/docs/wiki/2_Analysis_and_Visualization.md
+++ b/docs/wiki/2_Analysis_and_Visualization.md
@@ -3,9 +3,9 @@
-This section introduces some additional components, such as retrospective analysis and visualization methods, that are not essential steps in a Sequential Model-Based Optimization (SMBO) algorithm but are valuable in real-world applications for several reasons.
+This section introduces some additional components, such as retrospective analysis and visualization methods, that are not essential steps in a Algorithmic Process Optimization (APO) algorithm but are valuable in real-world applications for several reasons.
-* The technical details involved in SMBO algorithms, such as surrogate models and acquisition functions, may seem complicated to users with non-quantitative background. As a result, the entire workflow of suggesting new experimental conditions may appear to be a black box for users, which hampers the adoption of this this powerful process optimization technique. Various performance metrics and model interpretation methods help to bridge this gap by providing users with a better intuitive understanding of the underlying algorithms and revealing the decision-making processes involved.
+* The technical details involved in APO algorithms, such as surrogate models and acquisition functions, may seem complicated to users with non-quantitative background. As a result, the entire workflow of suggesting new experimental conditions may appear to be a black box for users, which hampers the adoption of this this powerful process optimization technique. Various performance metrics and model interpretation methods help to bridge this gap by providing users with a better intuitive understanding of the underlying algorithms and revealing the decision-making processes involved.
* The variable importance analysis and/or model interpretation tools can provide critical insights into the optimization process, aiding in a deeper understanding of the variables that influenced the selection of optimal solution and the relationships between input variables, which could be confirmed with additional experiments or scientific domain experts.
@@ -60,7 +60,7 @@ For multi-objective optimization, the evaluation is subjective to user preferenc
#### The Overall Optimization Performance Metrics
-To monitor the progress of SMBO workflow, we need to define a scalar evaluation metric to summarize performance over all the $N$ data points.
+To monitor the progress of APO workflow, we need to define a scalar evaluation metric to summarize performance over all the $N$ data points.
* Single-objective optimization: The optimal value (either max or min, depends on target specification) of measured experimental outcome.
diff --git a/docs/wiki/3_Data.md b/docs/wiki/3_Data.md
index c8daa1d..b906f5d 100644
--- a/docs/wiki/3_Data.md
+++ b/docs/wiki/3_Data.md
@@ -1,3 +1,114 @@
-# Data
+# Data Structure
-(TBA...)
\ No newline at end of file
+## Experimental design space $X_{space}$
+
+
+### Basic Syntax
+
+Each of the input varible is defined according to the variable type and domain.
+Continuous variable is specified by variable name, followed by lower and upper bounds.
+> Param_Continuous('varName', lower_bound, upper_bound)
+
+Discrete varaible is specified by variable name, followed by the (ordered) list of possible values in string format.
+> Param_Categorical('varName', ['level 1', 'level 2', 'level 3',...])
+
+An example list of input parameter specifications including commonly used variable types: continuous, categorical and ordinal:
+
+```python
+from obsidian.parameters import Param_Continuous, Param_Categorical, Param_Ordinal
+
+params = [
+ Param_Continuous('Temperature', -10, 30),
+ Param_Continuous('Concentration', 10, 150),
+ Param_Continuous('Enzyme', 0.01, 0.30),
+ Param_Categorical('Variant', ['MRK001', 'MRK002', 'MRK003']),
+ Param_Ordinal('StirRate', ['Low', 'Medium', 'High']),
+]
+```
+Then the $X_{space}$ is specified as a `ParamSpace` class object, initialized by the list of parameters.
+
+```python
+from obsidian import ParamSpace
+X_space = ParamSpace(params)
+```
+
+The `ParamSpace` class object can be exported into dictionary format to facilite save and reload for future usage:
+
+```python
+X_space_dict = X_space.save_state()
+X_space_reload = ParamSpace.load_state(X_space_dict)
+```
+
+### Additional Variable Types
+
+* Continuous observatioal variable
+
+ For example, an entire time course was measured during the experiment, and data at all the different timepoints ranging from 0 to 10 are used for fitting.
+ But during optimization, we are only interested in improving the results for a certain fixed time point at 6.
+
+ ```python
+ from obsidian.parameters import Param_Discrete_Numeric
+ Param_Observational(name = 'Time', min = 0, max = 10, design_point = 6)
+ ```
+
+
+* Discrete numerical variable
+
+ ```python
+ from obsidian.parameters import Param_Discrete_Numeric
+ Param_Discrete_Numeric('LightStage', [1, 2, 3, 4, 5])
+ ```
+
+* Task variable
+
+ ...
+
+## Initial experimental conditions, or seed experiments $X_0$
+
+When we start the APO workflow from scratch, the initial experimental conditions are usually generated by random sampling or design-of-experiments algorithms.
+For example, generate six input conditions $X_0$ according to previously specified $X_{space}$ using Latin hypercube sampling (LHS) method:
+
+```python
+from obsidian.experiment import ExpDesigner
+
+designer = ExpDesigner(X_space, seed = 0)
+X0 = designer.initialize(m_initial = 6, method='LHS')
+```
+
+
+| | Temperature | Concentration | Enzyme | Variant | StirRate |
+|---:|--------------:|----------------:|----------:|:----------|:-----------|
+| 0 | 13.3333 | 68.3333 | 0.2275 | MRK003 | High |
+| 1 | 6.66667 | 115 | 0.0825 | MRK003 | Low |
+| 2 | 26.6667 | 45 | 0.0341667 | MRK002 | Medium |
+| 3 | 20 | 91.6667 | 0.275833 | MRK001 | Low |
+| 4 | -6.66667 | 21.6667 | 0.179167 | MRK002 | Medium |
+| 5 | 0 | 138.333 | 0.130833 | MRK001 | High |
+
+
+The `designer` returns experimental conditions as a pandas dataframe, which is the default data format in various `obsidian` functions.
+
+
+
+## Experimental outcome variable(s) $Y$
+
+...
+
+```python
+from obsidian import Target
+target = Target('Yield', aim='max')
+```
+
+```python
+from obsidian import Target
+target = [
+ Target('Yield', aim='max'),
+ Target('Cost', aim='min')
+]
+```
+
+## Use campaign object to manage data
+
+The campaign class object serves as a convinient portal to access all components in APO workflow, including data management.
+
+...
From 574702d816d07c7efbc7dc7df5be7248b5eff8fd Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Wed, 14 Aug 2024 01:12:55 -0400
Subject: [PATCH 004/136] finish wiki/3_Data.md
---
docs/wiki/3_Data.md | 137 ++++++++++++++++++++++++++++++++++++++------
1 file changed, 121 insertions(+), 16 deletions(-)
diff --git a/docs/wiki/3_Data.md b/docs/wiki/3_Data.md
index b906f5d..08d0798 100644
--- a/docs/wiki/3_Data.md
+++ b/docs/wiki/3_Data.md
@@ -2,7 +2,6 @@
## Experimental design space $X_{space}$
-
### Basic Syntax
Each of the input varible is defined according to the variable type and domain.
@@ -32,13 +31,22 @@ from obsidian import ParamSpace
X_space = ParamSpace(params)
```
-The `ParamSpace` class object can be exported into dictionary format to facilite save and reload for future usage:
+The `ParamSpace` class object can be exported into dictionary format to facilite save to json files and reload for future usage:
```python
-X_space_dict = X_space.save_state()
-X_space_reload = ParamSpace.load_state(X_space_dict)
+import json
+
+with open('X_space.json', 'w') as f:
+ X_space_dict = X_space.save_state()
+ json.dump(X_space_dict, f)
+
+with open('X_space.json', 'r') as f:
+ X_space_dict = json.load(f)
+ X_space_reload = ParamSpace.load_state(X_space_dict)
```
+In addition, the `ParamSpace` class contains various instance methods for input variable transformation, which are implicitly called during the optimization but no need for direct access by the user.
+
### Additional Variable Types
* Continuous observatioal variable
@@ -51,7 +59,6 @@ X_space_reload = ParamSpace.load_state(X_space_dict)
Param_Observational(name = 'Time', min = 0, max = 10, design_point = 6)
```
-
* Discrete numerical variable
```python
@@ -61,11 +68,18 @@ X_space_reload = ParamSpace.load_state(X_space_dict)
* Task variable
- ...
+ Only one special 'task' categorical variable is allowed for encoding multiple tasks.
+ Distinct response will be predicted for each task.
+
+ ```python
+ from obsidian.parameters import Task
+ Task('TaskVar', ['Task_A', 'Task_B', 'Task_C', 'Task_D'])
+ ```
## Initial experimental conditions, or seed experiments $X_0$
When we start the APO workflow from scratch, the initial experimental conditions are usually generated by random sampling or design-of-experiments algorithms.
+
For example, generate six input conditions $X_0$ according to previously specified $X_{space}$ using Latin hypercube sampling (LHS) method:
```python
@@ -73,9 +87,9 @@ from obsidian.experiment import ExpDesigner
designer = ExpDesigner(X_space, seed = 0)
X0 = designer.initialize(m_initial = 6, method='LHS')
+print(X0.to_markdown())
```
-
| | Temperature | Concentration | Enzyme | Variant | StirRate |
|---:|--------------:|----------------:|----------:|:----------|:-----------|
| 0 | 13.3333 | 68.3333 | 0.2275 | MRK003 | High |
@@ -92,23 +106,114 @@ The `designer` returns experimental conditions as a pandas dataframe, which is t
## Experimental outcome variable(s) $Y$
-...
+### Basic Syntax
+
+Similar to the `ParamSpace` object for input variables, there is `Target` class object which handles the specification and preprocessing for experimental outcome variables.
+
+For each outcome measurement, there are three essential arguments to be specified:
+* name: Variable name, which is a required input by user
+* f_transform: Transformation function for preprocessing the raw response values, to facilitate the numerical computations during optimization.
+ - 'Identity': (default) No transformation
+ - 'Standard': Normalization into zero mean and unit standard deviation
+ - 'Logit_MinMax': Logit transofrmation with the range or scale automatically calculated based on data
+ - 'Logit_Percentage': Assuming input response is a percentage ranging between 0 to 100, apply logit transofrmation with scale 1/100.
+* aim: Either 'max'(default) or 'min', which specifies the desirable direction for improvement. Currently it only handles continuous outcome values.
+
+
+Depend on the number of outcomes, define one `Target` object or a list of multiple objects:
```python
from obsidian import Target
-target = Target('Yield', aim='max')
+
+target = Target(name = 'Yield', f_transform = 'Logit_Percentage', aim='max')
+
+target_multiple = [
+ Target(name = 'Yield', f_transform = 'Logit_Percentage', aim='max'),
+ Target(name = 'Cost', f_transform = 'Standard', aim='min')
+]
```
+### Example
+
+To demonstrate the usage of `Target` class, we simulate a single task experimental outcome $y_0$ using the previously generated $X_0$ and an analytical function 'shifted_parab'.
+
```python
-from obsidian import Target
-target = [
- Target('Yield', aim='max'),
- Target('Cost', aim='min')
-]
+from obsidian.experiment import Simulator
+from obsidian.experiment.benchmark import shifted_parab
+
+simulator = Simulator(X_space, shifted_parab, name='Yield')
+y0 = simulator.simulate(X0)
+print(y0.to_markdown())
+```
+
+| | Yield |
+|---:|--------:|
+| 0 | 47.8147 |
+| 1 | 62.5599 |
+| 2 | 60.7972 |
+| 3 | 39.1121 |
+| 4 | 83.0833 |
+| 5 | 52.2631 |
+
+If manually input $y_0$, it should be a pandas dataframe with the same variable name 'Yield' as specifed in the `target` definition.
+
+When the 'transform_f' function is called with 'fit=True' during the optimization workflow, the raw response will be saved as an attribute to `target` object
+```python
+y_transformed = target.transform_f(y0, fit = True)
+type(target.f_raw) # torch.Tensor
+```
+
+The `Target` class object, as well as the input response 'f_raw' (if exists), can be exported into dictionary format to facilite save to json files and reload for future usage:
+
+```python
+import json
+
+with open('target.json', 'w') as f:
+ target_dict = target.save_state()
+ json.dump(target_dict, f)
+
+with open('target.json', 'r') as f:
+ target_dict = json.load(f)
+ target_reload = Target.load_state(target_dict)
```
## Use campaign object to manage data
-The campaign class object serves as a convinient portal to access all components in APO workflow, including data management.
+The `Campaign` class object acts as the central hub, seamlessly connecting all components within the APO workflow, including data management, optimizer, and experimental designer.
+It is the recommended approach that offers a more streamlined workflow compared to utilizing each component separately.
+
+
+
+Here is an example of creating a `Campaign` class object and adding the initial dataset to its 'data' attribute:
+
+```python
+from obsidian.campaign import Campaign
+
+data_Iter0 = pd.concat([X0, y0], axis=1)
+my_campaign = Campaign(X_space, target, seed=0)
+my_campaign.add_data(data_Iter0)
+```
+
+The 'add_data' method will append each new batch of data to one pandas dataframe with incremental integer 'Iteration'. The new data should be a dataframe contains both the input experimental conditions and the target outcomes.
+
+
+There are various ways to retrieve data from `Campaign`:
-...
+```python
+print(my_campaign.data.to_markdown())
+```
+
+| Observation ID | Temperature | Concentration | Enzyme | Variant | StirRate | Yield | Iteration |
+|-----------------:|--------------:|----------------:|----------:|:----------|:-----------|--------:|------------:|
+| 0 | 13.3333 | 68.3333 | 0.2275 | MRK003 | High | 47.4471 | 0 |
+| 1 | 6.66667 | 115 | 0.0825 | MRK003 | Low | 61.3989 | 0 |
+| 2 | 26.6667 | 45 | 0.0341667 | MRK002 | Medium | 63.6213 | 0 |
+| 3 | 20 | 91.6667 | 0.275833 | MRK001 | Low | 43.4116 | 0 |
+| 4 | -6.66667 | 21.6667 | 0.179167 | MRK002 | Medium | 84.5542 | 0 |
+| 5 | 0 | 138.333 | 0.130833 | MRK001 | High | 51.8577 | 0 |
+
+and
+```python
+my_campaign.X
+my_campaign.y
+```
\ No newline at end of file
From 6f2e8836a5a042596be3683136632fa3993a4037 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Wed, 14 Aug 2024 18:31:15 -0400
Subject: [PATCH 005/136] fix the demo ipynb files and remove outputs to reduce
file sizes. Outputs could be found in gh pages
---
demo/Constrained multi objective.ipynb | 51 +-
...int optimization with constant param.ipynb | 32 +-
demo/Simple single objective.ipynb | 3417 +----------------
3 files changed, 82 insertions(+), 3418 deletions(-)
diff --git a/demo/Constrained multi objective.ipynb b/demo/Constrained multi objective.ipynb
index c83070b..56814c2 100644
--- a/demo/Constrained multi objective.ipynb
+++ b/demo/Constrained multi objective.ipynb
@@ -87,6 +87,27 @@
"Z0.plot(x='Response 1', y='Response 2', kind='scatter', figsize=(4,3))"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define the Target"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from obsidian.parameters import Target\n",
+ "\n",
+ "target = [\n",
+ " Target('Response 1', aim='max'),\n",
+ " Target('Response 2', aim='min')\n",
+ "]"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -109,7 +130,7 @@
"metadata": {},
"outputs": [],
"source": [
- "my_campaign = Campaign(X_space)\n",
+ "my_campaign = Campaign(X_space, target)\n",
"my_campaign.add_data(Z0)\n",
"my_campaign.data"
]
@@ -127,22 +148,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from obsidian.parameters import Target"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "target = [\n",
- " Target('Response 1', aim='max'),\n",
- " Target('Response 2', aim='min')\n",
- "]\n",
- "\n",
- "my_campaign.set_target(target)\n",
- "my_campaign.fit()\n"
+ "my_campaign.fit()"
]
},
{
@@ -168,10 +174,10 @@
"outputs": [],
"source": [
"# # X1 + X2 >= 2\n",
- "# optim_kwargs = {'m_batch':2, 'acquisition':[{'qNEHVI':{'ref_point':[-350,0]}}], 'ineq_constraints':[InConstraint_Generic(X_space, indices=[0,1], coeff=[1,1], rhs=2)]}\n",
+ "# optim_kwargs = {'m_batch':2, 'acquisition':[{'NEHVI':{'ref_point':[-350,0]}}], 'ineq_constraints':[InConstraint_Generic(X_space, indices=[0,1], coeff=[1,1], rhs=2)]}\n",
"\n",
"# X1 + X2 <= 6 aka -X1 - X2 >= -6\n",
- "optim_kwargs = {'m_batch':2, 'acquisition':[{'qNEHVI':{'ref_point':[-350,0]}}], 'ineq_constraints':[InConstraint_Generic(X_space, indices=[0,1], coeff=[-1,-1], rhs=-6)]}\n",
+ "optim_kwargs = {'m_batch':2, 'acquisition':[{'NEHVI':{'ref_point':[-350,0]}}], 'ineq_constraints':[InConstraint_Generic(X_space, indices=[0,1], coeff=[-1,-1], rhs=-6)]}\n",
"\n",
"X_suggest, eval_suggest = my_campaign.optimizer.suggest(**optim_kwargs)"
]
@@ -244,6 +250,13 @@
"my_campaign.data"
]
},
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
{
"cell_type": "code",
"execution_count": null,
diff --git a/demo/Joint optimization with constant param.ipynb b/demo/Joint optimization with constant param.ipynb
index b6b620e..8174785 100644
--- a/demo/Joint optimization with constant param.ipynb
+++ b/demo/Joint optimization with constant param.ipynb
@@ -88,6 +88,23 @@
"Z0.plot(x='Temperature', y='Yield', kind='scatter', figsize=(4,3))"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define the Target"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from obsidian.parameters import Target\n",
+ "target = Target('Yield', aim='max')"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {},
@@ -110,7 +127,7 @@
"metadata": {},
"outputs": [],
"source": [
- "my_campaign = Campaign(X_space)\n",
+ "my_campaign = Campaign(X_space,target)\n",
"my_campaign.add_data(Z0)\n",
"my_campaign.data"
]
@@ -128,18 +145,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from obsidian.parameters import Target"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "target = Target('Yield', aim='max')\n",
- "my_campaign.set_target(target)\n",
- "my_campaign.fit()\n"
+ "my_campaign.fit()"
]
},
{
diff --git a/demo/Simple single objective.ipynb b/demo/Simple single objective.ipynb
index dd31bfd..51f2b2c 100644
--- a/demo/Simple single objective.ipynb
+++ b/demo/Simple single objective.ipynb
@@ -2,17 +2,9 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "obsidian version: 0.7.10\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"import obsidian\n",
"print(f'obsidian version: ' + obsidian.__version__)\n",
@@ -32,7 +24,7 @@
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
@@ -42,141 +34,9 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": null,
"metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
In this tutorial, we will see how to use obsidian for multi-output optimization. To demonstrate the versatility of the approach, we will seek to maximize one response while minimizing the other.
GP model has been fit to data with a train-score of: 1 for response: Response 1
+GP model has been fit to data with a train-score of: 1 for response: Response 2
+
Note: It is a good idea to balance a set of acquisition functions with those that prefer design-space exploration. This helps to ensure that the optimizer is not severely misled by deficiencies in the dataset, particularly for small data. It also helps to ascertain a global optimum.
+
A simple choice is Space Filling (SF) although Negative Integrated Posterior Variance (NIPV) is available for single-output optimizations; and there are various other acquisiiton functions whose hyperparameters can be tuned to manage the "explore-exploit" balance.
+
+
+
+
+
+
+
+
+
In [9]:
+
+
+
X_suggest,eval_suggest=campaign.optimizer.suggest(acquisition=[{'NEHVI':{'ref_point':[-350,-20]}},'SF'],
+ # X1 + X2 <= 6, written as -X1 - X2 >= -6
+ ineq_constraints=[InConstraint_Generic(X_space,indices=[0,1],coeff=[-1,-1],rhs=-6)])
+
GP model has been fit to data with a train-score of: 0.998 for response: Response 1
+GP model has been fit to data with a train-score of: 0.999 for response: Response 2
+GP model has been fit to data with a train-score of: 1 for response: Response 1
+GP model has been fit to data with a train-score of: 1 for response: Response 2
+GP model has been fit to data with a train-score of: 1 for response: Response 1
+GP model has been fit to data with a train-score of: 1 for response: Response 2
+GP model has been fit to data with a train-score of: 0.999 for response: Response 1
+GP model has been fit to data with a train-score of: 1 for response: Response 2
+GP model has been fit to data with a train-score of: 1 for response: Response 1
+GP model has been fit to data with a train-score of: 1 for response: Response 2
+
+
+
+
+
+
+
+
+
+
+
In [13]:
+
+
+
optim_progress(campaign)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
In [14]:
+
+
+
surface_plot(campaign.optimizer,response_id=0)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
In [15]:
+
+
+
surface_plot(campaign.optimizer,response_id=1)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/_static/tutorials/Simple single objective.html b/docs/_static/tutorials/Simple single objective.html
index 4fdd762..f15cef3 100644
--- a/docs/_static/tutorials/Simple single objective.html
+++ b/docs/_static/tutorials/Simple single objective.html
@@ -7536,7 +7536,7 @@
-
obsidian version: 0.7.10
+
obsidian version: 0.8.0
@@ -7587,7 +7587,7 @@
Set up parameter space a
X_space=ParamSpace(params)target=Target('Yield',aim='max')campaign=Campaign(X_space,target,seed=0)
-X0=campaign.designer.initialize(10,'LHS')
+X0=campaign.initialize(m_initial=10,method='LHS')X0
In this tutorial, we will examine usage of obsidian for performing a cost-penalized optimization using a tailored objective function.
+
Often times, it is desireable to generate an objective function based on the input data X. Rather than build a model to a response calculated off-line, it is best to capture the analytical form where possible. The custom objective Feature_Objective simply allows the user to index the input variables and multiply them by coefficient(s) to generate a new objective function. In this example, we will create a Feature_Objective based off of "Enzyme" loading, to simulate an optimization where product yield might be weighed against an expensive input.
+
The optimization problem then becomes multi-output: "Product" and "Penalized Enzyme." However, we can combine these further to create a single objective function using a Scalarization. In the simplest case, we may want to add the two objectives with equal weights, which would be the default behavior of Scalarize_WeightedSum.
+
In obsidian, we combine a sequence of objectives using Objective_Sequence. Thus, finally, the final objective function is objective = Objective_Sequence([Feature_Objective, Scalarize_WeightedSum]) and single-output acquisition functions may be used to select optimal experiments.
Note: Objectives can be passed directly to an Optimizer, or set using campaign.set_objective() after which the Campaign will automatically use the objective durign campaign.suggest(). At any time, the objective can be re-set to a new objective, or deleted using campaign.clear_objective().
c:\Users\kevin\miniconda3\envs\obsidian-dev\lib\site-packages\botorch\optim\optimize.py:564: RuntimeWarning:
+
+Optimization failed in `gen_candidates_scipy` with the following warning(s):
+[OptimizationWarning('Optimization failed within `scipy.optimize.minimize` with status 2 and message ABNORMAL_TERMINATION_IN_LNSRCH.')]
+Trying again with a new set of initial conditions.
+
+c:\Users\kevin\miniconda3\envs\obsidian-dev\lib\site-packages\botorch\optim\optimize.py:564: RuntimeWarning:
+
+Optimization failed on the second try, after generating a new set of initial conditions.
+
+
Note: We can examine the output of various objectives within the sequence by passing them directly to optimizer.evaluate. Here, we can explicitly see the balance of Objective 1 (product response) and Objective 2 (cost penalty) before they are combined in the weighted sum.
GP model has been fit to data with a train-score of: 1 for response: Product
+GP model has been fit to data with a train-score of: 1 for response: Product
+GP model has been fit to data with a train-score of: 1 for response: Product
+
+
+
+
+
+
+
+
+
+
+
+
+
Examine the optimization progress from the context of different elements of the compositve objective function
+
First, the final objective - a weighted sum of product yield and a cost-penalized input.
Next, we can specifically examine the context of the multi-output optimization minimizing (maximizing negative) cost (Objective 2) and maximizing product (Objective 1).
"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "outputs": [],
"source": [
"campaign.clear_objective()\n",
"optim_progress(campaign)"
From 8dd3ca58a5b1e2621cd2577827a4dbb6b81b7906 Mon Sep 17 00:00:00 2001
From: Kevin Stone <91576158+kstone40@users.noreply.github.com>
Date: Thu, 15 Aug 2024 00:18:30 -0400
Subject: [PATCH 015/136] Move JS code to end of readme under collapsible
---
readme.md | 22 ++++++++++------------
1 file changed, 10 insertions(+), 12 deletions(-)
diff --git a/readme.md b/readme.md
index d506237..80e15ff 100644
--- a/readme.md
+++ b/readme.md
@@ -3,19 +3,7 @@ obsidian
ReadMe
-->
-
-
-
-
@@ -149,3 +137,13 @@ See [CONTRIBUTING](https://github.com/MSDLLCpapers/obsidian/blob/main/CONTRIBUTI
## License
__obsidian__ is licensed by the [GPLv3 license](https://github.com/MSDLLCpapers/obsidian/blob/main/LICENSE).
+
+JavaScript for ReadMe dark mode
+
+
From e0699cb1e0b8f97faf6b8518b5dd035f4db88903 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Thu, 15 Aug 2024 00:59:40 -0400
Subject: [PATCH 016/136] bugfix: change the SurrogateBoTorch class instance
attribute self.score to self.r2_score, so that it won't has the same name as
the instance method self.score() function
---
obsidian/surrogates/botorch.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/obsidian/surrogates/botorch.py b/obsidian/surrogates/botorch.py
index 9ddfaa0..c1e57ea 100644
--- a/obsidian/surrogates/botorch.py
+++ b/obsidian/surrogates/botorch.py
@@ -206,7 +206,7 @@ def score(self,
# Calculate a final loss and R2 train score
loss = self.loss = self.loss_fcn(self.torch_model(X_p), y_p).sum().detach().cpu().data.numpy().tolist()
- score = self.score = corr_matrix[0][1]**2
+ score = self.r2_score = corr_matrix[0][1]**2
return loss, score
From ea6ae06a2432aa727dbefe19f5b19cfe1ebc96f6 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Thu, 15 Aug 2024 01:25:42 -0400
Subject: [PATCH 017/136] finish revising docs/wiki/4_SurrogateModel.md to make
sure all examples are runnable and all statements are correct
---
docs/wiki/4_SurrogateModel.md | 175 ++++++++++++++++++++--------------
1 file changed, 103 insertions(+), 72 deletions(-)
diff --git a/docs/wiki/4_SurrogateModel.md b/docs/wiki/4_SurrogateModel.md
index d0f37d0..4ae7359 100644
--- a/docs/wiki/4_SurrogateModel.md
+++ b/docs/wiki/4_SurrogateModel.md
@@ -4,62 +4,103 @@
The `obsidian.surrogates` submodule is a key component of the Obsidian Bayesian optimization library. It provides a collection of surrogate models used to approximate the objective function in the optimization process. These surrogate models are essential for efficient exploration of the parameter space and for making informed decisions about which points to evaluate next.
-## 2. Available Surrogate Models
+## 2. How to Use Surrogate Models - Basic Syntax
-The `obsidian.surrogates` submodule offers several types of surrogate models:
+To use a surrogate model in your optimization process, you typically don't need to interact with it directly. The `obsidian.optimizer` submodule will handle the creation and management of the surrogate model. However, if you need to create a surrogate model manually, you can do so using the `SurrogateBoTorch` class.
-1. **Gaussian Process (GP)**: The default surrogate model, suitable for most optimization tasks.
-2. **Mixed Gaussian Process (MixedGP)**: A GP model that can handle mixed continuous and categorical input spaces.
-3. **Deep Kernel Learning GP (DKL)**: A GP model with a neural network feature extractor.
-4. **Flat GP**: A GP model with non-informative or no prior distributions.
-5. **Prior GP**: A GP model with custom prior distributions.
-6. **Multi-Task GP (MTGP)**: A GP model for multi-output optimization.
-7. **Deep Neural Network (DNN)**: A dropout neural network model.
+Below is a simple example using the default standard GP surrogate:
-## 3. How to Use Surrogate Models
-
-To use a surrogate model in your optimization process, you typically don't need to interact with it directly. The Obsidian optimizer will handle the creation and management of the surrogate model. However, if you need to create a surrogate model manually, you can do so using the `SurrogateBoTorch` class:
+### Define the parameter space:
+```python
+from obsidian.parameters import ParamSpace, Param_Continuous
+params = ParamSpace([Param_Continuous('X1', 0, 1),Param_Continuous('X2', 0, 1)])
+X_space = ParamSpace(params)
+```
+### Simulate training data:
```python
-from obsidian.surrogates import SurrogateBoTorch
-from obsidian.parameters import ParamSpace, Target
+from obsidian.experiment import ExpDesigner
+designer = ExpDesigner(X_space, seed = 789)
+X_train = designer.initialize(m_initial = 10, method='Sobol')
-# Define your parameter space
-param_space = ParamSpace([...]) # Define your parameters here
+from obsidian.parameters import Target
+target = Target(name = 'Y', f_transform = 'Standard', aim='max')
-# Create a surrogate model (default is GP)
-surrogate = SurrogateBoTorch(model_type='GP')
+from obsidian.experiment import Simulator
+from obsidian.experiment.benchmark import paraboloid
+simulator = Simulator(X_space, paraboloid, name='Y')
+y_train = simulator.simulate(X_train)
+y_train_transformed = target.transform_f(y_train, fit = True)
-# Fit the model to your data
-surrogate.fit(X, y)
+import pandas as pd
+print(pd.concat([X_train, y_train_transformed], axis=1).to_markdown())
+```
-# Make predictions
+| | X1 | X2 | Y Trans |
+|---:|---------:|----------:|----------:|
+| 0 | 0.709874 | 0.891838 | -0.692069 |
+| 1 | 0.316783 | 0.0374523 | -1.283 |
+| 2 | 0.102254 | 0.517022 | -0.229447 |
+| 3 | 0.99438 | 0.412149 | -1.33756 |
+| 4 | 0.80163 | 0.663557 | 0.252902 |
+| 5 | 0.162435 | 0.265744 | -0.351742 |
+| 6 | 0.385264 | 0.788242 | 0.507141 |
+| 7 | 0.523473 | 0.140918 | 0.113744 |
+| 8 | 0.588516 | 0.604981 | 1.423 |
+| 9 | 0.445465 | 0.465715 | 1.59703 |
+
+
+### Create a surrogate model (using the default 'GP' model), and fit the model to the training data:
+```python
+from obsidian.surrogates import SurrogateBoTorch
+surrogate = SurrogateBoTorch(model_type='GP', seed = 123)
+surrogate.fit(X_train, y_train_transformed,cat_dims=[],task_feature=None)
+```
+
+### Generate new input experimental conditions and make predictions:
+```python
+X_new = designer.initialize(m_initial = 3, method='Sobol')
mean, std = surrogate.predict(X_new)
+
+df = X_new.assign(pred_mean=mean,pred_std=std)
+print(df.to_markdown())
```
-## 4. Customization Options
+| | X1 | X2 | pred_mean | pred_std |
+|---:|---------:|----------:|------------:|-----------:|
+| 0 | 0.709874 | 0.891838 | -0.669406 | 0.120078 |
+| 1 | 0.316783 | 0.0374523 | -1.25515 | 0.119587 |
+| 2 | 0.102254 | 0.517022 | -0.217739 | 0.12007 |
+
-### 4.1 Model Selection
+## 3. Customization Options
-You can choose different surrogate models by specifying the `model_type` parameter when creating a `SurrogateBoTorch` instance. Available options are:
+### 3.1 Available Surrogate Models
-- `'GP'`: Standard Gaussian Process
-- `'MixedGP'`: Mixed input Gaussian Process
-- `'DKL'`: Deep Kernel Learning GP
-- `'GPflat'`: Flat (non-informative prior) GP
-- `'GPprior'`: Custom prior GP
-- `'MTGP'`: Multi-Task GP
-- `'DNN'`: Dropout Neural Network
+The `obsidian.surrogates` submodule offers several types of surrogate models.
+You can choose different surrogate models by specifying the `model_type` parameter when creating a `SurrogateBoTorch` instance.
+Available options are:
-### 4.2 Hyperparameters
+- `'GP'`: Standard Gaussian Process, which is the default surrogate model suitable for most optimization tasks.
+- `'MixedGP'`: Mixed input Gaussian Process, which is a GP model that can handle mixed continuous and categorical input spaces.
+- `'DKL'`: Deep Kernel Learning GP, which is a GP model with a neural network feature extractor.
+- `'GPflat'`: A GP model with non-informative or no prior distributions.
+- `'GPprior'`: A GP model with custom prior distributions.
+- `'MTGP'`: Multi-Task GP, which is a GP model for multi-output optimization.
+- `'DNN'`: Dropout Neural Network model.
-You can pass custom hyperparameters to the surrogate model using the `hps` parameter:
+### 3.2 Hyperparameters
+You can pass custom hyperparameters to the surrogate model using the `hps` argument as:
+> surrogate = SurrogateBoTorch(model_type='GP', hps={'custom_param_1': value_1, 'custom_param_2': value_2, ...})
+
+Some specific examples:
```python
-surrogate = SurrogateBoTorch(model_type='GP', hps={'your_custom_param': value})
+surrogate = SurrogateBoTorch(model_type='FlatGP', hps={'nu': 1.5})
+surrogate = SurrogateBoTorch(model_type='DNN', hps={'p_dropout': 0.1, 'h_width': 15, 'h_layers': 4, 'num_outputs': 2})
```
-### 4.3 Custom GP Models
+### 3.3 Custom GP Models
The submodule provides several custom GP implementations:
@@ -67,70 +108,60 @@ The submodule provides several custom GP implementations:
- `FlatGP`: A GP with non-informative or no prior distributions
- `DKLGP`: A GP with a neural network feature extractor
-### 4.4 Custom Neural Network Model
-
-The `DNN` class provides a customizable dropout neural network model. You can adjust parameters such as dropout probability, hidden layer width, and number of hidden layers.
-
-## 5. Examples
-
-### 5.1 Using a standard GP surrogate
-
-```python
-from obsidian.surrogates import SurrogateBoTorch
-from obsidian.parameters import ParamSpace, Target
-import pandas as pd
-
-# Define your parameter space
-param_space = ParamSpace([...]) # Define your parameters here
+### 3.4 Custom Neural Network Model
-# Assume X and y are your input features and target variables
-X = pd.DataFrame(...)
-y = pd.Series(...)
+The `DNN` class provides a customizable dropout neural network model.
+The 'hps' parameter allows you to customize the DNN architecture.
+You can adjust multiple DNN hyperparameters such as dropout probability, hidden layer width, and number of hidden layers.
-surrogate = SurrogateBoTorch(model_type='GP')
-surrogate.fit(X, y)
+## 4. Additional Examples
-# Make predictions
-X_new = pd.DataFrame(...)
-mean, std = surrogate.predict(X_new)
-```
-### 5.2 Using a Mixed GP for categorical and continuous variables
+### 4.1 Using a Mixed GP for categorical and continuous variables
```python
+# DO NOT RUN
surrogate = SurrogateBoTorch(model_type='MixedGP')
# cat_dims should be a list of indices for categorical variables in your input space
surrogate.fit(X, y, cat_dims=[0, 2]) # Assuming columns 0 and 2 are categorical
```
-### 5.3 Using a DNN surrogate
+### 4.2 Using a DNN surrogate
```python
-# The 'hps' parameter allows you to customize the DNN architecture
surrogate = SurrogateBoTorch(model_type='DNN', hps={'p_dropout': 0.1, 'h_width': 32, 'h_layers': 3})
-surrogate.fit(X, y)
+surrogate.fit(X_train, y_train_transformed,cat_dims=[],task_feature=None)
```
-## 6. Advanced Usage
+## 5. Advanced Usage
-### 6.1 Saving and Loading Models
+### 5.1 Saving and Loading Models
-You can save and load surrogate models using the `save_state()` and `load_state()` methods:
+You can save and load surrogate models to/from dictionary objects using the `save_state()` and `load_state()` methods,
+which enable saving the trained model to json files and reload for future usage:
```python
-# Save model state
-state = surrogate.save_state()
+import json
+
+# Save model state as dictionary to json file
+with open('surrogate.json', 'w') as f:
+ surrogate_dict = surrogate.save_state()
+ json.dump(surrogate_dict, f)
-# Load model state
-loaded_surrogate = SurrogateBoTorch.load_state(state)
+# Load model state dictionary from json file
+with open('surrogate.json', 'r') as f:
+ surrogate_dict = json.load(f)
+ surrogate_reload = SurrogateBoTorch.load_state(surrogate_dict)
```
-### 6.2 Model Evaluation
+### 5.2 Model Evaluation
You can evaluate the performance of a surrogate model using the `score()` method:
```python
-loss, r2_score = surrogate.score(X_test, y_test)
+y_new = simulator.simulate(X_new)
+y_new_transformed = target.transform_f(y_new, fit = False)
+loss, r2_score = surrogate.score(X_new, y_new_transformed)
```
This concludes the user guide for the `obsidian.surrogates` submodule. For more detailed information, please refer to the source code and docstrings in the individual files.
\ No newline at end of file
From bbb447fcbcdd55462b7686cdcb7f534b6a0f9fae Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Thu, 15 Aug 2024 01:45:01 -0400
Subject: [PATCH 018/136] Update botorch.py: revise attribute score to r2_score
to avoid conflict with instance method score()
---
obsidian/surrogates/botorch.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/obsidian/surrogates/botorch.py b/obsidian/surrogates/botorch.py
index c1e57ea..109954b 100644
--- a/obsidian/surrogates/botorch.py
+++ b/obsidian/surrogates/botorch.py
@@ -46,7 +46,7 @@ class SurrogateBoTorch(SurrogateModel):
mll (ExactMarginalLogLikelihood): The marginal log likelihood of the model.
torch_model (torch.nn.Module): The torch model for the surrogate.
loss (float): The loss of the model.
- score (float): The R2 score of the model.
+ r2_score (float): The R2 score of the model.
"""
def __init__(self,
model_type: str = 'GP',
From 8acc99a42b8adaa267a334eb217fb0ad8cab7c48 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Thu, 15 Aug 2024 01:47:36 -0400
Subject: [PATCH 019/136] Update bayesian.py: revise 'score' attribute in
surrogate to 'r2_score' to avoid conflict with instance method score()
---
obsidian/optimizer/bayesian.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/obsidian/optimizer/bayesian.py b/obsidian/optimizer/bayesian.py
index 7f8dfb0..343b26b 100644
--- a/obsidian/optimizer/bayesian.py
+++ b/obsidian/optimizer/bayesian.py
@@ -244,7 +244,7 @@ def fit(self,
if self.verbose > 0:
print(f'{self.surrogate_type[i]} model has been fit \
- to data with a train-score of: {self.surrogate[i].score:.3g} for response: {self.y_names[i]}')
+ to data with a train-score of: {self.surrogate[i].r2_score:.3g} for response: {self.y_names[i]}')
return
def save_state(self) -> dict:
From 4edb917b9585d3b511a6afd560d7e13d3bd9f777 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Thu, 15 Aug 2024 01:48:36 -0400
Subject: [PATCH 020/136] Update optimize.py: revise 'score' attribute in
surrogate to 'r2_score' to avoid conflict with instance method score()
---
obsidian/dash/optimize.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/obsidian/dash/optimize.py b/obsidian/dash/optimize.py
index cb45263..70cd171 100644
--- a/obsidian/dash/optimize.py
+++ b/obsidian/dash/optimize.py
@@ -117,7 +117,7 @@ def fit_statistics(opt_save, config, filename):
[
dbc.ListGroupItem(['Model Type: ', f'{optimizer.surrogate_type}']),
dbc.ListGroupItem(['Data Name: ', filename]),
- dbc.ListGroupItem(['R', html.Sup('2'), ' Score: ', f'{optimizer.surrogate[0].score: .4g}']), # for SOO only
+ dbc.ListGroupItem(['R', html.Sup('2'), ' Score: ', f'{optimizer.surrogate[0].r2_score: .4g}']), # for SOO only
dbc.ListGroupItem(['Marginal Log Likelihood: ', f'{optimizer.surrogate[0].loss: .4g}']), # for SOO only
], flush=True
)
From 010d60da364d4e8798fec5a9286760680d6c781e Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Thu, 15 Aug 2024 17:09:37 -0400
Subject: [PATCH 021/136] Add publications tab to document
---
docs/_static/publications.md | 14 ++++++++++++++
docs/index.rst | 2 ++
docs/stubs/publications.rst | 12 ++++++++++++
3 files changed, 28 insertions(+)
create mode 100644 docs/_static/publications.md
create mode 100644 docs/stubs/publications.rst
diff --git a/docs/_static/publications.md b/docs/_static/publications.md
new file mode 100644
index 0000000..749b5af
--- /dev/null
+++ b/docs/_static/publications.md
@@ -0,0 +1,14 @@
+# Papers
+
+* Christensen, M., Xu, Y., Kwan, E.E., Di Maso, M.J., Ji, Y., Reibarkh, M., Sun, A.C., Liaw, A., Fier, P.S., Grosser, S. and Hein, J.E., 2024.
+_Dynamic sampling in autonomous process optimization. Chemical Science, 15(19), pp.7160-7169._
+DOI: [10.1039/D3SC06884F](https://pubs.rsc.org/en/content/articlelanding/2024/sc/d3sc06884f)
+
+
+# Presentations
+
+* The 2024 Joint Statistical Meetings (JSM), Portland, Aug. 5, 2024.
+ Poster at Biopharmaceutical Section: _Algorithmic Process Optimization in Pharmaceutical Development_
+
+
+
diff --git a/docs/index.rst b/docs/index.rst
index 62a7540..c1d9270 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -16,6 +16,8 @@
stubs/tutorials
+ stubs/publications
+
stubs/reference
.. include:: ../readme.md
diff --git a/docs/stubs/publications.rst b/docs/stubs/publications.rst
new file mode 100644
index 0000000..7e99b3d
--- /dev/null
+++ b/docs/stubs/publications.rst
@@ -0,0 +1,12 @@
+Publications
+=========
+
+.. include:: ../_static/publications.md
+ :parser: myst_parser.sphinx_
+
+Indices
+~~~~~~~~~~~~~~~~~~
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
\ No newline at end of file
From 69791b28b4cc6cdb988e3cd94dbb17cf4dcdc49c Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Fri, 16 Aug 2024 10:46:00 -0400
Subject: [PATCH 022/136] modifying user guide of acquisition functions (in
progress)
---
docs/wiki/4_SurrogateModel.md | 4 +--
docs/wiki/5_AcquisitionFunction.md | 39 +++++++++++++++++++++++++-----
2 files changed, 35 insertions(+), 8 deletions(-)
diff --git a/docs/wiki/4_SurrogateModel.md b/docs/wiki/4_SurrogateModel.md
index 4ae7359..1a9c3e4 100644
--- a/docs/wiki/4_SurrogateModel.md
+++ b/docs/wiki/4_SurrogateModel.md
@@ -2,9 +2,9 @@
## 1. Introduction
-The `obsidian.surrogates` submodule is a key component of the Obsidian Bayesian optimization library. It provides a collection of surrogate models used to approximate the objective function in the optimization process. These surrogate models are essential for efficient exploration of the parameter space and for making informed decisions about which points to evaluate next.
+The `obsidian.surrogates` submodule is a key component of the Obsidian APO library. It provides a collection of surrogate models used to approximate the objective function in the optimization process. These surrogate models are essential for efficient exploration of the parameter space and for making informed decisions about which points to evaluate next.
-## 2. How to Use Surrogate Models - Basic Syntax
+## 2. Basic Syntax
To use a surrogate model in your optimization process, you typically don't need to interact with it directly. The `obsidian.optimizer` submodule will handle the creation and management of the surrogate model. However, if you need to create a surrogate model manually, you can do so using the `SurrogateBoTorch` class.
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 389b17d..e9a0c6d 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -2,13 +2,37 @@
## 1. Introduction
-The `obsidian.acquisition` submodule is a crucial component of the Obsidian Bayesian optimization library. It provides acquisition functions that guide the optimization process by determining which points in the parameter space should be evaluated next. These acquisition functions balance exploration of uncertain areas and exploitation of promising regions, which is key to efficient optimization.
+The [`obsidian.acquisition`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/acquisition) submodule is a crucial component of the Obsidian APO library. It provides acquisition functions that guide the optimization process by determining which points in the parameter space should be evaluated next. These acquisition functions balance exploration of uncertain areas and exploitation of promising regions, which is key to efficient optimization.
-## 2. Key Components
+## 2. Basic Syntax
-The acquisition submodule includes several acquisition functions, both standard and custom implementations:
+Typically, users don't need to interact with acquisition functions directly.
+The `BayesianOptimizer` class handles the selection and use of acquisition functions.
+The acquisition function, including its hyperparameters, could be specified as an input argument when calling the `suggest` method:
-### 2.1 Standard Acquisition Functions
+```python
+# DO NOT RUN
+from obsidian.optimizer import BayesianOptimizer
+
+optimizer = BayesianOptimizer(X_space=param_space)
+
+# Use one acquisition function EI per iteration
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI'])
+
+# Use two acquisition functions EI and UCB per iteration
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI','UCB'])
+
+# Use two acquisition functions EI and UCB per iteration, while specifying hyperparameters for UCB
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI',{'UCB':{'beta':0.1}}])
+```
+
+## 3. Customization Options
+
+### 3.1 Available Surrogate Models and Hyperparameters
+
+The acquisition submodule includes multiple acquisition functions, both standard `BoTorch` acquisition functions and custom implementations.
+
+#### Standard Acquisition Functions
- Expected Improvement (EI)
- Probability of Improvement (PI)
@@ -17,12 +41,15 @@ The acquisition submodule includes several acquisition functions, both standard
- Expected Hypervolume Improvement (EHVI)
- Noisy Expected Hypervolume Improvement (NEHVI)
-### 2.2 Custom Acquisition Functions
+#### Custom Acquisition Functions
- qMean: Optimizes for the maximum value of the posterior mean
- qSpaceFill: Optimizes for the maximum value of minimum distance between a point and the training data
-## 3. Understanding Acquisition Functions
+
+---------------
+
+## 4. Understanding Acquisition Functions
### 3.1 Expected Improvement (EI)
From 3f21b929f6db8bf93caff0c3d8f3fb1ee2e064b1 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Fri, 16 Aug 2024 11:53:14 -0400
Subject: [PATCH 023/136] continue editing 5_AcquisitionFunction.md
---
docs/wiki/5_AcquisitionFunction.md | 42 +++++++++++++++++++++---------
1 file changed, 29 insertions(+), 13 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index e9a0c6d..742046e 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -8,7 +8,9 @@ The [`obsidian.acquisition`](https://github.com/MSDLLCpapers/obsidian/tree/main/
Typically, users don't need to interact with acquisition functions directly.
The `BayesianOptimizer` class handles the selection and use of acquisition functions.
-The acquisition function, including its hyperparameters, could be specified as an input argument when calling the `suggest` method:
+The acquisition functions, as well as their hyperparameters, could be specified as an input argument when calling the `suggest` method.
+
+The format of `acquisition` should be a list of strings (if using default hyperparameters) and/or dictionaries, where the key is an acquisition function name and the value is a nested dictionary storing its hyperparameter:
```python
# DO NOT RUN
@@ -28,28 +30,42 @@ X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI',{'UCB':{'beta':0.1
## 3. Customization Options
-### 3.1 Available Surrogate Models and Hyperparameters
+### 3.1 Available Acquisition Function Names
-The acquisition submodule includes multiple acquisition functions, both standard `BoTorch` acquisition functions and custom implementations.
+The acquisition submodule includes multiple acquisition functions, both standard `BoTorch` acquisition functions and custom implementations.
+Below are several categories of options for the `acquisition` argument, each followed by a short description.
-#### Standard Acquisition Functions
+#### Standard Acquisition Functions from BoTorch
-- Expected Improvement (EI)
-- Probability of Improvement (PI)
-- Upper Confidence Bound (UCB)
-- Noisy Expected Improvement (NEI)
-- Expected Hypervolume Improvement (EHVI)
-- Noisy Expected Hypervolume Improvement (NEHVI)
+- EI: Expected Improvement
+- NEI: Noisy Expected Improvement
+- PI: Probability of Improvement
+- UCB: Upper Confidence Bound
+- SR: Simple Regret
+- EHVI: Expected Hypervolume Improvement
+- NEHVI: Noisy Expected Hypervolume Improvement
+- NParEGO: Random augmented chebyshev scalarization with Noisy Expected Improvement
+- NIPV: Integrated Negative Posterior Variance
#### Custom Acquisition Functions
-- qMean: Optimizes for the maximum value of the posterior mean
-- qSpaceFill: Optimizes for the maximum value of minimum distance between a point and the training data
+- Mean: qMean: Optimizes for the maximum value of the posterior mean
+- SF: qSpaceFill: Optimizes for the maximum value of minimum distance between a point and the training data
+
+#### Baseline: No Acquisition Function
+
+- RS: Random sampling from the parameter space
+
+
+### 3.2 Shared Hyperparameters
+
+There are some commonly used hyperparameters that could apply to multiple acquisition functions.
+
---------------
-## 4. Understanding Acquisition Functions
+## 4. Understanding Acquisition Functions and Hyperparameters
### 3.1 Expected Improvement (EI)
From 52727369187e6d6df20f9221d57b9c6b052558fe Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sat, 17 Aug 2024 01:41:35 -0400
Subject: [PATCH 024/136] minor changes to 5_AcquisitionFunction.md (in
progress)
---
docs/wiki/5_AcquisitionFunction.md | 73 +++++++++++++++---------------
1 file changed, 37 insertions(+), 36 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 742046e..8b9f3b5 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -4,68 +4,69 @@
The [`obsidian.acquisition`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/acquisition) submodule is a crucial component of the Obsidian APO library. It provides acquisition functions that guide the optimization process by determining which points in the parameter space should be evaluated next. These acquisition functions balance exploration of uncertain areas and exploitation of promising regions, which is key to efficient optimization.
-## 2. Basic Syntax
+## 2. Usage
-Typically, users don't need to interact with acquisition functions directly.
-The `BayesianOptimizer` class handles the selection and use of acquisition functions.
-The acquisition functions, as well as their hyperparameters, could be specified as an input argument when calling the `suggest` method.
-
-The format of `acquisition` should be a list of strings (if using default hyperparameters) and/or dictionaries, where the key is an acquisition function name and the value is a nested dictionary storing its hyperparameter:
-
-```python
-# DO NOT RUN
-from obsidian.optimizer import BayesianOptimizer
-
-optimizer = BayesianOptimizer(X_space=param_space)
-
-# Use one acquisition function EI per iteration
-X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI'])
+### 2.1 Available Acquisition Functions
-# Use two acquisition functions EI and UCB per iteration
-X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI','UCB'])
-
-# Use two acquisition functions EI and UCB per iteration, while specifying hyperparameters for UCB
-X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI',{'UCB':{'beta':0.1}}])
-```
-
-## 3. Customization Options
-
-### 3.1 Available Acquisition Function Names
-
-The acquisition submodule includes multiple acquisition functions, both standard `BoTorch` acquisition functions and custom implementations.
-Below are several categories of options for the `acquisition` argument, each followed by a short description.
+The acquisition submodule offers a range of acquisition functions, including standard `BoTorch` acquisition functions and custom implementations.
+Below, you will find various options for the acquisition argument, along with a brief description for each.
#### Standard Acquisition Functions from BoTorch
+_Single-Objective Optimization:_
+
- EI: Expected Improvement
- NEI: Noisy Expected Improvement
- PI: Probability of Improvement
- UCB: Upper Confidence Bound
- SR: Simple Regret
+- NIPV: Integrated Negative Posterior Variance
+
+_Multi-Objective Optimization:_
+
- EHVI: Expected Hypervolume Improvement
- NEHVI: Noisy Expected Hypervolume Improvement
- NParEGO: Random augmented chebyshev scalarization with Noisy Expected Improvement
-- NIPV: Integrated Negative Posterior Variance
#### Custom Acquisition Functions
-- Mean: qMean: Optimizes for the maximum value of the posterior mean
-- SF: qSpaceFill: Optimizes for the maximum value of minimum distance between a point and the training data
+- Mean: Seeks to optimize the maximum value of the posterior mean
+- SF: Space filling, aims to optimize the maximum value of the minimum distance between a point and the training data
#### Baseline: No Acquisition Function
- RS: Random sampling from the parameter space
+### 2.2 Basic Syntax
+
+Typically, users don't need to interact with acquisition functions directly.
+The `BayesianOptimizer` class handles the selection and use of acquisition functions.
+The acquisition functions, as well as their hyperparameters, could be specified as an input argument when calling the `suggest` method:
+
+```python
+# DO NOT RUN
+from obsidian.optimizer import BayesianOptimizer
+
+optimizer = BayesianOptimizer(X_space=param_space)
+
+# Use one acquisition function EI per iteration
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI'])
-### 3.2 Shared Hyperparameters
+# Use two acquisition functions EI and UCB per iteration
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI','UCB'])
+
+# Use two acquisition functions EI and UCB per iteration, while specifying hyperparameters for UCB
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI',{'UCB':{'beta':0.1}}])
+```
-There are some commonly used hyperparameters that could apply to multiple acquisition functions.
+The `acquisition` parameter should be always a list, containing either string or dictionary elements. When multiple elements are present in the list, the `suggest` function will propose new candidates sequentially using each acquisition function.
+* If using the default hyperparameters for an acquisition function, specify a string element using the name of the acquisition function (e.g., 'EI' for Expected Improvement).
+* If specifying custom hyperparameters for an acquisition function, use a dictionary element where the key is the acquisition function name and the value is a nested dictionary storing its hyperparameters (e.g., {'UCB': {'beta': 0.1}} for Upper Confidence Bound with a specified beta parameter).
----------------
+## 3. Understanding Acquisition Functions and Hyperparameters
-## 4. Understanding Acquisition Functions and Hyperparameters
### 3.1 Expected Improvement (EI)
@@ -111,7 +112,7 @@ X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEI'])
## 4. Advanced Usage
-### 4.1 Multi-Objective Optimization
+### 4.1 Additional Options for Multi-Objective Optimization
For multi-objective optimization problems, you can use specialized acquisition functions:
From 66ec24d462dadd026aae4445728eaa10b79b011b Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 11:56:25 -0400
Subject: [PATCH 025/136] revise 3.1 Expected Improvement (EI)
---
docs/wiki/4_SurrogateModel.md | 2 +-
docs/wiki/5_AcquisitionFunction.md | 53 ++++++++++++++++++------------
2 files changed, 33 insertions(+), 22 deletions(-)
diff --git a/docs/wiki/4_SurrogateModel.md b/docs/wiki/4_SurrogateModel.md
index 1a9c3e4..47fd74b 100644
--- a/docs/wiki/4_SurrogateModel.md
+++ b/docs/wiki/4_SurrogateModel.md
@@ -2,7 +2,7 @@
## 1. Introduction
-The `obsidian.surrogates` submodule is a key component of the Obsidian APO library. It provides a collection of surrogate models used to approximate the objective function in the optimization process. These surrogate models are essential for efficient exploration of the parameter space and for making informed decisions about which points to evaluate next.
+The [`obsidian.surrogates`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/surrogates) submodule is a key component of the Obsidian APO library. It provides a collection of surrogate models used to approximate the objective function in the optimization process. These surrogate models are essential for efficient exploration of the parameter space and for making informed decisions about which points to evaluate next.
## 2. Basic Syntax
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 8b9f3b5..3a49534 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -73,18 +73,39 @@ The `acquisition` parameter should be always a list, containing either string or
EI calculates the expected amount by which we will improve upon the current best observed value.
Mathematical formulation:
-```
-EI(x) = E[max(f(x) - f(x+), 0)]
-```
-where f(x+) is the current best observed value.
+\begin{equation*}
+EI(x) = E[max(\hat{f}(x) - y_{best}, 0)]
+\end{equation*}
-Example usage:
-```python
-from obsidian.optimizer import BayesianOptimizer
+where $y_{best}$ is the current best observed value.
+The expression $max(\hat{f}(x) - y_{best}$ captures the potential improvement over the current best observed value, and the expectation E[ ] calculates the average improvement over the posterior distribution of the surrogate model predictions.
-optimizer = BayesianOptimizer(X_space=param_space)
-X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI'])
-```
+_Optional hyperparameters:_
+
+* inflate: Increase the current best value $y_{best}$ to $(1+inflate)*y_{best}$, enabling a more flexible exploration-exploitation trade-off.
+
+ \begin{equation*}
+ EI(x) = E[max(\hat{f}(x) - (1+inflate)*y_{best}, 0)]
+ \end{equation*}
+
+ The default value is 0 (no inflation).
+
+**Example usage:**
+
+* Default:
+
+ ```python
+ from obsidian.optimizer import BayesianOptimizer
+
+ optimizer = BayesianOptimizer(X_space=param_space)
+ X_suggest, eval_suggest = optimizer.suggest(acquisition=['EI'])
+ ```
+
+* With all available hyperparameters:
+
+ ```python
+ X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'EI': {'inflate': 0.05}}])
+ ```
### 3.2 Upper Confidence Bound (UCB)
@@ -120,17 +141,7 @@ For multi-objective optimization problems, you can use specialized acquisition f
X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEHVI'])
```
-### 4.2 Customizing Acquisition Functions
-
-Some acquisition functions accept parameters to customize their behavior. These can be specified in the `suggest` method:
-
-```python
-X_suggest, eval_suggest = optimizer.suggest(
- acquisition=[{'EI': {'inflate': 0.01}}]
-)
-```
-
-### 4.3 Custom Acquisition Functions
+### 4.2 Custom Acquisition Functions
If you need to implement a custom acquisition function, you can extend the `MCAcquisitionFunction` class from BoTorch:
From aea5de53927a0a8e7af74b900d6a669719659177 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 14:52:04 -0400
Subject: [PATCH 026/136] Renamed help to infobar, updated logo specification
and version
---
app.py | 6 +++---
obsidian/dash/__init__.py | 2 +-
obsidian/dash/{help.py => infobar.py} | 17 +++++++++++------
3 files changed, 15 insertions(+), 10 deletions(-)
rename obsidian/dash/{help.py => infobar.py} (79%)
diff --git a/app.py b/app.py
index dbc89cf..524389d 100755
--- a/app.py
+++ b/app.py
@@ -1,6 +1,6 @@
from dash import html, Dash
import dash_bootstrap_components as dbc
-from obsidian.dash import setup_data, setup_config, setup_optimize, setup_plots, setup_predict, setup_help
+from obsidian.dash import setup_data, setup_config, setup_optimize, setup_plots, setup_predict, setup_infobar
import pandas as pd
from PIL import Image
@@ -15,7 +15,7 @@
dbc_css = "https://cdn.jsdelivr.net/gh/AnnMarieW/dash-bootstrap-templates/dbc.min.css" # For data tables
app.config.external_stylesheets = [dbc.themes.SANDSTONE, dbc_css, dbc.icons.BOOTSTRAP]
-logo = Image.open('docs/figures/obsidian_logo.png')
+logo = Image.open('docs/_static/obsidian_logo.png')
app_image = html.Div(html.Img(src=logo, style={'width': '5%', 'height': '5%'}), style={'textAlign': 'center'})
app_title = html.Div([html.H1(children='obsidian'),
@@ -45,7 +45,7 @@
default_data = pd.concat([X0, y0], axis=1)
# Set up each tab
-setup_help(app, app_infobar)
+setup_infobar(app, app_infobar)
setup_data(app, app_tabs, default_data)
setup_config(app, app_tabs)
setup_optimize(app, app_tabs)
diff --git a/obsidian/dash/__init__.py b/obsidian/dash/__init__.py
index 2738d7c..5812117 100644
--- a/obsidian/dash/__init__.py
+++ b/obsidian/dash/__init__.py
@@ -5,4 +5,4 @@
from .optimize import *
from .plots import *
from .predict import *
-from .help import *
+from .infobar import *
diff --git a/obsidian/dash/help.py b/obsidian/dash/infobar.py
similarity index 79%
rename from obsidian/dash/help.py
rename to obsidian/dash/infobar.py
index c7d03a7..b43f9fd 100644
--- a/obsidian/dash/help.py
+++ b/obsidian/dash/infobar.py
@@ -4,38 +4,43 @@
import pandas as pd
+import obsidian
from obsidian.experiment import ParamSpace
from obsidian.optimizer import BayesianOptimizer
from obsidian.plotting.plotly import parity_plot
from .utils import load_optimizer, center
-def setup_help(app, app_infobar):
-
+def setup_infobar(app, app_infobar):
+ # Infobar = 3 columns [Help, version, Contact]
app_infobar.children = dbc.Container([
+ html.Br(),
dbc.Row([
dbc.Col(dbc.Button('Help', outline='True', color='warning', className='me-1', id='button-help'),
style={'textAlign': 'left'}, width={'size': 2}),
- dbc.Col(html.Div(dbc.Badge('v0.1.1', color='primary', className='me-1'), style={'textAlign': 'center'}),
+ dbc.Col(html.Div(dbc.Badge(f'v{obsidian.__version__}', color='primary', className='me-1'), style={'textAlign': 'center'}),
width={'size': 2}),
dbc.Col(dbc.Button('Contact', outline='True', color='secondary', className='me-1', id='button-contact'),
style={'textAlign': 'right'}, width={'size': 2}),
], justify='center'),
+ # Pop-up for Contact Us
dbc.Modal([
dbc.ModalHeader(dbc.ModalTitle('Contact Us')),
dbc.ModalBody([
- dbc.Button('Email Support', href='', external_link=True,
+ dbc.Button('Email Developers', href='mailto:kevin.stone38@gmail.com', external_link=True,
color='primary', className='me-1'),
- dbc.Button('Visit Our Site', href='', target='_blank', external_link=True,
+ dbc.Button('Visit Our Site', href='https://msdllcpapers.github.io/obsidian/', target='_blank', external_link=True,
color='primary', className='me-1')
], style={'textAlign': 'center'}),
dbc.ModalFooter([])
], id='modal-contact', is_open=False, size='xl', centered=True),
+ # Pop-up for Help
dbc.Modal([
dbc.ModalHeader(dbc.ModalTitle('Help: APO with the obsidian Web App')),
dbc.ModalBody('Coming soon...'),
dbc.ModalFooter([])
- ], id='modal-help', is_open=False, size='xl', centered=True)
+ ], id='modal-help', is_open=False, size='xl', centered=True),
+ html.Br()
], fluid=True)
setup_help_callbacks(app)
From 1bf94e7565efeb5df565ad2e26b2d06e88d078bf Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 14:52:35 -0400
Subject: [PATCH 027/136] Fixed ordinal fitting
---
obsidian/dash/inputs_config.py | 3 +-
obsidian/dash/inputs_data.py | 10 +++----
obsidian/dash/optimize.py | 50 +++++++++++++++++++---------------
3 files changed, 34 insertions(+), 29 deletions(-)
diff --git a/obsidian/dash/inputs_config.py b/obsidian/dash/inputs_config.py
index f30fbd9..d0c21b1 100644
--- a/obsidian/dash/inputs_config.py
+++ b/obsidian/dash/inputs_config.py
@@ -164,8 +164,7 @@ def compile_config(param_xspaces, response_name, optimizer_seed, f_transform, su
config['response_name'] = response_name
config['optimizer_seed'] = int(optimizer_seed) if optimizer_seed is not None else None
config['surrogate_params'] = {'f_transform': f_transform, 'surrogate': surrogate}
- #config['aq_params'] = {'optim_sequential': optim_sequential, 'optim_restarts': optim_restarts,
- # 'm_batch': m_batch, 'acquisition': aq, 'alpha': alpha}
+ # TODO: Re-implement hyperparmeters based on selection in alpha
config['aq_params'] = {'optim_sequential': optim_sequential, 'optim_restarts': optim_restarts,
'm_batch': m_batch, 'acquisition': aq}
config['verbose'] = 0
diff --git a/obsidian/dash/inputs_data.py b/obsidian/dash/inputs_data.py
index a52d296..977a10d 100644
--- a/obsidian/dash/inputs_data.py
+++ b/obsidian/dash/inputs_data.py
@@ -53,6 +53,7 @@ def setup_data(app, app_tabs, default_data):
# Data store
storage_X0 = dcc.Store(id='store-X0', data=default_data.to_dict())
storage_X0_template = dcc.Store(id='store-X0_template', data=default_data.to_dict())
+ # storage_Xspace = dcc.Store(id='store-Xspace', data={})
# Parameter space table
xspace = html.Div([html.Div(id='div-xspace', children=[])])
@@ -143,11 +144,10 @@ def choose_col(data):
)
def update_xspace_types(data, ycol):
- df_xspace = pd.DataFrame(data)
- xcols = [x for x in df_xspace.columns if x != ycol]
+ df_X0 = pd.DataFrame(data)
+ xcols = [x for x in df_X0.columns if x != ycol]
- #param_types = ['Numeric', 'Categorical', 'Ordinal']
- param_types = ['Numeric', 'Categorical'] # Note: the Ordinal var may cause fitting issues
+ param_types = ['Numeric', 'Categorical', 'Ordinal']
cols = []
for i, x in enumerate(xcols):
@@ -207,7 +207,7 @@ def update_xspace_vals(param_type, param_id, data, ycol):
]
return choices
- # ategory management for categorical variables
+ # Category management for categorical variables
@app.callback(
Output({'type': 'store-param_categories', 'index': MATCH}, 'data'),
Output({'type': 'input-new_category', 'index': MATCH}, 'value'),
diff --git a/obsidian/dash/optimize.py b/obsidian/dash/optimize.py
index 70cd171..e6876b1 100644
--- a/obsidian/dash/optimize.py
+++ b/obsidian/dash/optimize.py
@@ -5,7 +5,7 @@
import pandas as pd
from obsidian.parameters import ParamSpace, Param_Categorical, Param_Ordinal, Param_Continuous
-#from obsidian.optimizer import BayesianOptimizer
+from obsidian.optimizer import BayesianOptimizer
from obsidian.campaign import Campaign
from obsidian.parameters import Target
@@ -54,10 +54,11 @@ def setup_optimize(app, app_tabs):
store_candidates = dcc.Store(id='store-candidates', data={})
# Suggested candidates download
- candidates_downloader = html.Div(children=[dbc.Button('Download Suggested Candidates', id='button-download_candidates',
- className='me-2', color='primary'),
- dcc.Download(id='downloader-candidates')],
- style={'textAlign': 'center','margin-top': '15px'})
+ candidates_downloader = html.Div(children=[
+ dbc.Button('Download Suggested Candidates', id='button-download_candidates',
+ className='me-2', color='primary'),
+ dcc.Download(id='downloader-candidates')],
+ style={'textAlign': 'center', 'margin-top': '15px'})
# Add all of these elements to the app
columns = dbc.Row([dbc.Col(fit_div, width=6), dbc.Col([predict_div, candidates_downloader], width=6)])
@@ -83,23 +84,27 @@ def fit_optimizer(fit_clicked, config, X0):
if config['xspace'] == {}:
return 0, None
- xspace = []
- for param_xspace in config['xspace']:
- if param_xspace['Type']=='Numeric':
- this_para = Param_Continuous(param_xspace['Name'], param_xspace['Low'], param_xspace['High'])
- elif param_xspace['Type']=='Categorical':
- this_para = Param_Categorical(param_xspace['Name'], param_xspace['Categories'])
- else:
- this_para = Param_Ordinal(param_xspace['Name'], param_xspace['Categories'])
- xspace.append(this_para)
- X_space = ParamSpace(xspace)
- my_campaign = Campaign(X_space)
- my_campaign.add_data(pd.DataFrame(X0))
+ param_list = []
+ for param_input in config['xspace']:
+ if param_input['Type'] == 'Numeric':
+ param = Param_Continuous(param_input['Name'], param_input['Low'], param_input['High'])
+ elif param_input['Type'] == 'Categorical':
+ param = Param_Categorical(param_input['Name'], param_input['Categories'])
+ elif param_input['Type'] == 'Ordinal':
+ param = Param_Ordinal(param_input['Name'], param_input['Categories'])
+ param_list.append(param)
+
+ X_space = ParamSpace(param_list)
target = Target(config['response_name'], aim='max')
- my_campaign.set_target(target)
- my_campaign.fit()
+ campaign = Campaign(X_space, target)
+ campaign.add_data(pd.DataFrame(X0))
+
+ optimizer = BayesianOptimizer(X_space=campaign.X_space,
+ surrogate=config['surrogate_params']['surrogate'])
+ campaign.set_optimizer(optimizer)
+ campaign.fit()
- return 0, my_campaign.optimizer.save_state()
+ return 0, campaign.optimizer.save_state()
@app.callback(
Output('div-fit', 'children'),
@@ -117,8 +122,9 @@ def fit_statistics(opt_save, config, filename):
[
dbc.ListGroupItem(['Model Type: ', f'{optimizer.surrogate_type}']),
dbc.ListGroupItem(['Data Name: ', filename]),
- dbc.ListGroupItem(['R', html.Sup('2'), ' Score: ', f'{optimizer.surrogate[0].r2_score: .4g}']), # for SOO only
- dbc.ListGroupItem(['Marginal Log Likelihood: ', f'{optimizer.surrogate[0].loss: .4g}']), # for SOO only
+ # TODO: Implement score for each surrogate, to support MOO
+ dbc.ListGroupItem(['R', html.Sup('2'), ' Score: ', f'{optimizer.surrogate[0].r2_score: .4g}']),
+ dbc.ListGroupItem(['Marginal Log Likelihood: ', f'{optimizer.surrogate[0].loss: .4g}']),
], flush=True
)
From 2ef065e0a299345e488c6006fd2f90ad5bfbcf80 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 14:52:47 -0400
Subject: [PATCH 028/136] Fixed linting
---
obsidian/dash/predict.py | 30 ++++++++++++------------------
obsidian/dash/utils.py | 2 +-
2 files changed, 13 insertions(+), 19 deletions(-)
diff --git a/obsidian/dash/predict.py b/obsidian/dash/predict.py
index e8f09e5..3784da4 100644
--- a/obsidian/dash/predict.py
+++ b/obsidian/dash/predict.py
@@ -3,10 +3,6 @@
import dash_bootstrap_components as dbc
from dash import dcc, html, Dash, dash_table, callback, Output, Input, State, ALL, MATCH
-#from obsidian.experiment import ParamSpace
-#from obsidian.optimizer import BayesianOptimizer
-#from obsidian.plotting.plotly_plotting import parity_plot
-
import pandas as pd
import base64
import io
@@ -36,7 +32,7 @@ def setup_predict(app, app_tabs):
html.Br(),
html.Br(),
dbc.Card([
- dbc.CardHeader('template'),
+ dbc.CardHeader('Example Prediction Input'),
dbc.CardBody([
html.Div(id='div-template', children=[], style={'overflow-x': 'scroll'})
]),
@@ -51,23 +47,23 @@ def setup_predict(app, app_tabs):
template_downloader = html.Div(children=[dbc.Button('Download Template Candidates', id='button-download_template',
className='me-2', color='primary'),
dcc.Download(id='downloader-template')],
- style={'textAlign': 'center','margin-top': '15px'})
+ style={'textAlign': 'center', 'margin-top': '15px'})
default_data = pd.DataFrame()
# Data upload
uploader_1 = dcc.Upload(id='uploader-X1',
- children=html.Div(['Upload Data: Drag and Drop or ',
- html.A('Select Files')]),
- style={
+ children=html.Div(['Upload Data: Drag and Drop or ',
+ html.A('Select Files')]),
+ style={
'width': '100%', 'height': '60px',
'lineHeight': '60px', 'borderWidth': '1px',
'borderStyle': 'dashed', 'borderRadius': '5px',
'textAlign': 'center', 'margin': '10px'
},
- multiple=False,
- filename='Example Data'
- )
+ multiple=False,
+ filename='Example Data'
+ )
# Data store
storage_X1 = dcc.Store(id='store-X1', data=default_data.to_dict())
@@ -89,7 +85,7 @@ def setup_predict(app, app_tabs):
]))
row1 = dbc.Row([dbc.Col([xspace_df_div], width=4),
- dbc.Col([template_div, template_downloader], width=8)],style={'margin-top': '15px'})
+ dbc.Col([template_div, template_downloader], width=8)], style={'margin-top': '15px'})
row2 = dbc.Row([uploader_1, preview_1], style={'margin-top': '15px'})
# Add all of these elements to the app
@@ -112,7 +108,7 @@ def setup_predict_callbacks(app):
def config_tableView(clicked, config):
if config is None:
return 0, None, {'overflow-x': 'scroll'}
- #print(config)
+
df_xspace = pd.DataFrame(config['xspace'])
tables = [center(make_table(df_xspace))]
return 0, tables, {'overflow-x': 'scroll'}
@@ -134,6 +130,7 @@ def config_InputTemplate(clicked, opt_save, config):
df_template = designer.initialize(3, 'LHS')
tables = [center(make_table(df_template))]
return 0, tables, df_template.to_dict()
+
# Download Template
@app.callback(
Output('downloader-template', 'data'),
@@ -145,7 +142,6 @@ def download_template(n_clicks, data):
df = pd.DataFrame(data)
return dcc.send_data_frame(df.to_csv, 'NewDataTemplate.csv', index=False)
-
# Save the uploaded data into the data-store
@app.callback(
Output('store-X1', 'data'),
@@ -168,7 +164,7 @@ def save_X1(contents):
State('store-fit', 'data'),
State('store-config', 'data'),
)
- def preview_X1(data, filename,opt_save, config):
+ def preview_X1(data, filename, opt_save, config):
df = pd.DataFrame(data)
optimizer = load_optimizer(config, opt_save)
preds = optimizer.predict(df)
@@ -176,5 +172,3 @@ def preview_X1(data, filename,opt_save, config):
return make_table(df_output, fill_width=True), filename
return
-
-
diff --git a/obsidian/dash/utils.py b/obsidian/dash/utils.py
index ee8b3ed..70ff30d 100644
--- a/obsidian/dash/utils.py
+++ b/obsidian/dash/utils.py
@@ -4,10 +4,10 @@
import dash_daq as daq
import pandas as pd
-#from obsidian.experiment import ParamSpace
from obsidian.parameters import ParamSpace, Param_Categorical, Param_Ordinal, Param_Continuous
from obsidian.optimizer import BayesianOptimizer
+
def center(element):
return html.Div(html.Div(element, style={'display': 'inline-block'}), style={'textAlign': 'center'})
From 4b815513058a270028726ba6bfedc1a73c90efd5 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 14:52:58 -0400
Subject: [PATCH 029/136] Ensure categorical parameters strip whitespace
---
obsidian/parameters/discrete.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/obsidian/parameters/discrete.py b/obsidian/parameters/discrete.py
index a286edd..4eea17d 100644
--- a/obsidian/parameters/discrete.py
+++ b/obsidian/parameters/discrete.py
@@ -95,6 +95,7 @@ def __init__(self,
super().__init__(name=name)
if isinstance(categories, str):
self.categories = categories.split(',')
+ self.categories = [c.rstrip().lstrip() for c in self.categories]
else:
self.categories = categories
for c in self.categories:
From b00d8877dd4b5f6177146a36eb7bcd6a141190cb Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 15:52:10 -0400
Subject: [PATCH 030/136] revise NEI, PI, UCB
---
docs/wiki/5_AcquisitionFunction.md | 100 +++++++++++++++++++++++------
1 file changed, 79 insertions(+), 21 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 3a49534..0e73dd6 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -65,7 +65,7 @@ The `acquisition` parameter should be always a list, containing either string or
* If specifying custom hyperparameters for an acquisition function, use a dictionary element where the key is the acquisition function name and the value is a nested dictionary storing its hyperparameters (e.g., {'UCB': {'beta': 0.1}} for Upper Confidence Bound with a specified beta parameter).
-## 3. Understanding Acquisition Functions and Hyperparameters
+## 3. Single-Objective Optimization: Acquisition Functions and Hyperparameters
### 3.1 Expected Improvement (EI)
@@ -82,13 +82,13 @@ The expression $max(\hat{f}(x) - y_{best}$ captures the potential improvement ov
_Optional hyperparameters:_
-* inflate: Increase the current best value $y_{best}$ to $(1+inflate)*y_{best}$, enabling a more flexible exploration-exploitation trade-off.
+* inflate: Increase the current best value $y_{best}$ to $(1+inflate)*y_{best}$, enabling a more flexible exploration-exploitation trade-off:
\begin{equation*}
EI(x) = E[max(\hat{f}(x) - (1+inflate)*y_{best}, 0)]
\end{equation*}
- The default value is 0 (no inflation).
+ The default value is 0 (no inflation). Recommended values: small numberic number 0~0.1
**Example usage:**
@@ -107,33 +107,91 @@ _Optional hyperparameters:_
X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'EI': {'inflate': 0.05}}])
```
-### 3.2 Upper Confidence Bound (UCB)
+### 3.2 Noisy Expected Improvement (NEI)
-UCB balances exploration and exploitation by selecting points with high predicted values or high uncertainty.
+NEI is a variant of EI that accounts for noise in the observations, making it more suitable for real-world problems with measurement uncertainty. It allows for more robust decision-making in selecting the next point for evaluation.
-Mathematical formulation:
-```
-UCB(x) = μ(x) + β * σ(x)
+Currently NEI doesn't accept additional hyperparameters.
+
+**Example usage:**
+
+```python
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEI'])
```
-where μ(x) is the predicted mean, σ(x) is the predicted standard deviation, and β is a parameter that controls the exploration-exploitation trade-off.
-Example usage:
+### 3.3 Probability of Improvement (PI)
+
+PI is designed to aid in the efficient selection of candidate points by quantifying the probability of improving upon the current best observed value.
+Different from EI, which measures the expected amount of improvement by integrating over the posterior distribution, PI directly evaluates the probability of outperforming the best value, emphasizing the likelihood of improvement rather than the magnitude of improvement.
+
+Mathematical formulation:
+$$
+PI(x) = P(\hat{f}(x) > y_{best})
+$$
+where $y_{best}$ is the current best observed value and the $\hat{f}$ is the trained surrogate function.
+
+_Optional hyperparameters:_
+* inflate: Increase the current best value $y_{best}$ to $(1+inflate)*y_{best}$, enabling a more flexible exploration-exploitation trade-off:
+
+ \begin{equation*}
+ PI(x) = P(\hat{f}(x) > (1+inflate)*y_{best})
+ \end{equation*}
+
+ The default value is 0 (no inflation). Recommended values: small numberic number 0~0.1
+
+**Example usage:**
```python
-X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'UCB': {'beta': 2.0}}])
+# Use the default inflate = 0
+X_suggest, eval_suggest = campaign.suggest(acquisition=['PI'])
+
+# Adjust the hyperparameter inflate
+X_suggest, eval_suggest = campaign.suggest(acquisition=[{'PI': {'inflate': 0.05}}])
```
-### 3.3 Noisy Expected Improvement (NEI)
+### 3.4 Upper Confidence Bound (UCB)
-NEI is a variant of EI that accounts for noise in the observations, making it more suitable for real-world problems with measurement uncertainty.
+UCB balances exploration and exploitation by selecting points with high predicted values or high uncertainty.
+
+Mathematical formulation:
+$$
+UCB(x) = \mu(x) + \beta * \sigma(x)
+$$
+
+where $\mu(x)$ is the predicted mean at the candidate point $x$, $\sigma(x)$ is the predicted standard deviation which is associated with uncertainty, and $\beta$ is a parameter that controls the exploration-exploitation trade-off.
+
+_Optional hyperparameters:_
+* $\beta$: By default $\beta = 1$. Recommended value range: 1~3
+
+**Example usage:**
-Example usage:
```python
-X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEI'])
+# Use the default beta = 1
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['UCB'])
+
+# Adjust the hyperparameter beta
+X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'UCB': {'beta': 2.0}}])
```
-## 4. Advanced Usage
+## 4. Multi-Objective Optimization: Acquisition Functions and Hyperparameters
+
+
+
+### 4.1 Expected Hypervolume Improvement (EHVI)
+
+
+
+### 4.2 Expected Hypervolume Improvement (NEHVI)
+
+
+
+### 4.3 Random augmented chebyshev scalarization with Noisy Expected Improvement (NParEGO)
+
+
+
+
+## 5. Advanced Usage
-### 4.1 Additional Options for Multi-Objective Optimization
+### 5.1 Additional Options for Multi-Objective Optimization
For multi-objective optimization problems, you can use specialized acquisition functions:
@@ -141,7 +199,7 @@ For multi-objective optimization problems, you can use specialized acquisition f
X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEHVI'])
```
-### 4.2 Custom Acquisition Functions
+### 5.2 Custom Acquisition Functions
If you need to implement a custom acquisition function, you can extend the `MCAcquisitionFunction` class from BoTorch:
@@ -157,7 +215,7 @@ class CustomAcquisition(MCAcquisitionFunction):
return (mean + 0.1 * std).sum(dim=-1) # Example custom acquisition logic
```
-## 5. Comparing Acquisition Functions
+## 6. Comparing Acquisition Functions
Different acquisition functions have different strengths:
@@ -167,7 +225,7 @@ Different acquisition functions have different strengths:
- qMean is purely exploitative and can be useful in the final stages of optimization.
- qSpaceFill is purely explorative and can be useful for initial space exploration.
-## 6. Best Practices
+## 7. Best Practices
1. Choose appropriate acquisition functions based on your problem characteristics (e.g., noise level, number of objectives).
2. For noisy problems, consider using noise-aware acquisition functions like NEI or NEHVI.
@@ -176,7 +234,7 @@ Different acquisition functions have different strengths:
5. For multi-objective problems, EHVI and NEHVI are often good choices.
6. Consider using a sequence of acquisition functions, starting with more exploratory ones and moving to more exploitative ones as the optimization progresses.
-## 7. Common Pitfalls
+## 8. Common Pitfalls
1. Using EI or PI in noisy problems, which can lead to overexploitation of noisy observations.
2. Setting UCB's beta parameter too high (over-exploration) or too low (over-exploitation).
From 84b0e00e6707ba481bb14ee4932761adc2283dad Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 16:38:16 -0400
Subject: [PATCH 031/136] minor formatting correction
---
docs/wiki/5_AcquisitionFunction.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 0e73dd6..66b4c70 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -125,9 +125,9 @@ PI is designed to aid in the efficient selection of candidate points by quantify
Different from EI, which measures the expected amount of improvement by integrating over the posterior distribution, PI directly evaluates the probability of outperforming the best value, emphasizing the likelihood of improvement rather than the magnitude of improvement.
Mathematical formulation:
-$$
+\begin{equation*}
PI(x) = P(\hat{f}(x) > y_{best})
-$$
+\end{equation*}
where $y_{best}$ is the current best observed value and the $\hat{f}$ is the trained surrogate function.
_Optional hyperparameters:_
@@ -142,10 +142,10 @@ _Optional hyperparameters:_
**Example usage:**
```python
# Use the default inflate = 0
-X_suggest, eval_suggest = campaign.suggest(acquisition=['PI'])
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['PI'])
# Adjust the hyperparameter inflate
-X_suggest, eval_suggest = campaign.suggest(acquisition=[{'PI': {'inflate': 0.05}}])
+X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'PI': {'inflate': 0.05}}])
```
### 3.4 Upper Confidence Bound (UCB)
@@ -153,9 +153,9 @@ X_suggest, eval_suggest = campaign.suggest(acquisition=[{'PI': {'inflate': 0.05}
UCB balances exploration and exploitation by selecting points with high predicted values or high uncertainty.
Mathematical formulation:
-$$
+\begin{equation*}
UCB(x) = \mu(x) + \beta * \sigma(x)
-$$
+\end{equation*}
where $\mu(x)$ is the predicted mean at the candidate point $x$, $\sigma(x)$ is the predicted standard deviation which is associated with uncertainty, and $\beta$ is a parameter that controls the exploration-exploitation trade-off.
From d2df76d648b38ca5fd1227dea6062d147ceec64a Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 18:20:26 -0400
Subject: [PATCH 032/136] revise Hypervolume Improvement for MOO in
5_AcquisitionFunction.md
---
docs/wiki/5_AcquisitionFunction.md | 49 +++++++++++++++++++-----------
1 file changed, 31 insertions(+), 18 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 66b4c70..0cfb2e2 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -65,10 +65,10 @@ The `acquisition` parameter should be always a list, containing either string or
* If specifying custom hyperparameters for an acquisition function, use a dictionary element where the key is the acquisition function name and the value is a nested dictionary storing its hyperparameters (e.g., {'UCB': {'beta': 0.1}} for Upper Confidence Bound with a specified beta parameter).
-## 3. Single-Objective Optimization: Acquisition Functions and Hyperparameters
+## 3. Single-Objective Optimizatio Acquisition Functions
-### 3.1 Expected Improvement (EI)
+### Expected Improvement (EI)
EI calculates the expected amount by which we will improve upon the current best observed value.
@@ -107,7 +107,7 @@ _Optional hyperparameters:_
X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'EI': {'inflate': 0.05}}])
```
-### 3.2 Noisy Expected Improvement (NEI)
+### Noisy Expected Improvement (NEI)
NEI is a variant of EI that accounts for noise in the observations, making it more suitable for real-world problems with measurement uncertainty. It allows for more robust decision-making in selecting the next point for evaluation.
@@ -119,7 +119,7 @@ Currently NEI doesn't accept additional hyperparameters.
X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEI'])
```
-### 3.3 Probability of Improvement (PI)
+### Probability of Improvement (PI)
PI is designed to aid in the efficient selection of candidate points by quantifying the probability of improving upon the current best observed value.
Different from EI, which measures the expected amount of improvement by integrating over the posterior distribution, PI directly evaluates the probability of outperforming the best value, emphasizing the likelihood of improvement rather than the magnitude of improvement.
@@ -148,7 +148,7 @@ X_suggest, eval_suggest = optimizer.suggest(acquisition=['PI'])
X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'PI': {'inflate': 0.05}}])
```
-### 3.4 Upper Confidence Bound (UCB)
+### Upper Confidence Bound (UCB)
UCB balances exploration and exploitation by selecting points with high predicted values or high uncertainty.
@@ -172,34 +172,46 @@ X_suggest, eval_suggest = optimizer.suggest(acquisition=['UCB'])
X_suggest, eval_suggest = optimizer.suggest(acquisition=[{'UCB': {'beta': 2.0}}])
```
-## 4. Multi-Objective Optimization: Acquisition Functions and Hyperparameters
+## 4. Multi-Objective Optimization Acquisition Functions
+### Hypervolume Improvement
+One of the most well-known and widely used acquisition functions for multi-objective optimization is the Hypervolume Improvement-based acquisition functions.
+The Hypervolume is a measure of the covered area in the objective space that is dominated by a given Pareto front; therefore, it is often used to quantify the quality of a Pareto front approximation. (See also Section [Additional Analysis](2_Analysis_and_Visualization.md))
-### 4.1 Expected Hypervolume Improvement (EHVI)
+There are two available options in `obsidian`:
+* **Expected Hypervolume Improvement (EHVI)**
+* **Noisy Expected Hypervolume Improvement (NEHVI)**
+The EHVI and NEHVI acquisition functions aim to select the next set of input points that would maximize the hypervolume of the Pareto front. They differ in the consideration of noise in the objective function evaluations, with NEHVI explicitly accounting for noisy observations while EHVI assumes noise-free evaluations.
-### 4.2 Expected Hypervolume Improvement (NEHVI)
+NEHVI enables more robust optimization in the presence of noisy observations, improving the reliability of the optimization process. However, the incorporation of noise modeling in NEHVI may lead to increased computational complexity, as it often requires a larger number of Monte Carlo samples to accurately capture the noise characteristics, potentially making it more computationally intensive than EHVI in scenarios with significant noise.
+_Hyperparameters:_
+* ref_point: The reference point for computing the hypervolume. Default value for each dimension is the minimum value minus 10% of the range.
+**Example usage:**
-### 4.3 Random augmented chebyshev scalarization with Noisy Expected Improvement (NParEGO)
-
+```python
+# Using default values for ref_point:
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['EHVI'])
+X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEHVI'])
+# Custom ref_point, assuming there are two outputs
+X_suggest, eval_suggest = optimizer.suggest(acquisition = [{'EHVI':{'ref_point':[5, 40]}}])
+X_suggest, eval_suggest = optimizer.suggest(acquisition = [{'NEHVI':{'ref_point':[-2, -30]}}])
+```
+### Weighted Response
-## 5. Advanced Usage
+* Random augmented chebyshev scalarization with Noisy Expected Improvement (NParEGO)
-### 5.1 Additional Options for Multi-Objective Optimization
+* Additional scalarization options for multi-objective problems
-For multi-objective optimization problems, you can use specialized acquisition functions:
-
-```python
-X_suggest, eval_suggest = optimizer.suggest(acquisition=['NEHVI'])
-```
+## 5. Advanced Usage
-### 5.2 Custom Acquisition Functions
+### Custom Acquisition Functions
If you need to implement a custom acquisition function, you can extend the `MCAcquisitionFunction` class from BoTorch:
@@ -224,6 +236,7 @@ Different acquisition functions have different strengths:
- NEI and NEHVI are robust to noisy observations.
- qMean is purely exploitative and can be useful in the final stages of optimization.
- qSpaceFill is purely explorative and can be useful for initial space exploration.
+- EHVI is advantageous in noise-free or low-noise settings due to its computational efficiency, while NEHVI is better suited for scenarios with significant noise, offering improved robustness at the cost of potentially higher computational demands.
## 7. Best Practices
From fcf82208c2572bd710899ee181580e2034caf0cd Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 21:17:54 -0400
Subject: [PATCH 033/136] bugfix for NParEGO: default scalarization_weights is
None (random weights); enable user input scalarization_weights (previous
missing in _parse_aq_kwargs
---
obsidian/acquisition/config.py | 2 +-
obsidian/optimizer/bayesian.py | 9 ++++++++-
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/obsidian/acquisition/config.py b/obsidian/acquisition/config.py
index 4665757..9404353 100644
--- a/obsidian/acquisition/config.py
+++ b/obsidian/acquisition/config.py
@@ -44,5 +44,5 @@
'NIPV': {},
'EHVI': {'ref_point': {'val': None, 'optional': True}},
'NEHVI': {'ref_point': {'val': None, 'optional': True}},
- 'NParEGO': {'scalarization_weights': {'val': [1], 'optional': True}},
+ 'NParEGO': {'scalarization_weights': {'val': None, 'optional': True}},
}
diff --git a/obsidian/optimizer/bayesian.py b/obsidian/optimizer/bayesian.py
index 343b26b..aab2e1a 100644
--- a/obsidian/optimizer/bayesian.py
+++ b/obsidian/optimizer/bayesian.py
@@ -459,7 +459,7 @@ def _validate_hypers(self,
if hps.get(key) is None:
if not defaults['optional']:
raise ValueError(f'Must specify hyperpameter value {key} for {aq_str}')
- if key in ['scalarization_weights', 'weights']:
+ if key in ['weights']: #['scalarization_weights', 'weights']:
aq_hps[key] = defaults['val'] * o_dim
else:
aq_hps[key] = defaults['val']
@@ -551,6 +551,13 @@ def _parse_aq_kwargs(self,
qmc_samples = draw_sobol_samples(bounds=X_bounds, n=128, q=m_batch)
aq_kwargs['mc_points'] = qmc_samples.squeeze(-2)
+ if aq == 'NParEGO':
+ w = hps['scalarization_weights']
+ if isinstance(w,list):
+ w = torch.tensor(w)
+ w = w/torch.sum(torch.abs(w))
+ aq_kwargs['scalarization_weights'] = w
+
return aq_kwargs
def suggest(self,
From 18e634e53cbff71e9de20344eebbcb37a2131caa Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 21:19:02 -0400
Subject: [PATCH 034/136] bugfix: make sure optim_sequential = False for aq_str
in ['NEHVI', 'EHVI']
---
obsidian/optimizer/bayesian.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/obsidian/optimizer/bayesian.py b/obsidian/optimizer/bayesian.py
index aab2e1a..6e16cdf 100644
--- a/obsidian/optimizer/bayesian.py
+++ b/obsidian/optimizer/bayesian.py
@@ -774,7 +774,7 @@ def suggest(self,
aq_kwargs.update(self._parse_aq_kwargs(aq_str, aq_hps, m_batch, target_locs, X_t_pending, objective))
# Hypervolume aqs fail with X_t_pending when optim_sequential=True
- if aq_i in ['NEHVI', 'EHVI']:
+ if aq_str in ['NEHVI', 'EHVI']:
optim_sequential = False
# If it's random search, no need to do optimization; Otherwise, initialize the aq function and optimize
From a898f35d80d3f7d943dca1ccfd8d4a9ffdcf75bd Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 22:07:10 -0400
Subject: [PATCH 035/136] revise MOO section in 5_AcquisitionFunction.md
---
docs/wiki/5_AcquisitionFunction.md | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 0cfb2e2..402b042 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -205,9 +205,29 @@ X_suggest, eval_suggest = optimizer.suggest(acquisition = [{'NEHVI':{'ref_point'
### Weighted Response
-* Random augmented chebyshev scalarization with Noisy Expected Improvement (NParEGO)
+Another widely used approach is aggregating multiple outcomes into a single objective function, which is usually a weighted sum of the individual outcomes plus some regularization or penalty terms.
-* Additional scalarization options for multi-objective problems
+* **Random augmented chebyshev scalarization with Noisy Expected Improvement (NParEGO)**
+
+ The mathematical formulation of augmented Chebyshev scalarization [[ref](https://github.com/pytorch/botorch/blob/main/botorch/utils/multi_objective/scalarization.py)]:
+ \begin{equation*}
+ objective = \max_j (w_j * y^j)+ \alpha * \sum_j(w_j * y^j)
+ \end{equation*}
+ where the constant $\alpha=0.05$, and the weights $\{w_j\}_{1\leq j\leq J}$ are samples from the unit simplex by default.
+
+_Optional hyperparameters:_
+* scalarization_weights: Default value is 'None', then random weights will be applied to search for global solution set on pareto front.
+
+**Example usage:**
+
+```python
+# Default, random weights
+X_suggest, eval_suggest = optimizer.suggest(acquisition = ['NParEGO'])
+# Fixed weights
+X_suggest, eval_suggest = optimizer.suggest(acquisition = [{'NParEGO':{'scalarization_weights':[0.75, 0.25]}}])
+```
+
+* Additional scalarization options for multi-objective optimization: Utilize various scalarization functions defined in [obsidian.objectives.scalarize](https://github.com/MSDLLCpapers/obsidian/blob/main/obsidian/objectives/scalarize.py) to combine multiple objectives into a single scalar value, then various single-objective optimization acquisition functions could be applied.
## 5. Advanced Usage
From 7bec63c505b780539595c8269c7d0b444c2b62f2 Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 22:15:25 -0400
Subject: [PATCH 036/136] minor changes
---
docs/wiki/5_AcquisitionFunction.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/wiki/5_AcquisitionFunction.md b/docs/wiki/5_AcquisitionFunction.md
index 402b042..47c0d36 100644
--- a/docs/wiki/5_AcquisitionFunction.md
+++ b/docs/wiki/5_AcquisitionFunction.md
@@ -223,7 +223,7 @@ _Optional hyperparameters:_
```python
# Default, random weights
X_suggest, eval_suggest = optimizer.suggest(acquisition = ['NParEGO'])
-# Fixed weights
+# Fixed weights, assuming there are two outputs
X_suggest, eval_suggest = optimizer.suggest(acquisition = [{'NParEGO':{'scalarization_weights':[0.75, 0.25]}}])
```
@@ -274,4 +274,4 @@ Different acquisition functions have different strengths:
3. Using single-objective acquisition functions for multi-objective problems.
4. Not accounting for constraints when selecting acquisition functions.
-This concludes the user guide for the `obsidian.acquisition` submodule. For more detailed information, please refer to the source code and docstrings in the individual files.
\ No newline at end of file
+This concludes the user guide for the [`obsidian.acquisition`](https://github.com/MSDLLCpapers/obsidian/tree/main/obsidian/acquisition) submodule. For more detailed information, please refer to the source code and docstrings in the individual files.
\ No newline at end of file
From caa6ea6c0ebf3a4dee025bd62d8684b6df806b7e Mon Sep 17 00:00:00 2001
From: Yuting Xu <12775874+xuyuting@users.noreply.github.com>
Date: Sun, 18 Aug 2024 22:25:26 -0400
Subject: [PATCH 037/136] add ACS 2024 presentation to publications.md
---
docs/_static/publications.md | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/docs/_static/publications.md b/docs/_static/publications.md
index 749b5af..0d73c6b 100644
--- a/docs/_static/publications.md
+++ b/docs/_static/publications.md
@@ -7,8 +7,11 @@ DOI: [10.1039/D3SC06884F](https://pubs.rsc.org/en/content/articlelanding/2024/sc
# Presentations
-* The 2024 Joint Statistical Meetings (JSM), Portland, Aug. 5, 2024.
- Poster at Biopharmaceutical Section: _Algorithmic Process Optimization in Pharmaceutical Development_
+* The ACS Fall 2024, Denver, CO, Aug. 19, 2024.
+ Oral Presentation: _Algorithmic process optimization (APO) for pharmaceutical development_ (Kevin Stone)
+
+* The 2024 Joint Statistical Meetings (JSM), Portland, OR, Aug. 5, 2024.
+ Poster Presentation at Biopharmaceutical Section: _Algorithmic Process Optimization in Pharmaceutical Development_ (Yuting Xu)
From efe5834175285d189ae4db0c5f42263eb51448ce Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 23:12:52 -0400
Subject: [PATCH 038/136] Simplified Xspace saving
---
app.py | 2 +-
obsidian/dash/inputs_config.py | 20 +------------
obsidian/dash/inputs_data.py | 54 ++++++++++++++++++++++++++++++----
obsidian/dash/optimize.py | 17 +++--------
obsidian/dash/predict.py | 12 +++++---
obsidian/dash/utils.py | 6 +++-
6 files changed, 68 insertions(+), 43 deletions(-)
diff --git a/app.py b/app.py
index 524389d..a758c89 100755
--- a/app.py
+++ b/app.py
@@ -46,7 +46,7 @@
# Set up each tab
setup_infobar(app, app_infobar)
-setup_data(app, app_tabs, default_data)
+setup_data(app, app_tabs, default_data, X_space)
setup_config(app, app_tabs)
setup_optimize(app, app_tabs)
setup_plots(app, app_tabs)
diff --git a/obsidian/dash/inputs_config.py b/obsidian/dash/inputs_config.py
index d0c21b1..86d499a 100644
--- a/obsidian/dash/inputs_config.py
+++ b/obsidian/dash/inputs_config.py
@@ -101,22 +101,6 @@ def toggle_collapse(n, is_open):
return not is_open
return is_open
- @app.callback(
- Output({'type': 'store-param_xspace', 'index': MATCH}, 'data'),
- Input({'type': 'store-param_xspace', 'index': MATCH}, 'id'),
- Input({'type': 'input-param_type', 'index': MATCH}, 'value'),
- Input({'type': 'input-param_min', 'index': MATCH}, 'value'),
- Input({'type': 'input-param_max', 'index': MATCH}, 'value'),
- Input({'type': 'store-param_categories', 'index': MATCH}, 'data'),
- prevent_initial_call=True # It takes a second for these input matches to show up
- )
- def update_param_xspace(param_id, param_type, param_min, param_max, param_cat):
-
- param_xspace = {'Name': param_id['index'], 'Type': param_type,
- 'Low': param_min, 'High': param_max, 'Categories': param_cat}
-
- return param_xspace
-
# Add acquisition function
@app.callback(
Output('div-acquisition_all', 'children'),
@@ -144,7 +128,6 @@ def func(del_clicked, aq_match):
# Store all of the input and settings into config
@app.callback(
Output('store-config', 'data'),
- Input({'type': 'store-param_xspace', 'index': ALL}, 'data'),
Input('input-response_name', 'value'),
Input('input-optimizer_seed', 'value'),
Input('input-f_transform', 'value'),
@@ -156,7 +139,7 @@ def func(del_clicked, aq_match):
Input({'type': 'input-alpha', 'index': ALL}, 'value'),
prevent_initial_call=True
)
- def compile_config(param_xspaces, response_name, optimizer_seed, f_transform, surrogate,
+ def compile_config(response_name, optimizer_seed, f_transform, surrogate,
m_batch, optim_sequential, optim_restarts, aq, alpha):
config = {}
@@ -168,7 +151,6 @@ def compile_config(param_xspaces, response_name, optimizer_seed, f_transform, su
config['aq_params'] = {'optim_sequential': optim_sequential, 'optim_restarts': optim_restarts,
'm_batch': m_batch, 'acquisition': aq}
config['verbose'] = 0
- config['xspace'] = param_xspaces
return config
diff --git a/obsidian/dash/inputs_data.py b/obsidian/dash/inputs_data.py
index 977a10d..ca5d157 100644
--- a/obsidian/dash/inputs_data.py
+++ b/obsidian/dash/inputs_data.py
@@ -6,9 +6,10 @@
import base64
import io
from dash.exceptions import PreventUpdate
+from obsidian.parameters import ParamSpace, Param_Categorical, Param_Ordinal, Param_Continuous, Target, Parameter
-def setup_data(app, app_tabs, default_data):
+def setup_data(app, app_tabs, default_data, default_Xspace):
# Data upload
uploader = dcc.Upload(id='uploader-X0',
@@ -53,7 +54,7 @@ def setup_data(app, app_tabs, default_data):
# Data store
storage_X0 = dcc.Store(id='store-X0', data=default_data.to_dict())
storage_X0_template = dcc.Store(id='store-X0_template', data=default_data.to_dict())
- # storage_Xspace = dcc.Store(id='store-Xspace', data={})
+ storage_Xspace = dcc.Store(id='store-Xspace', data=default_Xspace.save_state())
# Parameter space table
xspace = html.Div([html.Div(id='div-xspace', children=[])])
@@ -81,7 +82,7 @@ def setup_data(app, app_tabs, default_data):
# Add all of these elements to the app
elements = [html.Br(), preview_uploader, html.Hr(), ycol, xspace, storage_X0,
- storage_X0_template, html.Hr(), troubleshoot]
+ storage_X0_template, storage_Xspace, html.Hr(), troubleshoot]
add_tab(app_tabs, elements, 'tab-data', 'Data')
setup_data_callbacks(app)
@@ -112,7 +113,6 @@ def save_X0(contents):
)
def preview_X0(data, filename):
df = pd.DataFrame(data)
-
return make_table(df, fill_width=True), filename
# Download template data
@@ -157,7 +157,7 @@ def update_xspace_types(data, ycol):
id={'type': 'input-param_type', 'index': x},
kwargs={'value': param_types[0]}),
html.Div(id={'type': 'div-param_vals', 'index': x}, children=[]),
- dcc.Store(id={'type': 'store-param_xspace', 'index': x}, data={})],)
+ dcc.Store(id={'type': 'store-param_save', 'index': x}, data={})],)
],
color='primary', outline=True), width=2))
@@ -207,6 +207,33 @@ def update_xspace_vals(param_type, param_id, data, ycol):
]
return choices
+ @app.callback(
+ Output({'type': 'store-param_save', 'index': MATCH}, 'data'),
+ Input({'type': 'store-param_save', 'index': MATCH}, 'id'),
+ Input({'type': 'input-param_type', 'index': MATCH}, 'value'),
+ Input({'type': 'input-param_min', 'index': MATCH}, 'value'),
+ Input({'type': 'input-param_max', 'index': MATCH}, 'value'),
+ Input({'type': 'store-param_categories', 'index': MATCH}, 'data'),
+ prevent_initial_call=True # It takes a second for these input matches to show up
+ )
+ def update_param_save(param_id, param_type, param_min, param_max, param_cat):
+
+ name = param_id['index']
+
+ try:
+ if param_type == 'Numeric':
+ param = Param_Continuous(name, param_min, param_max)
+ elif param_type == 'Categorical':
+ param = Param_Categorical(name, param_cat)
+ elif param_type == 'Ordinal':
+ param = Param_Ordinal(name, param_cat)
+
+ single_param = ParamSpace([param])
+ return single_param.save_state()
+
+ except TypeError:
+ return None
+
# Category management for categorical variables
@app.callback(
Output({'type': 'store-param_categories', 'index': MATCH}, 'data'),
@@ -262,4 +289,21 @@ def preview_cats(current_cats):
def troubleshoot_config(data):
return
+ @app.callback(
+ Output('store-Xspace', 'data'),
+ Input({'type': 'store-param_save', 'index': ALL}, 'data'),
+ prevent_initial_call=True
+ )
+ def save_Xspace(param_saves):
+
+ param_list = []
+ for param_input in param_saves:
+ if param_input:
+ param_i = ParamSpace.load_state(param_input)
+ param_list += list(param_i.params)
+
+ X_space = ParamSpace(param_list)
+
+ return X_space.save_state()
+
return
diff --git a/obsidian/dash/optimize.py b/obsidian/dash/optimize.py
index e6876b1..23836de 100644
--- a/obsidian/dash/optimize.py
+++ b/obsidian/dash/optimize.py
@@ -77,24 +77,15 @@ def setup_optimize_callbacks(app):
Input('button-fit', 'n_clicks'),
State('store-config', 'data'),
State('store-X0', 'data'),
+ State('store-Xspace', 'data'),
prevent_initial_call=True
)
- def fit_optimizer(fit_clicked, config, X0):
+ def fit_optimizer(fit_clicked, config, X0, Xspace_save):
- if config['xspace'] == {}:
+ if not Xspace_save:
return 0, None
- param_list = []
- for param_input in config['xspace']:
- if param_input['Type'] == 'Numeric':
- param = Param_Continuous(param_input['Name'], param_input['Low'], param_input['High'])
- elif param_input['Type'] == 'Categorical':
- param = Param_Categorical(param_input['Name'], param_input['Categories'])
- elif param_input['Type'] == 'Ordinal':
- param = Param_Ordinal(param_input['Name'], param_input['Categories'])
- param_list.append(param)
-
- X_space = ParamSpace(param_list)
+ X_space = ParamSpace.load_state(Xspace_save)
target = Target(config['response_name'], aim='max')
campaign = Campaign(X_space, target)
campaign.add_data(pd.DataFrame(X0))
diff --git a/obsidian/dash/predict.py b/obsidian/dash/predict.py
index 3784da4..ce6338c 100644
--- a/obsidian/dash/predict.py
+++ b/obsidian/dash/predict.py
@@ -103,13 +103,17 @@ def setup_predict_callbacks(app):
Output('div-xspace_df', 'children'),
Output('div-xspace_df', 'style'),
Input('button-config', 'n_clicks'),
- State('store-config', 'data'),
+ State('store-Xspace', 'data'),
)
- def config_tableView(clicked, config):
- if config is None:
+ def config_tableView(clicked, Xspace_save):
+ if not Xspace_save:
return 0, None, {'overflow-x': 'scroll'}
- df_xspace = pd.DataFrame(config['xspace'])
+ Xspace_list = []
+ for param in Xspace_save.keys():
+ Xspace_list.append(Xspace_save[param]['state'])
+
+ df_xspace = pd.DataFrame(Xspace_list)
tables = [center(make_table(df_xspace))]
return 0, tables, {'overflow-x': 'scroll'}
diff --git a/obsidian/dash/utils.py b/obsidian/dash/utils.py
index 70ff30d..9bf06be 100644
--- a/obsidian/dash/utils.py
+++ b/obsidian/dash/utils.py
@@ -13,11 +13,15 @@ def center(element):
def load_optimizer(config, opt_save):
- # Note: it doesn't need config anymore?
optimizer = BayesianOptimizer.load_state(opt_save)
return optimizer
+def load_Xspace(config, Xspace_save):
+ X_space = ParamSpace.load_state(Xspace_save)
+ return X_space
+
+
def add_tab(target, elements, id, label):
target.children = list(target.children) if target.children else []
tab = dbc.Tab(elements, label=label, id=id)
From ac980f589fccccd82afc2a3be8344103743fa1cd Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 23:38:43 -0400
Subject: [PATCH 039/136] Parameter types inferred based on dtype
---
app.py | 2 +-
obsidian/dash/inputs_data.py | 25 ++++++++++++++++++++++---
obsidian/dash/predict.py | 8 +++++++-
3 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/app.py b/app.py
index a758c89..74ed3df 100755
--- a/app.py
+++ b/app.py
@@ -35,7 +35,7 @@
Param_Continuous('Concentration', 10, 150),
Param_Continuous('Enzyme', 0.01, 0.30),
Param_Categorical('Variant', ['MRK001', 'MRK002', 'MRK003']),
- Param_Ordinal('Stir Rate', ['Low', 'Medium', 'High']),
+ Param_Ordinal('Stir Rate', ['0 - Low', '1 - Medium', '2 - High']),
]
X_space = ParamSpace(params)
designer = ExpDesigner(X_space, seed=0)
diff --git a/obsidian/dash/inputs_data.py b/obsidian/dash/inputs_data.py
index ca5d157..bdfa024 100644
--- a/obsidian/dash/inputs_data.py
+++ b/obsidian/dash/inputs_data.py
@@ -7,6 +7,7 @@
import io
from dash.exceptions import PreventUpdate
from obsidian.parameters import ParamSpace, Param_Categorical, Param_Ordinal, Param_Continuous, Target, Parameter
+from pandas.api.types import is_numeric_dtype
def setup_data(app, app_tabs, default_data, default_Xspace):
@@ -140,9 +141,12 @@ def choose_col(data):
@app.callback(
Output('div-xspace', 'children'),
Input('store-X0', 'data'),
- Input('input-response_name', 'value')
+ Input('input-response_name', 'value'),
+ State('store-Xspace', 'data')
)
- def update_xspace_types(data, ycol):
+ def update_xspace_types(data, ycol, Xspace_save):
+
+ X_space = ParamSpace.load_state(Xspace_save)
df_X0 = pd.DataFrame(data)
xcols = [x for x in df_X0.columns if x != ycol]
@@ -151,11 +155,26 @@ def update_xspace_types(data, ycol):
cols = []
for i, x in enumerate(xcols):
+ if x in X_space.X_names:
+ x_idx = X_space.X_names.index(x)
+ param = X_space.params[x_idx]
+ if isinstance(param, Param_Continuous):
+ param_type = 'Numeric'
+ elif isinstance(param, Param_Categorical):
+ param_type = 'Categorical'
+ elif isinstance(param, Param_Ordinal):
+ param_type = 'Ordinal'
+ else:
+ if not is_numeric_dtype(df_X0[x]):
+ param_type = 'Categorical'
+ else:
+ param_type = 'Numeric'
+
cols.append(dbc.Col(children=dbc.Card([
dbc.CardHeader(f'{x}'),
dbc.CardBody([make_dropdown('Type', f'Select parameter type for {x}', param_types,
id={'type': 'input-param_type', 'index': x},
- kwargs={'value': param_types[0]}),
+ kwargs={'value': param_type}),
html.Div(id={'type': 'div-param_vals', 'index': x}, children=[]),
dcc.Store(id={'type': 'store-param_save', 'index': x}, data={})],)
],
diff --git a/obsidian/dash/predict.py b/obsidian/dash/predict.py
index ce6338c..9d55492 100644
--- a/obsidian/dash/predict.py
+++ b/obsidian/dash/predict.py
@@ -104,6 +104,7 @@ def setup_predict_callbacks(app):
Output('div-xspace_df', 'style'),
Input('button-config', 'n_clicks'),
State('store-Xspace', 'data'),
+ prevent_initial_call=True
)
def config_tableView(clicked, Xspace_save):
if not Xspace_save:
@@ -111,7 +112,12 @@ def config_tableView(clicked, Xspace_save):
Xspace_list = []
for param in Xspace_save.keys():
- Xspace_list.append(Xspace_save[param]['state'])
+ state_dict = Xspace_save[param]['state']
+ row = {'Parameter': state_dict['name']}
+ row['Min'] = state_dict.get('min', None)
+ row['Max'] = state_dict.get('max', None)
+ row['Categories'] = str(state_dict.get('categories', None))
+ Xspace_list.append(row)
df_xspace = pd.DataFrame(Xspace_list)
tables = [center(make_table(df_xspace))]
From 466b9e5dfd245f7e31d9f84e6845483b1479cf36 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Sun, 18 Aug 2024 23:43:28 -0400
Subject: [PATCH 040/136] Bump version
---
CHANGELOG.md | 6 ++++++
obsidian/__init__.py | 2 +-
pyproject.toml | 2 +-
3 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0e407ff..8eebd40 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,10 @@
# Changelog
+## [0.8.1]
+### Modified
+- Fixed infobar on dash app
+- Better handling of X_space on dash app
+- Bug fixes for optim_progress
+- Improved color and axes of parity_plot
## [0.8.0]
### Added
diff --git a/obsidian/__init__.py b/obsidian/__init__.py
index a01269c..dec54fe 100644
--- a/obsidian/__init__.py
+++ b/obsidian/__init__.py
@@ -1,5 +1,5 @@
"""obsidian: Automated experiment design and black-box optimization"""
-__version__ = '0.8.0'
+__version__ = '0.8.1'
# Import key objects
from obsidian.campaign import Campaign
diff --git a/pyproject.toml b/pyproject.toml
index 777221d..b60c9a9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "obsidian-apo"
-version = "0.8.0"
+version = "0.8.1"
description = "Automated experiment design and black-box optimization"
authors = [
"Kevin Stone ",
From 25d8bb1d0da1b80afb8bc4bf07a06ef925745144 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Mon, 19 Aug 2024 00:08:00 -0400
Subject: [PATCH 041/136] Bump version
---
CHANGELOG.md | 5 +++++
obsidian/__init__.py | 2 +-
pyproject.toml | 18 +++++++-----------
3 files changed, 13 insertions(+), 12 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8eebd40..0e3fb5b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,9 @@
# Changelog
+
+## [0.8.2]
+### Added
+- Project metadata properly captured on PyPI based on changes in pyproject.toml
+
## [0.8.1]
### Modified
- Fixed infobar on dash app
diff --git a/obsidian/__init__.py b/obsidian/__init__.py
index dec54fe..bb26ea7 100644
--- a/obsidian/__init__.py
+++ b/obsidian/__init__.py
@@ -1,5 +1,5 @@
"""obsidian: Automated experiment design and black-box optimization"""
-__version__ = '0.8.1'
+__version__ = '0.8.2'
# Import key objects
from obsidian.campaign import Campaign
diff --git a/pyproject.toml b/pyproject.toml
index b60c9a9..0649c4f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,19 +1,12 @@
[tool.poetry]
name = "obsidian-apo"
-version = "0.8.1"
+version = "0.8.2"
description = "Automated experiment design and black-box optimization"
authors = [
"Kevin Stone ",
"Yuting Xu "
]
-packages = [
- { include = "obsidian" },
-]
-
-[project]
readme = "readme.md"
-requires-python = ">=3.10"
-license = {text = "GPLv3"}
classifiers = [
"Development Status :: 4 - Beta",
"Programming Language :: Python :: 3.10",
@@ -28,11 +21,14 @@ keywords = [
"APO",
"DOE"
]
+homepage = "https://msdllcpapers.github.io/obsidian/"
+repository = "https://github.com/MSDLLCpapers/obsidian/"
+packages = [
+ { include = "obsidian" },
+]
-[project.urls]
-Homepage = "https://msdllcpapers.github.io/obsidian/"
+[tool.poetry.urls]
Documentation = "https://msdllcpapers.github.io/obsidian/stubs/api_docs.html"
-Repository = "https://github.com/MSDLLCpapers/obsidian/"
Changelog = "https://msdllcpapers.github.io/obsidian/stubs/changelog.html"
[tool.poetry.dependencies]
From e89140b3d8bee98b8c16e73f1aba228d878dafb7 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Mon, 19 Aug 2024 00:23:37 -0400
Subject: [PATCH 042/136] Removed javascript, fixed dark mode logo reaction
---
docs/index.rst | 9 ++++++++-
readme.md | 15 ++-------------
2 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/docs/index.rst b/docs/index.rst
index c1d9270..e04334b 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -20,5 +20,12 @@
stubs/reference
+.. raw:: html
+
+
+
+
+
.. include:: ../readme.md
- :parser: myst_parser.sphinx_
\ No newline at end of file
+ :parser: myst_parser.sphinx_
+
diff --git a/readme.md b/readme.md
index 80e15ff..0fe9174 100644
--- a/readme.md
+++ b/readme.md
@@ -3,9 +3,8 @@ obsidian
ReadMe
-->
-
-
+
@@ -136,14 +135,4 @@ See [CONTRIBUTING](https://github.com/MSDLLCpapers/obsidian/blob/main/CONTRIBUTI
- Kobi Felton (Merck & Co., Inc.)
## License
-__obsidian__ is licensed by the [GPLv3 license](https://github.com/MSDLLCpapers/obsidian/blob/main/LICENSE).
-
-JavaScript for ReadMe dark mode
-
-
+__obsidian__ is licensed by the [GPLv3 license](https://github.com/MSDLLCpapers/obsidian/blob/main/LICENSE).
\ No newline at end of file
From cd8e9ea973a750caf8b35de45693a62aa382192b Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Tue, 20 Aug 2024 19:57:08 -0400
Subject: [PATCH 043/136] Improved github actions
---
.github/workflows/CD.yml | 28 +++++++++++++++++++
.../workflows/{documentation.yml => CI.yml} | 23 +++++++++++++--
.github/workflows/codecov.yml | 21 --------------
pyproject.toml | 2 +-
4 files changed, 49 insertions(+), 25 deletions(-)
create mode 100644 .github/workflows/CD.yml
rename .github/workflows/{documentation.yml => CI.yml} (51%)
delete mode 100644 .github/workflows/codecov.yml
diff --git a/.github/workflows/CD.yml b/.github/workflows/CD.yml
new file mode 100644
index 0000000..7853825
--- /dev/null
+++ b/.github/workflows/CD.yml
@@ -0,0 +1,28 @@
+# Publishes to PyPi on release tag
+name: Continuous Deployment
+
+on:
+ push:
+ tags:
+ - "v*.*.*"
+
+jobs:
+ dev-publish:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - name: Build and publish to test pypi
+ uses: JRubics/poetry-publish@v2.0
+ with:
+ pypi_token: ${{ secrets.TEST_PYPI_TOKEN }}
+ repository_name: "testpypi"
+ repository_url: "https://test.pypi.org/legacy/"
+
+ prod-publish:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - name: Build and publish to pypi
+ uses: JRubics/poetry-publish@v2.0
+ with:
+ pypi_token: ${{ secrets.PYPI_TOKEN }}
diff --git a/.github/workflows/documentation.yml b/.github/workflows/CI.yml
similarity index 51%
rename from .github/workflows/documentation.yml
rename to .github/workflows/CI.yml
index b57de56..3787c66 100644
--- a/.github/workflows/documentation.yml
+++ b/.github/workflows/CI.yml
@@ -1,11 +1,28 @@
-name: documentation
+name: Continuous Integration
-on: [push, pull_request, workflow_dispatch]
+on: [push, pull_request]
permissions:
contents: write
jobs:
+ test:
+ runs-on: ubuntu-latest
+ name: PyTest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v2
+ with:
+ python-version: '3.10'
+ - name: Install requirements
+ run: pip install obsidian-apo[dev]
+ - name: Run tests and collect coverage
+ run: pytest --cov=.
+ - name: Upload coverage reports to Codecov with GitHub Action
+ uses: codecov/codecov-action@v4.2.0
+ env:
+ CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
docs:
runs-on: ubuntu-latest
steps:
@@ -24,4 +41,4 @@ jobs:
publish_branch: gh-pages
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: _build/
- force_orphan: true
+ force_orphan: true
\ No newline at end of file
diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml
deleted file mode 100644
index 43e2f6a..0000000
--- a/.github/workflows/codecov.yml
+++ /dev/null
@@ -1,21 +0,0 @@
-name: CI
-
-on: [push, pull_request]
-
-jobs:
- build:
- runs-on: ubuntu-latest
- name: Pytest
- steps:
- - uses: actions/checkout@v4
- - uses: actions/setup-python@v2
- with:
- python-version: '3.10'
- - name: Install requirements
- run: pip install obsidian-apo[dev]
- - name: Run tests and collect coverage
- run: pytest --cov=.
- - name: Upload coverage reports to Codecov with GitHub Action
- uses: codecov/codecov-action@v4.2.0
- env:
- CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
diff --git a/pyproject.toml b/pyproject.toml
index 0649c4f..059c39a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ readme = "readme.md"
classifiers = [
"Development Status :: 4 - Beta",
"Programming Language :: Python :: 3.10",
- "License :: OSI Approved :: GPLv3",
+ "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent"
]
keywords = [
From a8ea740bb8791a3b6f27cdaa3a5c406774a2956f Mon Sep 17 00:00:00 2001
From: Kevin Stone <91576158+kstone40@users.noreply.github.com>
Date: Tue, 20 Aug 2024 20:07:52 -0400
Subject: [PATCH 044/136] Removed "v" requirement from tag on CD
---
.github/workflows/CD.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/CD.yml b/.github/workflows/CD.yml
index 7853825..b286f14 100644
--- a/.github/workflows/CD.yml
+++ b/.github/workflows/CD.yml
@@ -4,7 +4,7 @@ name: Continuous Deployment
on:
push:
tags:
- - "v*.*.*"
+ - "*.*.*"
jobs:
dev-publish:
From 4be7df11eca1f5726bfcc1760beda182932b1680 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 10:51:58 -0400
Subject: [PATCH 045/136] Added default weights to NParEGO
---
obsidian/optimizer/bayesian.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/obsidian/optimizer/bayesian.py b/obsidian/optimizer/bayesian.py
index 6e16cdf..df7c122 100644
--- a/obsidian/optimizer/bayesian.py
+++ b/obsidian/optimizer/bayesian.py
@@ -459,8 +459,8 @@ def _validate_hypers(self,
if hps.get(key) is None:
if not defaults['optional']:
raise ValueError(f'Must specify hyperpameter value {key} for {aq_str}')
- if key in ['weights']: #['scalarization_weights', 'weights']:
- aq_hps[key] = defaults['val'] * o_dim
+ if key in ['weights', 'scalarization_weights']:
+ aq_hps[key] = [1] * o_dim
else:
aq_hps[key] = defaults['val']
@@ -553,7 +553,7 @@ def _parse_aq_kwargs(self,
if aq == 'NParEGO':
w = hps['scalarization_weights']
- if isinstance(w,list):
+ if isinstance(w, list):
w = torch.tensor(w)
w = w/torch.sum(torch.abs(w))
aq_kwargs['scalarization_weights'] = w
From 78a311e68d1ba30030c522c5495847e69f41cd76 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 09:00:32 -0600
Subject: [PATCH 046/136] Add pip-caching to CI workflow
---
.github/workflows/CI.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
index 3787c66..9e53232 100644
--- a/.github/workflows/CI.yml
+++ b/.github/workflows/CI.yml
@@ -14,6 +14,7 @@ jobs:
- uses: actions/setup-python@v2
with:
python-version: '3.10'
+ cache: 'pip'
- name: Install requirements
run: pip install obsidian-apo[dev]
- name: Run tests and collect coverage
@@ -28,6 +29,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
+ with:
+ python-version: '3.10'
+ cache: 'pip'
- name: Install dependencies
run: |
pip install obsidian-apo[docs]
From 19454a27f9f8d283b8a0b168e75b0f9c04659085 Mon Sep 17 00:00:00 2001
From: Kevin Stone <91576158+kstone40@users.noreply.github.com>
Date: Wed, 21 Aug 2024 09:11:00 -0600
Subject: [PATCH 047/136] Remove pip cache
---
.github/workflows/CI.yml | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
index 9e53232..1153a60 100644
--- a/.github/workflows/CI.yml
+++ b/.github/workflows/CI.yml
@@ -14,7 +14,6 @@ jobs:
- uses: actions/setup-python@v2
with:
python-version: '3.10'
- cache: 'pip'
- name: Install requirements
run: pip install obsidian-apo[dev]
- name: Run tests and collect coverage
@@ -31,7 +30,6 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- cache: 'pip'
- name: Install dependencies
run: |
pip install obsidian-apo[docs]
@@ -45,4 +43,4 @@ jobs:
publish_branch: gh-pages
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: _build/
- force_orphan: true
\ No newline at end of file
+ force_orphan: true
From 73efe83c7987c97e6e7c6465d5a02ea46c4c17d3 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 11:55:54 -0600
Subject: [PATCH 048/136] PDP ICE plot now works with categorical inputs
---
obsidian/campaign/explainer.py | 2 +-
obsidian/plotting/shap.py | 116 ++++++++++++++++++++++++---------
2 files changed, 87 insertions(+), 31 deletions(-)
diff --git a/obsidian/campaign/explainer.py b/obsidian/campaign/explainer.py
index b44c969..7bfca4b 100644
--- a/obsidian/campaign/explainer.py
+++ b/obsidian/campaign/explainer.py
@@ -141,7 +141,7 @@ def shap_summary_bar(self) -> Figure:
def shap_pdp_ice(self,
ind: int | tuple[int] = 0,
- ice_color_var: int = 0,
+ ice_color_var: int | None = None,
ace_opacity: float = 0.5,
npoints: int | None = None,
) -> Figure:
diff --git a/obsidian/plotting/shap.py b/obsidian/plotting/shap.py
index 6f11785..0a52672 100644
--- a/obsidian/plotting/shap.py
+++ b/obsidian/plotting/shap.py
@@ -1,6 +1,7 @@
"""Custom plots for SHAP analysis visualization"""
from .branding import obsidian_cm, obsidian_colors
+from obsidian.exceptions import UnsupportedError
from shap.plots._partial_dependence import compute_bounds
from shap.utils import convert_name
@@ -79,7 +80,7 @@ def one_shap_value(shap_value_new: np.ndarray,
def partial_dependence(ind: int | tuple[int],
model: Callable,
data: pd.DataFrame,
- ice_color_var: int | str | None = 0,
+ ice_color_var: int | str | None = None,
xmin: str | tuple[float] | float = "percentile(0)",
xmax: str | tuple[float] | float = "percentile(100)",
npoints: int | None = None,
@@ -149,10 +150,20 @@ def partial_dependence(ind: int | tuple[int],
if type(ind) is not tuple:
ind = convert_name(ind, None, feature_names)
xv = features[:, ind]
- xmin, xmax = compute_bounds(xmin, xmax, xv)
- npoints = 100 if npoints is None else npoints
- xs = np.linspace(xmin, xmax, npoints)
-
+ try:
+ xv = xv.astype(float)
+ except ValueError:
+ pass
+ if xv.dtype == np.dtype('O'):
+ xs = np.array(sorted(set(xv)))
+ npoints = len(xs)
+ xmin = xs[0]
+ xmax = xs[-1]
+ else:
+ xmin, xmax = compute_bounds(xmin, xmax, xv)
+ npoints = 100 if npoints is None else npoints
+ xs = np.linspace(xmin, xmax, npoints)
+
if ice:
features_tmp = features.copy()
ice_vals = np.zeros((npoints, features.shape[0]))
@@ -197,11 +208,25 @@ def partial_dependence(ind: int | tuple[int],
else:
if not isinstance(ice_color_var, int):
ice_color_var = convert_name(ice_color_var, None, feature_names)
+ if ice_color_var == ind:
+ raise UnsupportedError("Coloring by the feature(s) used in the PDP is not supported.")
colormap = obsidian_cm.obsidian_viridis
- color_vals = features[:, ice_color_var]
- colorbar_min = color_vals.min()
- colorbar_max = color_vals.max()
- color_vals = (color_vals - colorbar_min)/(colorbar_max-colorbar_min)
+ xc = features[:, ice_color_var]
+ # Try to convert to float for color mapping
+ try:
+ xc = xc.astype(float)
+ except ValueError:
+ pass
+ if xc.dtype == np.dtype('O'):
+ xc_cat_vals = sorted(set(xc))
+ colorbar_min = 0
+ colorbar_max = len(xc_cat_vals) - 1
+ color_vals = [xc_cat_vals.index(c)/colorbar_max for c in xc]
+ else:
+ color_vals = xc
+ colorbar_min = color_vals.min()
+ colorbar_max = color_vals.max()
+ color_vals = (color_vals - colorbar_min)/(colorbar_max-colorbar_min)
for i in range(ice_vals.shape[1]):
ax1.plot(xs, ice_vals[:, i], color=colormap(color_vals[i]),
linewidth=ace_linewidth, alpha=ace_opacity)
@@ -209,6 +234,9 @@ def partial_dependence(ind: int | tuple[int],
norm=plt.Normalize(
vmin=colorbar_min, vmax=colorbar_max)),
ax=ax1)
+ if xc.dtype == np.dtype('O'):
+ cbar.set_ticks(np.linspace(0, colorbar_max, len(xc_cat_vals)))
+ cbar.set_ticklabels(xc_cat_vals)
cbar.set_label('Color by ' + feature_names[ice_color_var])
# the line plot
@@ -222,6 +250,7 @@ def partial_dependence(ind: int | tuple[int],
ylabel = "E[f(x) | " + str(feature_names[ind]) + "]"
else:
ylabel = "f(x) | " + str(feature_names[ind])
+
ax1.set_ylabel(ylabel, fontsize=13)
ax1.xaxis.set_ticks_position('bottom')
ax1.yaxis.set_ticks_position('left')
@@ -246,39 +275,66 @@ def partial_dependence(ind: int | tuple[int],
else:
ind0 = convert_name(ind[0], None, feature_names)
ind1 = convert_name(ind[1], None, feature_names)
+ if ind0 == ind1:
+ raise UnsupportedError("The two features must be different for 2D PDP.")
xv0 = features[:, ind0]
+ try:
+ xv0 = xv0.astype(float)
+ except ValueError:
+ pass
+
xv1 = features[:, ind1]
+ try:
+ xv1 = xv1.astype(float)
+ except ValueError:
+ pass
+
+ xmin = list(xmin) if isinstance(xmin, tuple) else list([xmin, xmin])
+ xmax = list(xmax) if isinstance(xmax, tuple) else list([xmax, xmax])
+ xs = []
+ x = []
+ cat_list = []
+ npoints = [20, 20] if npoints is None else [npoints, npoints]
+
+ for i, xv in enumerate([xv0, xv1]):
+ if xv.dtype == np.dtype('O'):
+ cat_list.append(sorted(set(xv)))
+ xmin[i] = 0
+ xmax[i] = len(cat_list[i])
+ xs.append(np.linspace(xmin[i], xmax[i]-1e-6, npoints[i]))
+ x.append([cat_list[i][int(xi)] for xi in xs[i]])
+ npoints[i] = len(xs[i])
- xmin0 = xmin[0] if type(xmin) is tuple else xmin
- xmin1 = xmin[1] if type(xmin) is tuple else xmin
- xmax0 = xmax[0] if type(xmax) is tuple else xmax
- xmax1 = xmax[1] if type(xmax) is tuple else xmax
-
- xmin0, xmax0 = compute_bounds(xmin0, xmax0, xv0)
- xmin1, xmax1 = compute_bounds(xmin1, xmax1, xv1)
- npoints = 20 if npoints is None else npoints
- xs0 = np.linspace(xmin0, xmax0, npoints)
- xs1 = np.linspace(xmin1, xmax1, npoints)
-
- features_tmp = features.copy()
- x0 = np.zeros((npoints, npoints))
- x1 = np.zeros((npoints, npoints))
- vals = np.zeros((npoints, npoints))
- for i in range(npoints):
- for j in range(npoints):
- features_tmp[:, ind0] = xs0[i]
- features_tmp[:, ind1] = xs1[j]
- x0[i, j] = xs0[i]
- x1[i, j] = xs1[j]
+ else:
+ xmin[i], xmax[i] = compute_bounds(xmin[i], xmax[i], xv)
+ xs.append(np.linspace(xmin[i], xmax[i], npoints[i]))
+ x.append(np.linspace(xmin[i], xmax[i], npoints[i]))
+ cat_list.append(None)
+
+ features_tmp = pd.DataFrame(features, columns=feature_names)
+ vals = np.zeros((npoints[0], npoints[1]))
+ for i in range(npoints[0]):
+ for j in range(npoints[1]):
+ features_tmp.iloc[:, ind0] = x[0][i]
+ features_tmp.iloc[:, ind1] = x[1][j]
vals[i, j] = model(features_tmp).mean()
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
+ x0, x1 = np.meshgrid(xs[0], xs[1], indexing='ij')
ax.plot_surface(x0, x1, vals, cmap=obsidian_cm.obsidian_viridis)
ax.set_xlabel(feature_names[ind0], fontsize=13)
+ if xv0.dtype == np.dtype('O'):
+ ax.set_xticks(np.arange(len(cat_list[0])))
+ ax.set_xticklabels(cat_list[0])
+
ax.set_ylabel(feature_names[ind1], fontsize=13)
+ if xv1.dtype == np.dtype('O'):
+ ax.set_yticks(np.arange(len(cat_list[1])))
+ ax.set_yticklabels(cat_list[1])
+
ax.set_zlabel("E[f(x) | " + str(feature_names[ind0]) + ", " + str(feature_names[ind1]) + "]", fontsize=13)
if show:
From a97763c0c0b9ae2d9014ed71cf3f67e3989cda9d Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 11:57:28 -0600
Subject: [PATCH 049/136] Bump version
---
CHANGELOG.md | 8 ++++++++
obsidian/__init__.py | 2 +-
pyproject.toml | 2 +-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0e3fb5b..709c6f6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,13 @@
# Changelog
+## [0.8.3]
+### Added
+- Default values for NParEGO scalarization_weights
+- SHAP PDP ICE plots now work with categorical values
+
+### Modified
+- SHAP PDP ICE plots must now have color and x-axis indices that are distinct
+
## [0.8.2]
### Added
- Project metadata properly captured on PyPI based on changes in pyproject.toml
diff --git a/obsidian/__init__.py b/obsidian/__init__.py
index bb26ea7..e1be93d 100644
--- a/obsidian/__init__.py
+++ b/obsidian/__init__.py
@@ -1,5 +1,5 @@
"""obsidian: Automated experiment design and black-box optimization"""
-__version__ = '0.8.2'
+__version__ = '0.8.3'
# Import key objects
from obsidian.campaign import Campaign
diff --git a/pyproject.toml b/pyproject.toml
index 059c39a..8002663 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "obsidian-apo"
-version = "0.8.2"
+version = "0.8.3"
description = "Automated experiment design and black-box optimization"
authors = [
"Kevin Stone ",
From 9a6807a8d86114de81733ef7c099ab7286c96957 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 19:40:06 -0600
Subject: [PATCH 050/136] Greatly refactored shap PDP ICE for simplicity
---
obsidian/campaign/explainer.py | 7 +-
obsidian/plotting/shap.py | 156 ++++++++++++++-------------------
2 files changed, 70 insertions(+), 93 deletions(-)
diff --git a/obsidian/campaign/explainer.py b/obsidian/campaign/explainer.py
index 7bfca4b..c5b5142 100644
--- a/obsidian/campaign/explainer.py
+++ b/obsidian/campaign/explainer.py
@@ -142,6 +142,7 @@ def shap_summary_bar(self) -> Figure:
def shap_pdp_ice(self,
ind: int | tuple[int] = 0,
ice_color_var: int | None = None,
+ hist: bool = False,
ace_opacity: float = 0.5,
npoints: int | None = None,
) -> Figure:
@@ -151,7 +152,9 @@ def shap_pdp_ice(self,
Args:
ind (int): Index of the parameter to plot
ice_color_var (int): Index of the parameter to color the ICE lines
- ace_opacity (float): Opacity of the ACE line
+ hist (bool, optional): Show histogram of the feature values.
+ By default ``False``
+ ace_opacity (float, optional): Opacity of the ACE line. By default ``0.5``
npoints (int, optional): Number of points for PDP x-axis. By default
will use ``100`` for 1D PDP and ``20`` for 2D PDP.
@@ -167,7 +170,7 @@ def shap_pdp_ice(self,
model=self.shap['pred_func'],
data=self.shap['X_sample'],
ice_color_var=ice_color_var,
- hist=False,
+ hist=hist,
ace_opacity=ace_opacity,
show=False,
npoints=npoints
diff --git a/obsidian/plotting/shap.py b/obsidian/plotting/shap.py
index 0a52672..d30a768 100644
--- a/obsidian/plotting/shap.py
+++ b/obsidian/plotting/shap.py
@@ -4,7 +4,6 @@
from obsidian.exceptions import UnsupportedError
from shap.plots._partial_dependence import compute_bounds
-from shap.utils import convert_name
import matplotlib.pyplot as plt
from matplotlib.axes import Axes
@@ -12,6 +11,7 @@
import numpy as np
import pandas as pd
+from pandas.api.types import is_numeric_dtype
from typing import Callable
@@ -80,11 +80,10 @@ def one_shap_value(shap_value_new: np.ndarray,
def partial_dependence(ind: int | tuple[int],
model: Callable,
data: pd.DataFrame,
- ice_color_var: int | str | None = None,
+ ice_color_var: int | None = None,
xmin: str | tuple[float] | float = "percentile(0)",
xmax: str | tuple[float] | float = "percentile(100)",
npoints: int | None = None,
- feature_names: list[str] | None = None,
hist: bool = False,
ylabel: str | None = None,
ice: bool = True,
@@ -106,7 +105,7 @@ def partial_dependence(ind: int | tuple[int],
the partial dependence for.
model (Callable): The model used for prediction.
data (pd.DataFrame): The input data used for prediction.
- ice_color_var (int | str, optional): The index of the feature used for coloring
+ ice_color_var (int, optional): The index of the feature used for coloring
the ICE lines (for 1D partial dependence plot). Default is ``0``.
xmin (str | tuple | float, optional): The minimum value(s) for the feature(s) range.
Default is ``"percentile(0)"``.
@@ -114,8 +113,6 @@ def partial_dependence(ind: int | tuple[int],
Default is ``"percentile(100)"``.
npoints (int, optional): The number of points to sample within the feature(s) range.
By default, will use ``100`` points for 1D PDP and ``20`` points for 2D PDP.
- feature_names (list[str], optional): The names of the features. Will default to
- the names of the DataFrame columns.
hist (bool, optional): Whether to plot the histogram of the feature(s). Default
is ``False``.
ylabel (str, optional): The label for the y-axis. Default is ``None``.
@@ -133,28 +130,26 @@ def partial_dependence(ind: int | tuple[int],
tuple: A tuple containing the matplotlib figure and axis objects if `show` is False, otherwise None.
"""
- features = data
-
- # Convert from DataFrame if used
- use_dataframe = False
- if isinstance(features, pd.DataFrame):
- if feature_names is None:
- feature_names = features.columns
- features = features.values
- use_dataframe = True
+ def cast_to_float(x: np.ndarray) -> np.ndarray:
+ """
+ Attempts to cast array to float type
+ """
+ try:
+ return x.astype(float)
+ except ValueError:
+ return x
- if feature_names is None:
- feature_names = ["Feature %d" % i for i in range(features.shape[1])]
+ # Extract vals, names from data
+ df_features = data
+ feature_names = df_features.columns
# 1D PDP
- if type(ind) is not tuple:
- ind = convert_name(ind, None, feature_names)
- xv = features[:, ind]
- try:
- xv = xv.astype(float)
- except ValueError:
- pass
- if xv.dtype == np.dtype('O'):
+ if not isinstance(ind, tuple):
+ # xv = values, independent variable
+ # xs = scaled values
+ xv = df_features.iloc[:, ind]
+ df_test = df_features.copy()
+ if not is_numeric_dtype(xv):
xs = np.array(sorted(set(xv)))
npoints = len(xs)
xmin = xs[0]
@@ -162,27 +157,21 @@ def partial_dependence(ind: int | tuple[int],
else:
xmin, xmax = compute_bounds(xmin, xmax, xv)
npoints = 100 if npoints is None else npoints
- xs = np.linspace(xmin, xmax, npoints)
-
+ xs = np.linspace(xmin, xmax, npoints).astype(xv.dtype)
if ice:
- features_tmp = features.copy()
- ice_vals = np.zeros((npoints, features.shape[0]))
+ df_test = df_features.copy()
+ ice_vals = []
for i in range(npoints):
- features_tmp[:, ind] = xs[i]
- if use_dataframe:
- ice_vals[i, :] = model(pd.DataFrame(features_tmp, columns=feature_names))
- else:
- ice_vals[i, :] = model(features_tmp)
-
- features_tmp = features.copy()
- vals = np.zeros(npoints)
+ df_test.iloc[:, ind] = xs[i]
+ ice_vals.append(model(df_test))
+ ice_vals = np.array(ice_vals)
+
+ vals = []
for i in range(npoints):
- features_tmp[:, ind] = xs[i]
- if use_dataframe:
- vals[i] = model(pd.DataFrame(features_tmp, columns=feature_names)).mean()
- else:
- vals[i] = model(features_tmp).mean()
-
+ df_test.iloc[:, ind] = xs[i]
+ vals.append(model(df_test).mean())
+ vals = np.array(vals)
+
if ax is None:
fig = plt.figure()
ax1 = plt.gca()
@@ -192,32 +181,24 @@ def partial_dependence(ind: int | tuple[int],
ax2 = ax1.twinx()
- # the histogram of the data
+ # Histogram
if hist:
- # n, bins, patches =
- ax2.hist(xv, 50, density=False, facecolor='black', alpha=0.1, range=(xmin, xmax))
+ ax2.hist(sorted(xv), 50, density=False, facecolor='black', alpha=0.1)
- # ice line plot
+ # ICE line plot
if ice:
if ace_linewidth == "auto":
ace_linewidth = min(1, 50/ice_vals.shape[1])
-
if ice_color_var is None:
ax1.plot(xs, ice_vals, color=obsidian_colors.secondary.light_teal,
linewidth=ace_linewidth, alpha=ace_opacity)
else:
- if not isinstance(ice_color_var, int):
- ice_color_var = convert_name(ice_color_var, None, feature_names)
if ice_color_var == ind:
raise UnsupportedError("Coloring by the feature(s) used in the PDP is not supported.")
colormap = obsidian_cm.obsidian_viridis
- xc = features[:, ice_color_var]
- # Try to convert to float for color mapping
- try:
- xc = xc.astype(float)
- except ValueError:
- pass
- if xc.dtype == np.dtype('O'):
+ xc = df_features.iloc[:, ice_color_var]
+
+ if not is_numeric_dtype(xc):
xc_cat_vals = sorted(set(xc))
colorbar_min = 0
colorbar_max = len(xc_cat_vals) - 1
@@ -226,7 +207,8 @@ def partial_dependence(ind: int | tuple[int],
color_vals = xc
colorbar_min = color_vals.min()
colorbar_max = color_vals.max()
- color_vals = (color_vals - colorbar_min)/(colorbar_max-colorbar_min)
+ color_vals = ((color_vals - colorbar_min)/(colorbar_max-colorbar_min))
+
for i in range(ice_vals.shape[1]):
ax1.plot(xs, ice_vals[:, i], color=colormap(color_vals[i]),
linewidth=ace_linewidth, alpha=ace_opacity)
@@ -234,16 +216,14 @@ def partial_dependence(ind: int | tuple[int],
norm=plt.Normalize(
vmin=colorbar_min, vmax=colorbar_max)),
ax=ax1)
- if xc.dtype == np.dtype('O'):
+ if not is_numeric_dtype(xc):
cbar.set_ticks(np.linspace(0, colorbar_max, len(xc_cat_vals)))
cbar.set_ticklabels(xc_cat_vals)
cbar.set_label('Color by ' + feature_names[ice_color_var])
- # the line plot
ax1.plot(xs, vals, color="black", linewidth=pd_linewidth, alpha=pd_opacity)
- # Revised: PDP line from blue_rgb to black
- ax2.set_ylim(0, features.shape[0])
+ ax2.set_ylim(0, df_features.shape[0])
ax1.set_xlabel(feature_names[ind], fontsize=13)
if ylabel is None:
if not ice:
@@ -273,21 +253,14 @@ def partial_dependence(ind: int | tuple[int],
# 2D PDP
else:
- ind0 = convert_name(ind[0], None, feature_names)
- ind1 = convert_name(ind[1], None, feature_names)
- if ind0 == ind1:
+ # xv = values, independent variable
+ # xs = scaled values
+ # x = scaled values, ordered, then cast to original type
+ if ind[0] == ind[1]:
raise UnsupportedError("The two features must be different for 2D PDP.")
- xv0 = features[:, ind0]
- try:
- xv0 = xv0.astype(float)
- except ValueError:
- pass
-
- xv1 = features[:, ind1]
- try:
- xv1 = xv1.astype(float)
- except ValueError:
- pass
+ xv = []
+ for i in ind:
+ xv.append(df_features.iloc[:, i])
xmin = list(xmin) if isinstance(xmin, tuple) else list([xmin, xmin])
xmax = list(xmax) if isinstance(xmax, tuple) else list([xmax, xmax])
@@ -296,9 +269,9 @@ def partial_dependence(ind: int | tuple[int],
cat_list = []
npoints = [20, 20] if npoints is None else [npoints, npoints]
- for i, xv in enumerate([xv0, xv1]):
- if xv.dtype == np.dtype('O'):
- cat_list.append(sorted(set(xv)))
+ for i, xv_i in enumerate(xv):
+ if not is_numeric_dtype(xv_i):
+ cat_list.append(sorted(set(xv_i)))
xmin[i] = 0
xmax[i] = len(cat_list[i])
xs.append(np.linspace(xmin[i], xmax[i]-1e-6, npoints[i]))
@@ -306,18 +279,19 @@ def partial_dependence(ind: int | tuple[int],
npoints[i] = len(xs[i])
else:
- xmin[i], xmax[i] = compute_bounds(xmin[i], xmax[i], xv)
+ xmin[i], xmax[i] = compute_bounds(xmin[i], xmax[i], xv_i)
xs.append(np.linspace(xmin[i], xmax[i], npoints[i]))
- x.append(np.linspace(xmin[i], xmax[i], npoints[i]))
+ x.append(np.linspace(xmin[i], xmax[i], npoints[i]).astype(xv_i.dtype))
cat_list.append(None)
- features_tmp = pd.DataFrame(features, columns=feature_names)
- vals = np.zeros((npoints[0], npoints[1]))
+ df_test = df_features.copy()
+ vals = []
for i in range(npoints[0]):
for j in range(npoints[1]):
- features_tmp.iloc[:, ind0] = x[0][i]
- features_tmp.iloc[:, ind1] = x[1][j]
- vals[i, j] = model(features_tmp).mean()
+ df_test.iloc[:, ind[0]] = x[0][i]
+ df_test.iloc[:, ind[1]] = x[1][j]
+ vals.append(model(df_test).mean())
+ vals = np.array(vals).reshape(npoints[0], npoints[1])
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
@@ -325,17 +299,17 @@ def partial_dependence(ind: int | tuple[int],
x0, x1 = np.meshgrid(xs[0], xs[1], indexing='ij')
ax.plot_surface(x0, x1, vals, cmap=obsidian_cm.obsidian_viridis)
- ax.set_xlabel(feature_names[ind0], fontsize=13)
- if xv0.dtype == np.dtype('O'):
+ ax.set_xlabel(feature_names[ind[0]], fontsize=13)
+ if not is_numeric_dtype(xv[0]):
ax.set_xticks(np.arange(len(cat_list[0])))
ax.set_xticklabels(cat_list[0])
- ax.set_ylabel(feature_names[ind1], fontsize=13)
- if xv1.dtype == np.dtype('O'):
+ ax.set_ylabel(feature_names[ind[1]], fontsize=13)
+ if not is_numeric_dtype(xv[1]):
ax.set_yticks(np.arange(len(cat_list[1])))
ax.set_yticklabels(cat_list[1])
- ax.set_zlabel("E[f(x) | " + str(feature_names[ind0]) + ", " + str(feature_names[ind1]) + "]", fontsize=13)
+ ax.set_zlabel("E[f(x) | " + str(feature_names[ind[0]]) + ", " + str(feature_names[ind[1]]) + "]", fontsize=13)
if show:
plt.show()
From f13fd8385f5423946a7cb6104effdd0d11a36e4d Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 19:40:38 -0600
Subject: [PATCH 051/136] Improved testing coverage of SHAP PDP ICE
---
obsidian/tests/test_campaign.py | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/obsidian/tests/test_campaign.py b/obsidian/tests/test_campaign.py
index 31efd2e..a096b82 100644
--- a/obsidian/tests/test_campaign.py
+++ b/obsidian/tests/test_campaign.py
@@ -9,7 +9,7 @@
from obsidian.exceptions import IncompatibleObjectiveError, UnfitError
from obsidian.tests.param_configs import X_sp_cont_ndims, X_sp_default
-from obsidian.tests.utils import DEFAULT_MOO_PATH, equal_state_dicts
+from obsidian.tests.utils import DEFAULT_MOO_PATH, DEFAULT_SOO_PATH, equal_state_dicts
import pandas as pd
import pytest
@@ -72,6 +72,11 @@ def test_campaign_basics(X_space, sim_fcn, target):
def test_explain():
+ # SOO includes discrete variables
+ with open(DEFAULT_SOO_PATH) as json_file:
+ obj_dict = json.load(json_file)
+ campaign = Campaign.load_state(obj_dict)
+
# Standard usage
exp = Explainer(campaign.optimizer)
exp.shap_explain(n=50)
@@ -81,10 +86,12 @@ def test_explain():
exp.shap_summary()
exp.shap_summary_bar()
- # Test PDP-ICE, with options
- exp.shap_pdp_ice(ind=0, ice_color_var=None, npoints=10)
- exp.shap_pdp_ice(ind=0, npoints=10)
- exp.shap_pdp_ice(ind=(0, 1), npoints=5)
+ # Test PDP-ICE, with options and discrete variables
+ exp.shap_pdp_ice(ind=3, ice_color_var=None, npoints=10)
+ exp.shap_pdp_ice(ind=0, ice_color_var=3, npoints=None)
+ exp.shap_pdp_ice(ind=3, hist=True)
+ exp.shap_pdp_ice(ind=(0, 3), npoints=5)
+ exp.shap_pdp_ice(ind=(3, 0), npoints=None)
# Test pairwise SHAP analysis, with options
X_new = campaign.X.iloc[0, :]
From dea7b8c79478e2fe2a5541a176cd92529d69eace Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 19:50:53 -0600
Subject: [PATCH 052/136] Removed helper fcn from shap
---
obsidian/plotting/shap.py | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/obsidian/plotting/shap.py b/obsidian/plotting/shap.py
index d30a768..f186663 100644
--- a/obsidian/plotting/shap.py
+++ b/obsidian/plotting/shap.py
@@ -130,15 +130,6 @@ def partial_dependence(ind: int | tuple[int],
tuple: A tuple containing the matplotlib figure and axis objects if `show` is False, otherwise None.
"""
- def cast_to_float(x: np.ndarray) -> np.ndarray:
- """
- Attempts to cast array to float type
- """
- try:
- return x.astype(float)
- except ValueError:
- return x
-
# Extract vals, names from data
df_features = data
feature_names = df_features.columns
From 1ccaccc50ab482aabe15aa3e58f577a19d492014 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 20:28:08 -0600
Subject: [PATCH 053/136] Campaign clear data sets iter to 0
---
obsidian/campaign/campaign.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/obsidian/campaign/campaign.py b/obsidian/campaign/campaign.py
index b99e190..e3c05f2 100644
--- a/obsidian/campaign/campaign.py
+++ b/obsidian/campaign/campaign.py
@@ -99,6 +99,7 @@ def add_data(self, df: pd.DataFrame):
def clear_data(self):
"""Clears campaign data"""
self.data = pd.DataFrame()
+ self.iter = 0
@property
def optimizer(self) -> Optimizer:
From 93053f652e08d579e2b334ecd1aea856da0a1f14 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 20:28:25 -0600
Subject: [PATCH 054/136] Add MDS plot
---
CHANGELOG.md | 2 ++
obsidian/plotting/plotly.py | 50 +++++++++++++++++++++++++++++++++
obsidian/tests/test_plotting.py | 16 ++++++++++-
pyproject.toml | 1 +
4 files changed, 68 insertions(+), 1 deletion(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 709c6f6..1a186ce 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,8 @@
### Added
- Default values for NParEGO scalarization_weights
- SHAP PDP ICE plots now work with categorical values
+- Added scikit-learn to dependencies, for MDS
+- Added MDS plot
### Modified
- SHAP PDP ICE plots must now have color and x-axis indices that are distinct
diff --git a/obsidian/plotting/plotly.py b/obsidian/plotting/plotly.py
index 5dfccef..f789730 100644
--- a/obsidian/plotting/plotly.py
+++ b/obsidian/plotting/plotly.py
@@ -8,11 +8,61 @@
import plotly.graph_objects as go
from plotly.graph_objects import Figure
+from sklearn.manifold import MDS
import pandas as pd
import numpy as np
+def MDS_plot(campaign: Campaign) -> Figure:
+ """
+ Creates a Multi-Dimensional Scaling (MDS) plot of the campaign data,
+ colored by iteration.
+
+ This plot is helpful to visualize the convergence of the optimizer on a 2D plane.
+
+ Args:
+ campaign (Campaign): The campaign object containing the data.
+
+ Returns:
+ fig (Figure): The MDS plot
+ """
+ mds = MDS(n_components=2)
+ X_mds = mds.fit_transform(campaign.X_space.encode(campaign.X))
+
+ iter_max = campaign.data['Iteration'].max()
+ iter_vals = campaign.data['Iteration'].values
+
+ if campaign.data['Iteration'].nunique() == 1:
+ iter_vals = np.zeros_like(iter_vals)
+ iter_max = 0
+ cbar = None
+ else:
+ cbar = dict(title=dict(text='Iteration', font=dict(size=10)))
+
+ fig = go.Figure()
+
+ fig.add_trace(go.Scatter(x=X_mds[:, 0], y=X_mds[:, 1],
+ mode='markers',
+ name='Observations',
+ marker={'color': iter_vals, 'size': 10,
+ 'cmax': iter_max, 'cmin': 0,
+ 'colorscale': [[0, obsidian_colors.rich_blue],
+ [0.5, obsidian_colors.teal],
+ [1, obsidian_colors.lemon]],
+ 'colorbar': cbar
+ },
+ showlegend=False
+ ))
+
+ fig.update_xaxes(title_text='Component 1')
+ fig.update_yaxes(title_text='Component 2')
+ fig.update_layout(template='ggplot2', title='Multi-Dimensional Scaling (MDS) Plot',
+ autosize=False, height=400, width=500)
+
+ return fig
+
+
def parity_plot(optimizer: Optimizer,
f_transform: bool = False,
response_id: int = 0) -> Figure:
diff --git a/obsidian/tests/test_plotting.py b/obsidian/tests/test_plotting.py
index bc2cab5..b8a1eb0 100644
--- a/obsidian/tests/test_plotting.py
+++ b/obsidian/tests/test_plotting.py
@@ -6,7 +6,8 @@
factor_plot,
surface_plot,
visualize_inputs,
- optim_progress
+ optim_progress,
+ MDS_plot
)
from obsidian.objectives import Scalar_WeightedSum
@@ -75,6 +76,19 @@ def test_optim_progress_plot():
obj = Scalar_WeightedSum(weights=[1, 1])
campaign.set_objective(obj)
fig = optim_progress(campaign, X_suggest=X_suggest)
+
+
+@pytest.mark.fast
+def test_MDS_plot():
+ # Test with default data
+ fig = MDS_plot(campaign)
+
+ # Test with no iteration data
+ data = campaign.data
+ data = data.drop(columns='Iteration')
+ campaign.clear_data()
+ campaign.add_data(data)
+ fig = MDS_plot(campaign)
if __name__ == '__main__':
diff --git a/pyproject.toml b/pyproject.toml
index 8002663..3a487b2 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -58,6 +58,7 @@ sphinx = { version = "^7.3.7", optional = true}
myst-parser = { version = "^3.0.1", optional = true}
pydata-sphinx-theme = { version = "^0.15.4", optional = true}
linkify-it-py = { version = "^2.0.3", optional = true}
+scikit-learn = "^1.5.1"
[tool.poetry.extras]
From 42372b5c0adfc84e283e957f53a062a4a39e6ad8 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 21:00:02 -0600
Subject: [PATCH 055/136] Added best_X method to campaign
---
obsidian/campaign/campaign.py | 13 +++++++++++++
obsidian/tests/test_campaign.py | 1 +
2 files changed, 14 insertions(+)
diff --git a/obsidian/campaign/campaign.py b/obsidian/campaign/campaign.py
index e3c05f2..3c8b8bd 100644
--- a/obsidian/campaign/campaign.py
+++ b/obsidian/campaign/campaign.py
@@ -240,6 +240,19 @@ def out(self) -> pd.Series | pd.DataFrame:
else:
return self.y
+ @property
+ def X_best(self) -> pd.DataFrame:
+ """
+ Best performing X values
+ """
+ best_idx = self.out.idxmax().values
+
+ X_best = self.X.iloc[best_idx, :]
+ if isinstance(X_best, pd.Series):
+ X_best = X_best .to_frame().T
+
+ return X_best
+
@property
def X(self) -> pd.DataFrame:
"""
diff --git a/obsidian/tests/test_campaign.py b/obsidian/tests/test_campaign.py
index a096b82..436c3b1 100644
--- a/obsidian/tests/test_campaign.py
+++ b/obsidian/tests/test_campaign.py
@@ -55,6 +55,7 @@ def test_campaign_basics(X_space, sim_fcn, target):
campaign.y
campaign.fit()
campaign.response_max
+ campaign.X_best
# Serialize, deserialize, re-serialize
obj_dict = campaign.save_state()
From 259280d25e57e65fb0e9a5e741cd151d90226d3b Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 21:00:15 -0600
Subject: [PATCH 056/136] Added X_best attributes to optimizer
---
obsidian/optimizer/bayesian.py | 9 +++++++--
obsidian/tests/default_campaign_MOO.json | 2 +-
obsidian/tests/default_campaign_SOO.json | 2 +-
3 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/obsidian/optimizer/bayesian.py b/obsidian/optimizer/bayesian.py
index df7c122..ad5d42e 100644
--- a/obsidian/optimizer/bayesian.py
+++ b/obsidian/optimizer/bayesian.py
@@ -221,6 +221,10 @@ def fit(self,
self.y_train = pd.concat([Z_valid[t.name] for t in self.target], axis=1)
self.f_train = pd.concat([t.transform_f(Z_valid[t.name], fit=True) for t in self.target], axis=1)
+ # Extract the X which achieves the best sum_f
+ self.X_best_f_idx = self.f_train.sum(axis=1).idxmax()
+ self.X_best_f = self.X_train.iloc[self.X_best_f_idx, :].to_frame().T
+
# Instantiate and fit the model(s)
self.surrogate = []
for i in range(self.n_response):
@@ -270,7 +274,7 @@ def save_state(self) -> dict:
# Select some optimizer attributes to save directly
opt_attrs = ['X_train', 'y_train',
'y_names', 'n_response',
- 'seed']
+ 'seed', 'X_best_f_idx', 'X_best_f']
for attr in opt_attrs:
if isinstance(getattr(self, attr), (pd.Series, pd.DataFrame)):
@@ -317,11 +321,12 @@ def load_state(cls,
setattr(new_opt, k, v)
# Unpack and encode/transform the data objects if present
- data_objects = ['X_train', 'y_train']
+ data_objects = ['X_train', 'y_train', 'X_best_f']
if all(hasattr(new_opt, attr) for attr in data_objects):
new_opt.X_train = pd.DataFrame(new_opt.X_train)
new_opt.X_t_train = new_opt.X_space.encode(new_opt.X_train)
new_opt.y_train = pd.DataFrame(new_opt.y_train, columns=new_opt.y_names)
+ new_opt.X_best_f = pd.DataFrame(new_opt.X_best_f)
f_train = pd.DataFrame()
for t, y in zip(new_opt.target, new_opt.y_train.columns):
diff --git a/obsidian/tests/default_campaign_MOO.json b/obsidian/tests/default_campaign_MOO.json
index fc09166..6b578fd 100644
--- a/obsidian/tests/default_campaign_MOO.json
+++ b/obsidian/tests/default_campaign_MOO.json
@@ -1 +1 @@
-{"X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": 0, "max": 10}, "class": "Param_Continuous"}}, "optimizer": {"opt_attrs": {"X_train": {"Parameter 1": {"0": 2.5, "1": 0.8333333333333333, "2": 7.5, "3": 5.833333333333334, "4": 9.166666666666666, "5": 4.166666666666667, "6": 0.0, "7": 9.561359020262923, "8": 0.0, "9": 0.0}, "Parameter 2": {"0": 2.5, "1": 5.833333333333334, "2": 7.5, "3": 0.8333333333333333, "4": 4.166666666666667, "5": 9.166666666666666, "6": 8.44723404661871, "7": 9.862823941909266, "8": 0.0, "9": 10.0}}, "y_train": {"Response 1": {"0": 0.013856171114995977, "1": 0.03432975568022848, "2": 0.4189098892895902, "3": 0.003431693206423828, "4": 0.1643399024276901, "5": 0.18170428213463904, "6": 0.0, "7": 0.9575375984495041, "8": 0.0, "9": 0.0}, "Response 2": {"0": 0.5352280996541698, "1": 0.7663162272889814, "2": 0.308476724543929, "3": 0.37918523689244016, "4": 0.1246219695813563, "5": 0.28181351107250613, "6": 0.8534158017742339, "7": 0.06793137056494765, "8": 0.8534158017742339, "9": 0.8534158017742339}}, "y_names": ["Response 1", "Response 2"], "n_response": 2, "seed": null}, "X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": 0, "max": 10}, "class": "Param_Continuous"}}, "surrogate_spec": [{"GP": {}}, {"GP": {}}], "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.013856171114995977], [0.03432975568022848], [0.4189098892895902], [0.003431693206423828], [0.1643399024276901], [0.18170428213463904], [0.0], [0.9575375984495041], [0.0], [0.0]]}, {"init_attrs": {"name": "Response 2", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.5352280996541698], [0.7663162272889814], [0.308476724543929], [0.37918523689244016], [0.1246219695813563], [0.28181351107250613], [0.8534158017742339], [0.06793137056494765], [0.8534158017742339], [0.8534158017742339]]}], "model_states": [{"model_attrs": {"model_type": "GP", "seed": null, "hps": {}, "is_fit": true, "cat_dims": [], "task_feature": null, "X_order": ["Parameter 1", "Parameter 2"], "y_name": "Response 1 Trans"}, "train_X": {"Parameter 1": {"0": 0.25, "1": 0.08333333333333333, "2": 0.75, "3": 0.5833333333333334, "4": 0.9166666666666666, "5": 0.4166666666666667, "6": 0.0, "7": 0.9561359020262923, "8": 0.0, "9": 0.0}, "Parameter 2": {"0": 0.25, "1": 0.5833333333333334, "2": 0.75, "3": 0.08333333333333333, "4": 0.4166666666666667, "5": 0.9166666666666666, "6": 0.844723404661871, "7": 0.9862823941909266, "8": 0.0, "9": 1.0}}, "train_Y": {"0": -0.535669991872484, "1": -0.4686154775065579, "2": 0.7909506728074768, "3": -0.5698119517940827, "4": -0.042809863203040845, "5": 0.014061469943587476, "6": -0.5810513378760082, "7": 2.5550491552531267, "8": -0.5810513378760082, "9": -0.5810513378760082}, "torch_params": {"likelihood.noise_covar.raw_noise": [0.003398248045429932], "likelihood.noise_covar.noise_prior.concentration": 1.100000023841858, "likelihood.noise_covar.noise_prior.rate": 0.05000000074505806, "likelihood.noise_covar.raw_noise_constraint.lower_bound": 9.999999747378752e-05, "likelihood.noise_covar.raw_noise_constraint.upper_bound": "inf", "mean_module.raw_constant": 0.35957230077639185, "covar_module.raw_outputscale": 1.2891402290445992, "covar_module.base_kernel.raw_lengthscale": [[-0.22930632060853587, -0.1280487015578905]], "covar_module.base_kernel.lengthscale_prior.concentration": 3.0, "covar_module.base_kernel.lengthscale_prior.rate": 6.0, "covar_module.base_kernel.raw_lengthscale_constraint.lower_bound": 0.0, "covar_module.base_kernel.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.outputscale_prior.concentration": 2.0, "covar_module.outputscale_prior.rate": 0.15000000596046448, "covar_module.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.raw_outputscale_constraint.upper_bound": "inf"}}, {"model_attrs": {"model_type": "GP", "seed": null, "hps": {}, "is_fit": true, "cat_dims": [], "task_feature": null, "X_order": ["Parameter 1", "Parameter 2"], "y_name": "Response 2 Trans"}, "train_X": {"Parameter 1": {"0": 0.25, "1": 0.08333333333333333, "2": 0.75, "3": 0.5833333333333334, "4": 0.9166666666666666, "5": 0.4166666666666667, "6": 0.0, "7": 0.9561359020262923, "8": 0.0, "9": 0.0}, "Parameter 2": {"0": 0.25, "1": 0.5833333333333334, "2": 0.75, "3": 0.08333333333333333, "4": 0.4166666666666667, "5": 0.9166666666666666, "6": 0.844723404661871, "7": 0.9862823941909266, "8": 0.0, "9": 1.0}}, "train_Y": {"0": 0.1054446935687377, "1": 0.847300118950883, "2": -0.6224885826237413, "3": -0.3954951232741601, "4": -1.2127120997177576, "5": -0.7080847133049162, "6": 1.1269133234097133, "7": -1.394704263828187, "8": 1.1269133234097133, "9": 1.1269133234097133}, "torch_params": {"likelihood.noise_covar.raw_noise": [0.001348488460520536], "likelihood.noise_covar.noise_prior.concentration": 1.100000023841858, "likelihood.noise_covar.noise_prior.rate": 0.05000000074505806, "likelihood.noise_covar.raw_noise_constraint.lower_bound": 9.999999747378752e-05, "likelihood.noise_covar.raw_noise_constraint.upper_bound": "inf", "mean_module.raw_constant": -0.11554181160629078, "covar_module.raw_outputscale": 0.05017203782837847, "covar_module.base_kernel.raw_lengthscale": [[-1.0586594080313136, 0.5978649542247377]], "covar_module.base_kernel.lengthscale_prior.concentration": 3.0, "covar_module.base_kernel.lengthscale_prior.rate": 6.0, "covar_module.base_kernel.raw_lengthscale_constraint.lower_bound": 0.0, "covar_module.base_kernel.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.outputscale_prior.concentration": 2.0, "covar_module.outputscale_prior.rate": 0.15000000596046448, "covar_module.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.raw_outputscale_constraint.upper_bound": "inf"}}]}, "data": {"Parameter 1": {"0": 2.5, "1": 0.8333333333333333, "2": 7.5, "3": 5.833333333333334, "4": 9.166666666666666, "5": 4.166666666666667, "6": 0.0, "7": 9.561359020262923, "8": 0.0, "9": 0.0, "10": 8.05884810038377, "11": 6.8406804167176585}, "Parameter 2": {"0": 2.5, "1": 5.833333333333334, "2": 7.5, "3": 0.8333333333333333, "4": 4.166666666666667, "5": 9.166666666666666, "6": 8.44723404661871, "7": 9.862823941909266, "8": 0.0, "9": 10.0, "10": 10.0, "11": 10.0}, "Response 1": {"0": 0.013856171114995977, "1": 0.03432975568022848, "2": 0.4189098892895902, "3": 0.003431693206423828, "4": 0.1643399024276901, "5": 0.18170428213463904, "6": 0.0, "7": 0.9575375984495041, "8": 0.0, "9": 0.0, "10": 0.8191994146108325, "11": 0.6499071214288321}, "Response 2": {"0": 0.5352280996541698, "1": 0.7663162272889814, "2": 0.308476724543929, "3": 0.37918523689244016, "4": 0.1246219695813563, "5": 0.28181351107250613, "6": 0.8534158017742339, "7": 0.06793137056494765, "8": 0.8534158017742339, "9": 0.8534158017742339, "10": 0.25782685626481244, "11": 0.35190022724334613}, "Iteration": {"0": 0, "1": 0, "2": 0, "3": 0, "4": 0, "5": 0, "6": 1, "7": 1, "8": 2, "9": 2, "10": 3, "11": 3}, "Response 1 (pred)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.07260440423671881, "7": 0.2998109146830079, "8": 0.13553996363484694, "9": 0.0283643020909618, "10": 0.7895488620349673, "11": 0.5984855239765161}, "Response 1 lb": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.059246292549191615, "7": 0.15146042241994814, "8": -0.13616102654512918, "9": -0.09360519411923673, "10": 0.7119357066023786, "11": 0.49972885756864527}, "Response 1 ub": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.2044551224696099, "7": 0.4481614310769234, "8": 0.40724099801000757, "9": 0.15033381814085398, "10": 0.8671620300921978, "11": 0.697242206448256}, "Response 2 (pred)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.582326398459104, "7": 0.3408065670773579, "8": 0.5919322389003672, "9": 0.8108402926111451, "10": 0.23918866328259236, "11": 0.32933785504134366}, "Response 2 lb": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.3938802797683256, "7": 0.12880245692356157, "8": 0.37133988630877496, "9": 0.7274598657587373, "10": 0.17670696544989506, "11": 0.24887720423658888}, "Response 2 ub": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.7707725478027373, "7": 0.552810711715978, "8": 0.8125246273737563, "9": 0.8942207330263057, "10": 0.30167037127863194, "11": 0.40979851893391717}, "f(Response 1)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.4003359276706779, "7": 1.0322939538830744, "8": -0.2616209786702686, "9": -0.5868146567966284, "10": 2.0048571210574564, "11": 1.3790917937488958}, "f(Response 2)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.8168712025094681, "7": -0.2609086900401194, "8": 0.6239738174840637, "9": 1.3943418515102426, "10": -0.844922009589592, "11": -0.5555187084138095}, "Objective 1": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.4003359276706779, "7": 1.0322939538830744, "8": -0.2616209786702686, "9": -0.5868146567966284, "10": 2.0048571210574564, "11": 1.3790917937488958}, "Objective 2": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.8168712025094681, "7": -0.2609086900401194, "8": 0.6239738174840637, "9": 1.3943418515102426, "10": -0.844922009589592, "11": -0.5555187084138095}, "aq Value": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.7452203824605368, "7": -0.2832722625598487, "8": -0.6039225341357639, "9": -1.4415657461280458, "10": -0.3903933850999888, "11": -0.40909204451189396}, "aq Value (joint)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.1673485517948059, "7": 0.1673485517948059, "8": -0.3577695711270721, "9": -0.3577695711270721, "10": 0.015390517435002415, "11": 0.015390517435002415}, "aq Method": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": "NEHVI", "7": "NEHVI", "8": "NEHVI", "9": "NEHVI", "10": "NEHVI", "11": "NEHVI"}, "Expected Hypervolume (joint)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 2.45926678524061, "7": 2.45926678524061, "8": 1.4727612897691176, "9": 1.4727612897691176, "10": 2.1340993059181317, "11": 2.1340993059181317}, "Expected Pareto": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": false, "7": false, "8": true, "9": true, "10": true, "11": false}, "Hypervolume (iter)": {"0": 0.09053399549195458, "1": 0.09053399549195458, "2": 0.09053399549195458, "3": 0.09053399549195458, "4": 0.09053399549195458, "5": 0.09053399549195458, "6": 0.1164843458744553, "7": 0.1164843458744553, "8": 0.1164843458744553, "9": 0.1164843458744553, "10": 0.23092802448987865, "11": 0.23092802448987865}, "Pareto Front": {"0": false, "1": true, "2": false, "3": false, "4": false, "5": false, "6": true, "7": true, "8": false, "9": false, "10": true, "11": true}, "Response 1 (max) (iter)": {"0": 0.4189098892895902, "1": 0.4189098892895902, "2": 0.4189098892895902, "3": 0.4189098892895902, "4": 0.4189098892895902, "5": 0.4189098892895902, "6": 0.9575375984495041, "7": 0.9575375984495041, "8": 0.9575375984495041, "9": 0.9575375984495041, "10": 0.9575375984495041, "11": 0.9575375984495041}, "Response 2 (max) (iter)": {"0": 0.7663162272889814, "1": 0.7663162272889814, "2": 0.7663162272889814, "3": 0.7663162272889814, "4": 0.7663162272889814, "5": 0.7663162272889814, "6": 0.8534158017742339, "7": 0.8534158017742339, "8": 0.8534158017742339, "9": 0.8534158017742339, "10": 0.8534158017742339, "11": 0.8534158017742339}}, "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.013856171114995977], [0.03432975568022848], [0.4189098892895902], [0.003431693206423828], [0.1643399024276901], [0.18170428213463904], [0.0], [0.9575375984495041], [0.0], [0.0]]}, {"init_attrs": {"name": "Response 2", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.5352280996541698], [0.7663162272889814], [0.308476724543929], [0.37918523689244016], [0.1246219695813563], [0.28181351107250613], [0.8534158017742339], [0.06793137056494765], [0.8534158017742339], [0.8534158017742339]]}], "seed": null}
\ No newline at end of file
+{"X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": 0, "max": 10}, "class": "Param_Continuous"}}, "optimizer": {"opt_attrs": {"X_train": {"Parameter 1": {"0": 7.5, "1": 2.5, "2": 4.166666666666667, "3": 9.166666666666666, "4": 0.8333333333333333, "5": 5.833333333333334, "6": 8.528469786832659, "7": 3.8767420865729356, "8": 0.0, "9": 10.0}, "Parameter 2": {"0": 2.5, "1": 7.5, "2": 0.8333333333333333, "3": 5.833333333333334, "4": 4.166666666666667, "5": 9.166666666666666, "6": 10.0, "7": 10.0, "8": 6.6068112064542115, "9": 10.0}}, "y_train": {"Response 1": {"0": 0.04654554325439891, "1": 0.12470554003496379, "2": 0.001501688282104455, "3": 0.32210620875827256, "4": 0.017515181469504326, "5": 0.41523487797728315, "6": 0.876782893056231, "7": 0.19762289499997746, "8": 0.0, "9": 1.021018019467305}, "Response 2": {"0": 0.308476724543929, "1": 0.5352280996541698, "2": 0.28181351107250613, "3": 0.1246219695813563, "4": 0.7663162272889814, "5": 0.37918523689244016, "6": 0.20635437761236505, "7": 0.2833840135536786, "8": 0.8534158017742339, "9": 6.251932247062026e-17}}, "y_names": ["Response 1", "Response 2"], "n_response": 2, "seed": null, "X_best_f_idx": 8, "X_best_f": {"Parameter 1": {"8": 0.0}, "Parameter 2": {"8": 6.6068112064542115}}}, "X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": 0, "max": 10}, "class": "Param_Continuous"}}, "surrogate_spec": [{"GP": {}}, {"GP": {}}], "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.04654554325439891], [0.12470554003496379], [0.001501688282104455], [0.32210620875827256], [0.017515181469504326], [0.41523487797728315], [0.876782893056231], [0.19762289499997746], [0.0], [1.021018019467305]]}, {"init_attrs": {"name": "Response 2", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.308476724543929], [0.5352280996541698], [0.28181351107250613], [0.1246219695813563], [0.7663162272889814], [0.37918523689244016], [0.20635437761236505], [0.2833840135536786], [0.8534158017742339], [6.251932247062026e-17]]}], "model_states": [{"model_attrs": {"model_type": "GP", "seed": null, "hps": {}, "is_fit": true, "cat_dims": [], "task_feature": null, "X_order": ["Parameter 1", "Parameter 2"], "y_name": "Response 1 Trans"}, "train_X": {"Parameter 1": {"0": 0.75, "1": 0.25, "2": 0.4166666666666667, "3": 0.9166666666666666, "4": 0.08333333333333333, "5": 0.5833333333333334, "6": 0.8528469786832659, "7": 0.3876742086572936, "8": 0.0, "9": 1.0}, "Parameter 2": {"0": 0.25, "1": 0.75, "2": 0.08333333333333333, "3": 0.5833333333333334, "4": 0.4166666666666667, "5": 0.9166666666666666, "6": 1.0, "7": 1.0, "8": 0.6606811206454212, "9": 1.0}}, "train_Y": {"0": -0.6916561847607523, "1": -0.4802848891657192, "2": -0.8134701353270364, "3": 0.053553862344400065, "4": -0.7701641867415241, "5": 0.3054055547789411, "6": 1.5535888447610058, "7": -0.2830914855683647, "8": -0.8175312127431401, "9": 1.94364983242219}, "torch_params": {"likelihood.noise_covar.raw_noise": [0.0021111535446464783], "likelihood.noise_covar.noise_prior.concentration": 1.100000023841858, "likelihood.noise_covar.noise_prior.rate": 0.05000000074505806, "likelihood.noise_covar.raw_noise_constraint.lower_bound": 9.999999747378752e-05, "likelihood.noise_covar.raw_noise_constraint.upper_bound": "inf", "mean_module.raw_constant": -0.026229086617286235, "covar_module.raw_outputscale": 0.8058521136202499, "covar_module.base_kernel.raw_lengthscale": [[-0.09409776001235488, -0.32752547105197394]], "covar_module.base_kernel.lengthscale_prior.concentration": 3.0, "covar_module.base_kernel.lengthscale_prior.rate": 6.0, "covar_module.base_kernel.raw_lengthscale_constraint.lower_bound": 0.0, "covar_module.base_kernel.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.outputscale_prior.concentration": 2.0, "covar_module.outputscale_prior.rate": 0.15000000596046448, "covar_module.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.raw_outputscale_constraint.upper_bound": "inf"}}, {"model_attrs": {"model_type": "GP", "seed": null, "hps": {}, "is_fit": true, "cat_dims": [], "task_feature": null, "X_order": ["Parameter 1", "Parameter 2"], "y_name": "Response 2 Trans"}, "train_X": {"Parameter 1": {"0": 0.75, "1": 0.25, "2": 0.4166666666666667, "3": 0.9166666666666666, "4": 0.08333333333333333, "5": 0.5833333333333334, "6": 0.8528469786832659, "7": 0.3876742086572936, "8": 0.0, "9": 1.0}, "Parameter 2": {"0": 0.25, "1": 0.75, "2": 0.08333333333333333, "3": 0.5833333333333334, "4": 0.4166666666666667, "5": 0.9166666666666666, "6": 1.0, "7": 1.0, "8": 0.6606811206454212, "9": 1.0}}, "train_Y": {"0": -0.24134318384680478, "1": 0.5953922289455897, "2": -0.3397331271030966, "3": -0.9197857477018673, "4": 1.4481306953558992, "5": 0.019578348554502165, "6": -0.6181849298938439, "7": -0.33393781476491813, "8": 1.7695368979666997, "9": -1.37965336751216}, "torch_params": {"likelihood.noise_covar.raw_noise": [0.011779013752209843], "likelihood.noise_covar.noise_prior.concentration": 1.100000023841858, "likelihood.noise_covar.noise_prior.rate": 0.05000000074505806, "likelihood.noise_covar.raw_noise_constraint.lower_bound": 9.999999747378752e-05, "likelihood.noise_covar.raw_noise_constraint.upper_bound": "inf", "mean_module.raw_constant": -0.12048280254924493, "covar_module.raw_outputscale": 0.6806000911126695, "covar_module.base_kernel.raw_lengthscale": [[-0.9563125132165595, -0.12407709289267602]], "covar_module.base_kernel.lengthscale_prior.concentration": 3.0, "covar_module.base_kernel.lengthscale_prior.rate": 6.0, "covar_module.base_kernel.raw_lengthscale_constraint.lower_bound": 0.0, "covar_module.base_kernel.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.outputscale_prior.concentration": 2.0, "covar_module.outputscale_prior.rate": 0.15000000596046448, "covar_module.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.raw_outputscale_constraint.upper_bound": "inf"}}]}, "data": {"Parameter 1": {"0": 7.5, "1": 2.5, "2": 4.166666666666667, "3": 9.166666666666666, "4": 0.8333333333333333, "5": 5.833333333333334, "6": 8.528469786832659, "7": 3.8767420865729356, "8": 0.0, "9": 10.0, "10": 7.223740014374888, "11": 0.0}, "Parameter 2": {"0": 2.5, "1": 7.5, "2": 0.8333333333333333, "3": 5.833333333333334, "4": 4.166666666666667, "5": 9.166666666666666, "6": 10.0, "7": 10.0, "8": 6.6068112064542115, "9": 10.0, "10": 10.0, "11": 10.0}, "Response 1": {"0": 0.04654554325439891, "1": 0.12470554003496379, "2": 0.001501688282104455, "3": 0.32210620875827256, "4": 0.017515181469504326, "5": 0.41523487797728315, "6": 0.876782893056231, "7": 0.19762289499997746, "8": 0.0, "9": 1.021018019467305, "10": 0.7058262547664004, "11": 0.0}, "Response 2": {"0": 0.308476724543929, "1": 0.5352280996541698, "2": 0.28181351107250613, "3": 0.1246219695813563, "4": 0.7663162272889814, "5": 0.37918523689244016, "6": 0.20635437761236505, "7": 0.2833840135536786, "8": 0.8534158017742339, "9": 6.251932247062026e-17, "10": 0.3289273406006508, "11": 0.8534158017742339}, "Iteration": {"0": 0, "1": 0, "2": 0, "3": 0, "4": 0, "5": 0, "6": 1, "7": 1, "8": 2, "9": 2, "10": 3, "11": 3}, "Response 1 (pred)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.4117743113714847, "7": 0.30755192068269416, "8": 0.0958353188137184, "9": 0.8685195816251645, "10": 0.6785732165298491, "11": 0.00910231240977899}, "Response 1 lb": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.2848136558852717, "7": 0.21060473561128104, "8": -0.07482664160862634, "9": 0.7392106838423548, "10": 0.6447198087329133, "11": -0.16226796397383375}, "Response 1 ub": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.5387349875092586, "7": 0.404499121523644, "8": 0.26649730699612706, "9": 0.9978285004415032, "10": 0.7124266298334179, "11": 0.1804726166686711}, "Response 2 (pred)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.3085880244811958, "7": 0.433415080154813, "8": 0.6372412677391663, "9": 0.1915336082781775, "10": 0.34509289363598034, "11": 0.6937842405985601}, "Response 2 lb": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.11581070787955111, "7": 0.2811433699203987, "8": 0.48981248317747295, "9": 0.06904139838192591, "10": 0.2793691754405588, "11": 0.5451393229556163}, "Response 2 ub": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.5013653724402128, "7": 0.5856868151579127, "8": 0.7846700762817895, "9": 0.31402583809914797, "10": 0.41081662252209483, "11": 0.8424291824202512}, "f(Response 1)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 1.4808950126041587, "7": 0.8807443912057852, "8": -0.526996291857271, "9": 2.110038285822656, "10": 1.017562260158184, "11": -0.792915454731183}, "f(Response 2)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.40468362222161114, "7": 0.15235589656864806, "8": 1.3592382632807847, "9": -0.8312583491660362, "10": -0.10622583187829626, "11": 1.1804803589707014}, "Objective 1": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 1.4808950126041587, "7": 0.8807443912057852, "8": -0.526996291857271, "9": 2.110038285822656, "10": 1.017562260158184, "11": -0.792915454731183}, "Objective 2": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.40468362222161114, "7": 0.15235589656864806, "8": 1.3592382632807847, "9": -0.8312583491660362, "10": -0.10622583187829626, "11": 1.1804803589707014}, "aq Value": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.5076803605881066, "7": -0.6694488192529944, "8": -0.9182424523558645, "9": -1.1385797661125623, "10": -0.8802917384977791, "11": -1.859929304086834}, "aq Value (joint)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 0.03280142724219548, "7": 0.03280142724219548, "8": -0.29451063609365047, "9": -0.29451063609365047, "10": -0.5136458889486466, "11": -0.5136458889486466}, "aq Method": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": "NEHVI", "7": "NEHVI", "8": "NEHVI", "9": "NEHVI", "10": "NEHVI", "11": "NEHVI"}, "Expected Hypervolume (joint)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 3.2093699140448306, "7": 3.2093699140448306, "8": 2.4788339142543303, "9": 2.4788339142543303, "10": 3.0150519503445445, "11": 3.0150519503445445}, "Expected Pareto": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": true, "7": true, "8": true, "9": false, "10": true, "11": true}, "Hypervolume (iter)": {"0": 0.12824688245219812, "1": 0.12824688245219812, "2": 0.12824688245219812, "3": 0.12824688245219812, "4": 0.12824688245219812, "5": 0.12824688245219812, "6": 0.16597031314653288, "7": 0.16597031314653288, "8": 0.27620034892193324, "9": 0.27620034892193324, "10": 0.3118189949938207, "11": 0.3118189949938207}, "Pareto Front": {"0": false, "1": true, "2": false, "3": false, "4": true, "5": true, "6": true, "7": false, "8": true, "9": true, "10": true, "11": false}, "Response 1 (max) (iter)": {"0": 0.41523487797728315, "1": 0.41523487797728315, "2": 0.41523487797728315, "3": 0.41523487797728315, "4": 0.41523487797728315, "5": 0.41523487797728315, "6": 0.876782893056231, "7": 0.876782893056231, "8": 1.021018019467305, "9": 1.021018019467305, "10": 1.021018019467305, "11": 1.021018019467305}, "Response 2 (max) (iter)": {"0": 0.7663162272889814, "1": 0.7663162272889814, "2": 0.7663162272889814, "3": 0.7663162272889814, "4": 0.7663162272889814, "5": 0.7663162272889814, "6": 0.7663162272889814, "7": 0.7663162272889814, "8": 0.8534158017742339, "9": 0.8534158017742339, "10": 0.8534158017742339, "11": 0.8534158017742339}}, "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.04654554325439891], [0.12470554003496379], [0.001501688282104455], [0.32210620875827256], [0.017515181469504326], [0.41523487797728315], [0.876782893056231], [0.19762289499997746], [0.0], [1.021018019467305]]}, {"init_attrs": {"name": "Response 2", "aim": "max", "f_transform": "Standard"}, "f_raw": [[0.308476724543929], [0.5352280996541698], [0.28181351107250613], [0.1246219695813563], [0.7663162272889814], [0.37918523689244016], [0.20635437761236505], [0.2833840135536786], [0.8534158017742339], [6.251932247062026e-17]]}], "seed": null}
\ No newline at end of file
diff --git a/obsidian/tests/default_campaign_SOO.json b/obsidian/tests/default_campaign_SOO.json
index a4f3e03..2363f38 100644
--- a/obsidian/tests/default_campaign_SOO.json
+++ b/obsidian/tests/default_campaign_SOO.json
@@ -1 +1 @@
-{"X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": -20, "max": 0}, "class": "Param_Continuous"}, "Parameter 3": {"state": {"name": "Parameter 3", "min": 5, "max": 5}, "class": "Param_Continuous"}, "Parameter 7": {"state": {"name": "Parameter 7", "categories": ["A", "B", "C", "D"]}, "class": "Param_Categorical"}}, "optimizer": {"opt_attrs": {"X_train": {"Parameter 1": {"0": 5.833333333333334, "1": 7.5, "2": 9.166666666666666, "3": 2.5, "4": 4.166666666666667, "5": 0.8333333333333333, "6": 1.181027361160873, "7": 10.0, "8": 3.5671554309773494, "9": 3.8259819976910956}, "Parameter 2": {"0": -8.333333333333332, "1": -5.0, "2": -15.0, "3": -18.333333333333332, "4": -11.666666666666666, "5": -1.6666666666666679, "6": -1.3765871582395803, "7": -20.0, "8": -11.972662327181533, "9": -12.021715223130977}, "Parameter 3": {"0": 5.0, "1": 5.0, "2": 5.0, "3": 5.0, "4": 5.0, "5": 5.0, "6": 5.0, "7": 5.0, "8": 5.0, "9": 5.0}, "Parameter 7": {"0": "D", "1": "C", "2": "B", "3": "D", "4": "A", "5": "B", "6": "A", "7": "A", "8": "B", "9": "A"}}, "y_train": {"Response 1": {"0": 52.91999776955157, "1": 47.950545897489405, "2": 57.200296096064626, "3": 69.86459941931172, "4": 84.03902691676035, "5": 56.568255091930745, "6": 53.72360713169506, "7": 46.766642700990914, "8": 89.79791912501248, "9": 85.82078349307184}}, "y_names": ["Response 1"], "n_response": 1, "seed": null}, "X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": -20, "max": 0}, "class": "Param_Continuous"}, "Parameter 3": {"state": {"name": "Parameter 3", "min": 5, "max": 5}, "class": "Param_Continuous"}, "Parameter 7": {"state": {"name": "Parameter 7", "categories": ["A", "B", "C", "D"]}, "class": "Param_Categorical"}}, "surrogate_spec": [{"GP": {}}], "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[52.91999776955157], [47.950545897489405], [57.200296096064626], [69.86459941931172], [84.03902691676035], [56.568255091930745], [53.72360713169506], [46.766642700990914], [89.79791912501248], [85.82078349307184]]}], "model_states": [{"model_attrs": {"model_type": "GP", "seed": null, "hps": {}, "is_fit": true, "cat_dims": [3, 4, 5, 6], "task_feature": null, "X_order": ["Parameter 1", "Parameter 2", "Parameter 3", "Parameter 7^A", "Parameter 7^B", "Parameter 7^C", "Parameter 7^D"], "y_name": "Response 1 Trans"}, "train_X": {"Parameter 1": {"0": 0.5833333333333334, "1": 0.75, "2": 0.9166666666666666, "3": 0.25, "4": 0.4166666666666667, "5": 0.08333333333333333, "6": 0.1181027361160873, "7": 1.0, "8": 0.35671554309773496, "9": 0.3825981997691096}, "Parameter 2": {"0": 0.5833333333333334, "1": 0.75, "2": 0.25, "3": 0.0833333333333334, "4": 0.4166666666666667, "5": 0.9166666666666666, "6": 0.931170642088021, "7": 0.0, "8": 0.4013668836409233, "9": 0.3989142388434511}, "Parameter 3": {"0": 0.0, "1": 0.0, "2": 0.0, "3": 0.0, "4": 0.0, "5": 0.0, "6": 0.0, "7": 0.0, "8": 0.0, "9": 0.0}, "Parameter 7^A": {"0": 0.0, "1": 0.0, "2": 0.0, "3": 0.0, "4": 1.0, "5": 0.0, "6": 1.0, "7": 1.0, "8": 0.0, "9": 1.0}, "Parameter 7^B": {"0": 0.0, "1": 0.0, "2": 1.0, "3": 0.0, "4": 0.0, "5": 1.0, "6": 0.0, "7": 0.0, "8": 1.0, "9": 0.0}, "Parameter 7^C": {"0": 0.0, "1": 1.0, "2": 0.0, "3": 0.0, "4": 0.0, "5": 0.0, "6": 0.0, "7": 0.0, "8": 0.0, "9": 0.0}, "Parameter 7^D": {"0": 1.0, "1": 0.0, "2": 0.0, "3": 1.0, "4": 0.0, "5": 0.0, "6": 0.0, "7": 0.0, "8": 0.0, "9": 0.0}}, "train_Y": {"0": -0.6983359161594589, "1": -0.9989245473995874, "2": -0.43943230900326985, "3": 0.32659696334884825, "4": 1.183969540800228, "5": -0.4776627507524526, "6": -0.6497277709480181, "7": -1.0705356325892745, "8": 1.5323092713990505, "9": 1.2917431513039397}, "torch_params": {"likelihood.noise_covar.raw_noise": [9.999999974752427e-07], "likelihood.noise_covar.noise_prior.concentration": 0.8999999761581421, "likelihood.noise_covar.noise_prior.rate": 10.0, "likelihood.noise_covar.raw_noise_constraint.lower_bound": 9.999999974752427e-07, "likelihood.noise_covar.raw_noise_constraint.upper_bound": "inf", "mean_module.raw_constant": -0.2779732334765344, "covar_module.kernels.0.raw_outputscale": -0.09046509840992041, "covar_module.kernels.0.base_kernel.kernels.0.raw_lengthscale": [[-0.7876875598798336, -1.4605040475112987, 0.0]], "covar_module.kernels.0.base_kernel.kernels.0.active_dims": [0, 1, 2], "covar_module.kernels.0.base_kernel.kernels.0.raw_lengthscale_constraint.lower_bound": 9.999999747378752e-05, "covar_module.kernels.0.base_kernel.kernels.0.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.0.base_kernel.kernels.1.raw_outputscale": -3.411109706915788, "covar_module.kernels.0.base_kernel.kernels.1.active_dims": [3, 4, 5, 6], "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.raw_lengthscale": [[2188.6187430747477, -7630.060517320062, 648.5338363819815, 693.3724753872003]], "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.active_dims": [3, 4, 5, 6], "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.raw_lengthscale_constraint.lower_bound": 9.999999974752427e-07, "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.0.base_kernel.kernels.1.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.kernels.0.base_kernel.kernels.1.raw_outputscale_constraint.upper_bound": "inf", "covar_module.kernels.0.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.kernels.0.raw_outputscale_constraint.upper_bound": "inf", "covar_module.kernels.1.raw_outputscale": -3786.33741273289, "covar_module.kernels.1.base_kernel.kernels.0.raw_lengthscale": [[384.169413551668, 1250.3450776680072, 0.0]], "covar_module.kernels.1.base_kernel.kernels.0.active_dims": [0, 1, 2], "covar_module.kernels.1.base_kernel.kernels.0.raw_lengthscale_constraint.lower_bound": 9.999999747378752e-05, "covar_module.kernels.1.base_kernel.kernels.0.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.1.base_kernel.kernels.1.raw_lengthscale": [[1082.3676483903282, 615.165514751988, 172.3691505963996, -64.28745326902173]], "covar_module.kernels.1.base_kernel.kernels.1.active_dims": [3, 4, 5, 6], "covar_module.kernels.1.base_kernel.kernels.1.raw_lengthscale_constraint.lower_bound": 9.999999974752427e-07, "covar_module.kernels.1.base_kernel.kernels.1.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.1.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.kernels.1.raw_outputscale_constraint.upper_bound": "inf"}}]}, "data": {"Parameter 1": {"0": 5.833333333333334, "1": 7.5, "2": 9.166666666666666, "3": 2.5, "4": 4.166666666666667, "5": 0.8333333333333333, "6": 1.181027361160873, "7": 10.0, "8": 3.5671554309773494, "9": 3.8259819976910956, "10": 2.5436867612524203, "11": 1.943079972685201}, "Parameter 2": {"0": -8.333333333333332, "1": -5.0, "2": -15.0, "3": -18.333333333333332, "4": -11.666666666666666, "5": -1.6666666666666679, "6": -1.3765871582395803, "7": -20.0, "8": -11.972662327181533, "9": -12.021715223130977, "10": -12.190951336683044, "11": -11.695168668363625}, "Parameter 3": {"0": 5.0, "1": 5.0, "2": 5.0, "3": 5.0, "4": 5.0, "5": 5.0, "6": 5.0, "7": 5.0, "8": 5.0, "9": 5.0, "10": 5.0, "11": 5.0}, "Parameter 7": {"0": "D", "1": "C", "2": "B", "3": "D", "4": "A", "5": "B", "6": "A", "7": "A", "8": "B", "9": "A", "10": "B", "11": "B"}, "Response 1": {"0": 52.91999776955157, "1": 47.950545897489405, "2": 57.200296096064626, "3": 69.86459941931172, "4": 84.03902691676035, "5": 56.568255091930745, "6": 53.72360713169506, "7": 46.766642700990914, "8": 89.79791912501248, "9": 85.82078349307184, "10": 92.15225029636682, "11": 91.4972443201605}, "Iteration": {"0": 0, "1": 0, "2": 0, "3": 0, "4": 0, "5": 0, "6": 1, "7": 1, "8": 2, "9": 2, "10": 3, "11": 3}, "Response 1 (pred)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 82.87450053515634, "7": 82.73777771636757, "8": 83.20085855107834, "9": 84.41553343886545, "10": 89.68405430731173, "11": 87.53569795614557}, "Response 1 lb": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 74.82059925402734, "7": 74.25211765306716, "8": 78.12376026030807, "9": 81.59498033002811, "10": 86.43560965382999, "11": 81.72003667435482}, "Response 1 ub": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 90.92840312634186, "7": 91.22343915995488, "8": 88.27795766769508, "9": 87.2360870064971, "10": 92.9324994891891, "11": 93.35136018391825}, "f(Response 1)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 1.6187921375701637, "7": 1.608474260622717, "8": 1.9694820573716958, "9": 2.0668410833202255, "10": 1.525421898187632, "11": 1.3954736641291545}, "Objective 1": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 1.6187921375701637, "7": 1.608474260622717, "8": 1.9694820573716958, "9": 2.0668410833202255, "10": 1.525421898187632, "11": 1.3954736641291545}, "aq Value": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -1.6494456924070198, "7": -1.6094291851799092, "8": -2.0767808380705333, "9": -2.273620548613846, "10": -2.6296665707757936, "11": -2.5557710737282764}, "aq Value (joint)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -0.9759695445936654, "7": -0.9759695445936654, "8": -1.8489609014435588, "9": -1.8489609014435588, "10": -2.339551238036062, "11": -2.339551238036062}, "aq Method": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": "NEI", "7": "NEI", "8": "NEI", "9": "NEI", "10": "NEI", "11": "NEI"}, "Response 1 (max) (iter)": {"0": 84.03902691676035, "1": 84.03902691676035, "2": 84.03902691676035, "3": 84.03902691676035, "4": 84.03902691676035, "5": 84.03902691676035, "6": 84.03902691676035, "7": 84.03902691676035, "8": 89.79791912501248, "9": 89.79791912501248, "10": 92.15225029636682, "11": 92.15225029636682}}, "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[52.91999776955157], [47.950545897489405], [57.200296096064626], [69.86459941931172], [84.03902691676035], [56.568255091930745], [53.72360713169506], [46.766642700990914], [89.79791912501248], [85.82078349307184]]}], "seed": null}
\ No newline at end of file
+{"X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": -20, "max": 0}, "class": "Param_Continuous"}, "Parameter 3": {"state": {"name": "Parameter 3", "min": 5, "max": 5}, "class": "Param_Continuous"}, "Parameter 7": {"state": {"name": "Parameter 7", "categories": ["A", "B", "C", "D"]}, "class": "Param_Categorical"}}, "optimizer": {"opt_attrs": {"X_train": {"Parameter 1": {"0": 2.5, "1": 5.833333333333334, "2": 4.166666666666667, "3": 7.5, "4": 9.166666666666666, "5": 0.8333333333333333, "6": 0.0, "7": 0.0, "8": 0.8777418862425963, "9": 0.0}, "Parameter 2": {"0": -8.333333333333332, "1": -15.0, "2": -5.0, "3": -11.666666666666666, "4": -1.6666666666666679, "5": -18.333333333333332, "6": -12.440288396278948, "7": -11.997004768292857, "8": -17.090092065680093, "9": -20.0}, "Parameter 3": {"0": 5.0, "1": 5.0, "2": 5.0, "3": 5.0, "4": 5.0, "5": 5.0, "6": 5.0, "7": 5.0, "8": 5.0, "9": 5.0}, "Parameter 7": {"0": "B", "1": "A", "2": "D", "3": "D", "4": "B", "5": "C", "6": "A", "7": "B", "8": "B", "9": "B"}}, "y_train": {"Response 1": {"0": 82.53527201469105, "1": 79.49751326961999, "2": 50.06017298463221, "3": 50.06017298463221, "4": 34.310381079701294, "5": 85.4514145684633, "6": 85.9264174732847, "7": 88.46528887058383, "8": 94.35882493568113, "9": 88.47059049434836}}, "y_names": ["Response 1"], "n_response": 1, "seed": null, "X_best_f_idx": 8, "X_best_f": {"Parameter 1": {"8": 0.8777418862425963}, "Parameter 2": {"8": -17.090092065680093}, "Parameter 3": {"8": 5.0}, "Parameter 7": {"8": "B"}}}, "X_space": {"Parameter 1": {"state": {"name": "Parameter 1", "min": 0, "max": 10}, "class": "Param_Continuous"}, "Parameter 2": {"state": {"name": "Parameter 2", "min": -20, "max": 0}, "class": "Param_Continuous"}, "Parameter 3": {"state": {"name": "Parameter 3", "min": 5, "max": 5}, "class": "Param_Continuous"}, "Parameter 7": {"state": {"name": "Parameter 7", "categories": ["A", "B", "C", "D"]}, "class": "Param_Categorical"}}, "surrogate_spec": [{"GP": {}}], "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[82.53527201469105], [79.49751326961999], [50.06017298463221], [50.06017298463221], [34.310381079701294], [85.4514145684633], [85.9264174732847], [88.46528887058383], [94.35882493568113], [88.47059049434836]]}], "model_states": [{"model_attrs": {"model_type": "GP", "seed": null, "hps": {}, "is_fit": true, "cat_dims": [3, 4, 5, 6], "task_feature": null, "X_order": ["Parameter 1", "Parameter 2", "Parameter 3", "Parameter 7^A", "Parameter 7^B", "Parameter 7^C", "Parameter 7^D"], "y_name": "Response 1 Trans"}, "train_X": {"Parameter 1": {"0": 0.25, "1": 0.5833333333333334, "2": 0.4166666666666667, "3": 0.75, "4": 0.9166666666666666, "5": 0.08333333333333333, "6": 0.0, "7": 0.0, "8": 0.08777418862425963, "9": 0.0}, "Parameter 2": {"0": 0.5833333333333334, "1": 0.25, "2": 0.75, "3": 0.4166666666666667, "4": 0.9166666666666666, "5": 0.0833333333333334, "6": 0.3779855801860526, "7": 0.40014976158535714, "8": 0.14549539671599537, "9": 0.0}, "Parameter 3": {"0": 0.0, "1": 0.0, "2": 0.0, "3": 0.0, "4": 0.0, "5": 0.0, "6": 0.0, "7": 0.0, "8": 0.0, "9": 0.0}, "Parameter 7^A": {"0": 0.0, "1": 1.0, "2": 0.0, "3": 0.0, "4": 0.0, "5": 0.0, "6": 1.0, "7": 0.0, "8": 0.0, "9": 0.0}, "Parameter 7^B": {"0": 1.0, "1": 0.0, "2": 0.0, "3": 0.0, "4": 1.0, "5": 0.0, "6": 0.0, "7": 1.0, "8": 1.0, "9": 1.0}, "Parameter 7^C": {"0": 0.0, "1": 0.0, "2": 0.0, "3": 0.0, "4": 0.0, "5": 1.0, "6": 0.0, "7": 0.0, "8": 0.0, "9": 0.0}, "Parameter 7^D": {"0": 0.0, "1": 0.0, "2": 1.0, "3": 1.0, "4": 0.0, "5": 0.0, "6": 0.0, "7": 0.0, "8": 0.0, "9": 0.0}}, "train_Y": {"0": 0.4125354891352347, "1": 0.267182708937696, "2": -1.14135549673337, "3": -1.14135549673337, "4": -1.894962426391257, "5": 0.5520690937478134, "6": 0.5747973610698063, "7": 0.6962790346096207, "8": 0.9782770219897262, "9": 0.6965327103681006}, "torch_params": {"likelihood.noise_covar.raw_noise": [9.999999974752427e-07], "likelihood.noise_covar.noise_prior.concentration": 0.8999999761581421, "likelihood.noise_covar.noise_prior.rate": 10.0, "likelihood.noise_covar.raw_noise_constraint.lower_bound": 9.999999974752427e-07, "likelihood.noise_covar.raw_noise_constraint.upper_bound": "inf", "mean_module.raw_constant": -1.5147534000838652, "covar_module.kernels.0.raw_outputscale": 1.4311122496290651, "covar_module.kernels.0.base_kernel.kernels.0.raw_lengthscale": [[0.32933493883739173, 0.5001757052386704, 0.0]], "covar_module.kernels.0.base_kernel.kernels.0.active_dims": [0, 1, 2], "covar_module.kernels.0.base_kernel.kernels.0.raw_lengthscale_constraint.lower_bound": 9.999999747378752e-05, "covar_module.kernels.0.base_kernel.kernels.0.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.0.base_kernel.kernels.1.raw_outputscale": -2.1353680135909388, "covar_module.kernels.0.base_kernel.kernels.1.active_dims": [3, 4, 5, 6], "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.raw_lengthscale": [[15096.320859016314, 3.988947983686418, 1.2126492733150893, -19436.035077012253]], "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.active_dims": [3, 4, 5, 6], "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.raw_lengthscale_constraint.lower_bound": 9.999999974752427e-07, "covar_module.kernels.0.base_kernel.kernels.1.base_kernel.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.0.base_kernel.kernels.1.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.kernels.0.base_kernel.kernels.1.raw_outputscale_constraint.upper_bound": "inf", "covar_module.kernels.0.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.kernels.0.raw_outputscale_constraint.upper_bound": "inf", "covar_module.kernels.1.raw_outputscale": -10007.062975137847, "covar_module.kernels.1.base_kernel.kernels.0.raw_lengthscale": [[6663.697569267358, 3378.286071218982, 0.0]], "covar_module.kernels.1.base_kernel.kernels.0.active_dims": [0, 1, 2], "covar_module.kernels.1.base_kernel.kernels.0.raw_lengthscale_constraint.lower_bound": 9.999999747378752e-05, "covar_module.kernels.1.base_kernel.kernels.0.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.1.base_kernel.kernels.1.raw_lengthscale": [[7591.019679170469, 4626.571188777847, 2750.51933566292, -6364.228970927128]], "covar_module.kernels.1.base_kernel.kernels.1.active_dims": [3, 4, 5, 6], "covar_module.kernels.1.base_kernel.kernels.1.raw_lengthscale_constraint.lower_bound": 9.999999974752427e-07, "covar_module.kernels.1.base_kernel.kernels.1.raw_lengthscale_constraint.upper_bound": "inf", "covar_module.kernels.1.raw_outputscale_constraint.lower_bound": 0.0, "covar_module.kernels.1.raw_outputscale_constraint.upper_bound": "inf"}}]}, "data": {"Parameter 1": {"0": 2.5, "1": 5.833333333333334, "2": 4.166666666666667, "3": 7.5, "4": 9.166666666666666, "5": 0.8333333333333333, "6": 0.0, "7": 0.0, "8": 0.8777418862425963, "9": 0.0, "10": 2.313213846906087, "11": 2.344134252677562}, "Parameter 2": {"0": -8.333333333333332, "1": -15.0, "2": -5.0, "3": -11.666666666666666, "4": -1.6666666666666679, "5": -18.333333333333332, "6": -12.440288396278948, "7": -11.997004768292857, "8": -17.090092065680093, "9": -20.0, "10": -17.155920012756816, "11": -20.0}, "Parameter 3": {"0": 5.0, "1": 5.0, "2": 5.0, "3": 5.0, "4": 5.0, "5": 5.0, "6": 5.0, "7": 5.0, "8": 5.0, "9": 5.0, "10": 5.0, "11": 5.0}, "Parameter 7": {"0": "B", "1": "A", "2": "D", "3": "D", "4": "B", "5": "C", "6": "A", "7": "B", "8": "B", "9": "B", "10": "B", "11": "B"}, "Response 1": {"0": 82.53527201469105, "1": 79.49751326961999, "2": 50.06017298463221, "3": 50.06017298463221, "4": 34.310381079701294, "5": 85.4514145684633, "6": 85.9264174732847, "7": 88.46528887058383, "8": 94.35882493568113, "9": 88.47059049434836, "10": 95.42578063008993, "11": 91.97215812296726}, "Iteration": {"0": 0, "1": 0, "2": 0, "3": 0, "4": 0, "5": 0, "6": 1, "7": 1, "8": 2, "9": 2, "10": 3, "11": 3}, "Response 1 (pred)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 89.00676039170254, "7": 89.00292284974587, "8": 88.61091153957356, "9": 86.14499247393553, "10": 96.39280984122834, "11": 94.34539783336328}, "Response 1 lb": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 83.63269029163665, "7": 83.53890532835052, "8": 87.56797206296041, "9": 83.77485897341674, "10": 94.45329499483114, "11": 90.58064715863613}, "Response 1 ub": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 94.38083136592066, "7": 94.46694125992435, "8": 89.6538511858324, "9": 88.51512635998287, "10": 98.33232500310918, "11": 98.11014912046895}, "f(Response 1)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 1.1787512423113158, "7": 1.1785728302769922, "8": 0.8993847364615066, "9": 0.7831021102105586, "10": 1.0756005376839257, "11": 0.9776345527169232}, "Objective 1": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": 1.1787512423113158, "7": 1.1785728302769922, "8": 0.8993847364615066, "9": 0.7831021102105586, "10": 1.0756005376839257, "11": 0.9776345527169232}, "aq Value": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -1.6068393762851096, "7": -1.6011613591279685, "8": -3.792897051640378, "9": -4.7499083304917535, "10": -2.2670628784746567, "11": -2.6737066856622236}, "aq Value (joint)": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": -1.5757223145722694, "7": -1.5757223145722694, "8": -3.500561469513955, "9": -3.500561469513955, "10": -2.1523796091698157, "11": -2.1523796091698157}, "aq Method": {"0": NaN, "1": NaN, "2": NaN, "3": NaN, "4": NaN, "5": NaN, "6": "NEI", "7": "NEI", "8": "NEI", "9": "NEI", "10": "NEI", "11": "NEI"}, "Response 1 (max) (iter)": {"0": 85.4514145684633, "1": 85.4514145684633, "2": 85.4514145684633, "3": 85.4514145684633, "4": 85.4514145684633, "5": 85.4514145684633, "6": 88.46528887058383, "7": 88.46528887058383, "8": 94.35882493568113, "9": 94.35882493568113, "10": 95.42578063008993, "11": 95.42578063008993}}, "target": [{"init_attrs": {"name": "Response 1", "aim": "max", "f_transform": "Standard"}, "f_raw": [[82.53527201469105], [79.49751326961999], [50.06017298463221], [50.06017298463221], [34.310381079701294], [85.4514145684633], [85.9264174732847], [88.46528887058383], [94.35882493568113], [88.47059049434836]]}], "seed": null}
\ No newline at end of file
From 3ebce9e528ed0e17d529e427701e01bb9986a7df Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 21:03:51 -0600
Subject: [PATCH 057/136] Switched all X_ref from X_space.mean to
optimizer.X_best_f
---
obsidian/campaign/analysis.py | 2 +-
obsidian/campaign/explainer.py | 6 +++---
obsidian/plotting/plotly.py | 4 ++--
obsidian/tests/test_campaign.py | 4 ++--
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/obsidian/campaign/analysis.py b/obsidian/campaign/analysis.py
index 48e8ff2..ab25273 100644
--- a/obsidian/campaign/analysis.py
+++ b/obsidian/campaign/analysis.py
@@ -41,7 +41,7 @@ def calc_ofat_ranges(optimizer: Optimizer,
response_name = optimizer.target[response_id].name
if X_ref is None:
- X_ref = optimizer.X_space.mean()
+ X_ref = optimizer.X_best_f
if isinstance(X_ref, pd.Series):
X_ref = X_ref.to_frame().T
diff --git a/obsidian/campaign/explainer.py b/obsidian/campaign/explainer.py
index c5b5142..c64bed8 100644
--- a/obsidian/campaign/explainer.py
+++ b/obsidian/campaign/explainer.py
@@ -81,7 +81,7 @@ def shap_explain(self,
self.shap['responseid'] = responseid
if X_ref is None:
- X_ref = self.X_space.mean()
+ X_ref = self.optimizer.X_best_f
else:
if not all(x in X_ref.columns for x in self.X_space.X_names):
raise ValueError('X_ref must contain all parameters in X_space')
@@ -188,7 +188,7 @@ def shap_single_point(self,
Args:
X_new (pd.DataFrame | pd.Series): New data point to explain
X_ref (pd.DataFrame | pd.Series, optional): Reference data point
- for shap values. Default uses ``X_space.mean()``
+ for shap values. Default uses ``optimizer.X_best_f``
Returns:
pd.DataFrame: DataFrame containing SHAP values for the new data point
@@ -256,7 +256,7 @@ def sensitivity(self,
X_ref = X_ref.copy().to_frame().T
if X_ref is None:
- X_ref = self.optimizer.X_space.mean()
+ X_ref = self.optimizer.X_best_f
else:
if not all(x in X_ref.columns for x in self.optimizer.X_space.X_names):
raise ValueError('X_ref must contain all parameters in X_space')
diff --git a/obsidian/plotting/plotly.py b/obsidian/plotting/plotly.py
index f789730..ca7fe66 100644
--- a/obsidian/plotting/plotly.py
+++ b/obsidian/plotting/plotly.py
@@ -196,7 +196,7 @@ def factor_plot(optimizer: Optimizer,
# Create a dataframe of test samples for plotting
n_samples = 100
if X_ref is None:
- df_mean = optimizer.X_space.mean()
+ df_mean = optimizer.X_best_f
X_test = pd.concat([df_mean]*n_samples, axis=0).reset_index(drop=True)
else:
if not isinstance(X_ref, pd.DataFrame):
@@ -298,7 +298,7 @@ def surface_plot(optimizer: Optimizer,
# Create a dataframe of test samples for plotting
n_grid = 100
- df_mean = optimizer.X_space.mean()
+ df_mean = optimizer.X_best_f
X_test = pd.concat([df_mean]*(n_grid**2), axis=0).reset_index(drop=True)
# Create a mesh grid which is necessary for the 3D plot
diff --git a/obsidian/tests/test_campaign.py b/obsidian/tests/test_campaign.py
index 436c3b1..258a712 100644
--- a/obsidian/tests/test_campaign.py
+++ b/obsidian/tests/test_campaign.py
@@ -166,7 +166,7 @@ def test_explainer_validation():
# Unfit SHAP
with pytest.raises(UnfitError):
- exp.shap_single_point(X_new=campaign.X_space.mean())
+ exp.shap_single_point(X_new=campaign.optimizer.X_best_f)
random_data = pd.DataFrame(data={'A': [1], 'B': [4]})
long_data = pd.DataFrame(data={'Parameter 1': [1, 2], 'Parameter 2': [1, 2]})
@@ -187,7 +187,7 @@ def test_explainer_validation():
# Missing X names
with pytest.raises(ValueError):
- exp.shap_single_point(X_new=campaign.X_space.mean(), X_ref=random_data)
+ exp.shap_single_point(X_new=campaign.optimizer.X_best_f, X_ref=random_data)
# Missing X names
with pytest.raises(ValueError):
From 1c86a5094095e6bc491b3aa9cb4e73fe342dbed1 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 21:05:56 -0600
Subject: [PATCH 058/136] Bump version
---
CHANGELOG.md | 8 ++++++++
obsidian/__init__.py | 2 +-
pyproject.toml | 2 +-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1a186ce..7ba2c04 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,13 @@
# Changelog
+## [0.8.4]
+### Added
+- Campaign X_best method
+- Optimizer X_best_f attribute(s)
+
+### Modified
+- Switched all usages of X_ref = X_space.mean() to optimizer.X_best_f
+
## [0.8.3]
### Added
- Default values for NParEGO scalarization_weights
diff --git a/obsidian/__init__.py b/obsidian/__init__.py
index e1be93d..4769d04 100644
--- a/obsidian/__init__.py
+++ b/obsidian/__init__.py
@@ -1,5 +1,5 @@
"""obsidian: Automated experiment design and black-box optimization"""
-__version__ = '0.8.3'
+__version__ = '0.8.4'
# Import key objects
from obsidian.campaign import Campaign
diff --git a/pyproject.toml b/pyproject.toml
index 3a487b2..2e85a54 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "obsidian-apo"
-version = "0.8.3"
+version = "0.8.4"
description = "Automated experiment design and black-box optimization"
authors = [
"Kevin Stone ",
From db45780508e8509c0ae06793d1a818a9a9e1523a Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 22:15:35 -0600
Subject: [PATCH 059/136] Added color_list to branding
---
CHANGELOG.md | 1 +
obsidian/plotting/branding.py | 4 ++++
2 files changed, 5 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7ba2c04..d98a364 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
### Added
- Campaign X_best method
- Optimizer X_best_f attribute(s)
+- Sequence of colors "color_list" to branding
### Modified
- Switched all usages of X_ref = X_space.mean() to optimizer.X_best_f
diff --git a/obsidian/plotting/branding.py b/obsidian/plotting/branding.py
index a3f1afc..58fefb3 100644
--- a/obsidian/plotting/branding.py
+++ b/obsidian/plotting/branding.py
@@ -40,6 +40,10 @@ class Palette():
accent.vista_blue = '#688CE8'
accent.rich_blue = '#5450E4'
+# Create repeating list of colors for easy access
+obsidian_color_list = [primary.teal, secondary.blue, secondary.light_teal, accent.pastel_blue,
+ accent.lime, accent.vista_blue, accent.lemon, accent.rich_blue]
+
# Unused in palette, but used for color map
magenta = '#d04495'
From a96b484f4c59ff4c80e2f1ee5d62aa2f584b5c49 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Wed, 21 Aug 2024 22:16:18 -0600
Subject: [PATCH 060/136] Moved visualize_inputs from MPL to plotly
---
CHANGELOG.md | 1 +
obsidian/plotting/mpl.py | 40 ---------------------
obsidian/plotting/plotly.py | 72 +++++++++++++++++++++++++++++++++++++
3 files changed, 73 insertions(+), 40 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d98a364..cd74292 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@
### Modified
- Switched all usages of X_ref = X_space.mean() to optimizer.X_best_f
+- Refactored mpl "visualize_inputs" as plotly "visualize_inputs" for better interactivity
## [0.8.3]
### Added
diff --git a/obsidian/plotting/mpl.py b/obsidian/plotting/mpl.py
index 71b4767..729cb5d 100644
--- a/obsidian/plotting/mpl.py
+++ b/obsidian/plotting/mpl.py
@@ -1,54 +1,14 @@
"""Matplotlib figure-generating functions"""
-from obsidian.campaign import Campaign
from obsidian.optimizer import Optimizer
import matplotlib.pyplot as plt
-import matplotlib.gridspec as gridspec
from matplotlib.pyplot import Figure
import numpy as np
import pandas as pd
-def visualize_inputs(campaign: Campaign) -> Figure:
- """
- Visualizes the input variables of a campaign.
-
- Args:
- campaign (Campaign): The campaign object containing the input data.
-
- Returns:
- Figure: The matplotlib Figure object containing the visualization.
- """
- n_dim = campaign.X_space.n_dim
-
- fig = plt.figure(constrained_layout=True, figsize=(2*(n_dim+1), 2.5))
- gs = gridspec.GridSpec(ncols=n_dim+1, nrows=1, figure=fig)
-
- X0 = campaign.data[list(campaign.X_space.X_names)]
- X_u = campaign.X_space.unit_map(X0)
- m_exp = X0.shape[0]
-
- for i, param in enumerate(X0.columns):
- ax = fig.add_subplot(gs[0, i])
- ax.plot(X0[param], 'o')
- ax.set_xlabel('Experiment')
- ax.set_title(param)
- ax.set_xticks(np.around(np.linspace(0, m_exp, 5)))
-
- ax = fig.add_subplot(gs[0, -1])
- plt.imshow(abs(X_u.corr()), cmap='viridis')
- ax.set_xticks(range(n_dim))
- ax.set_yticks(range(n_dim))
- ax.set_xticklabels(X0.columns, rotation=90, ha='right')
- ax.set_yticklabels(X0.columns)
- plt.colorbar()
- plt.title('Correlation Plot')
-
- return fig
-
-
def plot_ofat_ranges(optimizer: Optimizer,
ofat_ranges: pd.DataFrame) -> Figure:
"""
diff --git a/obsidian/plotting/plotly.py b/obsidian/plotting/plotly.py
index ca7fe66..ed7e3d8 100644
--- a/obsidian/plotting/plotly.py
+++ b/obsidian/plotting/plotly.py
@@ -5,13 +5,85 @@
from obsidian.exceptions import UnfitError, UnsupportedError
from obsidian.parameters import Param_Continuous
from .branding import obsidian_colors
+from obsidian.plotting.branding import obsidian_color_list as colors
import plotly.graph_objects as go
from plotly.graph_objects import Figure
+from plotly.subplots import make_subplots
from sklearn.manifold import MDS
import pandas as pd
import numpy as np
+import math
+
+
+def visualize_inputs(campaign: Campaign) -> Figure:
+ """
+ Visualizes the input variables of a campaign.
+
+ Args:
+ campaign (Campaign): The campaign object containing the input data.
+
+ Returns:
+ Figure: The plotly Figure object containing the visualization.
+ """
+ n_dim = campaign.X_space.n_dim
+ X = campaign.X
+
+ # Enforce that there are 2 rows
+ # Determine the number of columns based on the number of dimensions
+ rows = 2
+ cols = math.ceil(n_dim / rows)
+
+ height = 200 * rows
+ width = 300 * cols
+ fontsize = 8
+
+ color_list = colors * 10
+
+ # Add an extra 2 cols for the correlation matrix
+ fig = make_subplots(
+ rows=rows, cols=cols + 2,
+ vertical_spacing=0.2,
+ horizontal_spacing=0.1,
+ specs=[[{}]*cols + [{"colspan": 2, "rowspan": 2}, None],
+ [{}]*cols + [None, None]],
+ subplot_titles=[X.columns[i] for i in range(cols)]
+ + ['Correlation Matrix']
+ + [X.columns[i] for i in range(cols, n_dim)]
+ )
+
+ for i, param in enumerate(X.columns):
+ row_i = i // cols + 1
+ col_i = i % cols + 1
+ fig.add_trace(go.Scatter(x=X.index, y=X[param],
+ mode='markers', name=param,
+ marker=dict(color=color_list[i]),
+ showlegend=False),
+ row=row_i, col=col_i)
+ fig.update_xaxes(tickvals=np.around(np.linspace(0, campaign.m_exp, 5)),
+ row=row_i, col=col_i)
+
+ # Calculate the correlation matrix
+ X_u = campaign.X_space.unit_map(X)
+ corr_matrix = X_u.corr()
+ fig.add_trace(go.Heatmap(z=corr_matrix.values,
+ x=corr_matrix.columns,
+ y=corr_matrix.columns,
+ colorscale=[[0, obsidian_colors.rich_blue],
+ [0.5, obsidian_colors.teal],
+ [1, obsidian_colors.lemon]],
+ name='Correlation'),
+ row=1, col=cols+1)
+
+ fig.update_yaxes(showticklabels=False, row=1, col=cols+1)
+ fig.update_xaxes(tickangle=-90, row=1, col=cols+1)
+
+ fig.update_layout(width=width, height=height, template='ggplot2',
+ font_size=fontsize, title_text='Campaign Data Visualization')
+ fig.update_annotations(font_size=fontsize)
+
+ return fig
def MDS_plot(campaign: Campaign) -> Figure:
From 97372b270bf131227c38e51e00142027cbc3776a Mon Sep 17 00:00:00 2001
From: Kevin Stone <91576158+kstone40@users.noreply.github.com>
Date: Wed, 21 Aug 2024 22:21:48 -0600
Subject: [PATCH 061/136] Add linting to CI
---
.github/workflows/CI.yml | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
index 1153a60..9108a51 100644
--- a/.github/workflows/CI.yml
+++ b/.github/workflows/CI.yml
@@ -44,3 +44,13 @@ jobs:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: _build/
force_orphan: true
+
+ lint:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.10"
+ - name: flake8 Lint
+ uses: py-actions/flake8@v2
From 1af47356995bf9d93153f73eb626de5952672a65 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 10:48:28 -0600
Subject: [PATCH 062/136] Added X hoverdata to MDS plot
---
obsidian/plotting/plotly.py | 23 ++++++++++++++++++-----
1 file changed, 18 insertions(+), 5 deletions(-)
diff --git a/obsidian/plotting/plotly.py b/obsidian/plotting/plotly.py
index ed7e3d8..d7fc8cf 100644
--- a/obsidian/plotting/plotly.py
+++ b/obsidian/plotting/plotly.py
@@ -116,7 +116,7 @@ def MDS_plot(campaign: Campaign) -> Figure:
fig.add_trace(go.Scatter(x=X_mds[:, 0], y=X_mds[:, 1],
mode='markers',
- name='Observations',
+ name='',
marker={'color': iter_vals, 'size': 10,
'cmax': iter_max, 'cmin': 0,
'colorscale': [[0, obsidian_colors.rich_blue],
@@ -124,8 +124,20 @@ def MDS_plot(campaign: Campaign) -> Figure:
[1, obsidian_colors.lemon]],
'colorbar': cbar
},
+ customdata=campaign.data[
+ list(campaign.X_space.X_names) + ['Iteration']],
showlegend=False
))
+
+ template = [""+str(param.name)+": "+" %{customdata["+str(i)+"]"
+ + (":.3G}"if isinstance(param, Param_Continuous) else "}") + " "
+ for i, param in enumerate(campaign.X_space)]
+
+ fig.update_traces(hovertemplate=''.join(template)
+ + 'Iteration'
+ + ": %{customdata["+str(len(campaign.X_space))+"]} "
+ + 'MDS C1' + ": %{x:.3G} "
+ + 'MDS C2' + ": %{y:.3G} ")
fig.update_xaxes(title_text='Component 1')
fig.update_yaxes(title_text='Component 2')
@@ -530,11 +542,12 @@ def optim_progress(campaign: Campaign,
customdata=campaign.data[X_names],
name='Data'))
- template = [""+str(param.name)+": "+" %{customdata["+str(i)+"]"
- + (":.3G}"if isinstance(param, Param_Continuous) else "}") + " "
+ template = [""+str(param.name)+": "+" %{customdata["+str(i)+"]"
+ + (":.3G}"if isinstance(param, Param_Continuous) else "}") + " "
for i, param in enumerate(campaign.X_space)]
- fig.update_traces(hovertemplate=''.join(template) + out_names[0]
- + ": %{x:.3G} " + out_names[1] + ": %{y:.3G} ")
+ fig.update_traces(hovertemplate=''.join(template)
+ + '' + out_names[0] + '' + ": %{x:.3G} "
+ + '' + out_names[1] + '' + ": %{y:.3G} ")
if X_suggest is not None:
if not all(x in X_suggest.columns for x in campaign.X.columns):
From 3c47edc2616b6763888b1344801a8b2d737ddc24 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 11:21:19 -0600
Subject: [PATCH 063/136] Created product objective
---
obsidian/objectives/config.py | 6 +++--
obsidian/objectives/custom.py | 50 +++++++++++++++++++++++++++++++----
2 files changed, 49 insertions(+), 7 deletions(-)
diff --git a/obsidian/objectives/config.py b/obsidian/objectives/config.py
index 7e9c926..84efdeb 100644
--- a/obsidian/objectives/config.py
+++ b/obsidian/objectives/config.py
@@ -5,7 +5,8 @@
Feature_Objective,
Utopian_Distance,
Bounded_Target,
- Index_Objective
+ Index_Objective,
+ Product_Objective
)
from .scalarize import (
@@ -21,4 +22,5 @@
'Scalar_WeightedSum': Scalar_WeightedSum,
'Scalar_WeightedNorm': Scalar_WeightedNorm,
'Scalar_Chebyshev': Scalar_Chebyshev,
- 'Index_Objective': Index_Objective}
+ 'Index_Objective': Index_Objective,
+ 'Product_Objective': Product_Objective}
diff --git a/obsidian/objectives/custom.py b/obsidian/objectives/custom.py
index 270d00f..3698b9b 100644
--- a/obsidian/objectives/custom.py
+++ b/obsidian/objectives/custom.py
@@ -27,6 +27,46 @@ def forward(self,
return self.output_shape(samples)
+class Product_Objective(Objective):
+ """
+ Objective function that computes the weighted products of other objectives
+
+ Args:
+ ind (tuple[int]): The indices of objectives to be used in a product
+ weights (tuple[float]): The weights corresponding to indexed objectives
+ const (float): A constant value that can be added to the product
+ new_dim (bool, optional): Whether to create a new objective dimension
+ from this product. Default is ``False``. Setting to ``True`` will
+ make this the only output objective.
+
+ """
+ def __init__(self,
+ ind: tuple[int],
+ weights: tuple[float],
+ const: float | int = 0,
+ new_dim: bool = True) -> None:
+ super().__init__(new_dim)
+ # Always MOO if dim is being added, always SOO otherwise
+ if len(weights) != len(ind):
+ raise ValueError('The length of weights and indices must be the same')
+ self.register_buffer('ind', torch.tensor(ind, dtype=torch.int))
+ self.register_buffer('weights', torch.tensor(weights, dtype=TORCH_DTYPE))
+ self.register_buffer('const', torch.tensor(const, dtype=TORCH_DTYPE))
+ self.register_buffer('new_dim', torch.tensor(new_dim, dtype=torch.bool))
+
+ def forward(self,
+ samples: Tensor,
+ X: Tensor | None = None) -> Tensor:
+ """
+ Evaluate the objective function on the candidate set samples, X
+ """
+ obj = (self.weights*samples[..., self.ind]).prod(dim=-1, keepdim=True) + self.const
+ if self.new_dim:
+ return torch.concat((samples, obj), dim=-1)
+ else:
+ return self.output_shape(obj)
+
+
class Feature_Objective(Objective):
"""
Creates an objective function which creates a new outcome as a linear combination of input features.
@@ -47,8 +87,8 @@ class Feature_Objective(Objective):
def __init__(self,
X_space: ParamSpace,
- indices: list[float | int],
- coeff: list[float | int | list]) -> None:
+ indices: list[int],
+ coeff: list[float | int]) -> None:
super().__init__(mo=True)
@@ -278,14 +318,14 @@ class Index_Objective(Objective):
Creates an objective function that returns the single-objective output
from a multi-output model, at the specified index.
- Always a single-output objective.
+ Single-objective when index is int; multi-objective if tuple.
Args:
index (int): The index of the value to be returned.
"""
def __init__(self,
- index: int = 0) -> None:
- super().__init__(mo=False)
+ index: int | tuple[int] = 0) -> None:
+ super().__init__(mo=not isinstance(index, int))
self.register_buffer('index', torch.tensor(index))
def __repr__(self):
From 222927b7ce2474c62f2f833db9ed5e8e587a4f27 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 11:30:38 -0600
Subject: [PATCH 064/136] Renamed index/indices to ind for consistency
---
obsidian/objectives/custom.py | 56 +++++++++++++++----------------
obsidian/tests/test_objectives.py | 2 +-
2 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/obsidian/objectives/custom.py b/obsidian/objectives/custom.py
index 3698b9b..ead99bd 100644
--- a/obsidian/objectives/custom.py
+++ b/obsidian/objectives/custom.py
@@ -34,7 +34,7 @@ class Product_Objective(Objective):
Args:
ind (tuple[int]): The indices of objectives to be used in a product
weights (tuple[float]): The weights corresponding to indexed objectives
- const (float): A constant value that can be added to the product
+ const (float | int): A constant value that can be added to the product
new_dim (bool, optional): Whether to create a new objective dimension
from this product. Default is ``False``. Setting to ``True`` will
make this the only output objective.
@@ -75,37 +75,37 @@ class Feature_Objective(Objective):
Args:
X_space (ParamSpace): The parameter space.
- indices (list[float | int], optional): The indices of the parameters in the real space
- to be used as features. Defaults to ``[0]``.
- coeff (list[float | int | list], optional): The coefficients corresponding to each feature.
+ ind (tuple[int]): The indices of the parameters in the real space
+ to be used as features.
+ coeff (tuple[float | int], optional): The coefficients corresponding to each feature.
Defaults to ``[0]``.
Raises:
- ValueError: If the length of `indices` and `coeff` are not the same.
+ ValueError: If the length of `ind` and `coeff` are not the same.
TypeError: If the indices for input objectives are not for continuous parameters.
"""
def __init__(self,
X_space: ParamSpace,
- indices: list[int],
- coeff: list[float | int]) -> None:
+ ind: tuple[int],
+ coeff: tuple[float | int]) -> None:
super().__init__(mo=True)
- if len(indices) != len(coeff):
+ if len(ind) != len(coeff):
raise ValueError("featureids and coeff must have the same length")
- for i in indices:
+ for i in ind:
if not isinstance(X_space.params[i], Param_Continuous):
raise TypeError('Indices for input objectives must be for continuous parameters only')
self.register_buffer('coeff', torch.tensor(coeff, dtype=TORCH_DTYPE))
- self.register_buffer('indices', torch.tensor(indices, dtype=torch.long))
+ self.register_buffer('ind', torch.tensor(ind, dtype=torch.long))
self.X_space = X_space
def __repr__(self):
"""String representation of object"""
- return f'{self.__class__.__name__} (indices={self.indices.tolist()}, coeff={self.coeff.tolist()})'
+ return f'{self.__class__.__name__} (ind={self.ind.tolist()}, coeff={self.coeff.tolist()})'
def forward(self,
samples: Tensor,
@@ -118,7 +118,7 @@ def forward(self,
# if q is 1, it is omitted
X_u_all = [] # Create unscaled X
- for i in self.indices:
+ for i in self.ind:
X_u_all.append(torch.tensor(
self.X_space[i].decode(X[..., i].detach().numpy()),
dtype=TORCH_DTYPE).unsqueeze(-1)) # Add output dimension
@@ -157,7 +157,7 @@ class Utopian_Distance(Objective):
and a most-desirable, ideal/utopian point.
Args:
- utopian (list[float]): A list of values representing the utopian point.
+ utopian (tuple[float | int]): A list of values representing the utopian point.
targets (Target | list[Target]): A single Target object or a list of Target objects.
Attributes:
@@ -170,7 +170,7 @@ class Utopian_Distance(Objective):
"""
def __init__(self,
- utopian: list[float],
+ utopian: list[float | int],
targets: Target | list[Target]) -> None:
if not isinstance(targets, list):
@@ -238,7 +238,7 @@ class Bounded_Target(Objective):
lb (torch.Tensor): The lower bounds for the targets.
ub (torch.Tensor): The upper bounds for the targets.
tau (torch.Tensor): The temperature parameter for the sigmoid function.
- indices (torch.Tensor): The indices of the non-None bounds.
+ ind (torch.Tensor): The indices of the non-None bounds.
"""
@@ -260,17 +260,17 @@ def __init__(self,
super().__init__(mo=len(targets) > 1)
# Only store and process the indicated bounds
- indices = []
+ ind = []
lb_t = []
ub_t = []
for i, (b, target) in enumerate(zip(bounds, targets)):
if b is not None:
lb_t.append(target.transform_f(b[0]).iloc[0])
ub_t.append(target.transform_f(b[1]).iloc[0])
- indices.append(i)
+ ind.append(i)
self.bounds = bounds
- self.indices = torch.tensor(indices)
+ self.ind = torch.tensor(ind)
self.lb = torch.tensor(lb_t, dtype=TORCH_DTYPE).flatten()
self.ub = torch.tensor(ub_t, dtype=TORCH_DTYPE).flatten()
tau = torch.tensor(tau, dtype=TORCH_DTYPE)
@@ -279,7 +279,7 @@ def __init__(self,
def __repr__(self):
"""String representation of object"""
- return f'{self.__class__.__name__} (indices={self.indices.tolist()}, bounds={self.bounds})'
+ return f'{self.__class__.__name__} (ind={self.ind.tolist()}, bounds={self.bounds})'
def forward(self,
samples: Tensor,
@@ -287,11 +287,11 @@ def forward(self,
"""
Evaluate the objective function on the candidate set samples, X
"""
- approx_lb = torch.sigmoid((samples[..., self.indices] - self.lb) / self.tau)
- approx_ub = torch.sigmoid((self.ub - samples[..., self.indices]) / self.tau)
+ approx_lb = torch.sigmoid((samples[..., self.ind] - self.lb) / self.tau)
+ approx_ub = torch.sigmoid((self.ub - samples[..., self.ind]) / self.tau)
product = approx_lb * approx_ub
out = samples.to(TORCH_DTYPE)
- out[..., self.indices] = product
+ out[..., self.ind] = product
return self.output_shape(out)
def save_state(self) -> dict:
@@ -321,16 +321,16 @@ class Index_Objective(Objective):
Single-objective when index is int; multi-objective if tuple.
Args:
- index (int): The index of the value to be returned.
+ ind (int | tuple[int]): The index of the value to be returned.
"""
def __init__(self,
- index: int | tuple[int] = 0) -> None:
- super().__init__(mo=not isinstance(index, int))
- self.register_buffer('index', torch.tensor(index))
+ ind: int | tuple[int] = 0) -> None:
+ super().__init__(mo=not isinstance(ind, int))
+ self.register_buffer('ind', torch.tensor(ind))
def __repr__(self):
"""String representation of object"""
- return f'{self.__class__.__name__} (index={self.index.item()})'
+ return f'{self.__class__.__name__} (ind={self.ind.item()})'
def forward(self,
samples: Tensor,
@@ -338,4 +338,4 @@ def forward(self,
"""
Evaluate the objective function on the candidate set samples, X
"""
- return samples[..., self.index]
+ return samples[..., self.ind]
diff --git a/obsidian/tests/test_objectives.py b/obsidian/tests/test_objectives.py
index 23dd6a8..2d3a07f 100644
--- a/obsidian/tests/test_objectives.py
+++ b/obsidian/tests/test_objectives.py
@@ -32,7 +32,7 @@
test_objs = [Identity_Objective(mo=len(target) > 1),
Scalar_WeightedNorm(weights=[1, 1]),
- Feature_Objective(X_space, indices=[0], coeff=[1]),
+ Feature_Objective(X_space, ind=(0,), coeff=(1,)),
Objective_Sequence([Utopian_Distance([1], target[0]), Index_Objective()]),
Bounded_Target(bounds=[(0, 1)]*len(target), targets=target),
None]
From dfe514cbd0be5517866cc56bd3e19d86a8035685 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 11:53:52 -0600
Subject: [PATCH 065/136] Created divide objective
---
obsidian/objectives/config.py | 6 ++--
obsidian/objectives/custom.py | 52 +++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+), 2 deletions(-)
diff --git a/obsidian/objectives/config.py b/obsidian/objectives/config.py
index 84efdeb..825ee76 100644
--- a/obsidian/objectives/config.py
+++ b/obsidian/objectives/config.py
@@ -6,7 +6,8 @@
Utopian_Distance,
Bounded_Target,
Index_Objective,
- Product_Objective
+ Product_Objective,
+ Divide_Objective
)
from .scalarize import (
@@ -23,4 +24,5 @@
'Scalar_WeightedNorm': Scalar_WeightedNorm,
'Scalar_Chebyshev': Scalar_Chebyshev,
'Index_Objective': Index_Objective,
- 'Product_Objective': Product_Objective}
+ 'Product_Objective': Product_Objective,
+ 'Divide_Objective': Divide_Objective}
diff --git a/obsidian/objectives/custom.py b/obsidian/objectives/custom.py
index ead99bd..71f29b9 100644
--- a/obsidian/objectives/custom.py
+++ b/obsidian/objectives/custom.py
@@ -67,6 +67,58 @@ def forward(self,
return self.output_shape(obj)
+class Divide_Objective(Objective):
+ """
+ Objective function that computes the weighted quotient of other objectives
+
+ (w_num * samples[..., ind_num])/(w_denom * samples[..., ind_denom]) + const
+
+ Args:
+ ind_num (int): The index of the objective to be used in the numerator
+ w_num (float | int, optional): The weights corresponding to numerator
+ objective. Defaults to ``1``.
+ ind_denom (int): The index of the objective to be used in the denominator
+ w_denom (float | int, optional): The weights corresponding to denominator
+ objective. Defaults to ``1``.
+ const (float | int): A constant value that can be added to the quotient
+ new_dim (bool, optional): Whether to create a new objective dimension
+ from this quotient. Default is ``False``. Setting to ``True`` will
+ make this the only output objective.
+
+ """
+ def __init__(self,
+ ind_num: int,
+ ind_denom: int,
+ w_num: float | int = 1,
+ w_denom: float | int = 1,
+ const: float | int = 0,
+ new_dim: bool = True) -> None:
+ super().__init__(new_dim)
+ # Always MOO if dim is being added, always SOO otherwise
+
+ self.register_buffer('ind_num', torch.tensor(ind_num, dtype=torch.int))
+ self.register_buffer('w_num', torch.tensor(w_num, dtype=TORCH_DTYPE))
+ self.register_buffer('ind_denom', torch.tensor(ind_denom, dtype=torch.int))
+ self.register_buffer('w_denom', torch.tensor(w_denom, dtype=TORCH_DTYPE))
+ self.register_buffer('const', torch.tensor(const, dtype=TORCH_DTYPE))
+ self.register_buffer('new_dim', torch.tensor(new_dim, dtype=torch.bool))
+
+ def forward(self,
+ samples: Tensor,
+ X: Tensor | None = None) -> Tensor:
+ """
+ Evaluate the objective function on the candidate set samples, X
+ """
+ w_num = (self.w_num*samples[..., self.ind_num]).unsqueeze(-1)
+ w_denom = (self.w_denom*samples[..., self.ind_denom]).unsqueeze(-1)
+ obj = w_num/w_denom + self.const
+
+ if self.new_dim:
+ return torch.concat((samples, obj), dim=-1)
+ else:
+ return self.output_shape(obj)
+
+
class Feature_Objective(Objective):
"""
Creates an objective function which creates a new outcome as a linear combination of input features.
From 3f591cc1ec1ee7b100ce0ba92307c02d74d7fc7b Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 11:54:33 -0600
Subject: [PATCH 066/136] Created Product and Divide objectives
---
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index cd74292..74a790e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@
- Campaign X_best method
- Optimizer X_best_f attribute(s)
- Sequence of colors "color_list" to branding
+- Created Product_Objective and Divide_Objective
### Modified
- Switched all usages of X_ref = X_space.mean() to optimizer.X_best_f
From 9a244e879987ade23935174b0b79f21d50e135c6 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 12:11:46 -0600
Subject: [PATCH 067/136] Added repr methods and testing
---
obsidian/objectives/custom.py | 27 ++++++++++++++++++++-------
obsidian/tests/test_objectives.py | 10 ++++++++--
2 files changed, 28 insertions(+), 9 deletions(-)
diff --git a/obsidian/objectives/custom.py b/obsidian/objectives/custom.py
index 71f29b9..3792fb3 100644
--- a/obsidian/objectives/custom.py
+++ b/obsidian/objectives/custom.py
@@ -33,7 +33,8 @@ class Product_Objective(Objective):
Args:
ind (tuple[int]): The indices of objectives to be used in a product
- weights (tuple[float]): The weights corresponding to indexed objectives
+ weights (list[float | int], optional): The weights corresponding to
+ indexed objectives. Defaults to ``(1,)*len(ind)``.
const (float | int): A constant value that can be added to the product
new_dim (bool, optional): Whether to create a new objective dimension
from this product. Default is ``False``. Setting to ``True`` will
@@ -42,18 +43,25 @@ class Product_Objective(Objective):
"""
def __init__(self,
ind: tuple[int],
- weights: tuple[float],
+ weights: list[float | int] | None = None,
const: float | int = 0,
new_dim: bool = True) -> None:
super().__init__(new_dim)
# Always MOO if dim is being added, always SOO otherwise
+ if weights is None:
+ weights = [1]*len(ind)
if len(weights) != len(ind):
raise ValueError('The length of weights and indices must be the same')
self.register_buffer('ind', torch.tensor(ind, dtype=torch.int))
self.register_buffer('weights', torch.tensor(weights, dtype=TORCH_DTYPE))
self.register_buffer('const', torch.tensor(const, dtype=TORCH_DTYPE))
self.register_buffer('new_dim', torch.tensor(new_dim, dtype=torch.bool))
-
+
+ def __repr__(self):
+ """String representation of object"""
+ return f'{self.__class__.__name__} (ind={self.ind.tolist()}, \
+ weights={self.weights.tolist()}, const={self.const})'
+
def forward(self,
samples: Tensor,
X: Tensor | None = None) -> Tensor:
@@ -75,10 +83,10 @@ class Divide_Objective(Objective):
Args:
ind_num (int): The index of the objective to be used in the numerator
- w_num (float | int, optional): The weights corresponding to numerator
+ w_num (float | int, optional): The weight corresponding to numerator
objective. Defaults to ``1``.
ind_denom (int): The index of the objective to be used in the denominator
- w_denom (float | int, optional): The weights corresponding to denominator
+ w_denom (float | int, optional): The weight corresponding to denominator
objective. Defaults to ``1``.
const (float | int): A constant value that can be added to the quotient
new_dim (bool, optional): Whether to create a new objective dimension
@@ -103,6 +111,11 @@ def __init__(self,
self.register_buffer('const', torch.tensor(const, dtype=TORCH_DTYPE))
self.register_buffer('new_dim', torch.tensor(new_dim, dtype=torch.bool))
+ def __repr__(self):
+ """String representation of object"""
+ return f'{self.__class__.__name__} (num={self.w_num} * {self.ind_num}, \
+ denom={self.w_denom} * {self.ind_denom}, const={self.const})'
+
def forward(self,
samples: Tensor,
X: Tensor | None = None) -> Tensor:
@@ -129,7 +142,7 @@ class Feature_Objective(Objective):
X_space (ParamSpace): The parameter space.
ind (tuple[int]): The indices of the parameters in the real space
to be used as features.
- coeff (tuple[float | int], optional): The coefficients corresponding to each feature.
+ coeff (list[float | int], optional): The coefficients corresponding to each feature.
Defaults to ``[0]``.
Raises:
@@ -140,7 +153,7 @@ class Feature_Objective(Objective):
def __init__(self,
X_space: ParamSpace,
ind: tuple[int],
- coeff: tuple[float | int]) -> None:
+ coeff: list[float | int]) -> None:
super().__init__(mo=True)
diff --git a/obsidian/tests/test_objectives.py b/obsidian/tests/test_objectives.py
index 2d3a07f..dce96b8 100644
--- a/obsidian/tests/test_objectives.py
+++ b/obsidian/tests/test_objectives.py
@@ -10,7 +10,9 @@
Objective_Sequence,
Utopian_Distance,
Index_Objective,
- Bounded_Target
+ Bounded_Target,
+ Product_Objective,
+ Divide_Objective
)
from obsidian.tests.utils import DEFAULT_MOO_PATH, equal_state_dicts
@@ -32,9 +34,13 @@
test_objs = [Identity_Objective(mo=len(target) > 1),
Scalar_WeightedNorm(weights=[1, 1]),
- Feature_Objective(X_space, ind=(0,), coeff=(1,)),
+ Feature_Objective(X_space, ind=(0,), coeff=[1]),
Objective_Sequence([Utopian_Distance([1], target[0]), Index_Objective()]),
Bounded_Target(bounds=[(0, 1)]*len(target), targets=target),
+ Objective_Sequence([Product_Objective(ind=(0,), weights=None, const=1, new_dim=True),
+ Divide_Objective(0, 1, new_dim=True),
+ Divide_Objective(0, 1, new_dim=False)]),
+ Product_Objective(ind=(0,), weights=[1], const=1, new_dim=False),
None]
utopian = Utopian_Distance(utopian=[10, 10], targets=target)
From 230b169701cce4006f96e7a25617ac84a43c75f0 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 12:13:17 -0600
Subject: [PATCH 068/136] Hoverdata for MDS plot
---
CHANGELOG.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index cd74292..22f78e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,10 +5,12 @@
- Campaign X_best method
- Optimizer X_best_f attribute(s)
- Sequence of colors "color_list" to branding
+- Informative hoverdata for MDS plot
### Modified
- Switched all usages of X_ref = X_space.mean() to optimizer.X_best_f
- Refactored mpl "visualize_inputs" as plotly "visualize_inputs" for better interactivity
+- Text formatting for some plotly hoverdata
## [0.8.3]
### Added
From 30203e05e270c401217834246717f004d51c6ee9 Mon Sep 17 00:00:00 2001
From: Kevin Stone
Date: Thu, 22 Aug 2024 12:59:32 -0600
Subject: [PATCH 069/136] Fixed browser tab header for ReadMe in GH docs
---
docs/index.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/docs/index.rst b/docs/index.rst
index e04334b..f91c6e5 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,3 +1,13 @@
+.. raw:: html
+
+
-
+
+