Skip to content

Commit

Permalink
update summary (#99)
Browse files Browse the repository at this point in the history
  • Loading branch information
aloctavodia authored Feb 4, 2025
1 parent ba9d1ae commit 4c01174
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 9 deletions.
6 changes: 3 additions & 3 deletions Chapters/Bayesian_workflow_example.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ This is a simple model, a small dataset. So we get a fast sampling.
We did not get any warning messages after sampling, and no divergences! Still, we want to quickly check that we have high ESS and low $\hat R$. Otherwise, this will be an indication of sampling issues and to solve them we may need to run more samples or most likely go back and change something in the model specification. We can see, with relief, that the effective sample size is very high and $\hat R$ is low enough.
```{python}
az.summary(idata, kind="diagnostics")
azs.summary(idata, kind="diagnostics")
```
Doing a visual check, just to be extra careful is also a good idea.
Expand Down Expand Up @@ -411,10 +411,10 @@ Nice, LOO agrees with the posterior predictive checks. We can see that the trunc
## Summarize results
We can summarize the results in a table. We can use the `az.summary` function to get the mean and the 94% HDI for the parameters.
We can summarize the results in a table. We can use the `azs.summary` function to get the mean and the 94% HDI for the parameters.
```{python}
az.summary(idata_t1, kind="stats")
azs.summary(idata_t1, kind="stats")
```
We can see that we got a positive slope indicating a positive correlation between the masses of predators and prey. In other words, as the mass of predators increases. This is in line with our discussion before the analysis. The magnitude of the slope is $\approx 0.3$, on the log-scale. On the original scale, this is $\approx 0.85$ on the original scale.
Expand Down
10 changes: 5 additions & 5 deletions Chapters/MCMC_diagnostics.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ $\hat R$ is a numerical diagnostic that answers the question [@vehtari_2021]. Di

The version implemented in ArviZ does several things under the hood, but the central idea is that it compares the variance between chains with the variance within each chain. Ideally, we should get $\hat R = 1$, in practice $\hat R \lessapprox 1.01$ are considered safe and in the first modeling phases, even higher values like $\hat R \approx 1.1$ may be fine.

Using ArviZ we can get the $\hat R$ with `azs.rhat(⋅)`, `az.summary(⋅)` and `az.plot_forest(⋅, r_hat=True)`
Using ArviZ we can get the $\hat R$ with `azs.rhat(⋅)`, `azs.summary(⋅)` and `az.plot_forest(⋅, r_hat=True)`

```{python}
azs.rhat(sample)
Expand Down Expand Up @@ -265,7 +265,7 @@ ax.plot(1, 0, ">k", transform=ax.get_yaxis_transform(), clip_on=False)
ax.plot(48, 1, "^k", transform=ax.get_xaxis_transform(), clip_on=False)
```

With ArviZ we can get `azs.ess(⋅)`, `az.summary(⋅)` and `az.plot_forest(⋅, ess=True)`
With ArviZ we can get `azs.ess(⋅)`, `azs.summary(⋅)` and `az.plot_forest(⋅, ess=True)`

```{python}
azs.ess(sample)
Expand All @@ -277,10 +277,10 @@ One way to use the ESS is as a minimum requirement for trustworthy MCMC samples.
The ESS can also be used as a metric of the efficiency of MCMC sampling methods. For instance, we may want to measure the ESS per sample (ESS/n), a sampler that generates a ESS/n closer to 1 is more efficient than a sampler that generates values closer to 0. Other common metrics are the ESS per second, and the ESS per likelihood evaluation.
:::

We see that `az.summary(⋅)` returns two ESS values, `ess_bulk` and `ess_tail`. This is because different regions of the parameter space may have different ESS values since not all regions are sampled with the same efficiency. Intuitively, one may think that when sampling a distribution like a Gaussian it is easier to obtain better sample quality around the mean than around the tails, simply because we have more samples from that region. For some models, it could be the other way around, but the take-home message remains, not all regions are necessarily sampled with the same efficiency
We see that `azs.summary(⋅)` returns two ESS values, `ess_bulk` and `ess_tail`. This is because different regions of the parameter space may have different ESS values since not all regions are sampled with the same efficiency. Intuitively, one may think that when sampling a distribution like a Gaussian it is easier to obtain better sample quality around the mean than around the tails, simply because we have more samples from that region. For some models, it could be the other way around, but the take-home message remains, not all regions are necessarily sampled with the same efficiency

```{python}
az.summary(sample, kind="diagnostics")
azs.summary(sample, kind="diagnostics")
```

If we are going to use the MCMC samples to calculate central values such as means or medians then we have to make sure that the `ess_bulk` is sufficiently large, however, if we want to calculate intervals such as an HDI 94% we have to make sure that `ess_tail` be appropriate.
Expand Down Expand Up @@ -311,7 +311,7 @@ azp.plot_ess_evolution(sample,
An advantage of the ESS is that it is scale-free, it does not matter if one parameter varies between 0.1 and 0.2 and another between -2000 and 0, an ESS of 400 has the same meaning for both parameters. In models with many parameters, we can quickly identify which parameters are most problematic. However, when reporting results it is not very informative to know whether the ESS was 1372 or 1501. Instead, we would like to know the order of the errors we are making when approximating the posterior. This information is given by the **Monte Carlo standard error** (MCSE). Like the ESS, the MCSE takes into account the autocorrelation of the samples. This error should be below the desired precision in our results. That is, if for a parameter the MCSE is 0.1, it does not make sense to report that the mean of that parameter is 3.15. Since the correct value could easily be between 3.4 and 2.8.


With ArviZ we can get the MCSE with `azs.mcse(⋅)` or `az.summary(⋅)`.
With ArviZ we can get the MCSE with `azs.mcse(⋅)` or `azs.summary(⋅)`.

```{python}
azs.mcse(sample)
Expand Down
2 changes: 1 addition & 1 deletion Chapters/Sensitivity_checks.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ Overall, the power-scaling sensitivity analysis on the adjusted model shows that
### Bacteria treatment


## Interpreting sensitivity diagnostics (Summary)
## Interpreting sensitivity diagnostics: Summary

As with other diagnostics we need to interpret the results of the sensitivity analysis in the context of the model, the data and the problem we are trying to solve. Context and modelling purpose should always be part of an analysis.

Expand Down

0 comments on commit 4c01174

Please sign in to comment.