Skip to content

Commit

Permalink
vignette updates
Browse files Browse the repository at this point in the history
  • Loading branch information
florianhartig committed Sep 8, 2022
1 parent 955bcf2 commit 1204468
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 28 deletions.
27 changes: 0 additions & 27 deletions DHARMa/vignettes/DHARMa.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -759,33 +759,6 @@ plot(res3)

**Conclusions:** if you see overdispersion or a pattern after grouping, it highlights a model error that is structured by group. As the pattern usually highlights a model misfit, rather than a dispersion problem akin to what happens in an overdispersed binomial (which has major impacts on p-values and CIs), I view this binomial grouping pattern as less critical. Likely, most conclusions will not change if you ignore the problem. Nevertheless, you should try to understand the reason for it. When the group is spatial, it could be the sign of residual spatial autocorrelation which could be addressed by a spatial RE or a spatial model. When grouped by a continuous variable, it could be the sign of a nonlinear effect.

## Bayesian vs. MLE quantile residuals

A common question is if there are differences between Bayesian and MLE quantile residuals.

First of all, note that MLE and Bayesian quantile residuals are not identical. The main difference is in how the simulation of the data under the fitted model are performed:

* For models fitted by MLE, simulations in DHARMa are done with H0 = the true model is the fitted model with the MLE (point estimate)

* For models fitted with Bayes, simulations are practically always performed by additionally drawing from the posterior parameter uncertainty (as a point estimate is not available).

From this we can directly conclude that Bayesian and MLE quantile residuals are asymptotically identical (and via the usual arguments uniformly distributed).

The more interesting question is what happens in the low data situation. Let's imagine that we start with a situation of infinite data. In this case, we have a "sharp" posterior that can be viewed as identical to the MLE.

If we reduce the number of data, there are two things happening

1. The posterior gets wider, with the likelihood componet being normally distributed, at least initially

2. The influence of the prior increases, the faster the stronger the prior is.

Thus, if we reduce the data, for weak / uninformative priors, we will simulate data while sampling parameters from a normal distribution around the MLE, while for strong priors, we will effectively sample data while drawing parameters of the model from the prior.

In particular in the latter case (prior dominates, which can be checked via prior sensitivity analysis), you may see residual patterns that are caused by the prior, even though the model structure is correct. In some sense, you could say that the residuals check if the combination of prior + structure is compatible with the data. It's a philosophical debate how to react on such a deviation, as the prior is not really negotiable in a Bayesian analysis.

Of course, also the MLE distribution might get problems in low data situations, but I would argue that MLE is usually only used anyway if the MLE is reasonably sharp. In practice, I have seldom experienced problems with MLE estimates. It's a bit different in the Bayesian case, where it is possible and often done to fit very complex models with limited data. In this case, many of the general issues in defining null distributions for Bayesian p-values (as, e.g., reviewed in
[Conn et al., 2018](https://doi.org/10.1002/ecm.1314)) apply.

# Supported packages and frameworks

## lm and glm
Expand Down
2 changes: 1 addition & 1 deletion DHARMa/vignettes/DHARMaForBayesians.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ Thus, if we reduce the data, for weak / uninformative priors, we will simulate d
In particular in the latter case (prior dominates, which can be checked via prior sensitivity analysis), you may see residual patterns that are caused by the prior, even though the model structure is correct. In some sense, you could say that the residuals check if the combination of prior + structure is compatible with the data. It's a philosophical debate how to react on such a deviation, as the prior is not really negotiable in a Bayesian analysis.

Of course, also the MLE distribution might get problems in low data situations, but I would argue that MLE is usually only used anyway if the MLE is reasonably sharp. In practice, I have self experienced problems with MLE estimates. It's a bit different in the Bayesian case, where it is possible and often done to fit very complex models with limited data. In this case, many of the general issues in defining null distributions for Bayesian p-values (as, e.g., reviewed in
[Conn et al., 2018](https://doi.org/10.1002/ecm.1314)) apply.
[Conn et al., 2018](https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1314)) apply.

I would add though that while I find it important that users are aware of those differences, I have found that in practice these issues are small, and usually overruled by the much stronger effects of model error.

Expand Down

0 comments on commit 1204468

Please sign in to comment.