Uncertainty breakdown

[Tomoya-Iizawa]

Hi,

Is there some option to obtain a table showing the uncertainty breakdown on the parameter of interest?

[alexander-held]

There are different ways to perform such breakdowns, so it depends on what you are looking for exactly. There is fit.ranking which can be used together with visualize.ranking to produce a figure with one way to quantify contributions per parameter. This is technically not a table, but the information could also be shown in table format using the data in the RankingResults object.

https://arxiv.org/abs/2307.04007 proposes another method with useful properties, that one is not implemented within cabinetry so far.

[Tomoya-Iizawa]

Thank you.
Yes, ranking plot is one of the way, but what I want is, for example, the contribution from ttbar modeling is XX%, Z modeling is XX%, then total modeling uncertainty is XX%, total CP systematics uncertainty is XX%, statistical uncertainty is XX%, as in p.9 (marked as p.7 in the slide) of this slides. Is there any way as this in the cabinetry?

[alexander-held]

Yes, to evaluate this what you need is one fit per category in which you hold the parameters within a given category constant at their best-fit value. The example below shows how to do that:

import cabinetry

cabinetry.set_logging()

spec = cabinetry.workspace.load("workspaces/example_workspace.json")
model, data = cabinetry.model_utils.model_and_data(spec)

nominal_fit_result = cabinetry.fit.fit(model, data)


def contribution_from_category(model, data, parameter_names, nominal_fit_results):
    fixed = model.config.suggested_fixed()
    inits = model.config.suggested_init()
    for par_name in parameter_names:
        idx = cabinetry.model_utils._parameter_index(par_name, model.config.par_names)
        fixed[idx] = True  # set everything in category to constant
        inits[idx] = nominal_fit_results.bestfit[idx]  # set to best-fit value
    category_fit_result = cabinetry.fit.fit(
        model, data, init_pars=inits, fix_pars=fixed
    )

    # quadrature subtraction to get relative contribution of category
    poi_unc_nom = nominal_fit_results.uncertainty[model.config.poi_index]
    poi_unc_cat = category_fit_result.uncertainty[model.config.poi_index]

    return (poi_unc_nom**2 - poi_unc_cat**2) ** 0.5 / poi_unc_nom


# define the names of everything falling into a category
PARAMETERS_IN_CATEGORY = ["Modeling", "WeightBasedModeling"]

cat_contrib = contribution_from_category(
    model, data, PARAMETERS_IN_CATEGORY, nominal_fit_result
)
print(
    f"category with parameters {PARAMETERS_IN_CATEGORY} contributes "
    f"{cat_contrib:.1%} to total uncertainty"
)

When defining categories of parameters like that, there is no guarantee that the individual contributions will sum to 100% due to correlations between parameters in different groups.