About signal statistical uncertainty

[Tomoya-Iizawa]

Hi,
Can I ask how the signal statistical error is implemented in the fit?
I think it can be implemented if I do NOT add "DisableStaterror: True", and it appears as "staterror_Region[bin]" in the results (e.g. ranking, correlation), is it right?
In that case, it is convoluted with background statistical error?

[alexander-held]

By default cabinetry creates a single staterror parameter per bin, which includes contributions from all the samples in the bin (unless you use DisableStaterror), with the name like you mention. This does not distinguish between signal and background in any way and acts on both.

If you want to split out contributions, then staterror is probably not a good modifier type. See also scikit-hep/pyhf#1830 for that. The easiest way to handle this is to change the modifier type for the relevant samples to shapesys and also update the modifier name (to not clash with the staterror names). A shapesys modifier looks quite similar to staterror, but uses a Poisson constraint term instead of a Gaussian term (in practice this usually does not make a huge difference).

There is no automated way to do this in cabinetry currently, for now this would have to be done as a manual workspace post-processing step. It is certainly something we could consider, would require a new config option to control the behavior and possibly a way to correlate MC statistical uncertainties between samples manually to create groups and have flexibility.

[Tomoya-Iizawa]

I see, thank you. hm, now I am not sure how manually treat workspace post-processing step, but I will try. At least, maybe if I disable staterror for signal, I can compare the impact between with and without signal stat error.

[alexander-held]

Here is an example for how to implement this:

import cabinetry

spec = cabinetry.workspace.load("workspaces/example_workspace.json")

# decorrelate MC statistical uncertainty of "Signal" samples
for channel in spec["channels"]:
    for sample in channel["samples"]:
        if sample["name"] != "Signal":  # pick the sample you need
            continue

        for modifier in sample["modifiers"]:
            if modifier["type"] == "staterror":
                modifier["type"] = "shapesys"
                modifier["name"] = f"{modifier['name'].replace('staterror', 'shapesys')}_{sample['name']}"

cabinetry.workspace.save(spec, "ws_modified.json")  # optionally save again

model, data = cabinetry.model_utils.model_and_data(spec)  # continue as usual
_ = cabinetry.fit.fit(model, data)

which uses the example workspace. This identifies the sample where you want to update the treatment and then changes the modifier type and parameter name.

It is also often not unreasonable to just exclude signal MC statistical uncertainties, since they tend to be small given that analyses look at regions of phase space where signal acceptance is high anyway.

[Tomoya-Iizawa]

Thank you for showing an example.
I have a question, that we are using custom modified signal scaling as discussed in #382.
In our case, we are using g2 as POI, and BSM signal sample is proportional to g2^2, and interference signal sample is proportional to g2, and final signal yields should be the some of the two components, as attached file
custom.txt
. In this case, how the signal stat uncertainty is considered (g2=1 or else?), and how it is combined with background stat uncertainties?

[alexander-held]

The MC statistical uncertainty contribution per sample is relative to the sample yield, so it does not matter by what you scale the sample. I cannot think of any reason for why scaling with custom expressions would conceptually change something here. The same thing holds if you scale a signal linearly by some parameter of interest: its statistical uncertainty is relative to its yield and that relative contribution is unaffected by scaling.

There is nothing special about how contributions from different samples enter, it is the same situation as described above. By default, all samples contribute (again, including their scaling by whatever parameters act on them). You can still exclude samples with DisableStaterror or change the modifiers to create new decorrelated MC statistical uncertainty terms with shapesys parameters.

[Tomoya-Iizawa]

Thank you! The example you showed seems to be the one we want. And thank you for the explanation.

[cspollard]

Hi. Would it be possible to separate out the MC stat uncertainties by process (and give them their own "gammas")? I think this would solve our problem.