Model Interface to include Threshold

[JamesYang007]

Adaptive T-Test is giving a new example of a test where we might need the threshold information in the sim_batch function. While I can technically implement this using the current setup, it would be nice to be able to support "return the first time the T-statistic is above the threshold". Of course, we can just return negative of the maximum of the T-statistic, but I would also like to just plot the histogram of the first t-statistic that rejects for fun. Right now, I can't use this model because it has to return the -maximum.

Solution: pass the threshold into the sim_batch but keep the same convention of returning some kind of a test statistic.

[tbenthompson]

Let's talk about this!

There are a few things here:

1. Do you want to run gather this info at the same time that you're calibrating or validating? Or do you want to just run the same simulations separately and get the info out? If it's the latter, you can add as many different methods to your model as you want and have those call some shared simulation method. Then, to get the interim number of the first rejection, you can call the Model method directly instead of calling it through ip.validate.
2. If you want "extra" information in tandem with a val/cal run.... I think this is what the detailed parameter is intended for in the Model interface. Basically it would let the caller flip a flag to ask for extra information that can be provided by the Model on a per-tile per-sim level. It's only stubbed in right now so it doesn't actually do anything, but it shouldn't be a huge job to implement. Would that achieve your goals?
3. But, there's actually a similar situation that this raises... What if the threshold changes the computational path? Like suppose that we have 100 interims. Then, if we don't know the threshold, we need to keep computing all the way until the end! But if we do know the threshold, we could stop at an early interim. This might be very important for some models.

So I'm think that we actually do need a second method to handle this situation #3 above:

• stat_batch - (renaming of current sim_batch), returns test stats
• rej_batch - returns rejections given a threshold. Default implementation in the Model superclass just calls stat_batch internally. But the Model writer can override if they have a situation where they want to validate and would have computational benefits from knowing the threshold. This would only work for validation since calibration needs the test stats.
• we might also have calibrate_batch that handles the calibration directly too. Again, this would have a default implementation but I could imagine there are situations where the simulation logic could be simplified by passing the calibration index into the Model method. I'm not sure!

[JamesYang007]

Actually, you make a good point that we can make the model classes as complex as we want. I think that definitely suffices for my application. Yes 3. is also a good point, but also, the added benefit of looking at all 100 interims and taking like the max of the T-statistics is also nice in that it becomes lambda-agnostic: you don't have to recompute the test statistic if your threshold changes. I don't know if that optimization is ever assumed/needed in the current framework, but something to keep in mind.

I ended up not needing this for my application - I decided to plot the max of T-statistics. We can hold off on this until we have an example that actually needs this change.