Testing & Validation Objectives

[pearsonca]

Per the organizing call, looks like I'm wearing the Testing & Validation hat.

Let's figure out what that means in this thread.

My rough ideas at this point:

• re Testing:
  • my focus will be on integration testing, vice unit testing. Briefly, unit testing concerns piece-part testing - its function/class scale interface / contract confirmation, and should be part of an automated suite. Integration testing concerns the user-scale capabilities, and is typically less amenable to automation - e.g. can a new user on mac os follow the documentation and get an example SIR-inference + projection pipeline working on their cluster?
  • I am going to need a lot of perspective on what we want the user-level capabilities to be. I think "given a model + data, infer parameters, and project scenarios" is one example. But I imagine there are some alternative overall pipelines (e.g. model selection => projection), as well as some sub-process end points (e.g. given config, make me a model implementation to use elsewhere)
  • I am also thinking of the testing here in terms of things like "is it easy to ...", "how long does it take to ...", etc checks
• re Validation:
  • I generally think of "Validation" as confidence-building products. So e.g. checking that new model versions continue to provide the same quantitative answers as historical versions (or that we understand why historical answers were wrong). Also comparing against alternative model building tools - e.g. our SIR + data gives the same projections as, say, the odin-dust SIR + data projections.
  • Other confidence building products are making externally checkable intermediate products - e.g. the equation emitter or a model diagram or a model code object that could tossed into an independent python / R / etc stepper
• I think both of these will need a building example model world. I'm thinking starting from basic SIR, iteratively increasingly complexity for testing / demonstrating model features and pipeline capabilities we want to ensure. Clearly, there's some age and location strata increments, typical interventions, and so on. There does seem to be a bit of these in the existing examples, but we'll want to reframe (and extend) those a bit for T&V purposes (though they should still be useful as a examples as well).

Asking y'all: what are the big picture capabilities (both current and in development) we want to be integration testing that flepimop can provide? What the comparisons / outputs that we want flepimop to produce that would build our confidence in the tool (or that we think would convince other people that its useful / correct / doing what it says on the label)? What are the model world features we need to build up for our demonstration cases?

[twallema]

Thanks for starting this discussion @pearsonca. I would ideally want a piece of software to build an epidemiological model that can do the following things:

• Specify a set of disease compartments and the flows of individuals between them.
• "Stratify" every disease compartment in strata such as "age", "spatial patch", "race", "sex", "immune status" etc.
• Have a way to build a force of infection term that allows me to specify two network structures in my compartmental model: one for social contacts, one for mobility between spatial patches. This is accomplished using square, origin-destination social contact/mobility matrices. I want population groups to be able to make contacts differently based on their presence in their residency versus visited patch (f.i. to implement that commuters make "work" contacts in their visited patches and their "home" contacts on the home patch).
• Have a way to broadcast social/mobility network structure over other "strata". I.e. If a model has two dimensions: "age" and "location", with the following coordinates: "age": old & young, "location": A, B, C. I want to have a way to broadcast my social contact network over the 3 locations. For this 2D case, this could be accomplished with a vector broadcast = [1, 2, 0.5], which multiplies the amount of contacts people in make in 'A', 'B' and 'C'. If you add in a third dimension "race", this broadcasting vector becomes either a 2D matrix, or 2 vectors, specifying how social contact varies for (space, race) combinations.
• Have a way to vary model parameters (including network structure) through the time course of a simulation. F.i. I'd like to use a different social contact matrix varying during holidays versus non-holidays.
• Model outcomes which derive from model states and may have reporting delay.
• Have simulation output in an appropriate format to handle all these strata (xarray.DataSet/pd.DataFrame with pd.Multiindex).
• Bring all of the above to data & calibrate.

I think a big step forward would be to find a way to construct a force-of-infection term when the model has $n$ strata, and most importantly and most difficult, to compute it without looping over all the model's compartments (we will have to figure out a good tensor calculation and Einstein summation). Aside from the force-of-infection term, most transitions in epi models are holding states. In my experience it's going from 2 --> 3 dimensions on the FOI where the matrix algebra becomes massively abstract but I would really love to generalize that.

Really like the idea of integration testing and validations. Perhaps a chain SIR (no strata) --> SIR (strata: space) --> SIR (strata: age + space) --> SIR (strata: age + space + other) would provide a good way to build confidence in the end result. Perhaps such step-by-step approach can be leveraged to compute a generic n-d FOI term?

[pearsonca]

Out of conversation with @twallema : mentioned that for SIR incremental buildup, would want to show that can do model as SIR+V or as SIR, with a vaccinated vs not strata and get identical outcomes. Likewise asymptomatic vs symptomatic infectious compartments.

[pearsonca]

Capturing some discussion with @saraloo @jcblemai : we want the fitting / interference stage to:

• flexibly generate the inference equations (and therefore potentially adjust the model outputs, or at the very least deal with filtering / transforming them to the appropriate measures) based on the data provided.
• that would mean interpretting NAs (e.g. days with non-reported incidence would be interpretted to mean accumulate model incidence from that day until the next actually reported day)
• ... not just for time series, but for compartmental strata (ages, places, infection variants, ...)
• the NA interpretation should also provide a concise, intuitive syntax that allow people to differentiate between incidence accumulators (e.g. we should roll reported cases) vs not-measured-snapshots (e.g. we shouldn't roll a prevalence measure for comparison to a serosurvey)