Lei example

[tbenthompson]

High-level decomposition of what needs to happen:

• the general structure of the problem.
• computing the probability that a given arm is the best arm. This can be done with sampling as discussed on zoom and in slack. But, also might be nice to build a lookup/interpolation table.
• computing the expected power. For this, we probably will need to build a lookup/interpolation table

Piece-wise:

• we should run with just a single treatment arm and a control arm and small n and no lookup/interp.
• then, move to a lookup table and extend to slightly larger problems.
• then, move to an interpolation table and extend to the full problem.

[tbenthompson]

Mike's notes:

“A randomized, double-blind, placebo-controlled two-stage adaptive design intended to identify an optimal treatment regimen from three possible regimens (for example, different dosages or different combinations of agents) and to assess the efficacy of that regimen with respect to a primary binary response endpoint measured at month 6.
In the first stage, one of four experimental regimens will be selected, or the trial will stop for futility. In this stage, a minimum of 200 and a maximum of 400 will be randomized 1:1:1:1 to one of the three experimental arms or one placebo arm. Interim analyses will be conducted after 200, 300 and 400 subjects have been enrolled to select the best experimental regimen and to potentially stop the trial for futility. If an experimental regimen is dropped for futility at an interim analysis, the next 100 subjects to be randomized will be allocated equally among the remaining arms in the study. At each of these three analysis time points (N = 200, 300, 400), the probabilities of being the best regimen (PrBest) and predictive probability of success (PPS) are calculated for each experimental regimen using a Bayesian approach, and the trial will either stop for futility, continue to the next interim analysis, or proceed to Stage 2 depending on the results of these PrBest and PPS calculations.
In Stage 2, a minimum of 200 and a maximum of 400 additional subjects will be randomized 1:1 to the chosen regimen or placebo. The two groups (pooling both Stage 1 and Stage 2 subjects) will be compared for efficacy and futility assessment at an interim analysis after 200 subjects have been enrolled in Stage 2, and for efficacy at a final analysis after 400 subjects have been enrolled in Stage 2 and fully followed-up for response. The study may be stopped for futility or efficacy based on PPS at the interim analysis. If the study continues to the final analysis, the posterior distribution of the difference in response rates between placebo and the chosen experimental arm will be evaluated against a pre-specified decision criterion.
· Scenario 1: interim analyses are based on available data on the primary endpoint (measured at month 6)
· Scenario 2: interim analyses are based on available data on a secondary endpoint (measured at month 3) ”
For conditional power:
Conditional power_1t is defined as the posterior probability of success (at time t), given that treatment_1 is selected to continue to phase III against treatment_0 (control).
There is some ambiguity here about what “the power of the trial” is. Here, I’m taking the interpretation that we’re doing this in terms of the FINAL outcome of the the Phase III, rather than the interim of the phase III. I’m also assuming that the final analysis will NOT be FULLY Bayesian, so that it only depends on y_1, y_0
If power_1t < .2, drop the arm
If power_it > .8 for any arm, stop the phase II and go with the arm with highest power_it for phase III
This should be accounting for the Bayes posterior at time t (which is based on n_successes_i for i = 1, 2, 3,4)
But, we can split this computational problem into 2 steps:
(1) Pre-computation: Given p_1 (selected arm), y_1, p_0, y_0 , that’s all we need to simulate forward the phase-III.
This is now completely sufficient to completely determine the conditional power with these inputs
This is still a 4-D problem, so it’s nontrivial: we will later need to interpolate over p_1, p_0 (and I’m hoping that we can do this with tools Ben was experimenting with)
Said another way, we need to integrate over p_0 and p_1 using weights from the INLA posterior, and can optionally interpolate over y_0, y_1 to save compute.
(2) ^That was a “conditional” -conditional power, which had fixed p_0 and p_1. To do the Bayesian conditional power, we need to integrate those quantities over the posterior using a combination of interpolation + INLA over the posterior p_0, p_1. ( the other dims p_2 p3 don’t matter here )

[tbenthompson]

Finishing this up:

1. play with the existing results enough to get my bearings
2. run a 3D lei + adagrid job
3. run a 4D lei + tuning + adagrid job.
4. figure out how to plot/portray the results effectively
5. write the paper section to flow around the new figures.

[tbenthompson]

The sticking point currently for 4D is figuring out how to handle the memory requirements of storing the whole 100+ million tiles. Potential solutions:

1. Modify the splitting rule. Most of the tiles are coming from cases where floor((K+1)*alpha_0) == 0. In that situation there are two solutions. One is to increase alpha_0 by splitting the tile and reducing the continuous simulation extension bound component. That is what I am doing now. Instead, we can increase K until the tuning is meaningful (K + 1 > 1/alpha_0). This might be cheaper overall in some situations. It would undoubtedly reduce memory requirements.
2. Upgrade to using the exponential holder bound. This might help a lot.
3. Improve the gridding code.
   • chunk up the grid like in the validation setting: find the most conservative critical value for each chunk separately and then take a minimum of the chunks at the end. This might work but might also be very inefficient if it results in a lot more tiles than necessary. This inefficiency is testable but would probably also be problem dependent!
   • write an out-of-core version of adagrid that temporarily stores most of the grid on disk and loads it up when needed, possibly storing only the grid points and re-generating tiles at load time. This is a long-term option since it seems like load/save times are short enough compared to the cost of simulation that this would be practical.
   • distributed compute. Just use a bunch of machines/GPUs so that we can store only part of the grid on each machine and have more memory available. This would be great (also for runtime!!) but will be a bit of work that I don't think we should tackle right at this instant.
   • The largest memory usage is the tile vertices. We can avoid storing tile vertices until they are necessary. A sparse vertices array could only stores vertices for those tiles that have been split. Or we can even skip the tile splitting and just treat every tile as unsplit&&. This would get us to 5-10x higher tile counts before needing to do anything involving chunking or out-of-core, bringing memory usage down to a level equivalent to the C++ code. Eventually we'll still need to do some chunking or out-of-core stuff!
4. Back off and just use the 3D results for now.

&&: I actually think this would result in no loss. The relevant distance for computing a holder-odi bound on tile is the furthest vertex from the grid point. When a square tile is split, the furthest vertex is just as far away as before the split.

[tbenthompson]

An important remaining task is to decide when to add more simulations:

• can we estimate the approximate simulation cost with Clopper-Pearson?
• how about the Poisson argument from the paper?
• when alpha is impossible (<= 2 / k + 1), we should check whether simulating more would be cheaper than refining the grid.
• should we fix the alpha-impossible cases before ever simulating?
  • if we simulate and none of the simulations reach the final analysis, can we infer anything useful about that tile?

[tbenthompson]

An important remaining task is to decide when to add more simulations

We're now using a bootstrapping approach. Lots more to explore here though.

[tbenthompson]

3D job is done. We should try to run a 4D job.