Skip to content

Commit

Permalink
Merge pull request #213 from JohnLCaron/refactors
Browse files Browse the repository at this point in the history 
Rename sampling package to estimate.
  • Loading branch information
@JohnLCaron
JohnLCaron authored Feb 21, 2025
2 parents 5b14227 + afb6b6b commit dbd9c40
Show file tree
Hide file tree
Showing 104 changed files with 296 additions and 298 deletions.
45 changes: 23 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ You can also read this document on [github.io](https://johnlcaron.github.io/rlau
* [Audit Types](#audit-types)
* [Card Level Comparison Audits (CLCA)](#card-level-comparison-audits-clca)
* [Polling Audits](#polling-audits)
* [Stratified Audits using OneAudit](#stratified-audits-using-oneaudit)
* [Stratified Audits using OneAudit (In Progress)](#stratified-audits-using-oneaudit-in-progress)
* [Measuring Samples Needed](#measuring-samples-needed)
* [Samples needed with no errors](#samples-needed-with-no-errors)
* [Samples needed when there are errors](#samples-needed-when-there-are-errors)
Expand All @@ -49,34 +49,34 @@ You can also read this document on [github.io](https://johnlcaron.github.io/rlau
<!-- TOC -->

# Reference Papers
````
P2Z Limiting Risk by Turning Manifest Phantoms into Evil Zombies. Banuelos and Stark. July 14, 2012
P2Z Limiting Risk by Turning Manifest Phantoms into Evil Zombies. Banuelos and Stark. July 14, 2012

RAIRE Risk-Limiting Audits for IRV Elections. Blom, Stucky, Teague 29 Oct 2019
https://arxiv.org/abs/1903.08804

SHANGRLA Sets of Half-Average Nulls Generate Risk-Limiting Audits: SHANGRLA. Stark, 24 Mar 2020
https://github.com/pbstark/SHANGRLA
RAIRE Risk-Limiting Audits for IRV Elections. Blom, Stucky, Teague 29 Oct 2019
https://arxiv.org/abs/1903.08804
MoreStyle More style, less work: card-style data decrease risk-limiting audit sample sizes Glazer, Spertus, Stark; 6 Dec 2020
SHANGRLA Sets of Half-Average Nulls Generate Risk-Limiting Audits: SHANGRLA. Stark, 24 Mar 2020
https://github.com/pbstark/SHANGRLA
ALPHA: Audit that Learns from Previously Hand-Audited Ballots. Stark, Jan 7, 2022
https://github.com/pbstark/alpha.
MoreStyle More style, less work: card-style data decrease risk-limiting audit sample sizes. Glazer, Spertus, Stark; 6 Dec 2020
BETTING Estimating means of bounded random variables by betting. Waudby-Smith and Ramdas, Aug 29, 2022
https://github.com/WannabeSmith/betting-paper-simulations
ALPHA: Audit that Learns from Previously Hand-Audited Ballots. Stark, Jan 7, 2022
https://github.com/pbstark/alpha.
COBRA: Comparison-Optimal Betting for Risk-limiting Audits. Jacob Spertus, 16 Mar 2023
https://github.com/spertus/comparison-RLA-betting/tree/main
BETTING Estimating means of bounded random variables by betting. Waudby-Smith and Ramdas, Aug 29, 2022
https://github.com/WannabeSmith/betting-paper-simulations
ONEAudit: Overstatement-Net-Equivalent Risk-Limiting Audit. Stark 6 Mar 2023.
https://github.com/pbstark/ONEAudit
COBRA: Comparison-Optimal Betting for Risk-limiting Audits. Jacob Spertus, 16 Mar 2023
https://github.com/spertus/comparison-RLA-betting/tree/main
STYLISH Stylish Risk-Limiting Audits in Practice. Glazer, Spertus, Stark 16 Sep 2023
https://github.com/pbstark/SHANGRLA
ONEAudit: Overstatement-Net-Equivalent Risk-Limiting Audit. Stark 6 Mar 2023.
https://github.com/pbstark/ONEAudit
VERIFIABLE Publicly Verifiable RLAs. Alexander Ek, Aresh Mirzaei, Alex Ozdemir, Olivier Pereira, Philip Stark, Vanessa Teague
STYLISH Stylish Risk-Limiting Audits in Practice. Glazer, Spertus, Stark 16 Sep 2023
https://github.com/pbstark/SHANGRLA
VERIFIABLE Publicly Verifiable RLAs. Alexander Ek, Aresh Mirzaei, Alex Ozdemir, Olivier Pereira, Philip Stark, Vanessa Teague
````

# SHANGRLA framework

Expand All @@ -99,7 +99,8 @@ in a risk-limiting audit with risk limit α:
| risk | we want to confirm or reject the null hypothesis with risk level α. |
| assorter | assigns a number between 0 and upper to each ballot, chosen to make assertions "half average". |
| assertion | the mean of assorter values is > 1/2: "half-average assertion" |
| estimator | estimates the true population mean from the sampled assorter values. |
| estimator | estimates the true population mean from the sampled assorter values. (AlphaMart) |
| bettingFn | decides how much to bet for each sample. (BettingMart) |
| riskTestingFn | is the statistical method to test if the assertion is true. |
| audit | iterative process of picking ballots and checking if all the assertions are true. |

Expand Down Expand Up @@ -255,7 +256,7 @@ A few representative plots showing the effect of d are at [meanDiff plots](https
See [ALPHA testing statistic](docs/AlphaMart.md) for more details and plots.


## Stratified Audits using OneAudit
## Stratified Audits using OneAudit (In Progress)

OneAudit is a CLCA audit that uses AlphaMart instead of BettingMart.

Expand Down
2 changes: 1 addition & 1 deletion core/src/main/kotlin/org/cryptobiotic/rlauxe/core/Assorter.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ data class AuditRoundResult(
val samplesNeeded: Int, // first sample when pvalue < riskLimit
val samplesUsed: Int, // sample count when testH0 terminates
val status: TestH0Status, // testH0 status
val errorRates: ErrorRates? = null, // measured error rates (clca only)
val errorRates: ClcaErrorRates? = null, // measured error rates (clca only)
) {
override fun toString() = "round=$roundIdx estSampleSize=$estSampleSize maxBallotsUsed=$maxBallotsUsed " +
" pvalue=$pvalue samplesNeeded=$samplesNeeded samplesUsed=$samplesUsed status=$status"
Expand Down
41 changes: 30 additions & 11 deletions ...obiotic/rlauxe/workflow/ClcaErrorRates.kt → ...ryptobiotic/rlauxe/core/ClcaErrorRates.kt
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,38 @@
package org.cryptobiotic.rlauxe.workflow
package org.cryptobiotic.rlauxe.core

import org.cryptobiotic.rlauxe.core.ClcaAssorterIF
import org.cryptobiotic.rlauxe.core.Cvr
import org.cryptobiotic.rlauxe.core.ErrorRates
import org.cryptobiotic.rlauxe.core.PrevSamplesWithRates
import org.cryptobiotic.rlauxe.util.df

data class ClcaErrorRates(val p2o: Double, val p1o: Double, val p1u: Double, val p2u: Double) {
init {
require(p2o in 0.0..1.0) {"p2o out of range $p2o"}
require(p1o in 0.0..1.0) {"p1o out of range $p1o"}
require(p1u in 0.0..1.0) {"p1u out of range $p1u"}
require(p2u in 0.0..1.0) {"p2u out of range $p2u"}
}
override fun toString(): String {
return "[${df(p2o)}, ${df(p1o)}, ${df(p1u)}, ${df(p2u)}]"
}
fun toList() = listOf(p2o, p1o, p1u, p2u)
fun areZero() = (p2o == 0.0 && p1o == 0.0 && p1u == 0.0 && p2u == 0.0)

companion object {
fun fromList(list: List<Double>): ClcaErrorRates {
require(list.size == 4) { "ErrorRates list must have 4 elements"}
return ClcaErrorRates(list[0], list[1], list[2], list[3])
}
}
}

//////////////////////////////////////////////////////////////////////////////////
// the idea is that the errorRates are proportional to fuzzPct
// Then p1 = fuzzPct * r1, p2 = fuzzPct * r2, p3 = fuzzPct * r3, p4 = fuzzPct * r4.
// margin doesnt matter (TODO show this)

object ClcaErrorRates {
object ClcaErrorTable {
val rrates = mutableMapOf<Int, List<Double>>() // errorRates / FuzzPct
val standard = ErrorRates(.01, 1.0e-4, 0.01, 1.0e-4)
val standard = ClcaErrorRates(.01, 1.0e-4, 0.01, 1.0e-4)

fun getErrorRates(ncandidates: Int, fuzzPct: Double?): ErrorRates {
fun getErrorRates(ncandidates: Int, fuzzPct: Double?): ClcaErrorRates {
if (fuzzPct == null) return standard

val useCand = when {
Expand All @@ -22,13 +41,13 @@ object ClcaErrorRates {
else -> ncandidates
}
val rr = rrates[useCand]!!.map { it * fuzzPct }
return ErrorRates(rr[0], rr[1], rr[2], rr[3])
return ClcaErrorRates(rr[0], rr[1], rr[2], rr[3])
}

fun calcErrorRates(contestId: Int,
cassorter: ClcaAssorterIF,
cvrPairs: List<Pair<Cvr, Cvr>>, // (mvr, cvr)
) : ErrorRates {
) : ClcaErrorRates {
require(cvrPairs.size > 0)
val samples = PrevSamplesWithRates(cassorter.noerror()) // accumulate error counts here
cvrPairs.filter { it.first.hasContest(contestId) }.forEach { samples.addSample(cassorter.bassort(it.first, it.second)) }
Expand All @@ -37,7 +56,7 @@ object ClcaErrorRates {
}

// given an error rate, what fuzz pct does it corresond to ?
fun calcFuzzPct(ncandidates: Int, errorRates: ErrorRates ) : List<Double> {
fun calcFuzzPct(ncandidates: Int, errorRates: ClcaErrorRates ) : List<Double> {
val useCand = when {
ncandidates < 2 -> 2
ncandidates > 10 -> 10
Expand Down
8 changes: 4 additions & 4 deletions core/src/main/kotlin/org/cryptobiotic/rlauxe/core/OptimalComparison.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ class AdaptiveComparison(
val withoutReplacement: Boolean = true,
val a: Double, // compareAssorter.noerror
val d: Int, // weight
errorRates: ErrorRates, // ? = null, // a priori estimate of the error rates
errorRates: ClcaErrorRates, // ? = null, // a priori estimate of the error rates
val eps: Double = .00001
): BettingFn {
val p2o: Double = if (errorRates == null) -1.0 else errorRates.p2o // apriori rate of 2-vote overstatements; set < 0 to remove consideration
Expand All @@ -87,7 +87,7 @@ class AdaptiveComparison(
val p2uest = if (p2u < 0.0 || lastj == 0) 0.0 else estimateRate(d, p2u, prevSamples.countP2u().toDouble() / lastj, lastj, eps)

val mui = populationMeanIfH0(Nc, withoutReplacement, prevSamples)
val kelly = OptimalLambda(a, ErrorRates(p2oest, p1oest, p1uest, p2uest), mui)
val kelly = OptimalLambda(a, ClcaErrorRates(p2oest, p1oest, p1uest, p2uest), mui)
return kelly.solve()
}

Expand All @@ -106,7 +106,7 @@ class AdaptiveComparison(
// We know the true rate of all errors
class OracleComparison(
val a: Double, // noerror
val errorRates: ErrorRates,
val errorRates: ClcaErrorRates,
): BettingFn {
val lam: Double
init {
Expand Down Expand Up @@ -135,7 +135,7 @@ class OracleComparison(
* p1u := #{xi = 3a/2}/N is the rate of 1-vote understatements.
* p2u := #{xi = 2a}/N is the rate of 2-vote understatements.
*/
class OptimalLambda(val a: Double, val errorRates: ErrorRates, val mui: Double = 0.5) {
class OptimalLambda(val a: Double, val errorRates: ClcaErrorRates, val mui: Double = 0.5) {
val p2o = errorRates.p2o
val p1o = errorRates.p1o
val p1u = errorRates.p1u
Expand Down
28 changes: 4 additions & 24 deletions core/src/main/kotlin/org/cryptobiotic/rlauxe/core/SampleTracker.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ interface SampleTracker {
fun sum(): Double // sum of samples so far
fun mean(): Double // average of samples so far
fun variance(): Double // variance of samples so far
fun errorRates(): ErrorRates // only for clca
fun errorRates(): ClcaErrorRates // only for clca
}

/**
Expand All @@ -30,7 +30,7 @@ class PrevSamples : SampleTracker {
override fun sum() = sum
override fun mean() = welford.mean
override fun variance() = welford.variance()
override fun errorRates() = ErrorRates(0.0, 0.0, 0.0, 0.0, )
override fun errorRates() = ClcaErrorRates(0.0, 0.0, 0.0, 0.0, )

fun addSample(sample : Double) {
last = sample
Expand Down Expand Up @@ -81,33 +81,13 @@ class PrevSamplesWithRates(val noerror: Double) : SampleTracker {
}

fun errorCounts() = listOf(countP0,countP2o,countP1o,countP1u,countP2u) // canonical order
override fun errorRates(): ErrorRates {
override fun errorRates(): ClcaErrorRates {
val p = errorCounts().map { it / numberOfSamples().toDouble() /* skip p0 */ }
return ErrorRates(p[1], p[2], p[3], p[4])
return ClcaErrorRates(p[1], p[2], p[3], p[4])
}
fun errorRatesList(): List<Double> {
val p = errorCounts().map { it / numberOfSamples().toDouble() /* skip p0 */ }
return listOf(p[1], p[2], p[3], p[4])
}
}

data class ErrorRates(val p2o: Double, val p1o: Double, val p1u: Double, val p2u: Double) {
init {
require(p2o in 0.0..1.0) {"p2o out of range $p2o"}
require(p1o in 0.0..1.0) {"p1o out of range $p1o"}
require(p1u in 0.0..1.0) {"p1u out of range $p1u"}
require(p2u in 0.0..1.0) {"p2u out of range $p2u"}
}
override fun toString(): String {
return "[${df(p2o)}, ${df(p1o)}, ${df(p1u)}, ${df(p2u)}]"
}
fun toList() = listOf(p2o, p1o, p1u, p2u)
fun areZero() = (p2o == 0.0 && p1o == 0.0 && p1u == 0.0 && p2u == 0.0)

companion object {
fun fromList(list: List<Double>): ErrorRates {
require(list.size == 4) { "ErrorRates list must have 4 elements"}
return ErrorRates(list[0], list[1], list[2], list[3])
}
}
}
5 changes: 3 additions & 2 deletions ...obiotic/rlauxe/sampling/ClcaSimulation.kt → ...obiotic/rlauxe/estimate/ClcaSimulation.kt
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.core.*
import org.cryptobiotic.rlauxe.util.df
import org.cryptobiotic.rlauxe.workflow.Sampler
import kotlin.math.max
import kotlin.random.Random

Expand All @@ -15,7 +16,7 @@ class ClcaSimulation(
rcvrs: List<Cvr>, // may have phantoms
val contest: ContestIF,
val cassorter: ClcaAssorterIF,
val errorRates: ErrorRates,
val errorRates: ClcaErrorRates,
): Sampler {
val Ncvrs = rcvrs.size
val maxSamples = rcvrs.count { it.hasContest(contest.id) }
Expand Down
2 changes: 1 addition & 1 deletion ...tic/rlauxe/sampling/ConsistentSampling.kt → ...tic/rlauxe/estimate/ConsistentSampling.kt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.core.*
import org.cryptobiotic.rlauxe.util.roundToInt
Expand Down
2 changes: 1 addition & 1 deletion ...otic/rlauxe/sampling/ContestSimulation.kt → ...otic/rlauxe/estimate/ContestSimulation.kt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.core.Contest
import org.cryptobiotic.rlauxe.core.ContestInfo
Expand Down
15 changes: 6 additions & 9 deletions ...tic/rlauxe/sampling/EstimateSampleSize.kt → ...tic/rlauxe/estimate/EstimateSampleSize.kt
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,12 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.concur.*
import org.cryptobiotic.rlauxe.core.*
import org.cryptobiotic.rlauxe.oneaudit.OneAuditComparisonAssorter
import org.cryptobiotic.rlauxe.oneaudit.OneAuditContestUnderAudit
import org.cryptobiotic.rlauxe.util.df
import org.cryptobiotic.rlauxe.util.margin2mean
import org.cryptobiotic.rlauxe.workflow.AuditConfig
import org.cryptobiotic.rlauxe.workflow.AuditType
import org.cryptobiotic.rlauxe.workflow.ClcaErrorRates
import org.cryptobiotic.rlauxe.workflow.ClcaStrategyType
import org.cryptobiotic.rlauxe.workflow.*
import kotlin.math.min
import kotlin.math.max

Expand Down Expand Up @@ -191,13 +188,13 @@ fun simulateSampleSizeClcaAssorter(
}
(clcaConfig.strategy == ClcaStrategyType.phantoms || clcaConfig.strategy == ClcaStrategyType.previous) -> {
val phantomRate = contest.phantomRate()
val errorRates = ErrorRates(0.0, phantomRate, 0.0, 0.0)
val errorRates = ClcaErrorRates(0.0, phantomRate, 0.0, 0.0)
if (debugErrorRates) println("phantoms simulate round $roundIdx using errorRates=$errorRates")
errorRates
}
(clcaConfig.simFuzzPct != null && clcaConfig.simFuzzPct != 0.0) -> {
if (debugErrorRates) println("simFuzzPct simulate round $roundIdx using simFuzzPct=${clcaConfig.simFuzzPct} errorRate=${ClcaErrorRates.getErrorRates(contest.ncandidates, clcaConfig.simFuzzPct)}")
ClcaErrorRates.getErrorRates(contest.ncandidates, clcaConfig.simFuzzPct)
if (debugErrorRates) println("simFuzzPct simulate round $roundIdx using simFuzzPct=${clcaConfig.simFuzzPct} errorRate=${ClcaErrorTable.getErrorRates(contest.ncandidates, clcaConfig.simFuzzPct)}")
ClcaErrorTable.getErrorRates(contest.ncandidates, clcaConfig.simFuzzPct)
}
(clcaConfig.errorRates != null) -> {
if (debugErrorRates) println("simulate apriori round $roundIdx using clcaConfig.errorRates =${clcaConfig.errorRates}")
Expand Down Expand Up @@ -238,7 +235,7 @@ fun simulateSampleSizeClcaAssorter(
Nc = contest.Nc,
a = cassorter.noerror(),
d = clcaConfig.d,
errorRates = ErrorRates(0.0, 0.0, 0.0, 0.0)
errorRates = ClcaErrorRates(0.0, 0.0, 0.0, 0.0)
)
)
}
Expand Down
3 changes: 2 additions & 1 deletion ...yptobiotic/rlauxe/sampling/FuzzSampler.kt → ...yptobiotic/rlauxe/estimate/FuzzSampler.kt
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.core.*
import org.cryptobiotic.rlauxe.oneaudit.OneAuditComparisonAssorter
import org.cryptobiotic.rlauxe.oneaudit.OneAuditContestUnderAudit
import org.cryptobiotic.rlauxe.util.*
import org.cryptobiotic.rlauxe.workflow.Sampler
import kotlin.random.Random

// this takes a list of cvrs and fuzzes them
Expand Down
2 changes: 1 addition & 1 deletion ...ptobiotic/rlauxe/sampling/MakePhantoms.kt → ...ptobiotic/rlauxe/estimate/MakePhantoms.kt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.workflow.Ballot
import org.cryptobiotic.rlauxe.core.Contest
Expand Down
2 changes: 1 addition & 1 deletion ...c/rlauxe/sampling/MultiContestTestData.kt → ...c/rlauxe/estimate/MultiContestTestData.kt
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.core.*
import org.cryptobiotic.rlauxe.util.*
Expand Down
3 changes: 2 additions & 1 deletion ...yptobiotic/rlauxe/sampling/RunRepeated.kt → ...yptobiotic/rlauxe/estimate/RunRepeated.kt
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
package org.cryptobiotic.rlauxe.sampling
package org.cryptobiotic.rlauxe.estimate

import org.cryptobiotic.rlauxe.core.RiskTestingFn
import org.cryptobiotic.rlauxe.core.TestH0Status
import org.cryptobiotic.rlauxe.util.*
import org.cryptobiotic.rlauxe.workflow.Sampler
import kotlin.math.sqrt

// single threaded, used for estimating sample size
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/kotlin/org/cryptobiotic/rlauxe/oneaudit/OneAuditContest.kt
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package org.cryptobiotic.rlauxe.oneaudit

import org.cryptobiotic.rlauxe.core.*
import org.cryptobiotic.rlauxe.sampling.ContestSimulation
import org.cryptobiotic.rlauxe.estimate.ContestSimulation
import org.cryptobiotic.rlauxe.util.df
import org.cryptobiotic.rlauxe.util.margin2mean
import org.cryptobiotic.rlauxe.util.mean2margin
Expand Down
4 changes: 2 additions & 2 deletions core/src/main/kotlin/org/cryptobiotic/rlauxe/workflow/AuditConfig.kt
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
package org.cryptobiotic.rlauxe.workflow

import org.cryptobiotic.rlauxe.core.ErrorRates
import org.cryptobiotic.rlauxe.core.ClcaErrorRates
import org.cryptobiotic.rlauxe.util.secureRandom

enum class AuditType { POLLING, CARD_COMPARISON, ONEAUDIT }
Expand Down Expand Up @@ -50,7 +50,7 @@ enum class ClcaStrategyType { oracle, noerror, fuzzPct, apriori, previous, phant
data class ClcaConfig(
val strategy: ClcaStrategyType,
val simFuzzPct: Double? = null, // use to generate apriori errorRates for simulation
val errorRates: ErrorRates? = null, // use as apriori
val errorRates: ClcaErrorRates? = null, // use as apriori
val d: Int = 100, // shrinkTrunc weight for error rates
)

Expand Down
Loading

0 comments on commit dbd9c40

Please sign in to comment.