new fieller interval documentation

Author: Liz Obermaier

cspell.json

@@ -658,7 +658,9 @@
     "akin",
     "vineeth",
     "vijaye",
-    "brock"
+    "brock",
+    "Fieller",
+    "Fieller's"
   ],
   "ignorePaths": [
     "node_modules/**",

docs/stats-engine/confidence-intervals.mdx

@@ -22,36 +22,41 @@ Once we've established the variance of the delta, it's straightforward to comput
 
 For the **absolute metric delta**, the confidence interval is given by:
 
-$$\Large
-CI(\Delta \overline{X}) = \Delta \overline{X} \pm Z_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}  
+$$
+\Large
+CI(\Delta \overline{X}) = \Delta \overline{X} \pm Z_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}
 $$
 
 where:
 
 - $Z_{\alpha/2}$ is the z-critical value for the desired significance level (1.96 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a two-sided test
 - $var(\Delta \overline{X})$ is the variance of the absolute delta (details [here](/stats-engine/variance))
 
-Similarly, the confidence interval for the **relative metric delta** is:
+The confidence interval for the **relative metric delta** is calculated using Fieller's Theorem:
+
+First the constant g must be calculated.
 
-$$\Large
-CI(\Delta \overline X\%)
-= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}}
-= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot\sqrt{(\frac{\overline X_t}{\overline X_c})^{2} \cdot (\frac{var(\overline X_c)}{n_c \cdot \overline X_c^2} + \frac{var(\overline X_t)}{n_t \cdot \overline X_t^2})} \cdot 100\% 
+$$
+\Large
+g = \frac{Z_{\alpha/2}^2 \cdot var(X_C)}{(n-1) \cdot \overline{X_C}^2}
 $$
 
-If the control mean is not significantly away from zero, then
+When g < 1, the control mean is significantly different from 0. Since the control and test group results are independent of each other, covariance terms in Fieller's Theorem can be dropped.
 
-$$\Large
+$$
+\Large
 CI(\Delta \overline X\%)
-= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}}
-= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot \frac{\sqrt{{var\left(\Delta \overline X\right)}}}{\overline X_c} \cdot 100\% 
+= \frac{1}{1-g} ( \frac{\overline{X_T}}{\overline{X_C}} - 1 \pm \frac{Z_{\alpha/2}}{sqrt{n_C} \cdot \overline{X_C}} sqrt{(1-g) \cdot \frac{var(X_T)}{n_T(n_T-1)} + \frac{\overline{X_T} var(X_C)}{\overline{X_C} n_C (n_C-1)}})
 $$
 
+If the control mean is not significantly different from zero, then a confidence interval for relative metric delta is not well defined, and a point estimate is surfaced instead.
+
 ### One-Sided Tests
 
 When running one-sided tests, the form of the confidence interval calculation changes slightly to account for a redistribution of desired false positive rate when looking for increases or decreases in the metric:
 
-$$\Large
+$$
+\Large
 CI(\Delta \overline{X}) = \begin{cases}
 \left[\Delta \overline{X} - Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}}, \quad +\infty  \right) & \text{if right-hand test}\\
 \\
@@ -71,11 +76,13 @@ For small sample sizes, we use Welch's t-test instead of a standard z-test. This
 
 For a two-sided test, the confidence interval is therefore:
 
-$$\Large
-CI(\Delta \overline{X}) = \Delta \overline{X} \pm t_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}  
+$$
+\Large
+CI(\Delta \overline{X}) = \Delta \overline{X} \pm t_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}}
 $$
 
-$$\Large
+$$
+\Large
 \nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}
 = \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}
 $$
@@ -88,6 +95,7 @@ Sometimes we want to answer questions like "Does my test variant lead to a click
 
 The confidence interval is calculated by
 
-$$\Large
+$$
+\Large
 CI(\Delta \overline X) = (\overline X_{group} - fixed \ value) \pm Z \cdot\sqrt{{var( \overline X_{group})}}
 $$