Address feedback from PETs artifact evaluation.

Thank you anon reviewer!

README.md

@@ -2,14 +2,35 @@
 [![devcontainer](https://github.com/secret-snail/localization-server/actions/workflows/devcontainer.yml/badge.svg)](https://github.com/secret-snail/localization-server/actions/workflows/devcontainer.yml)
 [![container](https://github.com/secret-snail/localization-server/actions/workflows/docker-image.yml/badge.svg)](https://github.com/secret-snail/localization-server/actions/workflows/docker-image.yml)
 
-Privacy preserving localization based on secure multiparty computation (MPC).
+**Privacy preserving localization using secure multiparty computation.**
+
+Visual localization is a computer vision task by which the position and
+orientation of a camera is determined from an image and environmental map. This
+project implements a secure localization server that uses MPC to compute a
+camera pose without revealing the image or map. For more information, see the
+paper on [arxiv](https://arxiv.org/abs/2403.14916), published at
+[PETs 2023](https://petsymposium.org/2024/paperlist.php).
+
+## Repository Structure
+
+This repository stores external dependencies as submodules in the `extern`
+directory. The core localization libraries for each MPC protocol are stored in
+the `src` directory. The `test` directory contains unit tests and benchmarks for
+the localization libraries. The `.vscode` and `.devcontainer` directories
+automatically configure a working development environment and continuous
+deployment is set up with GitHub Actions in `.github`.
 
 ## Build via Container
+
+Building the container requires compiling OpenCV, which may take 30+
+minutes.
+
 ```bash
 docker/build.sh
 ```
 
 Run an interactive shell inside the conatiner:
+
 ```bash
 docker run -it --rm --init \
   --net=host \
@@ -18,36 +39,71 @@ docker run -it --rm --init \
 ```
 
 ## Build From Source
-ABY Dependencies:
-`sudo apt install g++ make cmake libgmp-dev libssl-dev libboost-all-dev`
 
-EMP Dependencies:
-`sudo apt install software-properties-common cmake git build-essential libssl-dev`
+If not using the provided devcontainer, install the following dependencies:
 
-Build and test the code:
+```bash
+sudo apt install g++ make cmake libgmp-dev libssl-dev libboost-all-dev  # ABY
+sudo apt install software-properties-common cmake git build-essential libssl-dev  # EMP
 ```
+
+Build and test the code:
+
+```bash
 git submodule update --init --recursive
-mkdir build && cd build
-cmake ..
+mkdir build
+cd build
+cmake -DCMAKE_BUILD_TYPE=Release ..
 make
 ctest
 ```
 
-ABY testing requires -DCMAKE_BUILD_TYPE=Release, tests fail when the sanitizers
-are turned on because there appears to be memory leaks in the ABY library.
-
-Note - circuits cannot be built on the fly. must be fully specified then executed.
-This means if control flow requires some secret data, circuit must be broken and
-intermediate ciphertext stored as secret share.
+Special notes about ABY, one of the MPC libraries used in this project:
+- ABY tests and experiments are memory intensive, requiring up to 19 GB of
+memory (resident set size).
+- ABY tests require `-DCMAKE_BUILD_TYPE=Release`. The tests fail when the
+sanitizers are turned on because there appear to be memory leaks in the external
+ABY library. Due to this, tracking memory leaks in the ABY localization
+library in *this* repo is difficult thus likely also leaks, though an effort was
+made to avoid this. This is not true for the EMP localization library.
+- ABY circuits cannot be built on the fly. They must be fully specified then
+executed. This means if control flow requires some secret data, circuit must be
+broken and intermediate ciphertext stored as secret share.
 [Developer Guide](https://www.informatik.tu-darmstadt.de/media/encrypto/encrypto_code/abydevguide.pdf)
 [Reusing Computation](https://github.com/encryptogroup/ABY/issues/167)
 
-## Experimental Evaluation
+Special notes about EMP, the other MPC library used in this project:
+- A machine with AES hardware acceleration (for example AES-NI on Intel
+hardware) is required for performance reasons.
+
+### External Dependencies
+
+External source dependencies in `extern`:
+
+- [ABY](https://github.com/encryptogroup/ABY)
+- [Catch2](https://github.com/catchorg/Catch2)
+- [EMP-Toolkit](https://github.com/emp-toolkit)
+- [OpenCV](https://github.com/opencv/opencv)
+- [Fixed Point Math Library](https://github.com/MikeLankamp/fpm) (direct import at `src/common/fixed_point.h`)
+- [Singular Value Decomposition](https://numerical.recipes/) (direct import at `test/common-test/cleartext-ref/svd.hpp`)
+
+Other dependencies:
+
+- docker
+- cmake
+- gcc
+- clang-format
+- python3
+- matplotlib
+
+## Reproduce Results
+
 First, download the
 [eth3d dataset](https://www.eth3d.net/datasets#high-res-multi-view) with the
 following commands on the host (outside the container). These will specifically
 download the high-res multi-view undistorted images and ground truth scan
-evaluation from eth3d.
+evaluation from eth3d. Note this requires ~20GB of storage.
+
 ```bash
 sudo apt-get install p7zip-full
 mkdir ./data-eth3d
@@ -63,7 +119,7 @@ The easiest way to run the experiments is via the provided Docker container.
 Build and start the container with the following commands.
 
 ```bash
-docker/build.sh
+docker/build.sh  # 30 minutes
 
 mkdir -p results
 
@@ -75,8 +131,23 @@ docker run -it --rm --init \
   snail-server bash
 ```
 
-Run the experiments. Note some of the commands must be run outside the container
-due to network setup requirements. They are marked with `# outside container`.
+Run `ctest` inside the `build` directory of the container to verify the build.
+The tests should pass in roughly 15 minutes. Machine with less than 19 GB of
+memory may fail some of the ABY tests due to memory constraints. Machines with
+sufficient memory may still occasionally fail the `ABY large SVD` test due to
+memory errata.
+
+Run the experiments with the following commands. Note some of the commands must
+be run outside the container due to network setup requirements. They are marked
+with `# outside container`. Each experiment outputs information to the console
+for example the image being tested, how many points are being used, the ground
+truth pose and the pose as estimated by OpenCV vs. the MPC protocol. The
+console output is piped to a file in `results` and lines prefixed by `SeNtInAl`
+are used to parse the results in subsequent plotting. The characters which look
+like `...:::555/44/33/2/1` correspond to the different phases and iterations of
+the SVD algorithm. Lines which look like "Trial x with y randomly selected
+points..." show the progress of the experiment.
+
 
 ```bash
 # ABY and EMP single-iteration localization tests.
@@ -109,7 +180,12 @@ Install dependencies for the plotter scripts (requires python3 and pip3).
 ./plotter_deps.sh
 ```
 
-Plot the results.
+Plot the results. The outputs are stored in `plots/` and can be matched to
+figures in the paper (see [claims](claims.md)). The time required to exactly
+reproduce the experiments from the paper is large thus the number of averaged
+trials for each experiment has been reduced from results shown in the paper.
+To run an exact reproduction, the scripts in `scripts/*_run.sh` may be edited to
+run 5 trials and 8 frames.
 
 ```bash
 mkdir plots
@@ -122,6 +198,7 @@ scripts/mult_add_fixed_float_time_plot.sh
 ```
 
 ## Robotic Snail Demo
+
 *Requires raspberry pi, see [snail repo](https://github.com/secret-snail/snail)*
 
 Currently, the demo requires `privacy_conf.h` to have this set:
@@ -133,11 +210,9 @@ To run the snail demo, first `docker/build.sh` the container. Then run
 `docker/alice.sh` and `docker/bob.sh` in two terminals on the server(s).
 On the [snail](https://github.com/secret-snail/snail), run
 `sudo ./build/bin/visp_snail --secure`.
-Make sure the snail can see the marker in the first 10 seconds, otherwise the
-snail will need to be restarted.
-
 
 ## Running on Separate Machines
+
 Set the machines IP addresses in `test/emp-float/client.cpp` and
 `src/emp-float-server/server-lm.cpp` replacing the localhost IP.
 

claims.md

@@ -0,0 +1,51 @@
+# Main Results and Claims
+
+The results and accompanying claims are enumerated below.
+
+1. Figure 2 compares the runtime of localization using two MPC libraries, EMP
+and ABY. Claim: EMP is better suited to localization than ABY.
+2. Figure 3 measures the runtime of arithmetic operations on various data
+representations. Claim: 64 bit fixed point multiplication is slower than 32 bit
+floating.
+3. Figure 4 counts the number of arithmetic operations performed during
+localization. Claim: Multiplication is the dominant operation.
+4. Figure 6 compares the time to localize using data oblivious (DO) vs. single
+iteration localization (SIL). Claim: SIL localizes faster than DO with the
+Levenburg Marquardt (LM) optimization algorithm being faster than Gauss Newton
+(GN).
+5. Figure 7 measures runtime and network IO for different localization
+configurations at large input sizes. Claim: LM better scales to large input
+sizes than GN, making LM the better approach of the two.
+
+## Main Result 1: Figure 2
+
+The EMP MPC library is better suited to localization than the ABY MPC library.
+Open the file `plots/emp_vs_aby.pdf` and compare it to Figure 2 in the paper.
+The EMP LM and EMP GN bars should be smaller than the ABY bars.
+
+## Main Result 2: Figure 3
+
+64 bit fixed point multiplication is slower than 32 bit floating point. Open
+`plots/emp_float_vs_fixed_benchmark_add_mul.pdf` and verify the size of these
+two bars noting the y axis for multiplication is is on the right.
+
+## Main Result 3: Figure 4
+
+Multiplication is the dominant operation in LM-based localization.
+Open `plots/emp_arith_ops.pdf` and verify the green multiplication bars are
+larger than all other bars at any of the measured number of features.
+
+## Main Result 4: Figure 6
+
+Single iteration localization (SIL) localizes faster than the data oblivious
+(DO) adaptation with LM being the faster optimization algorithm. Open
+`plots/loopleak_vs_dataobl.pdf` and verify the DO lines are (much) higher than
+the SIL lines. Also check the purple LM SIL line is the lowest in the figure
+for all the numbers of features.
+
+## Main Result 5: Figure 7
+
+LM better scales to large input size i.e. large numbers of input features.
+Open `plots/emp_float_runtime_long.pdf` and verify the GN lines are higher than
+their respective LM lines for the same latency. Next open `plots/netio.pdf`
+and verify the LM lines are below their respective GN lines.