build and explainer docs (#70)

* delete unused inputs * codegen: change template name * change circuits path * add explainer docs * add comments to extractor test
pluto · Aug 30, 2024 · b3bd8da · b3bd8da
1 parent 2af0bcd
commit b3bd8da
Show file tree

Hide file tree

Showing 8 changed files with 310 additions and 159 deletions.
diff --git a/README.md b/README.md
@@ -3,11 +3,15 @@
 
 ## Repo Structure
 The repository is currently new and being organized as follows:
- - `src/`
-    - Example Rust code to test against circom circuits.
-    - Used for doing witness generation
- - `circuits/`
-    - Has current implementation of circuits
+- `src/bin`: binaries
+  - `witness`: Used for doing witness generation
+  - `codegen`: Used for generating extractor circuits based on input
+- `circuits/`: Has current implementation of circuits
+  - `http`: HTTP parser and extractor
+  - `json`: JSON parser and extractor
+  - `utils`: utility circuits
+  - `test`: circuit tests
+- `examples`: reference examples for JSON and HTTP parsers
 
 ## Instructions
 
@@ -57,25 +61,94 @@ npx circomkit clean extract
 
 All of the above should be ran from repository root.
 
-## Rust Example Witness JSON Creation
-To generate example input JSON files for the Circom circuits, you can 
-```
+## Binaries
+
+### Rust Example Witness JSON Creation
+To generate example input JSON files for the Circom circuits, run:
+
+```bash
 cargo install --path .
 ```
-to install the `witness` binary. 
-To get the basic idea, run `witness --help`. 
-It can process and generate JSON files to be used for the circuits.
+
+to install the `witness` binary.
+
+To get the basic idea, run `witness --help`. It can process and generate JSON files to be used for the circuits.
 For example, if we have a given JSON file we want to parse such as `examples/json/test/example.json` for the `extract` circuit (see `circuits.json`), then we can:
-```
+
+```bash
 witness json --input-file examples/json/test/example.json --output-dir inputs/extract --output-filename input.json
 ```
 
 For an HTTP request/response, you can generate a JSON input via:
-```
+```bash
 witness http --input-file examples/http/get_request.http --output-dir inputs/get_request --output-filename input.json
 ```
+
 Afterwards, you can run `circomkit compile get_request` then `circomkit witness get_request input`.
 
+### Codegen
+
+JSON extractor circuit is generated using rust to handle arbitrary keys and array indices.
+
+Run:
+```bash
+cargo run --bin codegen -- --help
+```
+to get options:
+```
+Usage: codegen [OPTIONS] --json-file <JSON_FILE>
+
+Options:
+  -j, --json-file <JSON_FILE>              Path to the JSON file
+  -o, --output-filename <OUTPUT_FILENAME>  Output circuit file name [default: extractor]
+```
+Takes input 2 arguments:
+- `json-file`: input json file. Examples are located in [codegen](./examples/json/test/codegen/)
+- `output-filename`: circuit filename to save. Located in [circuits/main](./circuits/main/). If not given, defaults to `extractor.circom`.
+
+To test an end-to-end JSON extraction proof:
+- Run codegen to generate circuits. Replace `value_string` with input filename.
+   ```bash
+   cargo run --bin codegen -- --json-file ./examples/json/test/codegen/value_string.json --output-filename value_string
+   ```
+
+- Compile circom circuit using
+   ```
+   circom ./circuits/main/value_string.circom --r1cs --wasm
+   ```
+
+- To use circomkit: add circuit config to [circuits.json](./circuits.json). and input file to [inputs](./inputs/)
+
+- Generate witness:
+   ```bash
+   node build/json_extract_value_string/json_extract_value_string_js/generate_witness inputs/json_extract_value_string/value_string.json build/json_extract_value_string/witness/
+   ```
+   or generate using circomkit:
+   ```bash
+   npx circomkit witness json_extract_value_string value_string
+   ```
+
+- create trusted setup:
+   ```bash
+   npx circomkit setup json_extract_value_string
+   # OR
+   snarkjs groth16 setup build/json_extract_value_string/json_extract_value_string.r1cs ptau/powersOfTau28_hez_final_14.ptau build/json_extract_value_string/groth16_pkey.zkey
+   ```
+
+- create proof:
+   ```bash
+   npx circomkit prove json_extract_value_string value_string
+   # OR
+   snarkjs groth16 prove build/json_extract_value_string/groth16_pkey.zkey build/json_extract_value_string/value_string/witness.wtns build/json_extract_value_string/value_string/groth16_proof.json inputs/json_extract_value_string/value_string.json
+   ```
+
+- verify proof:
+   ```bash
+   npx circomkit verify json_extract_value_string value_string
+   # OR
+   snarkjs groth16 verify build/json_extract_value_string/groth16_vkey.json inputs/json_extract_value_string/value_string.json build/json_extract_value_string/value_string/groth16_proof.json
+   ```
+
 ## Testing
 To test, you can just run
 ```

diff --git a/circuits.json b/circuits.json
@@ -1,62 +1,62 @@
 {
     "extract": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             157,
             13
         ]
     },
     "value_string": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             12,
             1
         ]
     },
     "value_number": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             12,
             2
         ]
     },
     "value_array": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             18,
             2
         ]
     },
     "value_array_nested": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             24,
             4
         ]
     },
     "value_array_object": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             25,
             4
         ]
     },
     "value_array_object_array": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             31,
             5
         ]
     },
     "value_object": {
-        "file": "parser_json/parser",
+        "file": "json/parser/parser",
         "template": "Parser",
         "params": [
             21,
@@ -71,9 +71,22 @@
             10
         ]
     },
-    "get_request": {
-        "file": "parser_http_request/parser",
+    "http_get_request": {
+        "file": "http/http_request/parser",
         "template": "Parser",
-        "params": [158]
+        "params": [
+            158
+        ]
+    },
+    "json_extract_value_string": {
+        "file": "main/value_string",
+        "template": "ExtractStringValue",
+        "params": [
+            12,
+            1,
+            1,
+            0,
+            1
+        ]
     }
 }
diff --git a/circuits/test/common/index.ts b/circuits/test/common/index.ts
@@ -25,7 +25,7 @@ export function generateDescription(input: any): string {
         .join(", ");
 }
 
-export function readInputFile(filename: string, key: any[]): [number[], number[][], number[]] {
+export function readJSONInputFile(filename: string, key: any[]): [number[], number[][], number[]] {
     const valueStringPath = join(__dirname, "..", "..", "..", "examples", "json", "test", filename);
 
     let input: number[] = [];

diff --git a/circuits/test/json/extractor/extractor.test.ts b/circuits/test/json/extractor/extractor.test.ts
@@ -1,4 +1,4 @@
-import { circomkit, WitnessTester, readInputFile } from "../../common";
+import { circomkit, WitnessTester, readJSONInputFile } from "../../common";
 import { join } from "path";
 import { spawn } from "child_process";
 
@@ -32,8 +32,12 @@ describe("ExtractValue", async () => {
 
     it("value_string: {\"a\": \"b\"}", async () => {
         let filename = "value_string";
+
+        // generate extractor circuit using codegen
         await executeCodegen(`${filename}.json`, filename);
-        let [input, keyUnicode, output] = readInputFile(`${filename}.json`, ["k"]);
+
+        // read JSON input file into bytes
+        let [input, keyUnicode, output] = readJSONInputFile(`${filename}.json`, ["k"]);
 
         circuit = await circomkit.WitnessTester(`Extract`, {
             file: `circuits/main/${filename}`,
@@ -42,6 +46,7 @@ describe("ExtractValue", async () => {
         });
         console.log("#constraints:", await circuit.getConstraintCount());
 
+        // match circuit output to original JSON value
         await circuit.expectPass({
             data: input, key1: keyUnicode,
         }, {
@@ -52,7 +57,7 @@ describe("ExtractValue", async () => {
     it("two_keys: {\"key1\": \"abc\", \"key2\": \"def\" }", async () => {
         let filename = "two_keys"
         await executeCodegen(`${filename}.json`, filename);
-        let [input, keyUnicode, output] = readInputFile(`${filename}.json`, ["key2"]);
+        let [input, keyUnicode, output] = readJSONInputFile(`${filename}.json`, ["key2"]);
 
         circuit = await circomkit.WitnessTester(`Extract`, {
             file: `circuits/main/${filename}`,
@@ -67,7 +72,7 @@ describe("ExtractValue", async () => {
     it("value_number: {\"k\": 69 }", async () => {
         let filename = "value_number";
         await executeCodegen(`${filename}.json`, filename);
-        let [input, keyUnicode, output] = readInputFile(`${filename}.json`, ["k"]);
+        let [input, keyUnicode, output] = readJSONInputFile(`${filename}.json`, ["k"]);
 
         circuit = await circomkit.WitnessTester(`Extract`, {
             file: `circuits/main/${filename}`,
@@ -86,7 +91,7 @@ describe("ExtractValue", async () => {
         await executeCodegen(`${filename}.json`, filename);
 
         for (let i = 0; i < 4; i++) {
-            let [input, keyUnicode, output] = readInputFile("value_array.json", ["b", i]);
+            let [input, keyUnicode, output] = readJSONInputFile("value_array.json", ["b", i]);
 
             circuit = await circomkit.WitnessTester(`Extract`, {
                 file: `circuits/main/${filename}`,
@@ -104,7 +109,7 @@ describe("ExtractValue", async () => {
         await executeCodegen(`${filename}.json`, filename);
 
         for (let i = 0; i < 4; i++) {
-            let [input, keyUnicode, output] = readInputFile("value_array.json", ["k", i]);
+            let [input, keyUnicode, output] = readJSONInputFile("value_array.json", ["k", i]);
 
             circuit = await circomkit.WitnessTester(`Extract`, {
                 file: `circuits/main/${filename}`,
@@ -123,7 +128,7 @@ describe("ExtractValue", async () => {
         await executeCodegen(`${filename}.json`, filename);
         let index_0 = 1;
         let index_1 = 0;
-        let [input, keyUnicode, output] = readInputFile(`${filename}.json`, ["a", index_0, index_1]);
+        let [input, keyUnicode, output] = readJSONInputFile(`${filename}.json`, ["a", index_0, index_1]);
 
         circuit = await circomkit.WitnessTester(`Extract`, {
             file: `circuits/main/${filename}`,
@@ -147,7 +152,7 @@ describe("ExtractValueMultiDepth", () => {
 
         await executeCodegen(`${filename}.json`, filename);
 
-        let [input, keyUnicode, output] = readInputFile(`${filename}.json`, ["e", "e"]);
+        let [input, keyUnicode, output] = readJSONInputFile(`${filename}.json`, ["e", "e"]);
 
         circuit = await circomkit.WitnessTester(`Extract`, {
             file: `circuits/main/${filename}`,
@@ -158,7 +163,7 @@ describe("ExtractValueMultiDepth", () => {
 
         await circuit.expectPass({ data: input, key1: keyUnicode[0], key2: keyUnicode[1] }, { value: output });
 
-        let [input1, keyUnicode1, output1] = readInputFile("value_object.json", ["e", "f"]);
+        let [input1, keyUnicode1, output1] = readJSONInputFile("value_object.json", ["e", "f"]);
         await circuit.expectPass({ data: input1, key1: keyUnicode1[0], key2: keyUnicode1[1] }, { value: output1 });
     });
 
@@ -175,7 +180,7 @@ describe("ExtractValueArrayObject", () => {
 
         let index_0 = 0;
         let index_1 = 0;
-        let [input, keyUnicode, output] = readInputFile(`${filename}.json`, ["a", index_0, "b", index_1]);
+        let [input, keyUnicode, output] = readJSONInputFile(`${filename}.json`, ["a", index_0, "b", index_1]);
 
         circuit = await circomkit.WitnessTester(`Extract`, {
             file: `circuits/main/${filename}`,

diff --git a/circuits/test/json/extractor/interpreter.test.ts b/circuits/test/json/extractor/interpreter.test.ts
@@ -1,4 +1,4 @@
-import { circomkit, WitnessTester, generateDescription, readInputFile } from "../../common";
+import { circomkit, WitnessTester, generateDescription, readJSONInputFile } from "../../common";
 import { PoseidonModular } from "../../common/poseidon";
 
 describe("Interpreter", async () => {
@@ -283,7 +283,7 @@ describe("Interpreter", async () => {
             });
         }
 
-        let input = readInputFile("value_array_object.json", ["a"]);
+        let input = readJSONInputFile("value_array_object.json", ["a"]);
         const concatenatedInput = input[1][0].concat(input[0]);
         const hashResult = PoseidonModular(concatenatedInput);
 
@@ -324,7 +324,7 @@ describe("Interpreter", async () => {
             });
         }
 
-        let input = readInputFile("value_array_object.json", ["a", 0, "b", 0]);
+        let input = readJSONInputFile("value_array_object.json", ["a", 0, "b", 0]);
         const concatenatedInput = input[1][0].concat(input[0]);
         const hashResult = PoseidonModular(concatenatedInput);