proc macro to generate memory-efficient gate #695

taikiy · 2023-06-09T08:47:54Z

Procedural macro to implement the Step and StepNarrow traits and generate a memory-efficient gate.

The goal is to generate a state transition graph and the corresponding StepNarrow implementations for the IPA protocol. This macro assumes that a complete IPA steps file exists in the repo at the location specified by STEPS_FILE. The steps file can be generated by running collect_steps.py.

The steps file contains a list of narrowed steps, where each line represents a hierarchy of narrowed steps delimited by "/". For example, the following lines represent a hierarchy of narrowed steps ("=> x" parts are not in the file. They are added in here as comments to indicate their potential IDs when the macro parses the file):

    RootStep                                => 0
    RootStep/StepA::A1                      => 1
    RootStep/StepA::A1/StepB::B1            => 2
    RootStep/StepA::A1/StepB::B2            => 3
    RootStep/StepC::C1                      => 4
    RootStep/StepC::C1/StepD::D1            => 5
    RootStep/StepC::C1/StepD::D1/StepA::A2  => 6
    RootStep/StepC::C2                      => 7

From these lines, we want to generate StepNarrow implementations for each step.

    impl StepNarrow<StepA> for Compact {
        fn narrow(&self, step: &StepA) -> Self {
            Self(match (self.0, step.as_ref()) {
                (0, "A1") => 1,
                (5, "A2") => 6,
                _ => panic!("invalid state transition"),
            })
        }
    }
    impl StepNarrow<StepB> for Compact {
        fn narrow(&self, step: &StepB) -> Self {
            Self(match (self.0, step.as_ref()) {
                (1, "B1") => 2,
                (1, "B2") => 3,
                _ => panic!("invalid state transition"),
            })
        }
    }
    ...

Currently, this derive macro assumes it annotates the Compact struct defined in src/protocol/step/compact.rs. The Compact struct is a wrapper around a u16 value that represents the current state of the IPA protocol.

In the future, we might change the macro to annotate each step in the IPA protocol. The macro will then generate both Descriptive and Compact implementations for the step. However, that kind of derive macro needs to be able to analyze the fully qualified module path of the step at runtime. This is because there are many locally-defined Step enums in IPA, and we need to disambiguate them. There are two ways to solve this. One is to use a feature to resolve the fully qualified module path of a Step enum/struct, but the feature is in nightly as of June 2023. Another way is to define globally unique step names.

martinthomson · 2023-06-09T10:37:02Z

One comment, just based on your commentary (not the patch, I am out of energy for the week).

    impl StepNarrow<StepA> for Compact {
        fn narrow(&self, step: &StepA) -> Self {
            Self(match (self.0, step.as_ref()) {
                (0, "A1") => 1,

This would be better as:

    impl StepNarrow<StepA> for Compact {
        fn narrow(&self, step: &StepA) -> Self {
            Self(match (self.0, step) {

And, to confirm. Do you have annotations for enum arms that take integers, like this:

enum StepC {
  DoX,
  DoY(u32),
}

It would be OK if you only have code for BitOpStep and friends.

martinthomson · 2023-06-09T10:38:13Z

ipa-macros/tests/derive_gate/common/mod.rs

+            StepB::Bar => "bar",
+            StepB::Baz => "baz",


Suggested change

StepB::Bar => "bar",

StepB::Baz => "baz",

Self::Bar => "bar",

Self::Baz => "baz",

martinthomson · 2023-06-09T10:42:32Z

ipa-macros/src/derive_gate/mod.rs

+//     config file, and generate the steps file accordingly. However, the value could change after the steps
+//     file is generated, so we need to make sure that the steps file is always up to date somehow.
+//
+// 6. Root step


We might need a separate type and proc macro for that.

alternatively we could come up with a oneshot run of IPA that initializes TestWorld in a way that does not allow to use it in more than one run

This was more about whether we need to dynamically generate a string representation of root step(s), i.e., does query ID "X" must be generated by Compact gate's AsRef<str> implementation? If we do not require the root step in a real world scenario, we don't need to worry about this. Descriptive gate has id field which embeds protocol/run-0 as the root step, so I was wondering if that was also needed for Compact gate.

taikiy · 2023-06-12T12:11:43Z

And, to confirm. Do you have annotations for enum arms that take integers, like this:
enum StepC {
  DoX,
  DoY(u32),
}

We don't have macros or annotations for a macro to deal with this yet, but I cover these cases in #695. That said, we should create an improved version of the macro with the annotation for this. I could do this together with what we discussed about using strum to directly use enum arms as step representations.

…he end of Cargo.toml

akoshelev · 2023-06-13T19:27:32Z

ipa-macros/Cargo.toml

@@ -0,0 +1,23 @@
+[package]


there are a lot of benefits using workspaces, we should consider using it at some point: https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html

akoshelev · 2023-06-13T19:29:43Z

ipa-macros/src/derive_gate/mod.rs

+};
+use syn::{parse_macro_input, punctuated::Punctuated, DeriveInput, PathArguments, PathSegment};
+
+// TODOs:


lots of good stuff here, lets make sure we have a tracking issue for those. Likely this code will not go through a lot of changes so we may just forget about them

Filed #704, #705, #706, #707, #708. Deleted the comments in the source file.

akoshelev · 2023-06-14T08:40:42Z

ipa-macros/src/derive_gate/mod.rs

+//   - We could use https://doc.rust-lang.org/proc_macro/struct.Span.html#method.source_file, but it's in nightly.
+//   - This will also allow us to generate state transition match statements from int to int rather than str to int.
+//
+// 2. Enable compile-time detection of steps that are `narrow`ed but never trigger `send` in the protocol.


nit: steps need to be inert in order to be "invisible". If they are created to generate shared randomness, that's fine

akoshelev · 2023-06-14T08:47:59Z

ipa-macros/src/derive_gate/mod.rs

+//         a case where there are child steps branching off from the conditional step.
+//     (b) Forcibly execute conditional steps when `step-trace` feature is on to generate the steps file.
+//         Not sure what is the best way to do this, or whether this will work though.
+//  - Currently, we are doing (a-1). It could let unexpected state transitions happen without panicking, but


not sure if it is a great option, but we could annotate such steps with a macro that must enumerate all the valid "parent" steps for conditionally executed one. Then use that information to generate "narrow"

Yes, we could attribute such a step with #[conditional_step] (name TBD). This assumes we have a solution to #704.

akoshelev · 2023-06-14T12:02:33Z

ipa-macros/src/derive_gate/mod.rs

+//     we could apply the same conditions used to call `multiply` using these contexts to where we `narrow`
+//     the context. The code becomes a bit messy, but it's doable.
+//   - crate::protocol::context::UpgradeStep
+//     This steps is executed in both malicious and semi-honest contexts, but the `narrow` call in semi-honest


is it possible to solve it with putting the step narrow inside the upgrade?

Yes, this is basically what I did. Instead of the template implementation in UpgradedContext trait, I created specific implementations of upgrade for semi-honest and malicious.

akoshelev · 2023-06-14T15:04:42Z

ipa-macros/src/tree.rs

+        if let Some(my_parent_ref) = my_parent_weak.upgrade() {
+            Some(Node {
+                data: my_parent_ref,
+            })
+        } else {
+            None
+        }


could be wrong but I think this should work

Suggested change

if let Some(my_parent_ref) = my_parent_weak.upgrade() {

Some(Node {

data: my_parent_ref,

})

} else {

None

}

my_parent_weak.upgrade().clone()

The difference here is that my_parent_ref encloses InnerNode but the return type is Node. I could implement From<> but this is the only place we would use that.

akoshelev · 2023-06-14T15:07:45Z

ipa-macros/src/tree.rs

+        new_child
+    }
+
+    pub fn get_children(&self) -> Vec<Node<T>> {


returning borrowed data may be nicer, but not sure if it is easy to do.

pub fn get_children(&self) -> impl Iterator<Item = Node<T>>

Let me take this as a homework. Not sure how to return an iterator while keeping borrow().iter().

akoshelev · 2023-06-14T19:34:36Z

ipa-macros/src/tree.rs

+}
+
+impl<T> Node<T> {
+    pub fn new(value: T) -> Node<T> {


Suggested change

pub fn new(value: T) -> Node<T> {

pub fn new(value: T)-> Self {

akoshelev · 2023-06-14T19:45:52Z

ipa-macros/src/tree.rs

+
+// We want to make sure that once a parent node is dropped, all its children are dropped as well.
+// To do that, we use `Rc<>` for children, and `Weak<>` for parent. (`Rc` because it's single-threaded)
+type InnerNodeRef<T> = Rc<InnerNode<T>>;


it feels that it should be reversed, that is: Rc<RefCell<Node<T>>> for strong reference and Weak<RefCell<Node<T>>> for weak

Yes, it's a bit confusing...

The type of Node::data::parent is

Rc<RefCell<Weak<InnerNode<T>>>>

and Node::data::children is

Rc<RefCell<Vec<Rc<InnerNode<T>>>>>,

but Node::data::value is

Rc<T>.

We can keep the value immutable while able to mutate parent/child edges via add_child.

akoshelev · 2023-06-14T19:48:26Z

ipa-macros/src/tree.rs

+        Rc::clone(&self.data)
+    }
+
+    pub fn add_child(&self, value: T) -> Node<T> {


You could make it a method on Rc and make a Node constructor that takes Weak as an argument for parent (pass Weak::new() for none).

pub fn add_child(self: Rc<Self>, value: T) -> Self { let child = Rc::new(Node::new(value, self.downgrade()); self.children.push(Rc::clone(child)); child }

Yes, that would make this part of the code a bit more readable. I think hiding all Rc/Weak stuff from the users of Node is a pretty nice feature though.

proc macro to generate memory-efficient gate

e31680a

taikiy requested review from martinthomson and akoshelev June 9, 2023 08:53

update comments

7fe3c8b

martinthomson reviewed Jun 9, 2023

View reviewed changes

taikiy added 2 commits June 12, 2023 13:24

address PR comments

76b877a

Add Cargo.lock. Remove Cargo.lock from gitignore. Add a new line at t…

3c80251

…he end of Cargo.toml

akoshelev reviewed Jun 14, 2023

View reviewed changes

taikiy added 2 commits June 15, 2023 17:39

Address PR comments

da72267

Simplify get_parent()

4c09f00

akoshelev merged commit f078c9c into private-attribution:main Jun 16, 2023

taikiy deleted the ipa-macros branch June 16, 2023 10:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proc macro to generate memory-efficient gate #695

proc macro to generate memory-efficient gate #695

taikiy commented Jun 9, 2023 •

edited

Loading

martinthomson commented Jun 9, 2023

martinthomson Jun 9, 2023

martinthomson Jun 9, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

taikiy commented Jun 12, 2023

akoshelev Jun 13, 2023

akoshelev Jun 13, 2023

taikiy Jun 15, 2023

akoshelev Jun 14, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

taikiy Jun 16, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

akoshelev Jun 14, 2023

taikiy Jun 15, 2023

	pub fn new(value: T) -> Node<T> {
	pub fn new(value: T)-> Self {

proc macro to generate memory-efficient gate #695

proc macro to generate memory-efficient gate #695

Conversation

taikiy commented Jun 9, 2023 • edited Loading

martinthomson commented Jun 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taikiy commented Jun 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taikiy commented Jun 9, 2023 •

edited

Loading