Implement macro meta-variable expressions #93545

c410-f3r · 2022-02-01T14:20:04Z

Based on https://github.com/markbt/rust/tree/metavariable_expressions with some refactoring, additional tests, explicit feature declaration and minor documentation enhancement.

With this feature, declarative macros will be much more powerful and more flexible to work with.

#![feature(macro_metavar_expr)]

macro_rules! build_and_modify_tuple {
    ( ( $( $l:literal ),* ) ) => {{
        let mut tuple = ( $( $l ),* );
        $( ${ignore(l)} tuple.${index()} *= 2; )*
        tuple
    }};
}

macro_rules! count {
    ( $( $e:stmt ),* ) => {
        ${ count(e) }
    };
}

fn main() {
    assert_eq!(build_and_modify_tuple!((0u8, 2i32, 4u128)), (0u8, 4i32, 8u128));
    assert_eq!(count!(let _a = 1i32, let _ = (), if true {}), 3);
}

Thank you very much @markbt for writing the RFC and for starting the implementation.

cc #83527
r? @petrochenkov

c410-f3r

If more documentation needs to be created, then I will do so in a following PR.

src/test/ui/rfc-3086-metavar-expr/nested-macro-declarations.rs

compiler/rustc_expand/src/mbe/transcribe.rs

compiler/rustc_expand/src/mbe/metavar_expr.rs

compiler/rustc_expand/src/mbe/macro_check.rs

compiler/rustc_expand/src/mbe/transcribe.rs

src/test/ui/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs

joshtriplett · 2022-02-01T20:49:47Z

Could you please separate out the change from '{}' to `{}` and the corresponding test changes, for separate review and FCP? That seems orthogonal to this change, and it'd reduce the amount that needs to be reviewed simultaneously.

src/test/ui/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs

compiler/rustc_expand/src/mbe/macro_check.rs

compiler/rustc_expand/src/mbe/metavar_expr.rs

src/test/ui/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs

compiler/rustc_expand/src/mbe/quoted.rs

compiler/rustc_expand/src/mbe/transcribe.rs

petrochenkov · 2022-02-05T14:49:24Z

I still don't understand two things:

which of the PR implements ignore, I don't see where variables used inside ${...} are marked as used, but apparently the variables used inside ignore are indeed counted as used because there's no warning.
what is the difference between length and count.

petrochenkov · 2022-02-05T14:51:46Z

We need tests making sure that all kinds of erroneous conditions produce errors:

The variables named inside meta-variable expressions are defined.
The indices used inside meta-variable expressions are in range, I don't think there should be any silent saturating or wrapping behavior.

petrochenkov · 2022-02-05T14:52:58Z

Also, parsing functions returning Option are error-prone.
Cases in which the syntactic element is truly optional, and in which it's expected but not found should be clearly separated.

c410-f3r · 2022-02-05T14:55:43Z

Now it is coding time!

Thank you very much for the review @petrochenkov! I will address everything and reach back as soon as possible

petrochenkov · 2022-02-05T14:56:05Z

Could you please separate out the change from '{}' to `{}` and the corresponding test changes, for separate review and FCP? That seems orthogonal to this change, and it'd reduce the amount that needs to be reviewed simultaneously.

+1, the 'x' -> `x` changes are off-topic for this PR.

petrochenkov · 2022-02-05T16:25:22Z

This also needs a second look from someone who is actually familiar with all this macro repetition logic, because I'm not.
cc @mark-i-m maybe

c410-f3r · 2022-02-05T22:05:18Z

@petrochenkov

I will probably finish the remaining stuff tomorrow

mark-i-m · 2022-02-06T15:22:47Z

I'll try to take a look soon.

c410-f3r · 2022-02-06T22:13:12Z

@petrochenkov

which of the PR implements ignore, I don't see where variables used inside ${...} are marked as used, but apparently the variables used inside ignore are indeed counted as used because there's no warning.

My understanding is that ignored tokens are indeed counted but don't produce any output (https://github.com/rust-lang/rust/pull/93545/files#diff-727806e582042b1e7f196258a14707cc155d7ba36ab925066d3f7b43e6097b6fR511).

what is the difference between length and count.

https://github.com/rust-lang/rust/pull/93545/files#diff-23d875c394e16567144626114846b378fc49bbb8feedf5871399860ac5538744 has an extensive set of comments that hopefully will help understanding the difference between both.

The variables named inside meta-variable expressions are defined.

The indices used inside meta-variable expressions are in range, I don't think there should be any silent saturating or wrapping behavior.

Did a fair amount of rewriting to improve these errors

+1, the 'x' -> x changes are off-topic for this PR.

Removed

mark-i-m

@c410-f3r Nice work navigating this very tricky and confusing code!

I left a bunch of comments in various places. My main concerns:

I think one of the changes to macro_check is incomplete. I left a comment below.
I think count_repetitions needs more comments, as it is not obvious what it does or why at a glance. I also left comments/suggestions there.

@petrochenkov

which of the PR implements ignore, I don't see where variables used inside ${...} are marked as used, but apparently the variables used inside ignore are indeed counted as used because there's no warning.

If you mean linting for unused meta-variables, I don't think we have any. It might be a good lint to add...

EDIT: I mean it doesn't exist for macros in general, and we should add it as a separate feature in a different PR.

If you mean checking for the number of iterations (e.g. so that $(${ignore(x)} a)* outputs a the number of times $x matched), that is done by lockstep_iter_size. The call to lockstep_iter_size from transcribe gets the number of repetitions matched by all variables repeating at that depth (and checks that all variables repeat the same amount).

compiler/rustc_expand/src/mbe/macro_parser.rs

compiler/rustc_expand/src/mbe/quoted.rs

compiler/rustc_expand/src/mbe/macro_check.rs

compiler/rustc_expand/src/mbe/transcribe.rs

c410-f3r · 2022-02-11T12:54:01Z

Thank you very much for the review, @mark-i-m! I really should have put more comments :)

mark-i-m

The core logic looks good to me!

I left some comments about minor concerns below, most notably the error messages. I think this PR can be merged, but those things should be fixed before the feature is stabilized.

@petrochenkov feel free to r=me

r? @petrochenkov

compiler/rustc_expand/src/mbe/transcribe.rs

src/tools/tidy/src/ui_tests.rs

src/test/ui/rfc-3086-metavar-expr/syntax-errors.stderr

mark-i-m · 2022-02-22T19:59:14Z

Right... forgot bors doesn't look in reviews...

r? @petrochenkov

c410-f3r · 2022-02-22T21:46:00Z

@mark-i-m Thank you very much for the review and if desired, feel free to suggest more changes

petrochenkov · 2022-02-24T11:22:50Z

I've noticed one unaddressed comment here - #93545 (comment).

c410-f3r · 2022-02-24T11:25:44Z

I've noticed one unaddressed comment here - #93545 (comment).

Based on #93545 (comment) and other comments, this case will be handled in a following PR

src/test/ui/macros/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs

compiler/rustc_expand/src/mbe/macro_check.rs

src/test/ui/macros/rfc-3086-metavar-expr/requiring-feature.rs

compiler/rustc_expand/src/mbe/macro_parser.rs

compiler/rustc_expand/src/mbe/metavar_expr.rs

petrochenkov · 2022-02-25T15:41:22Z

I started reviewing this, but didn't finish - the PR is too large.

Could you split it into 2 PRs:

The first one introduces the meta-variable expression infra, and one of the expression kinds, e.g. ignore as the simplest one.
The second one implements the remaining expression kinds.

c410-f3r · 2022-02-25T19:11:37Z

Addressed the comments here and created #94368 as requested

…aaaa, r=petrochenkov [1/2] Implement macro meta-variable expressions See rust-lang#93545 (comment) The logic behind `length`, `index` and `count` was removed but the parsing code is still present, i.e., everything is simply ignored like `ignored`. r? `@petrochenkov`

…aaaa, r=petrochenkov [1/2] Implement macro meta-variable expressions See rust-lang#93545 (comment) The logic behind `length`, `index` and `count` was removed but the parsing code is still present, i.e., everything is simply ignored like `ignored`. r? ``@petrochenkov``

[2/2] Implement macro meta-variable expression Final part of rust-lang#93545 (comment) r? `@petrochenkov`

Fix remaining meta-variable expression TODOs As promised on rust-lang#93545. cc rust-lang#83527 cc `@mark-i-m` cc `@petrochenkov`

…lett Stabilize `$$` in Rust 1.63.0 # Stabilization proposal This PR proposes the stabilization of a subset of `#![feature(macro_metavar_expr)]` or more specifically, the stabilization of dollar-dollar (`$$`). Tracking issue: rust-lang#83527 Version: 1.63 (2022-06-28 => beta, 2022-08-11 => stable). ## What is stabilized ```rust macro_rules! foo { () => { macro_rules! bar { ( $$( $$any:tt )* ) => { $$( $$any )* }; } }; } fn main() { foo!(); } ``` ## Motivation For more examples, see the [RFC](https://github.com/markbt/rfcs/blob/macro_metavar_expr/text/0000-macro-metavar-expr.md). Users must currently resort to a tricky and not so well-known hack to declare nested macros with repetitions. ```rust macro_rules! foo { ($dollar:tt) => { macro_rules! bar { ( $dollar ( $any:tt )* ) => { $dollar ( $any )* }; } }; } fn main() { foo!($); } ``` As seen above, such hack is fragile and makes work with declarative macros much more unpleasant. Dollar-dollar (`$$`), on the other hand, makes nested macros more intuitive. ## What isn't stabilized `count`, `ignore`, `index` and `length` are not being stabilized due to the lack of consensus. ## History * 2021-02-22, [RFC: Declarative macro metavariable expressions](rust-lang/rfcs#3086) * 2021-03-26, [Tracking Issue for RFC 3086: macro metavariable expressions](rust-lang#83527) * 2022-02-01, [Implement macro meta-variable expressions](rust-lang#93545) * 2022-02-25, [[1/2] Implement macro meta-variable expressions](rust-lang#94368) * 2022-03-11, [[2/2] Implement macro meta-variable expressions](rust-lang#94833) * 2022-03-12, [Fix remaining meta-variable expression TODOs](rust-lang#94884) * 2019-03-21, [[macro-metavar-expr] Fix generated tokens hygiene](rust-lang#95188) * 2022-04-07, [Kickstart the inner usage of macro_metavar_expr](rust-lang#95761) * 2022-04-07, [[macro_metavar_expr] Add tests to ensure the feature requirement](rust-lang#95764) ## Non-stabilized expressions rust-lang#83527 lists several concerns about some characteristics of `count`, `index` and `length` that effectively make their stabilization unfeasible. `$$` and `ignore`, however, are not part of any discussion and thus are suitable for stabilization. It is not in the scope of this PR to detail each concern or suggest any possible converging solution. Such thing should be restrained in this tracking issue. ## Tests This list is a subset of https://github.com/rust-lang/rust/tree/master/src/test/ui/macros/rfc-3086-metavar-expr * [Ensures that nested macros have correct behavior](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/dollar-dollar-has-correct-behavior.rs) * [Compares produced tokens to assert expected outputs](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs) * [Checks the declarations of the feature](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/required-feature.rs) * [Verifies all possible errors that can occur due to incorrect user input](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/syntax-errors.rs) ## Possible future work Once consensus is achieved, other nightly expressions can be stabilized. Thanks `@markbt` for creating the RFC and thanks to `@petrochenkov` and `@mark-i-m` for reviewing the implementations.

…lett Stabilize `$$` in Rust 1.63.0 # Stabilization proposal This PR proposes the stabilization of a subset of `#![feature(macro_metavar_expr)]` or more specifically, the stabilization of dollar-dollar (`$$`). Tracking issue: rust-lang#83527 Version: 1.63 (2022-06-28 => beta, 2022-08-11 => stable). ## What is stabilized ```rust macro_rules! foo { () => { macro_rules! bar { ( $$( $$any:tt )* ) => { $$( $$any )* }; } }; } fn main() { foo!(); } ``` ## Motivation For more examples, see the [RFC](https://github.com/markbt/rfcs/blob/macro_metavar_expr/text/0000-macro-metavar-expr.md). Users must currently resort to a tricky and not so well-known hack to declare nested macros with repetitions. ```rust macro_rules! foo { ($dollar:tt) => { macro_rules! bar { ( $dollar ( $any:tt )* ) => { $dollar ( $any )* }; } }; } fn main() { foo!($); } ``` As seen above, such hack is fragile and makes work with declarative macros much more unpleasant. Dollar-dollar (`$$`), on the other hand, makes nested macros more intuitive. ## What isn't stabilized `count`, `ignore`, `index` and `length` are not being stabilized due to the lack of consensus. ## History * 2021-02-22, [RFC: Declarative macro metavariable expressions](rust-lang/rfcs#3086) * 2021-03-26, [Tracking Issue for RFC 3086: macro metavariable expressions](rust-lang#83527) * 2022-02-01, [Implement macro meta-variable expressions](rust-lang#93545) * 2022-02-25, [[1/2] Implement macro meta-variable expressions](rust-lang#94368) * 2022-03-11, [[2/2] Implement macro meta-variable expressions](rust-lang#94833) * 2022-03-12, [Fix remaining meta-variable expression TODOs](rust-lang#94884) * 2019-03-21, [[macro-metavar-expr] Fix generated tokens hygiene](rust-lang#95188) * 2022-04-07, [Kickstart the inner usage of macro_metavar_expr](rust-lang#95761) * 2022-04-07, [[macro_metavar_expr] Add tests to ensure the feature requirement](rust-lang#95764) ## Non-stabilized expressions rust-lang#83527 lists several concerns about some characteristics of `count`, `index` and `length` that effectively make their stabilization unfeasible. `$$` and `ignore`, however, are not part of any discussion and thus are suitable for stabilization. It is not in the scope of this PR to detail each concern or suggest any possible converging solution. Such thing should be restrained in this tracking issue. ## Tests This list is a subset of https://github.com/rust-lang/rust/tree/master/src/test/ui/macros/rfc-3086-metavar-expr * [Ensures that nested macros have correct behavior](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/dollar-dollar-has-correct-behavior.rs) * [Compares produced tokens to assert expected outputs](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs) * [Checks the declarations of the feature](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/required-feature.rs) * [Verifies all possible errors that can occur due to incorrect user input](https://github.com/rust-lang/rust/blob/master/src/test/ui/macros/rfc-3086-metavar-expr/syntax-errors.rs) ## Possible future work Once consensus is achieved, other nightly expressions can be stabilized. Thanks ``@markbt`` for creating the RFC and thanks to ``@petrochenkov`` and ``@mark-i-m`` for reviewing the implementations.

rust-highfive assigned petrochenkov Feb 1, 2022

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Feb 1, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Feb 1, 2022

c410-f3r commented Feb 1, 2022

View reviewed changes

fmease reviewed Feb 1, 2022

View reviewed changes

compiler/rustc_expand/src/mbe/transcribe.rs Outdated Show resolved Hide resolved

fmease reviewed Feb 1, 2022

View reviewed changes

compiler/rustc_expand/src/mbe/transcribe.rs Outdated Show resolved Hide resolved

c410-f3r force-pushed the meta-vars branch from 1eafd76 to 4f2ec05 Compare February 1, 2022 14:34

This comment has been minimized.

Sign in to view

c410-f3r force-pushed the meta-vars branch from 4f2ec05 to 661d520 Compare February 1, 2022 14:57

c410-f3r commented Feb 1, 2022

View reviewed changes

src/test/ui/rfc-3086-metavar-expr/feature-gate-macro_metavar_expr.rs Outdated Show resolved Hide resolved

petrochenkov reviewed Feb 5, 2022

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 5, 2022

c410-f3r force-pushed the meta-vars branch from 661d520 to b6607d6 Compare February 5, 2022 22:04

c410-f3r force-pushed the meta-vars branch 2 times, most recently from 583c234 to 49645c0 Compare February 6, 2022 22:04

mark-i-m reviewed Feb 11, 2022

View reviewed changes

c410-f3r mentioned this pull request Feb 11, 2022

Check if meta-variable expressions in the RHS are declared in the LHS #93902

Closed

c410-f3r force-pushed the meta-vars branch from 49645c0 to e3de6ec Compare February 11, 2022 12:38

mark-i-m approved these changes Feb 22, 2022

View reviewed changes

compiler/rustc_expand/src/mbe/transcribe.rs Outdated Show resolved Hide resolved

src/tools/tidy/src/ui_tests.rs Outdated Show resolved Hide resolved

src/test/ui/rfc-3086-metavar-expr/syntax-errors.stderr Outdated Show resolved Hide resolved

rust-highfive assigned petrochenkov and unassigned mark-i-m Feb 22, 2022

c410-f3r force-pushed the meta-vars branch from df8a975 to a575175 Compare February 22, 2022 21:41

Implement macro meta-variable expressions

9ca8946

c410-f3r force-pushed the meta-vars branch from a575175 to 9ca8946 Compare February 24, 2022 12:05

petrochenkov reviewed Feb 25, 2022

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 25, 2022

c410-f3r mentioned this pull request Feb 25, 2022

[1/2] Implement macro meta-variable expressions #94368

Merged

c410-f3r closed this Feb 25, 2022

c410-f3r mentioned this pull request Mar 11, 2022

[2/2] Implement macro meta-variable expression #94833

Merged

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Mar 12, 2022

Rollup merge of rust-lang#94833 - c410-f3r:meta-take-2, r=petrochenkov

8c87132

[2/2] Implement macro meta-variable expression Final part of rust-lang#93545 (comment) r? `@petrochenkov`

c410-f3r mentioned this pull request Mar 12, 2022

Fix remaining meta-variable expression TODOs #94884

Merged

c410-f3r mentioned this pull request Mar 15, 2022

Tracking Issue for RFC 3086: macro metavariable expressions #83527

Open

4 tasks

c410-f3r mentioned this pull request Apr 9, 2022

Stabilize $$ in Rust 1.63.0 #95860

Merged

c410-f3r mentioned this pull request Mar 21, 2024

Stabilize count, ignore, index, and len (macro_metavar_expr) #122808

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement macro meta-variable expressions #93545

Implement macro meta-variable expressions #93545

c410-f3r commented Feb 1, 2022 •

edited

Loading

c410-f3r left a comment

This comment has been minimized.

joshtriplett commented Feb 1, 2022

petrochenkov commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

c410-f3r commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

c410-f3r commented Feb 5, 2022

mark-i-m commented Feb 6, 2022

c410-f3r commented Feb 6, 2022

mark-i-m left a comment •

edited

Loading

c410-f3r commented Feb 11, 2022

mark-i-m left a comment

mark-i-m commented Feb 22, 2022

c410-f3r commented Feb 22, 2022

petrochenkov commented Feb 24, 2022

c410-f3r commented Feb 24, 2022 •

edited

Loading

petrochenkov commented Feb 25, 2022

c410-f3r commented Feb 25, 2022 •

edited

Loading

Implement macro meta-variable expressions #93545

Implement macro meta-variable expressions #93545

Conversation

c410-f3r commented Feb 1, 2022 • edited Loading

c410-f3r left a comment

Choose a reason for hiding this comment

This comment has been minimized.

joshtriplett commented Feb 1, 2022

petrochenkov commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

c410-f3r commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

petrochenkov commented Feb 5, 2022

c410-f3r commented Feb 5, 2022

mark-i-m commented Feb 6, 2022

c410-f3r commented Feb 6, 2022

mark-i-m left a comment • edited Loading

Choose a reason for hiding this comment

c410-f3r commented Feb 11, 2022

mark-i-m left a comment

Choose a reason for hiding this comment

mark-i-m commented Feb 22, 2022

c410-f3r commented Feb 22, 2022

petrochenkov commented Feb 24, 2022

c410-f3r commented Feb 24, 2022 • edited Loading

petrochenkov commented Feb 25, 2022

c410-f3r commented Feb 25, 2022 • edited Loading

c410-f3r commented Feb 1, 2022 •

edited

Loading

mark-i-m left a comment •

edited

Loading

c410-f3r commented Feb 24, 2022 •

edited

Loading

c410-f3r commented Feb 25, 2022 •

edited

Loading