Stabilize count, ignore, index, and len (macro_metavar_expr)

[c410-f3r]

Stabilization proposal

This PR proposes the stabilization of a subset of #![feature(macro_metavar_expr)] or more specifically, the stabilization of count, ignore, index and len.

Tracking issue: #83527
Version: 1.80 (June 13 2024 => beta, July 25 2024 => stable).

What is stabilized

Count

The number of times a meta variable repeats in total.

The output of count depends on where it is placed as well the provided index. If no index is provided, then it will always start at the innermost level.

macro_rules! count_idents {
    ( $( $i:ident ),* ) => {
        ${count($i)}
    };
}

fn main() {
    assert_eq!(count_idents!(a, b, c), 3);
}

Ignore

Binds a meta variable for repetition, but expands to nothing.

macro_rules! count {
    ( $( $i:stmt ),* ) => {{
        0 $( + 1 ${ignore($i)} )*
    }};
}

fn main() {
    assert_eq!(count!(if true {} else {}, let _: () = (), || false), 3);
}

Index

The current index of the inner-most repetition.

index is a counter that starts from 0 until the end of the repetition.

trait Foo {
    fn bar(&self) -> usize;
}

macro_rules! impl_tuple {
    ( $( $name:ident ),* ) => {
        impl<$( $name, )*> Foo for ($( $name, )*)
        where
            $( $name: AsRef<[u8]>, )*
        {
            fn bar(&self) -> usize {
                let mut sum: usize = 0;
                $({
                    const $name: () = ();
                    sum = sum.wrapping_add(self.${index()}.as_ref().len());
                })*
                sum
            }
        }
    };
}

impl_tuple!(A, B, C, D);

fn main() {
}

len

The current index starting from the inner-most repetition.

len indicates the sum of meta variable elements, aka length, of the current repetition.

macro_rules! array_3d {
    ( $( $( $number:literal ),* );* ) => {
        [
            $(
                [
                    $( $number + ${len()}, )*
                ],
            )*
        ]
    };
}

fn main() {
    assert_eq!(array_3d!(0, 1; 2, 3; 4, 5), [[2, 3], [4, 5], [6, 7]]);
}

Motivation

Meta variable expressions not only facilitate the use of macros but also allow things that can't be done today like in the $index example.

An initial effort to stabilize this feature was made in #111908 but ultimately reverted because of possible obstacles related to syntax and expansion.

Nevertheless, #83527 (comment) tried to address some questions and fortunately the lang team accept #117050 the unblocking suggestions.

Here we are today after ~4 months so everything should be mature enough for wider use.

After RFC changes

In order to unblock progress, some changes were applied in #118958. These changes did not trigger any opposition debates and were quickly integrated.

1. The depth direction of count changed from outermost-to-innermost to innermost-to-outermost.
2. Metavariables were originally referenced by their names like in count(some_metavariable) but now they must be preceded by $. For example, count($some_metavariable).

What isn't stabilized

$$ is not being stabilized due to unresolved concerns.

History

• 2021-02-22, RFC: Declarative macro metavariable expressions
• 2021-03-26, Tracking Issue for RFC 3086: macro metavariable expressions
• 2022-02-01, Implement macro meta-variable expressions
• 2022-02-25, [1/2] Implement macro meta-variable expressions
• 2022-03-11, [2/2] Implement macro meta-variable expressions
• 2022-03-12, Fix remaining meta-variable expression TODOs
• 2019-03-21, [macro-metavar-expr] Fix generated tokens hygiene
• 2022-04-07, Kickstart the inner usage of macro_metavar_expr
• 2022-04-07, [macro_metavar_expr] Add tests to ensure the feature requirement
• 2023-05-24, [RFC-3086] Restrict the parsing of count
• 2023-10-22, [RFC 3086] Attempt to try to resolve blocking concerns

Tests

This list is a subset of https://github.com/rust-lang/rust/tree/master/tests/ui/macros/rfc-3086-metavar-expr.

• Ensures that nested macros have correct behavior

• Compares produced tokens to assert expected outputs

• Checks the declarations of the features

• Verifies all possible errors that can occur due to incorrect user input

Possible future work

With enough consensus, other nightly expressions can be added for experimentation and possibly stabilized in the future. For example, #118958.

Thanks @markbt for creating the RFC and thanks to @petrochenkov and @mark-i-m for reviewing the implementations.

[rustbot]

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[fmease]

@c410-f3r Has this been documented in the Reference already? The box is not ticked over at the tracking issue. If not, stabilization is blocked on adding sufficient documentation.

[joshtriplett]

@rfcbot merge

@rfcbot concern document-in-reference

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

Concerns:

• depth levels are confusing (Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment))
• document-in-reference resolved by Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment)
• hygiene-handled? resolved by Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[RalfJung]

I tend to agree with @tgross35 -- having to refer to capture groups by index is a really awkward and fragile API, I don't think we should stabilize that if better alternatives exist. After all we generally let the user name things they need to refer to (variables, functions, types, even loops to break out of) instead of using opaque numbers and counting.

[workingjubilee]

Is there consensus for ${ignore($var)} and ${count($var)} at least? Those would make certain patterns much easier to handle.

[RalfJung]

ignore seems like a no-brainer to me. (Not sure this is the best possible name, but I can't come up with a better one either.)

count does seem to have very surprising interaction with repetitions. Judging from the reference PR:

macro_rules! count_value_nested {
    ( $( $name:ident: $( $value:literal ),* );+ ) => {
        [ $(
            // using `count` within a repetition group will return the number
            // of times `$value` is matched _within that group_.
            ${count($value)},
        )+ ]
    };
}

So, this is a repetition group that only contains $value. And somehow that becomes a once-per-$name repetition? That's not how repetition behaves anywhere else, is it? I would have expected that I must use $name somewhere in this repetition group to make this a once-per-$name group, and then I can count the $value in it -- the argument of $count would not be part of the automatic detection of the repetition group in my mental model, that has to already be clear without.

Which repetition group does this count?

macro_rules! count_test {
    ( $( $supername:ident { $( $name:ident: $( $value:ident )* ),+ } )* ) => {
        [ $(
            ${count($value)},
        )+ ]
    };
}

This could reasonably be for i in supername { i.total_numer_of_value() } but also for i in supername { for j in i.name { j.total_numer_of_value() } }.

[RalfJung]

Trying this out, it seems like like it is the former, counting the total number of value for each supername.

Usually a metavar "must appear in exactly the same number, kind, and nesting order of repetitions in the transcriber as it did in the matcher". With count that seems to be different. Is it something like "must appear in strictly fewer repetitions than it did in the matcher, and then it counts the number of inner repetitions in the current instance of the outer repetitions it occurs in"?

[workingjubilee]

looks kinda like group.iter().flat_map(|expansion| expansion.get(metavar)).count()

[c410-f3r]

As previously commented, the suggestion around named placeholders is orthogonal to the current behavior approved in the RFC and both can co-exist in the future.

Fate imposed yet another syntax-related blocking concern in a stabilization PR, as such, I ask the responsible party to decide the final outcome.

@rustbot labels +I-lang-nominated

[tmandry]

@rfcbot concern depth levels are confusing

Before reading this thread I had a strong reaction to the RFC exactly in line with @RalfJung's, and I had thoughts along the lines of what @tgross35 is proposing. So I think there's something worth paying attention to here. I see it as a major downside, quite possibly a failure, of the lang team process that we got this far without such syntax concerns being surfaced. With that said, even though the concern is a late breaking one, it feels significant enough to me that I don't think it should be minimized.

Rust's declarative macro syntax can already be quite intimidating to the uninitiated, and we should be careful in how we spend our "readability budget" here. This emphasis on readability is the reason I'm not that swayed by the argument of future compatibility: Even if we add named expansions later, the syntax for unnamed depth levels will always exist if we stabilize it now, and people who read Rust macro code will have to know about it.

To make my concern more concrete: I think adding recursion depths is a significant step up in both capability and complexity, but I would want that to be equally matched with stronger usability as well as motivation for the feature. The usability piece is quite lacking, in my opinion, in that the recursion depths are unlabeled integers without an obvious meaning. I certainly think they are non-obvious to the reader, and in some cases even to the writer who already knows about this feature (I speculate that many users might forget how count works with nested metavars and depths, even after using it a few times, because it doesn't seem to work like anything else in the language). I also don't see a motivation for these specific capabilities laid out in the RFC or the stabilization report.

---

In the interest of making forward progress, I'll put forward a proposal that I frame as a question. The implied question in each of these is how much the proposal limits the use cases enabled by these features. If the answer is "not that much" I think we should go forward with stabilizing this subset for now; otherwise, I hope it's a useful starting point for getting things moving.

• Can the stabilization proposal be subsetted not to support depth levels on any of these operators?
• Can count specifically be limited to supporting only metavars with a single recursion level?
  • If that's too limiting: What is the motivation for counting all nested expansions instead of only counting the outermost level (the one that would be expanded in the next use of $()*)? If hypothetically we switched it to instead only count the outermost level (without supporting explicit depths but holding open the possibility to support them in the future), would that enable significantly more use cases? edit: Nevermind, see Stabilize count, ignore, index, and len (macro_metavar_expr) #122808 (comment)

[RalfJung]

If that's too limiting: What is the motivation for counting all nested expansions instead of only counting the outermost level (the one that would be expanded in the next use of $()*)?

I don't exactly know what you mean here, could you give an example?

FWIW, I experimented with count a bit and here are the results. Without named repetitions, I don't think there is another interpretation that makes sense. Possibly there is a subset that makes sense, like only supporting count($x) when x is exactly one level inside the current nesting -- but it's not clear that that actually makes it more understandable.

The only point that gives me pause around count is -- if/when we have named repetition groups, how will they work with count, and is that compatible with today's behavior?

[tmandry]

I don't exactly know what you mean here, could you give an example?

What I meant is, why not make count($x) equivalent to count($x, 1) instead of count($x, ∞)? We would then have to invent syntax for the latter if we wanted to preserve expressiveness, but this default aligns more with my expectations.

[RalfJung]

AFAIK it is currently equivalent to count($x, 0), which does align with my intuition once I re-familiarized myself with how nested capture groups work. I don't know what infinite depth would even mean.

[tmandry]

I'm not sure what count(x, 0) would mean, based on this from the RFC:

If repetitions are nested, then an optional depth parameter can be used to limit the number of nested repetitions that are counted. For example, a macro expansion like:

${count(x, 1)} ${count(x, 2)} ${count(x, 3)} $( a $( b $( $x )* )* )*

The three values this expands to are the number of outer-most repetitions (the number of times a would be generated), the sum of the number of middle repetitions (the number of times b would be generated), and the total number of repetitions of $x.

Reading this over again though, I no longer think it makes sense to limit to 1 by default. ${count(x)} should of course count the number of expansions for $x, not the number of expansions for the next-expandable metavariable that $x happens to be nested in. We could be conservative in only allowing it in the same contexts where $x can already be expanded, but I can see it being useful outside of those contexts.

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

 ${count(a)} ${count(b)} ${count(x)} $( a $( b $( $x )* )* )*

[RalfJung]

I'm not sure what count(x, 0) would mean, based on this from the RFC:

I have linked this already above, see here. That's what I reverse engineered based on my experiments, anyway. The default is definitely 0, according to that PR.

Maybe the semantics changed since the RFC was accepted?

Would be good to get an up=to-date description of the proposed semantics confirmed by the people involved in the implementation -- Cc @c410-f3r , not sure who else?

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

I think the issue is around matchers like $( [[ $( {{ $( $x, )* }} )* ]] )* where there is no variable in the outer repetition groups that one can refer to.

[c410-f3r]

I will try to address all questions this weekend in my free time.

cc rust-lang/reference#1485 (comment)

[c410-f3r]

I see it as a major downside, quite possibly a failure, of the lang team process that we got this far without such syntax concerns being surfaced.

Thank you. macro_metavar_expr has been around for +3 years and its implementation has been available for +2 years. Although part of the game, it is surprising and demotivating to see syntax concerns in stabilizing PRs (#95860).

The only point that gives me pause around count is -- if/when we have named repetition groups, how will they work with count, and is that compatible with today's behavior?

It is more complex but not impossible IMO. One way is to attach a corresponding matcher depth to each named repetition and then evaluate which ones can be associated based on the current transcriber position.

Syntax-wise, the distinction between numbers and strings can create appropriated implementation branches.

What I meant is, why not make count($x) equivalent to count($x, 1) instead of count($x, ∞)? We would then have to invent syntax for the latter if we wanted to preserve expressiveness, but this default aligns more with my expectations.

AFAIK it is currently equivalent to count($x, 0),

Before #117050 count($x) was equal to count($x, N) with N being the number of nested repetitions minus one. Now count($x) is equal to count($x, 0) which can also be seen as the innermost repetition.

rust-lang/reference#1485 (comment) will probably provide a better overview.

I still think specifying a max depth level is non-obvious and of dubious value, since you could always rewrite the excerpt above as something like the following; are there cases where you can't do this?

${count(a)} ${count(b)} ${count(x)} $( a $( b $( $x )* )* )*

Unfortunately it is not possible to count without a metavariable. AFAICT, a and b are just references used in the RFC to illustrative how the algorithm works.

But yeah, the specification of a depth is non-obvious as it depends on the number of nested matcher repetitions as well as the positioning.

the people involved in the implementation

@markbt, @petrochenkov and @mark-i-m

[tmandry]

I see, I was confused by reading the RFC for more detail, but that is now outdated. @c410-f3r Can you please include a list of differences from the RFC in the stabilization report (with rationale; feel free to summarize and link to more context)?

[c410-f3r]

I see, I was confused by reading the RFC for more detail, but that is now outdated. @c410-f3r Can you please include a list of differences from the RFC in the stabilization report (with rationale; feel free to summarize and link to more context)?

Sure. The report has been updated.

[alex-semenyuk]

@c410-f3r
From triage team. Does it ready for review? if so could you please fix merge conflicts before

[c410-f3r]

Well, I can rebase but I don't think it will make much of a difference. The actual issue lies within the decision around the use of indices in count, index and len.

This PR has been waiting for more than 2 months for a response from the responsible team. AFAICT, all questions posted here as well as in rust-lang/reference#1485 have already been answered to the best of my ability.