Simplifying parsing - eliminating EOI

[ZachJHansen]

Currently the grammar and parsing relies on "something_eoi" rules to parse "something." This could be simplified by adding "~ !ANY" to the end of "something" rules and eliminating the EOI rules. See the definition of keywords in the FOL grammar for an example.

[janheuer]

I have tried this approach a bit now, only for the ASP grammar, see the branch jan/parsing_eoi. Here are my findings:

• I have kept the EOI rules for now but just started with replacing EOI in them with !ANY
• This causes all ASP parsing test to fail
• I think the call pairs.next_back() in parsing/mod.rs line 51 needs to be removed then, as there is no EOI anymore and the comment in that line states that the purpose is to remove EOI
• Then almost all parsing test for ASP pass, only parse_program fails
• For parse_program it fails because of the test case "a.\n". Here the trailing newline is not parsed anymore.
• If the program rule still uses EOI instead of !ANY this would work again if the pairs.next_back() is added again which of course causes all the component tests to fail again. I'm not sure if it would be possible to only have this call when parsing a program?

[teiesti]

Your remarks make me confident that we can actually implement this. Nice work.

I think the call pairs.next_back() in parsing/mod.rs line 51 needs to be removed then, as there is no EOI anymore and the comment in that line states that the purpose is to remove EOI

FYI, this was the last piece I was missing. Good catch.

Then almost all parsing test for ASP pass, only parse_program fails

Once you integrate

program = { rule* }
program_eoi = _{ program ~ !ANY }

into

program = { rule* ~ !ANY }

and adjust the rule used in the ProgramParser, the problem vanishes. I think the reason for this behavior is hidden somewhere in here. Maybe you can think about this. I have pushed my changes to tobias/parsing_eoi so that you have the details.

I have kept the EOI rules for now but just started with replacing EOI in them with !ANY

Can you finish all the chances for the ASP and FOL parsing and prepare a pull request? This includes removing all the EOI rules.

[janheuer]

I was able to remove the EOI rules for all the top-level rules, i.e. program, theory, specification and user guide. However I can not remove them for all the other rules. As it is right now we have e.g. a rule for formula and then the formula_eoi rule. The formula rule is the one that is then used to define theories and merging formula_eoi and formula breaks the test for program as now theory is

{ (formula ~ !ANY ~ ".")* ~ !ANY }

as opposed to what we have with the separate rules

{ (formula ~ ".")* ~ !ANY }