Merge pull request #71 from tunnelvisionlabs/update-ref

Update code to the 4.9 release of ANTLR 4

antlr4-maven-plugin/pom.xml

@@ -13,7 +13,7 @@
     <parent>
         <groupId>com.tunnelvisionlabs</groupId>
         <artifactId>antlr4-master</artifactId>
-        <version>4.7.5-SNAPSHOT</version>
+        <version>4.9.0-SNAPSHOT</version>
     </parent>
 
     <artifactId>antlr4-maven-plugin</artifactId>
@@ -184,7 +184,7 @@
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-plugin-plugin</artifactId>
-                <version>3.3</version>
+                <version>3.6.0</version>
                 <configuration>
                     <!-- see http://jira.codehaus.org/browse/MNG-5346 -->
                     <skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>

antlr4-testgen-maven-plugin/pom.xml

@@ -9,7 +9,7 @@
     <parent>
         <groupId>com.tunnelvisionlabs</groupId>
         <artifactId>antlr4-master</artifactId>
-        <version>4.7.5-SNAPSHOT</version>
+        <version>4.9.0-SNAPSHOT</version>
     </parent>
 
     <artifactId>antlr4-testgen-maven-plugin</artifactId>

contributors.txt

@@ -1,5 +1,9 @@
 ANTLR Project Contributors Certification of Origin and Rights
 
+NOTE: This tool is mature and Terence is mostly occupied elsewhere. We
+can't accept any changes that could have widespread effects on thousands
+of existing projects. Sorry!
+
 All contributors to ANTLR v4 must formally agree to abide by this
 certificate of origin by signing on the bottom with their github
 userid, full name, email address (you can obscure your e-mail, but it
@@ -204,6 +208,9 @@ YYYY/MM/DD, github id, Full name, email
 2018/06/16, EternalPhane, Zongyuan Zuo, [email protected]
 2018/07/03, jgoppert, James Goppert, [email protected]
 2018/07/27, Maksim Novikov, [email protected]
+2018/08/03, ENDOH takanao, [email protected]
+2018/10/08, xsIceman, Andreas Skaar, [email protected]
+2018/10/18, edirgarcia, Edir García Lazo, [email protected]
 2018/07/31, Lucas Henrqiue, [email protected]
 2018/08/03, ENDOH takanao, [email protected]
 2018/10/29, chrisaycock, Christopher Aycock, chris[at]chrisaycock[dot]com
@@ -213,3 +220,64 @@ YYYY/MM/DD, github id, Full name, email
 2018/11/29, hannemann-tamas, Ralf Hannemann-Tamas, [email protected]
 2018/12/20, WalterCouto, Walter Couto, [email protected]
 2018/12/23, youkaichao, Kaichao You, [email protected]
+2019/01/02, wkhemir, Wail Khemir, [email protected]
+2019/01/16, kuegi, Markus Zancolo, [email protected]
+2019/02/06, ralucado, Cristina Raluca Vijulie, ralucris.v[at]gmail[dot]com
+2019/02/23, gedimitr, Gerasimos Dimitriadis, [email protected]
+2019/03/13, base698, Justin Thomas, [email protected]
+2019/03/18, carlodri, Carlo Dri, [email protected]
+2019/05/02, askingalot, Andy Collins, [email protected]
+2019/05/13, mapio, Massimo Santini, [email protected]
+2019/07/11, olowo726, Olof Wolgast, [email protected]
+2019/07/16, abhijithneilabraham, Abhijith Neil Abraham, [email protected]
+2019/07/26, Braavos96, Eric Hettiaratchi, [email protected]
+2019/08/02, thron7, Thomas Herchenroeder, thron7[at]users[dot]sourceforge[dot]net
+2019/08/12, easonlin404, Eason Lin, [email protected]
+2019/08/23, akaJes, Oleksandr Mamchyts, [email protected]
+2019/08/27, wurzelpeter, Markus Franke, markus[hyphen]franke[at]web[dot]de
+2019/09/10, ImanHosseini, Iman Hosseini, [email protected]
+2019/09/03, João Henrique, [email protected]
+2019/09/10, neko1235, Ihar Mokharau, [email protected]
+2019/09/10, yar3333, Yaroslav Sivakov, [email protected]
+2019/09/10, marcospassos, Marcos Passos, [email protected]
+2019/09/10, amorimjuliana, Juliana Amorim, [email protected]
+2019/09/15, sullis, Sean Sullivan, [email protected]
+2019/09/17, kaz, Kazuki Sawada, [email protected]
+2019/09/28, lmy269, Mingyang Liu, [email protected]
+2019/10/29, tehbone, Tabari Alexander, [email protected]
+2019/10/31, a-square, Alexei Averchenko, [email protected]
+2019/11/05, listba, Ben List, [email protected]
+2019/11/11, foxeverl, Liu Xinfeng, liuxf1986[at]gmail[dot]com
+2019/11/17, felixn, Felix Nieuwenhuizhen, [email protected]
+2019/11/18, mlilback, Mark Lilback, [email protected]
+2020/01/19, lingyv-li, Lingyu Li, [email protected]
+2020/02/02, carocad, Camilo Roca, [email protected]
+2020/02/10, julibert, Julián Bermúdez Ortega, [email protected]
+2020/02/17, quantumsheep, Nathanael Demacon, [email protected]
+2020/02/21, StochasticTinkr, Daniel Pitts, [email protected]
+2020/03/17, XsongyangX, Song Yang, [email protected]
+2020/04/07, deniskyashif, Denis Kyashif, [email protected]
+2020/04/08, lwehmeier, Leon Wehmeier, [email protected]
+2020/04/10, agrabski, Adam Grabski, [email protected]
+2020/04/23, martinvw, Martin van Wingerden, [email protected]
+2020/04/30, TristonianJones, Tristan Swadell, [email protected]
+2020/05/06, iammosespaulr, Moses Paul R, [email protected]
+2020/05/10, gomerser, Erik Gomersbach, [email protected]
+2020/05/22, keywan-ghadami-oxid, Keywan Ghadami, [email protected]
+2020/05/25, graknol, Sindre van der Linden, [email protected]
+2020/05/31, d-markey, David Markey, [email protected]
+2020/06/02, cohomology, Kilian Kilger, kkilger AT gmail.com
+2020/06/04, IohannRabeson, Iohann Rabeson, [email protected]
+2020/06/04, sigmasoldi3r, Pablo Blanco, [email protected]
+2020/07/01, sha-N, Shan M Mathews, [email protected]
+2020/08/22, stevenjohnstone, Steven Johnstone, [email protected]
+2020/09/06, ArthurSonzogni, Sonzogni Arthur, [email protected]
+2020/09/10, Khailian, Arunav Sanyal, [email protected]
+2020/09/12, Clcanny, Charles Ruan, [email protected]
+2020/09/15, rmcgregor1990, Robert McGregor, [email protected]
+2020/09/16, trenki2, Markus Trenkwalder, trenki2[at]gmx[dot]net
+2020/10/08, Marti2203, Martin Mirchev, [email protected]
+2020/10/16, adarshbhat, Adarsh Bhat, [email protected]
+2020/10/20, adamwojs, Adam Wójs, adam[at]wojs.pl
+2020/10/24, cliid, Jiwu Jang, [email protected]
+2020/11/05, MichelHartmann, Michel Hartmann, [email protected]

doc/actions.md

@@ -86,7 +86,7 @@ Using a rule label looks like this:
 returnStat : 'return' e=expr {System.out.println("matched "+e.text);} ;
 ```
 
-You can also use `$ followed by the name of the attribute to access the value associated with the currently executing rule. For example, `$start` is the starting token of the current rule.
+You can also use `$` followed by the name of the attribute to access the value associated with the currently executing rule. For example, `$start` is the starting token of the current rule.
 
 ```
 returnStat : 'return' expr {System.out.println("first token "+$start.getText());} ;

doc/case-insensitive-lexing.md

@@ -16,7 +16,7 @@ For the 4.7.1 release, we discussed both approaches in [detail](https://github.c
 
 ## Case-insensitive grammars
 
-As a prime example of a grammar that specifically describes case insensitive keywords, see the 
+As a prime example of a grammar that specifically describes case insensitive keywords, see the
 [SQLite grammar](https://github.com/antlr/grammars-v4/blob/master/sqlite/SQLite.g4).  To match a case insensitive keyword, there are rules such as
 
 ```
@@ -74,4 +74,8 @@ Lexer lexer = new SomeSQLLexer(upper);
 
 Here are implementations of `CaseChangingCharStream` in various target languages:
 
-* [Java](https://github.com/parrt/antlr4/blob/case-insensitivity-doc/doc/resources/CaseChangingCharStream.java)
+* [C#](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingCharStream.cs)
+* [Go](https://github.com/antlr/antlr4/blob/master/doc/resources/case_changing_stream.go)
+* [Java](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingCharStream.java)
+* [JavaScript](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingStream.js)
+* [Python2/3](https://github.com/antlr/antlr4/blob/master/doc/resources/CaseChangingStream.py)

doc/creating-a-language-target.md

@@ -11,12 +11,14 @@ Creating a new target involves the following key elements:
 1. For the tool, create class *X*Target as a subclass of class `Target` in package `org.antlr.v4.codegen.target`. This class describes language specific details about escape characters and strings and so on. There is very little to do here typically.
 1. Create *X*.stg in directory tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg. This is a [StringTemplate](http://www.stringtemplate.org/) group file (`.stg`) that tells ANTLR how to express all of the parsing elements needed to generate code.  You will see templates called `ParserFile`, `Parser`, `Lexer`, `CodeBlockForAlt`, `AltBlock`, etc... Each of these must be described how to build the indicated chunk of code. Your best bet is to find the closest existing target, copy that template file, and tweak to suit.
 1. Create a runtime library to support the parsers generated by ANTLR. Under directory runtime/*X*, you are in complete control of the directory structure as dictated by common usage of that target language. For example, Java has: `runtime/Java/lib` and `runtime/Java/src` directories. Under `src`, you will find a directory structure for package `org.antlr.v4.runtime` and below.
-1. Create a template file for runtime tests. All you have to do is provide a few simple templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code in a new target and check the results. All it needs to know is how to generate a test rig (i.e., a `main` program), how to define various class fields, compare members and so on. You must create a *X* directory underneath `runtime-testsuite/resources/org/antlr/v4/test/runtime`. Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
+1. Create a template file for runtime tests. All you have to do is provide a few templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code using these templates for each target and check the test results.  It needs to know how to define various class fields, compare members and so on. You must create a *X*.test.stg file underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime). Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
+1. Create test files under [/runtime-testsuite/test/org/antlr/v4/test/runtime](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime). They will load defined test cases in each test descriptor. Also add the `/runtime-testsuite/test/org/antlr/v4/test/runtime/X/BaseXTest.java` which defines how test cases will execute and output.
+1. Create/edit shell scripts in [/.travis](https://github.com/antlr/antlr4/blob/master/.travis) and [/appveyor.yml](https://github.com/antlr/antlr4/blob/master/appveyor.yml) to run tests in CI pipelines.
 
 ## Getting started
 
 1. Fork the `antlr/antlr4` repository at github to your own user so that you have repository `username/antlr4`.
-2. Clone `username/antlr4`, forked repository, to your local disk.  Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/).
+2. Clone `username/antlr4`, the forked repository, to your local disk.  Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/).
 3. Try to build it before doing anything
 ```bash
 $ mvn compile

doc/faq/index.md

@@ -38,8 +38,8 @@ This is the main landing page for the ANTLR 4 FAQ. The links below will take you
 
 ## Translation
 
-* [ASTs vs parse trees](parse-trees.md)
-* [Decoupling input walking from output generation](parse-trees.md)
+* [ASTs vs parse trees](translation.md)
+* [Decoupling input walking from output generation](translation.md)
 
 ## Actions and semantic predicates
 

doc/faq/parse-trees.md

@@ -50,7 +50,7 @@ For writing a compiler, either generate [LLVM-type static-single-assignment](htt
 
 ### XPath
 
-XPath works great when you need to find specific nodes, possibly in certain contexts. The context is limited to the parents on the way to the root of the tree. For example, if you want to find all ID nodes, use path `//ID`. If you want all variable declarations, you might use path `//vardecl`.  If you only want fields declarations, then you can use some context information via path `/classdef/vardecl`, which would only find vardecls that our children of class definitions. You can merge the results of multiple XPath `findAll()`s simulating a set union for XPath. The only caveat is that the order from the original tree is not preserved when you union multiple `findAll()` sets.
+XPath works great when you need to find specific nodes, possibly in certain contexts. The context is limited to the parents on the way to the root of the tree. For example, if you want to find all ID nodes, use path `//ID`. If you want all variable declarations, you might use path `//vardecl`.  If you only want fields declarations, then you can use some context information via path `/classdef/vardecl`, which would only find vardecls that are children of class definitions. You can merge the results of multiple XPath `findAll()`s simulating a set union for XPath. The only caveat is that the order from the original tree is not preserved when you union multiple `findAll()` sets.
 
 ### Tree pattern matching
 
@@ -70,4 +70,4 @@ scopeStack.peek().define(new VariableSymbol("foo"))
 
 That way each listener function does not have to compute its appropriate scope.
 
-Examples: [DefScopesAndSymbols.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/DefScopesAndSymbols.java) and [SetScopeListener.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/SetScopeListener.java) and [VerifyListener.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/VerifyListener.java)
+Examples: [DefScopesAndSymbols.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/DefScopesAndSymbols.java) and [SetScopeListener.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/SetScopeListener.java) and [VerifyListener.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/VerifyListener.java)

doc/getting-started.md

@@ -8,46 +8,46 @@ Hi and welcome to the version 4 release of ANTLR! It's named after the fearless
 
 ANTLR is really two things: a tool that translates your grammar to a parser/lexer in Java (or other target language) and the runtime needed by the generated parsers/lexers. Even if you are using the ANTLR Intellij plug-in or ANTLRWorks to run the ANTLR tool, the generated code will still need the runtime library. 
 
-The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers.  In what follows, I talk about antlr-4.7.1-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3).
+The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers.  In what follows, I talk about antlr-4.9-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3).
 
-If you are going to integrate ANTLR into your existing build system using mvn, ant, or want to get ANTLR into your IDE such as eclipse or intellij, see Integrating ANTLR into Development Systems.
+If you are going to integrate ANTLR into your existing build system using mvn, ant, or want to get ANTLR into your IDE such as eclipse or intellij, see [Integrating ANTLR into Development Systems](https://github.com/antlr/antlr4/blob/master/doc/IDEs.md).
 
 ### UNIX
 
-0. Install Java (version 1.6 or higher)
+0. Install Java (version 1.7 or higher)
 1. Download
 ```
 $ cd /usr/local/lib
-$ curl -O https://www.antlr.org/download/antlr-4.7.1-complete.jar
+$ curl -O https://www.antlr.org/download/antlr-4.9-complete.jar
 ```
 Or just download in browser from website:
     [https://www.antlr.org/download.html](https://www.antlr.org/download.html)
 and put it somewhere rational like `/usr/local/lib`.
 
-2. Add `antlr-4.7.1-complete.jar` to your `CLASSPATH`:
+2. Add `antlr-4.9-complete.jar` to your `CLASSPATH`:
 ```
-$ export CLASSPATH=".:/usr/local/lib/antlr-4.7.1-complete.jar:$CLASSPATH"
+$ export CLASSPATH=".:/usr/local/lib/antlr-4.9-complete.jar.jar:$CLASSPATH"
 ```
 It's also a good idea to put this in your `.bash_profile` or whatever your startup script is.
 
 3. Create aliases for the ANTLR Tool, and `TestRig`.
 ```
-$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.7.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool'
-$ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.7.1-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'
+$ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.9-complete.jar.jar:$CLASSPATH" org.antlr.v4.Tool'
+$ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.9-complete.jar.jar:$CLASSPATH" org.antlr.v4.gui.TestRig'
 ```
 
 ### WINDOWS
 
 (*Thanks to Graham Wideman*)
 
-0. Install Java (version 1.6 or higher)
-1. Download antlr-4.7.1-complete.jar (or whatever version) from [https://www.antlr.org/download/](https://www.antlr.org/download/)
+0. Install Java (version 1.7 or higher)
+1. Download antlr-4.9-complete.jar (or whatever version) from [https://www.antlr.org/download/](https://www.antlr.org/download/)
 Save to your directory for 3rd party Java libraries, say `C:\Javalib`
-2. Add `antlr-4.7.1-complete.jar` to CLASSPATH, either:
+2. Add `antlr-4.9-complete.jar.jar` to CLASSPATH, either:
   * Permanently: Using System Properties dialog > Environment variables > Create or append to `CLASSPATH` variable
   * Temporarily, at command line:
 ```
-SET CLASSPATH=.;C:\Javalib\antlr-4.7.1-complete.jar;%CLASSPATH%
+SET CLASSPATH=.;C:\Javalib\antlr-4.9-complete.jar;%CLASSPATH%
 ```
 3. Create short convenient commands for the ANTLR Tool, and TestRig, using batch files or doskey commands:
   * Batch files (in directory in system PATH) antlr4.bat and grun.bat
@@ -73,7 +73,7 @@ Either launch org.antlr.v4.Tool directly:
 
 ```
 $ java org.antlr.v4.Tool
-ANTLR Parser Generator Version 4.7.1
+ANTLR Parser Generator Version 4.9
 -o ___ specify output directory where all output is generated
 -lib ___ specify location of .tokens files
 ...
@@ -82,8 +82,8 @@ ANTLR Parser Generator Version 4.7.1
 or use -jar option on java:
 
 ```
-$ java -jar /usr/local/lib/antlr-4.7.1-complete.jar
-ANTLR Parser Generator Version 4.7.1
+$ java -jar /usr/local/lib/antlr-4.9-complete.jar
+ANTLR Parser Generator Version 4.9
 -o ___ specify output directory where all output is generated
 -lib ___ specify location of .tokens files
 ...
@@ -114,8 +114,11 @@ Now test it:
 
 ```
 $ grun Hello r -tree
+(Now enter something like the string below)
 hello parrt
+(now,do:)
 ^D
+(The output:)
 (r hello parrt)
 (That ^D means EOF on unix; it's ^Z in Windows.) The -tree option prints the parse tree in LISP notation.
 It's nicer to look at parse trees visually.
@@ -137,3 +140,4 @@ The book has lots and lots of examples that should be useful too. You can downlo
 Also, there is a large collection of grammars for v4 at github:
 
 [https://github.com/antlr/grammars-v4](https://github.com/antlr/grammars-v4)
+/