Merge pull request #3 from mpsijm/rewrite-lexer

Rewrite lexer

README.md

@@ -1,5 +1,5 @@
-# Spoofax Latex tools
-> A collection of tools for writing about Spoofax in a Latex document
+# Spoofax LaTeX tools
+> A collection of tools for writing about Spoofax in a LaTeX document
 
 
 ## sg2tikz
@@ -26,53 +26,66 @@ Can be executed using `python sg2tikz.py sg.txt`, with the contents of `sg.txt`
 
 ## Spoofax-pygments
 A [Pygments](http://pygments.org/) lexer implementation for Spoofax languages.
-These lexers can be invoked within a `minted`-environment in Latex to color code snippets of Spoofax languages.
+This lexer can be invoked within a `minted`-environment in LaTeX to color code snippets of Spoofax languages.
 
-New lexers can easily be added by providing a parsetable and a mapping from the parsetree-nodes to a Pygments token type.
-For a full list of available tokens, see [this page](http://pygments.org/docs/tokens/#keyword-tokens).
-If no token type could be found for a parsetree-node, the type of its parent is used recursively.
+### Usage
+> I have not been able to get this to work on Overleaf. Presumably, they do not have Java installed.
 
+Add the files `spoofax-pygmentize-core.jar` and `spoofax_lexer.py` in the source root of your LaTeX project.
+You can download these files on the releases page, or take the Python file from the source and generate the JAR yourself by running `./release.sh` in the `spoofax-pygments/`-folder.
+After copying the files, you can use the `spoofax_lexer` in the following way inside your LaTeX document.
+Make sure to run the LaTeX compiler using `-shell-escape` for Minted lexers to work at all.
 
-Available lexers:
-- `DynamixLexer`
-- `FrameVMLexer`
-- `SDF3Lexer`
+```latex
+\documentclass{minimal}
+\usepackage{minted}
+\begin{document}
 
-### Usage
-> I have not been able to get this to work on Overleaf. Presumably they do not have Java installed
+\begin{figure}
+  \begin{minted}{spoofax_lexer.py -O "language=org/metaborg/org.metaborg.meta.lang.template" -x}
+module test
 
-Extract the files in `Spoofax-pygments-release.zip` in the root of your Latex project.
-You can download this file on the releases page, or generate it yourself by running `python release.py` in the `spoofax-pygments/`-folder.
-After unpacking the files, you can use the `spoofax-pygments-lexer` in the folowing way inside your Latex document:
-```Latex
-\documentclass[twoside,a4paper,11pt]{memoir}
+context-free syntax
 
-\usepackage{minted}
+    Start = "Hello World!"
+  \end{minted}
+  \caption{Parsing and coloring using a language definition from Maven}
+\end{figure}
 
-\begin{document}
+\begin{figure}
+  \inputminted[firstline=5, lastline=9]{spoofax_lexer.py -O "language=org/metaborg/org.metaborg.meta.lang.template" -x}{grammar.sdf3}
+  \caption{Parsing and coloring of just a few lines of the file grammar.sdf3}
+\end{figure}
 
-\chapter{My Spoofax implementation}
 \begin{figure}
-  \begin{minted}{spoofax-lexer.py:FrameVMLexer -x}
+  \begin{minted}{spoofax_lexer.py -O "parseTable=fvm/sdf3.tbl,esv=fvm/editor.esv.af,preset=layoutSensitive" -x}
 BLOCK:
   ipush 0 // This is a comment
   return // Also a comment
 BLOCK2:
   ipush 0
   return
   \end{minted}
-  \caption{Some nicely colored code example}
-\end{figure}
-
-\begin{figure}
-  \inputminted[firstline=5, lastline=9]{spoofax-lexer.py:FrameVMLexer -x}{test.stc}
-  \caption{Just a few lines this time}
+  \caption{Parsing and coloring using a parse table and ESV definition in the fvm/ directory and a non-standard parser preset}
 \end{figure}
 
 \end{document}
-
 ```
-The Spoofax lexer does only supports valid programs as input. If you want to have a colorized snippet of just a few lines from the middle of a program, you therefore have to use the `firstline` and `lastline` arguments to get those lines from a valid program.
-Lastly, make sure to run the Latex compiler using `-shell-escape` for Minted lexers to work at all.
 
-Latex swallows a lot of errors in its log with respect to the custom lexers. If you have trouble setting op the tool, you could try running pygments directly on the command-line, in order to identify the problem. The command for this is: `pygmentize -l spoofax-lexer.py:FrameVMLexer -x <SOME STRING TO PARSE>`
+The possible options for the lexer (passed via the `-O` argument) are:
+- `preset` - One of the parser presets supported by JSGLR2 (currently `standard` (default), `dataDependent`, `layoutSensitive`, `incremental`). Note that this setting is not automatically read from `metaborg.yaml` in case you use the `language` option.
+- `language` - The identifier of a Spoofax language in the local Maven repository, in the format `[groupId]/[artifactId]/[version]`, where the `groupId` is slash-separated, the `artifactId` is period-separated, and the `version` is optional. Examples: `org/metaborg/lang.java`, `org/metaborg/lang.java/1.1.0-SNAPSHOT`.
+- `parseTable` - A file reference to a parse table relative to the directory of `spoofax_lexer.py`. After building a language with Spoofax, this file can be grabbed from `target/metaborg/sdf.tbl`.
+- `esv` - A file reference to an ESV definition relative to the directory of `spoofax_lexer.py`. After building a language with Spoofax, this file can be grabbed from `target/metaborg/editor.esv.af`.
+
+Either the `language` or the `parseTable` option is required for the lexer to work. If only a `parseTable` is given, [this default ESV definition](https://github.com/metaborg/spoofax/blob/master/meta.lib.spoofax/editor/libspoofax/color/default.esv) is used for coloring.
+
+### Limitations
+
+The Spoofax lexer does only supports complete, valid programs as input. This gives two limitations:
+- If you want to have a colorized snippet of just a few lines from the middle of a program, you have to use the `firstline` and `lastline` options of Minted to get those lines from a valid program.
+- The `escapeinline` option of Minted is not supported, because this causes Pygments to split up the file in multiple (incomplete) parts.
+
+### Debugging
+
+LaTeX swallows a lot of errors in its log with respect to the custom lexers. If you have trouble setting op the tool, you could try running Pygments directly on the command-line, in order to identify the problem. The command for this is: `pygmentize -l spoofax_lexer.py -O "language=..." -x -f latex <SOME FILE TO PARSE>` (note that the custom lexer only supports the LaTeX formatter; other formatters like `html` or `terminal` are not supported)

spoofax-pygments/.gitignore

@@ -1 +1,12 @@
-Spoofax-pygments-release.zip
+/spoofax-pygmentize-core.jar
+__pycache__/
+*.pyc
+test/*
+!test/dnx.editor.esv.af
+!test/dnx.tbl
+!test/inverted.esv.af
+!test/sdf3.tbl
+!test/spoofax_lexer.py
+!test/spoofax-pygmentize-core.jar
+!test/test.sh
+!test/test.tex

spoofax-pygments/core/pom.xml

@@ -17,13 +17,14 @@
     <dependencies>
         <dependency>
             <groupId>org.metaborg</groupId>
-            <artifactId>org.spoofax.jsglr2</artifactId>
+            <artifactId>org.spoofax.jsglr2.cli</artifactId>
             <version>2.6.0-SNAPSHOT</version>
         </dependency>
+
         <dependency>
-            <groupId>info.picocli</groupId>
-            <artifactId>picocli</artifactId>
-            <version>4.0.0-beta-2</version>
+            <groupId>org.json</groupId>
+            <artifactId>json</artifactId>
+            <version>20190722</version>
         </dependency>
     </dependencies>
 
@@ -55,10 +56,22 @@
                         </goals>
                         <configuration>
                             <shadedArtifactAttached>true</shadedArtifactAttached>
+                            <filters>
+                                <filter>
+                                    <artifact>org.metaborg:org.spoofax.jsglr2.cli</artifact>
+                                    <excludes>
+                                        <!-- Exclude all integration resources from the CLI project -->
+                                        <exclude>*.tbl</exclude>
+                                        <exclude>grammars/**</exclude>
+                                        <exclude>parsetables/**</exclude>
+                                        <exclude>samples/**</exclude>
+                                    </excludes>
+                                </filter>
+                            </filters>
                         </configuration>
                     </execution>
                 </executions>
             </plugin>
         </plugins>
     </build>
-</project>
+</project>