Alien

Compiler frontend generator


Project documentation · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Usage
3. Roadmap
4. Contributing
5. License
6. Contact

About The Project

Alien is a compiler frontend generator, which makes creating lexers and parsers for your new language an easy task. There are many generators out there; however, I tried to make this one comfortable to work with.

Here are the main features:

• Single configuration file for both lexer and parser
• Default templates are customisable with help of %code directives
• If default templates aren't what you want, you can always create your own
• Unicode support; ASCII-only lexers are also possible

(back to top)

Built With

Alien is written in C++ and uses CMake as the build system. CPM is the dependency manager being used.

Dependencies:

• CxxOpts
• Utf8proc
• Inja
• Nlohmann Json

(back to top)

Documentation:

• mdBook

Getting Started

You can set up the generator following these steps

Prerequisites

You need to have cmake installed on your system

Installation

You shouldn't install any of the dependencies yourself since CPM handles the ones that are not included in the ./deps folder

1. Clone the repo

   git clone https://github.com/evonicgu/alien.git && cd alien

2. Let cmake do its work

   cmake -DCMAKE_BUILD_TYPE=Release -G <your generator (Ninja, VS, etc.)> -S . -B ./cmake-build-release
   cmake --build ./cmake-build-release --target alien

3. Create a config file

   Config file

   #generation.noutf8 = true       // ascii-only lexer is generated
   #generation.track_lines = false // for performance reasons
   #token.namespace = "lexer"      // to show parser where custom token classes are
   
   %code {
       struct digit_token : public token<token_type> {
           long long value;
       
           digit_token(long long value)
               : token(token_type::digit),
                 value(value) {}
       };
   }
       
   {
       digit = digit_token,
       plus,
       minus,
       mul,
       div,
       exp,
       paren_open,
       paren_close
   }
       
   %left plus minus
   %left mul div
   %right exp
       
   %%
       
   \+:             {}[plus];
   \-:             {}[minus];
   \*:             {}[mul];
   /:              {}[div];
   \^:             {}[exp];
   \(:             {}[paren_open];
   \):             {}[paren_close];
   [1-9][0-9]*:    {
                       return new digit_token(std::stoll(gettext()));
   };
   \s:             {};
       
   %%
       
   #generation.symbol_type = expr_t
       
   %code-top {
       enum class expr_type {
           DIGIT,
           BINARY
       };
       
       enum class op_type {
           PLUS,
           MINUS,
           MUL,
           DIV,
           EXP
       };
       
       struct expr_t {
           expr_type etype;
       
           expr_t(expr_type etype)
               : etype(etype) {}
       
           virtual ~expr_t() = default;
       };
       
       struct digit_expr : public expr_t {
           long long value;
       
           digit_expr(long long value)
               : expr_t(expr_type::DIGIT),
                 value(value) {}
       };
       
       struct binary_expr : public expr_t {
           expr_t *lhs, *rhs;
           op_type otype;
       
           binary_expr(expr_t* lhs, expr_t* rhs, op_type otype)
               : expr_t(expr_type::BINARY),
                 lhs(lhs),
                 rhs(rhs),
                 otype(otype) {}
   
           ~binary_expr() override {
               delete lhs;
               delete rhs;
           }
       };
   }
       
   {
       expr = expr_t
   }
       
   %%
       
   expr:
   %digit                          {$$ = new digit_expr($0->value);}
   | expr %plus expr               {$$ = new binary_expr($0, $2, op_type::PLUS);}
   | expr %minus expr              {$$ = new binary_expr($0, $2, op_type::MINUS);}
   | expr %mul expr                {$$ = new binary_expr($0, $2, op_type::MUL);}
   | expr %div expr                {$$ = new binary_expr($0, $2, op_type::DIV);}
   | expr %exp expr                {$$ = new binary_expr($0, $2, op_type::EXP);}
   | %paren_open expr %paren_close {$$ = $1;}
   ;
   
   %%
   

4. Generate your lexer and parser

   ./alien <config file> -o <output file>

5. Use them in your code

   #include <iostream>
   #include <fstream>
   #include "parser.out.h"
   
   int main() {
       lexer::lexer<std::ifstream> l("../content/f.txt");
   
       parser::parser p(l);
   
       auto* tree = p.parse(); // pointer to root of the produced AST
   }

6. Parse some text

   Produced parser is able to parse simple mathematical expressions such as

   5 + 5           -> ((5) + (5))
   18 ^ 3 * 3      -> (((18) ^ (3)) * (3))
   2 + 2 / 2       -> ((2) + ((2) / (2)))
   

   Here every parenthesized digit or expression represents a single instance of expr_t class generated by one of the actions the parser will execute while building the AST.
   
   In fact our code can parse correctly any expression which uses + - / * and ^ as operators.

(back to top)

Roadmap

• Separate source and header files
• Lexer documentation
• Parser documentation
• Proper error messages
• GLR Parsing
• Parser generation for other languages

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Anton - [email protected]

Project Link: https://github.com/evonicgu/alien

(back to top)