Kent.hall/tokens parser

[kentjhall]

The structure is there for setting up parsing logic.

[JamesYang007]

Great design!!

General comments:

• Inline definitions should always be placed in header file. Otherwise, compiler will only be able to inline this function whenever it gets called within current .cpp file.
• I like to name arrays with _arr so instead of handlers_t I would write handler_arr_t. Gives less confusion to those who read your code for the first time (like how it did to me)
• It would be better if you wrote all the using ..._t = ...; inside each of your classes. This is not code duplication. I know I didn't do this in my lexer stuff and I'm still thinking about fixing it. My issue is that I don't have a class for these routines so the "closest" place I can put these aliases are in the same scope as my functions (docgen::core).

[JamesYang007]

I think it's worth having an empty struct like

struct RoutineTraits
{
using routine_t = void (*)(const token_t&, ParseFeeder&);
};

in a separate header file, which you can include into your tags_parser.hpp, ignore_parser.hpp, comments_parser.hpp and have each of your _routines struct derive from it.

SymbolHandler should not own the actual definition for this routine_t. It can then just reference this routine_t (see comment in parser_worker.hpp).

Also, once this restructure works, try compiling with

using routine_t =std::function<void(const token_t&, ParserFeeder&)>;

inside RoutineTraits. It should theoretically work. You may have to make all of your routines non-constexpr and just const. The performance disadvantage of std::function will not be an issue because all of your lambda functions are empty structs (there's a thing called small-object optimization - your lambda function will be constructed directly inside std::function object, i.e. no heap-allocation). With this one-line fix, you provide possibility of inlining your lambdas now, so you should see a noticeable performance difference with larger files.

[JamesYang007]

Coooool 💃 :D

[JamesYang007]

Fantastic work :')