Lexer routines to support token-by-token processing

[JamesYang007]

Lexer processes all tokens at once and stores into vector. We want to support processing per token whether we want to single-process or batch-process for the parser.

The lexer routine logic take advantage of the fact that we are simply pushing to a vector of tokens. There will be major restructuring to get rid of this dependency.

[JamesYang007]

Waiting for merge: @ #13