The lexer should only return a span and a token-kind

[jyn514]

This is the traditional way to write a lexer, used by yacc, logos, and clang. The lexer identifies the boundaries of the token and the token kind and nothing else. This means for instance 1000000000000000000000000 would not be a lex error, but instead a parse error (and I'd probably want to delay it even further into constant folding).

This would make it significantly easier to switch to logos and support rust-lang/rust-bindgen#1762.

This requires a fairly significant refactor.

[hdamron17]

Why should the lexer only provide span and token kind? It seems this restriction is only necessary for the preprocessor.

[jyn514]

Look at the linked issues for examples; #385 and #383 in particular would be tricky without this, I'd have to move types all the way into the lexer.

[Stupremee]

After I finished my current PRs, I will try to implement this.

[hdamron17]

I am working on this in #474 because I needed it for #456.