test(textmate): Show used theme colour

[CrossR]

Just a test to help look into stuff like #1933 / #1879.

We can look at the colours being produced more easily now and see the difference between themes:

I.e. OneDark always produces the same #bbbbbb colour for everything here. This was just a quick test to check it was reproducible, so I'll now extend it to tokens : (___), colour: (__) from (__) so we know which scope is being used to give us that colour.

[CrossR]

Other point of interest/nice comparison here is the VSCode Ctrl-Shift-P Inspect Editor tokens/scopes:

The scopes are the exact same*, so they are all valid and complete (ignore the text.html.derivative vs basic, that is just the syntax file I loaded).

Gruvbox managed to match against text.html entity.name.tag, which does exist as a HTML/XML specific part and is the same colour shown above.
OneDark however matches against entity.name.tag, and should have a colour....but we don't have that.

Looking into it and with help from the comment over in #1933, it looks like a lower priority scope is causing the higher priority ones to be ignored. In that case it was a general "scope" one, but for OneDark and HTML, its the "meta.tag" one. Removing that gives me slightly better results.

Its then possible that is causing further issues like the text to not be matched correctly etc. But removing that option does fix the div to start reporting correctly.

Still trying to get my head around what that means need changing though!

(* The scopes loaded aren't quite the exact same...we have $1 etc in ours. I haven't done enough digging to check if that is just a result of the CLI tool, or if we are infact not replacing those variables with what is needed).

[bryphe]

Awesome, thanks for investigating this @CrossR ! Huge help.

This is the module where we do the matching: https://github.com/onivim/oni2/blob/master/src/reason-textmate/TokenTheme.re

And we have some basic test cases around the matching here:

oni2/test/reason-textmate/TokenThemeTests.re

Line 68 in c6b7772

describe("match", ({test, _}) => {

The scopes loaded aren't quite the exact same...we have $1 etc in ours.

This is an interesting observation - I think it might be a bug with the textmate parsing - we might be missing a case where we need to replace a part of the token name with a match group in the regex.

So it looks like two separate issues - these are possible the root cause of several highlighting issues, as well:

• Syntax Highlighting: Terraform - Not getting full highlights #2220
• incorrect syntax highlighting for tsx files #2006
• Typescript syntax highlighting broken after constructors #2144

[CrossR]

I've had a poke around the HTML grammar, and this is the part we are looking at:

oni2/extensions/html/syntaxes/html.tmLanguage.json

Lines 2150 to 2171 in 94e9fa2

	"begin": "(?i)(</)(address|article|aside|blockquote|body|button|caption|colgroup|datalist|dd|details|dialog|div|dl|dt|fieldset|figcaption|figure|footer|form|head|header|hgroup|html|h[1-6]|label|legend|li|main|map|menu|meter|nav|ol|optgroup|option|output|p|pre|progress|section|select|slot|summary|table|tbody|td|template|textarea|tfoot|th|thead|tr|ul)(?=\\s|/?>)",
	"beginCaptures": {
	"1": {
	"name": "punctuation.definition.tag.begin.html"
	},
	"2": {
	"name": "entity.name.tag.html"
	}
	},
	"end": ">",
	"endCaptures": {
	"0": {
	"name": "punctuation.definition.tag.end.html"
	}
	},
	"name": "meta.tag.structure.$2.end.html",
	"patterns": [
	{
	"include": "#attribute"
	}
	]
	},

However, I don't really see anywhere that the capture groups are really dealt with as part of the regex creation, it looks like its a TODO that might have gotten lost/an open issue?

oni2/src/reason-textmate/RegExpFactory.re

Lines 134 to 143 in 94e9fa2

	{
	captureGroups: None,
	raw: str,
	regex,
	anchorCache,
	compiledAnchorCache,
	hasAnchorA: anchorA,
	hasAnchorG: anchorG,
	hasUnresolvedBackReferences,
	};

captureGroups is only referenced twice, that None and the type def.

I would guess, since we are missing those capture groups, we then can't replace the $2 with the contents of that capture group.

I'll loop back to that later, and try and pin down what is also causing the scope matching issue as well, since this is two separate issues of making scopes and matching scopes. I've added a failing test for this specific part at least now.