Why are the TextMate token blocks not coloring syntax as I expect them to?

Question

I am trying to customize vscode colors by using TextMate tokens, and the "editor's inspector tool". It doesn't seem to be working though. The syntax is not highlighting, and the tool shows the scope property as being crossed out.

pictures:

Answer 1

Semantic Highlighting

Your problem is that you're using Semantic Tokens and don't realize it. Semantic tokens override TextMate tokens, and they even implement themselves, using your TextMate tokens configuration.

I included an excerpt explaining how to disable the semantic tokens, and another one that explains a work around to stop them from overriding your TextMate tokens. I also included a link above that takes you to the VS Code documentation for semantic tokens. There are benefits to using them when you understand them.

Here's a simple, but very good example that's perfect for explaining how the two token types behave inside of an actively equipped theme.

{
  "semanticTokenColors": {
    "function": "#77BB55"
  }
    
  "tokenColors": [
    {
      "scope": ["entity.name.function"],
      "settings": {
        "foreground": "#22AAFF"
      }
    },
    {
      "scope": ["source meta.function-call entity.name.function"],
      "settings": {
        "foreground": "#CCAA44"
      }
    }, 
  ]    
} // eof

The example has three different token blocks.

1. A semantic token block that colors all functions green
2. A TextMate token block that colors all functions blue
3. A TextMate token block that colors function-calls yellow

Obviously a function can't be green, yellow, & blue. The snippet is easy enough to interpret though: All functions in the theme will be green.

The reason all functions will be green is because TextMate tokens are colored first. You may have noticed that some of the syntax is colored a few seconds after a theme loads. The delay is caused by the wait for the language server to start running in the background. Once running it still has to parse your code. The process takes about 2-3 seconds on my machine, therefore, I see a 2-3 second delay before semantic highlighting changes the colors of syntax.

TextMate tokes, as I stated above, still color their syntax, as they are not aware that they will be overridden by the semantic highlighting language server. The language server logs information about the tokes it highlights. If we equipped our snippet above and loaded an active Python script with Functions in it, we would see the function definitions as being blue (#2), and the function calls as being yellow (#3), before they all turn green after the language server kicks in (which, as I said, happens fast). This is an important detail though, because it allows us to optically observe the tokens behavior, which gives us an easy means to gaining a solid understanding of what is happening.

Now, here is where the Crossed-out (or stroked-through) properties come from that you see in the syntax-inspector-tool. When ever you write a TextMate token into your theme, and a semantic token overrides it, the SemanticHighlighting Language server logs that information, and you are able too access (which you already have, so GJ) with the Syntax-Inspector-Tool.

---

The culprit is a feature that uses a language server to improve how theme & extension developers highlight syntax in VS Code. This feature is called semantic highlighting, and the problem you are experiencing causes many new theme-developers issues when first using Semantic Highlighting. Fixing the issue is simple, but you will have to make a couple thought out decisions, as there are 3 different ways you can resolve the issue. I have outlined all 3 solutions below, you can decide which of the three best suits your needs.

<br

Disabling 'Semantic Highlighting' for a Theme

This is how you turn off 'Semantic Highlighting' for a theme, such that anyone who uses the theme will not be able to use the 'semantic highlighting feature'. In other words; this method targets the theme itself, rather than your personal VS Code configuration. If you publish your theme, your theme will be published without the option for your theme's users to use 'semantic highlighting'.

To disable semantic highlighting for a theme, open up the theme's JSON file and add the setting below to it.

• "semanticHighlighting": false

Make sure that the "semanticHighlighting" property is placed in the top-most level of the theme's json file, and that you don't accidentally add it to the "colors": { ... }, "tokenColors": [ ... ], or "semanticTokenColors": [ ... ] properties. Personally, I prefer to place it at the top of the theme file, however, from what I have seen, most theme-developers place it in the middle of the theme, right above the semanticTokenColors property. Either way is fine, so long as it is scoped properly.

Turning of SemanticHighlighting for a Workspace or User

This method only turns semanticHighlighting of for an individual user, and does not require accessing the theme's source JSON file as the last method for deactivating semantic highlighting did.

You can deactivate semantic highlighting for your current workspace, or for all themes you use with method. Open up your settings.json file. If you want to disable semantic highlighting in all themes, open up your "user-level settings.json file", to turn it off for a specific project (aka a specific workspace), open up ./.vscode/settings.json. After you open a setting.json file, add the following setting to the file:

"editor.semanticHighlighting.enabled": false

If you need to know more about the various settings.json files available to you, you can find the official documentation on the subject by clicking HERE.

---

Alternate Solution: Use Semantic Tokens Intermittently

---

Semantic tokens have a ton of advantages, and are a powerful feature at the disposal of Theme developers. I must admit though, they can be too powerful at times. You should never try to color a theme using semantic tokens alone. The most important aspect to semantic tokens that gives theme-developers a major advantage that is not had without them, is their ability to color parameters, properties, methods, constructors, and many other entities, in a way that TextMate tokes can not. The problem with this is that sometimes they are too greedy, and setting one semantic token can end up coloring several entities that you didn't intend to color. Furthermore, semantic tokens don't have to be set using semantic token blocks, if you author a textmate block, like the the all-functions TextMate block in the example at the top of this answer (if you dont remember, it has the scope entity.name.function) semantic highlighting will use that as a semantic token, if there is not a function block already set in the "semanticTokenColors" property.

There is a work around for this though!

To use semantic tokens for the syntax that you like to use it with, and if you play with it, you will find instances you like to use it in, you can trick semantic tokens into not coloring syntax based on the TextMate tokens you write, by prepending each TextMate token block's scope property with "source" (or "text" for Markup languages).

...so instead of writing your block like this...

  "tokenColors": [
    {
      "scope": ["entity.name.function"],
      "settings": {
        "foreground": "#22AAFF"
      }
    },
]

---

Write it like this instead:

  "tokenColors": [
    {
      "scope": ["source entity.name.function"],
      "settings": {
        "foreground": "#22AAFF"
      }
    },
]

See the source prepended to the scope?

This workaround tames semantic highlighting, and puts you in control, while getting the advantages that come from semantic highlighting.

---