Tree Sitter in Review Board: Introduction

Part 1: Introduction

Pygments

Since the very early days of Review Board, we’ve used the Pygments library to handle syntax highlighting. Pygments takes an interesting approach: it doesn’t do any parsing. Pygments lexers are essentially a big pile of regular expressions and a state machine.

Using Pygments has always had some trade-offs:

On the “pro” side, it’s a pure-Python library with no compatibility headaches. It has pretty broad language support, and it generally does a pretty good job for most of those languages. The output is quite easy to style with CSS to fit our color schemes.

On the “con” side, because Pygments isn’t actually parsing, it’s limited to what a lexer can do. It has no idea if a { character is opening a class body, a function, or just a block. If it sees an identifier, it doesn’t have a good way of telling if that’s a variable, a method call, or something else. When it makes a mistake about what language a file is, the results can be terrible (to the point where we had to add an escape hatch for server admins to be able to manually override the language based on file extension).

Tree Sitter

The new kid on the block is Tree Sitter. Unlike Pygments, Tree Sitter is a parsing system. TS grammars compile into actual parsers which process a file and return an abstract syntax tree.

Having a parse tree gives us several things that Pygments can’t:

• Highlighting that knows foo is a method call and not a variable.
• Injections: the content inside of a <script> tag in a .html file should be highlighted as JavaScript, not just plain text.
• Structure that we can query and process for more than just syntax highlighting.

The one major downside is that both the Tree Sitter framework as well as individual grammars compile to native code. We’re using the py-tree-sitter bindings and tree-sitter-language-pack for the grammars. Both of these packages ship pre-built binary wheels that cover all our supported platforms, but any time we pull in code that isn’t pure Python, we have to acknowledge the possibility that we’re going to get support calls from our customers because their web server is segfaulting.

Optimistically switching over

Review Board renders diffs for whatever files people happen to upload. We don’t want to rip out Pygments and then find out later that a user no longer has any syntax highlighting for whatever obscure language their entire business is based on.

We also want to be able to (as much as we can) gracefully handle malformed input. A syntax error shouldn’t cause highlighting to completely disappear. Many Tree Sitter grammars do a remarkable job of parsing partial or malformed input, but some cases will cause it to fail entirely. Because Pygments is operating with regexes, even if there’s unparsable garbage in the input, it will be able to give us something.

We’re therefore going to try first with Tree Sitter, and if we don’t get any result from that, we fall back to the existing Pygments implementation:

if tree and language:
    try:
        if highlighted := ts_highlight(data, lines, tree, language):
            return highlighted
    except Exception as e:
        logger.exception('Tree sitter highlighting failed: %s', e)

return apply_pygments(data, filename)

Making it seamless

Review Board’s syntax highlighting CSS actually handles two separate highlighting systems already. Anything coming from the server is highlighted via Pygments, but then we also use CodeMirror for editing in the browser. CodeMirror contains its own syntax highlighting subsystem that generates different HTML element classes. Adding Tree Sitter just means adding additional classes to our highlight CSS for the new match names:

/*
 * Pygments: Comment.Single
 * CodeMirror: Comment:
 * Tree Sitter: comment
 */
.c, .com,
.cm,
.c1,
.cm-s-rb, .cm-comment,
.ts-comment {
  color:
    var(--if-light, #0000C0)
    var(--if-dark, #05f3f2);
  font-style: italic;
}

Putting together the new styles actually surfaced one place where the Tree Sitter implementation can do something that Pygments got wrong:

/*
 * Pygments: Generic.Traceback
 * Tree Sitter: function.call
 * Tree Sitter: function.method.call
 */
 .gt,
 .ts-function-call,
 .ts-function-method-call {
   ...
 }

Keeping the size guards

Review Board has two heuristics that we use to decide when to skip syntax highlighting entirely:

Any file over 200KB: This is a pure performance cutoff: both Pygments and Tree Sitter start to get very slow as the file size increases.

Any file where any individual line is more than 1000 characters long: This heuristic exists purely because the Pygments regex state machine exhibits terrible performance when dealing with very long lines. In reality, this condition only tends to match “minified” files which are virtually impossible for humans to review whether or not they are highlighted. Because of this, even though Tree Sitter could highlight these files without the performance penalty of Pygments, we’ve decided to just apply the same limits no matter which subsystem ends up doing the highlighting.