At Fireside, the Engineering team is allowed to experiment with tools they find helpful to get work done. In the early teens when we tried writing code on OSX for our .Net application, Engineering staff variously used TextMate and Sublime Text along with Vim. You couldn’t build the app (no, Mono didn’t support all the APIs), but it shortened the cycle for some kinds of development.
The Web team, in particular, really clamped onto Sublime as the editor of choice. It was cross-platform, fairly resource-light, and suggested completions from other project files. It didn’t know what types the completions had, but that saved a lot of overhead. The trouble was that Sublime didn’t know how to display ASP.Net files. There was a language syntax for Classic ASP, but it tried to highlight code blocks with Visual Basic. (Yes, that same Visual Basic that Stack Overflow’s Developer Survey rated “Most Dreaded” for the third year in a row.)
I thought surely someone must be out there wanting to glue C# to HTML, but it seemed that everyone thought the state of affairs was good enough. A year or two went by before I gave up and set out to do it myself on my own time. After some investigation, I discovered that syntax definitions fall broadly into 3 categories:
- Keyword identification
- Stack-based contexts
- Full IDE-level language comprehension
Keyword identification is kind of boring. It’s effectively just a few lists of reserved words and symbols (if, for, true, =, etc.) that you tell the editor to put in certain symbol classes called scopes to be colored by your editor’s theme. You can improve highlighting beyond plaintext for a small toy language with those, but it can’t do something like HTML. Full language comprehension isn’t supported by Sublime, so that route was dead from the get-go, leaving the stack method. By and large, every code editor has its own engine that was first designed to take TextMate grammars (tmLanguage XML files) convert them to a more readable format, and then bolt extra bells and whistles on top.
Briefly, this is how they work: When a recognized pattern is encountered (say if), the engine will assign it a scope, and can optionally push a new context onto the stack, potentially with a different set of patterns to look for. Then, when a sentinel pattern is encountered, the stack is popped back to a previous state.
if (true) { var f = 3; }
// First, `if` is encountered, and an expression in parentheses is expected
// ^ The parenthesis pushes an additional context onto the stack
// ^^^^ The `true` keyword is inside this context
// ^ The closing parenthesis pops the context off of the stack
// ^ The open brace pushes a new context, and so on
My first attempt unsurprisingly came up short. I was trying to model my syntax definition after PHP, another language that injects itself into an HTML file. The trouble is that the Sublime Text PHP code syntax definition has references to the PHP HTML syntax buried in its guts. Maintainers of the C# syntax definition file would have looked unkindly upon an extension developer asking to special-case their extension. After all, C# is not so heavily associated with HTML that it makes sense to pollute the default C# syntax with references to HTML contexts.
The second attempt, modeled after HTML’s injection of Javascript worked better. Due to engine limitations, there are some bugs with expectations for certain contexts, but these have fairly limited effects.
Web folks loved it, and 2.5 years later the package has a modest install base outside Fireside of 5 or 6 thousand with a few more each day.
But in the meantime, the Engineering team expanded, with new faces and new tools. When a project brought some of the team into contact with ASP.Net files, they discovered the same problems with Visual Studio Code that we had earlier with Sublime Text. VS Code didn’t have the Classic ASP stopgap that eased the pain for so long in Sublime. You had to use the main HTML syntax or nothing. On certain files, some construct would catastrophically break the HTML syntax and the engine would mis-highlight the rest of the file.
I was already familiar with the concepts of syntax definitions, so I ported the YAML-based Sublime plugin to the JSON-based VS Code format on a Friday afternoon. As always, the devil is in the details, and I discovered rather quickly that the bells and whistles of Sublime were not the same as the ones in VS Code. Where Sublime has a corner case (never seen in practice) that prematurely reverts from C# back to HTML, VS Code had a very common occurrence where C# refused to pop the stack to use HTML contexts.
Ironically, if the VS Code C# syntax definition maintainers were as verbose with their scope contexts as the Sublime Text maintainers, the VS Code extension could use its engine to be more accurate than it is possible in Sublime. VS Code lets a language definition inject patterns into arbitrary locations in another language definition, by scope name. This allows an extension like mine to do what I tried with the PHP-style definition in Sublime–without changing files in the C# definition. The trouble is that the would-be targets for injection in C# do not have named scopes. Sublime adds a meta.block.cs every time they nest another set of curly braces, but VS Code went minimalist.
Even without a perfect marriage, we can still prevent cascading highlighting failures from an unexpected pattern in the file. The engineer who asked about it was delighted with a zipped pre-release and wanted to know how everyone else could get a copy. I said I’d look into publishing, but I had written the VS Code version on the clock. My supervisor was happy to let it go out to the world under the company banner. Pause for a second. Our little software company is willing to let us release tools and source code developed at company expense for the use of the software engineering community.
That took care of distribution, but I was surprised to see the adoption levels outside the company.
With Sublime Text, you had to go looking for a plugin to do something. VS Code has plugins come looking for you. If you open a file with a file extension that is unhandled by any of your existing language definitions, VS Code will look in its database for any plugins that claim to highlight those files and suggest that you peruse them. With Microsoft to serve as a pusher, I was seeing 30 downloads a day averaged over the first week, and the rate was increasing. By the time a month had gone by, the total downloads divided by the time the plugin had been available put the average at 60 downloads a day. The current count is up to 
.
Fireside is usually busy making legislative office processes run smoothly, but I’m pleased we can take some time for software development processes as well.