I suppose this would be ideal as a svelte dialect for the html parser?

This kind of thing can be implemented separately, using mixed-language parsing. The idea, illustrated in that link with a similar templating language, is that you write a parser for Svelte component notation that leaves 'gaps' where HTML should be parsed, and then defers the parsing of those to the HTML parser.

I don't intend to maintain something like this myself, but it'd be very welcome if someone published a package for it.

Context: I wrote the Textmate grammar used in the official Svelte VSCode extension.

Yeah, so I gave a shot at it. Here's some proof:

Doing this with mixed-language parsing isn't practical, I tried (what you see above isn't mixed). Svelte doesn't parse HTML in a "mixed" approach anyways and has both an understanding of HTML and JS at the same time, kind of like JSX. It has a full AST, including tags and JS expressions.

As an example, parsing attributes in a mixed way would be really messy:

<form on:submit|preventDefault={handleSubmit}>
</form>

The grammar would have to pretty fully parse HTML because it has to figure out what is and isn't an attribute, as on:submit|preventDefault all being parsed as the "attribute name" is absurd. I suppose even if you did, you could still overlay the HTML parser in afterwards but that sounds so messy.

Unfortunately, even with forking the HTML grammar there are problems. The interpolation in {} blocks in Svelte are always expressions, not statements. So if you do attr={{ foo: "bar" }}, the text inside of the outer pair of braces should be parsed as an object expression. The issue is that, as it stands, I can't nest the TypeScript parser in a way where it correctly interprets that text as an object. I can't change the TopDeclaration to be SequenceExpression, which I figured would be the way to go but since there is only one @top in the grammar there is nothing I can do.

Another related issue is demonstrated by how Svelte blocks work, like {#if foo} {/if}. foo in that start block is an expression, you could do foo.bar() === bar.foo({ foo: "asdf" }). Parsing something like this is a nightmare:

{#each items as { id, name, qty }, i (id)}
	<li>{i + 1}: {name} x {qty}</li>
{/each}

In the {#each} block, there are three expressions: items, { id, name, qty }, and id. This becomes a nightmare because you need to figure out where those expressions end before you continue on. How do I know the qty } part of the text isn't actually the end of the {#each} block? Right now in the grammar I made I'm using an external tokenizer that is doing some crappy JS parsing to try and figure that out, so that the JS parser can be nested later. This sucks, but the right solution is to include most of the JS grammar, which sucks even more.

I guess my point here is that doing this is just really frustrating and difficult, and I feel like it shouldn't be this hard. I don't know what to suggest to help here, maybe some way of yielding to another parser for a token or something? If I could use rules from the JS grammar it would be helpful.

The interpolation in {} blocks in Svelte are always expressions,

This patch adds a @top rule to the JS grammar to let you parse single expressions with it (using the top config option). I.e. javascriptLanguage.parser.configure({top: "SingleExpression"}). Would that help?

As for finding the end of a JS expression, that is indeed not easy. Lezer had a feature like this at some point, but then realized all the languages I was actually going to implement used a crude end token to delimit the end of nested regions, so I scrapped that. I guess Svelte does provide a use case for this, but I'm hesitant to add that complexity in again.

Thanks for the patch. It does help, although I worry that you'll have to constantly add new @top declarations to languages.

If you want another motivating example, Marko does the same thing to an even more extreme degree.

<for|color| of=["red", "green", "blue"]>
    <li style=`color:${color}`>
        ${color.toUpperCase()}
    </li>
</for>

The value of an attribute here is literally a JS expression, which is not like Svelte where those expressions are at least wrapped in {}, "", or "{}".

Actually, Marko provides yet another @top declaration issue:

class {
  onCreate() {
    this.state = { count: 0 };
  }
  increment() {
    this.state.count++;
  }
}
<div>${state.count}</div>
<button on-click("increment")>
  Click me!
</button>

That class {} syntax is fairly self-explanatory - the inside of that block is supposed to be an ordinary JS class. I suspect figuring out where that block ends is a problem too, actually.

To be fair to Lezer and your implementation of it - these template languages are kind of absurd in concept. Of course when you're actually using the template language the design makes sense.

I worry that you'll have to constantly add new @top declarations to languages.

It seems most languages have only a few ways in which they are reasonably parsed, and these are simple patches, but indeed, it's generally open-ended.

Marko looks scary. Hadn't seen that before.

How is your Svelte parser coming along? Anything I can help with?

It's going well! You can take a look at it in this Repl, if you want to. It's a bit messy still.

The parse is pretty detailed now:

I think it pretty much supports all of the syntax that Svelte has, at this point. I'm working on polishing the parse in certain places (e.g. it does not at all handle quote marks in interpolation), and language features like making indentation work well. I think the most fragile part of this has to do with the external tokenizers that figure out where expressions are supposed to end. Expressions aren't using the SingleExpression thing as I don't think that's been published yet.

I don't think I need help yet, but there are things I'm still figuring out. One is how to parse unknown blocks (e.g. {#fake}) reliably, mainly to avoid having the parser trying to recover and treat an unknown block as a {/if}, or something.

Maybe we can re-open this issue?

Is the plan to have this as a separate @codemirror/lang-svelte package? Or is this something that would be part of lang-html?

I've published @lezer/javascript 1.2.0 with the new top rule.

Maybe we can re-open this issue?

No, this is not something I myself plan to publish or support—I'm just seeing if there's anything I can help @Monkatraz with, in the hope that they publish a package for this.

I've published @lezer/javascript 1.2.0 with the new top rule.

Works great.

No, this is not something I myself plan to publish or support—I'm just seeing if there's anything I can help @Monkatraz with, in the hope that they publish a package for this.

Yeah, I plan to publish this. The biggest issue I'm running into now is definitely the JS expression thing, specifically dealing with strings. I can handle cases like this now:

If it was just single quote strings or double quote strings I think I would be okay, but...

Template strings are another layer of screwy. I think I could get them to work too, but I'm basically writing a crappy JavaScript expression parser at this point, and what I have now is probably full of edge cases I have no idea about.

Even if it's inefficient, I wonder if I can use the JavaScript Lezer parser to figure out where the expression ends in an external tokenizer? idk. Might just have to accept this as a known issue for now.

EDIT: I totally forgot about this pattern and it's absolutely going to cause problems.

Well, I did try to harden the external tokenizers a bit without doing anything too extreme (like recursively parsing interpolation in template strings) and it survived this:

I think yielding to the JavaScript grammar is not practically necessary for Svelte, but without doing so the Svelte grammar will never be exactly "correct".

Nice work. I may at one point look into adding something like yielding parses again, but for the moment I don't have the time—and in general, it'd involve a bunch of additional complexity.

Alright, I have an initial version of the extension here: https://github.com/replit/codemirror-lang-svelte

It's published under Replit and we'll be maintaining it (well, mostly me, but still). The NPM package name is @replit/codemirror-lang-svelte.

Wonderful. Linked from the community language package list.

Wow, this is pretty amazing. Thank you!

I'm going to hook this up to Chrome DevTools now, will likely be part of M110.

@Monkatraz Would you like also implement this for .vue
@yyx990803