replacing ly.slexer and ly.lex

[wbsoft]

(related to #3, #55, and possibly others).

Dear Friends,

on the Salzburg conference I announced that I am writing a new lexer that wil replace ly.slexer and the ly.lex.xxx definition modules.

The reason is that it is almost impossible to inherit from, say, ly.lex.lilypond to implement small changes in lexing between versions of LilyPond, because when one changes a class, all references still point to the old one. Also, the tokens generated by the slexer module are instances of different classes, making it necessary to import the lexing definitions to be able to refer to classes and e.g. find instances. And finally, the linear structure of the tokens in the current ly module makes it tedious to find ranges in a text where e.g. a certain parser state is active (e.g. find the start of a lyrics expression, etc). ly.document.Source and Runner try to adress that a little but basically they are a PITA to work with (I can say that because I wrote them 7 years ago:-) )

The module that will replace ly.slexer and ly.lex will be a generic parsing package which I'm writing right now; I first called it LiveLex, but because that coincides with a German company name, I renamed it parce.

parce addresses all of the aforementioned shortcomings:

• A language definition is just a very simple class instead of a module global namespace. What slexer calls a Parser (or a context) is a lexicon (which is basically a class method yielding the parsing rules).
  It is very simple to inherit from a language definition and override just one lexicon to make a few adjustments. Because a lexicon is basically a class method, targets in other lexicons automatically refer to the inherited version of the lexicon. So supporting multiple versions of LilyPond becomes possible.
• What slexer calls a Token instance is a token, but the "type" is stored in an action attribute. So there are no classes anymore for every possible token, and manipulating tokens does not require one to import a language module to get all the definitions. And all the isinstance(token, ly.lex.lilypond.BlaBla) calls can go!
• Instead of a linear stream, the parce module creates a tree structure of the tokens, nested in contexts, and can update the structure when the user types in the document. So starting from a token, it is as simple as accessing its parent attribute to get the encompassing context. You can easily navigate to sibling or child tokens and there is a very powerful query method on every node, to search for anything in the tree structure.

This parce package already works very well, and I created good quality documentation and a web site for it, see https://parce.info/ . To get an idea of what a language definition looks like, compare ly.lex.lilypond (1659 lines) with parce.lang.lilypond (currently 534 lines).

Parsing and updating can be done in a background thread. There is already a small package bringing parce to Qt's QTextDocument, which can be used later to enable highlighting of text in Frescobaldi using the new lexer.

Right now, in ly and Frescobaldi, the music manipulation functionality uses either plain slexer tokens (such as the rhythm functions) or the ly.music tree structure. But also code such as the MusicXml output uses both the Music tree and also the original tokens.

Both can be ported to the new parser/lexer. But because that causes a rewrite of almost the whole of the ly package, I propose to create a new package from scratch, providing the same (and more) functionality, and that can be developed until the moment that it is possible to port Frescobaldi to it.

It is still desirable to have a tree structure like ly.music, but now the parce tree structure can probably form a solid base of such a structure.

(How/whether parce can support a custom data structure attached to a parce tree is yet to be determined. But we could also opt for an independent tree, which can replace ly.music and, while at it, also has a simpler structure (with fewer classes as current ly.music). Because parce can exactly report which part of the tree got updated, it is easy to invalidate just the modified part of a music tree instead of rebuilding it completely after every change, as is the case now.)

So, we could think of a name for the new ly module. We could use ly2 but I don't like the number in the name. I am currently thinking of quickly. :-)

[wbsoft]

Screenshot of the debugger that's part of parceqt

[dliessi]

I'm no expert but I like what you're doing!

So, we could think of a name for the new ly module. We could use ly2 but I don't like the number in the name. I am currently thinking of quickly.

I would avoid ly2.
I like quickly.

[PaulMorris]

Sounds like a big leap forward, nice work! If quickly is too broad or generic, another idea would be: parcely

[wbsoft]

I think the new ly module will provide more than only functionality based on parce, but just use parce for parsing (and the Document functionality that's also in parce).

[wbsoft]

I settled on the name quickly, the package is already created, at version 0.0.0.0.0.1 :-)

I added a transform module to parce, and it is shaping up very well to be able to have a living music expression as-you-type in Frescobaldi. When typing, parce quickly updates the tokenized/lexed tree structure and calls the transformer (like current ly.music.read) with only the modified region, which rebuilds any musical structure for that fragment, and integrates it in the whole document. Probably the new music object module in quickly will be a combination of the ideas in ly.dom and ly.music, so that both the score wizard and all music manipulation functions can use the same abstract model for a LilyPond source document.