Support terminal lookbehind

[msujew]

Closes #1355

Allows adopters to write positive/negative lookbehind in both EBNF and Regex notation, with full support in the lexer/parser phase.

[snarkipus]

Yay - this cleans up my request from #591!

[sailingKieler]

Looks pretty good to me! 🚀

[sailingKieler]

Am I right that this distinction and the corresponding RegExp wrapping in regexPatternFunction is just there to prevent Chevrotain from applying validation rules to the given regex pattern?

[msujew]

It does have something to do with Chevrotain, yes. Basically, Chevrotain will apply a few optimizations on token types based on the regex you pass. If you pass a non-trivial regex (usually those with the /u flag or lookbehind), those optimizations cannot be performed. Instead of ignoring optimizations, it will instead throw an error. To disable those errors, we wrap the regex inside of an custom pattern function.

[szarnekow]

Do you plan to filter code completion based on the look-behind patterns, too?

[msujew]

@szarnekow Can you give an example? I don't think the completion takes the specific token pattern into account at all right now.

[szarnekow]

I see. The completion engine doesn't use the terminal rules to check inserted tokens for validity at all.

[msujew]

Yes, it will currently just insert whatever the scope provider returns, ignoring any potential parser/lexer error that this might result in.

[szarnekow]

grammar test
entry Main: things=AorB+;
AorB: A|B;
A: ('A' | 'C') b=[B:B_TERMINAL]? ';';
B: name=B_TERMINAL ';'
terminal B_TERMINAL: (?<!'A')'B';
hidden terminal WS: /\\s+/;

Model with cursor positions <1> and <2> where I'd invoke CA:

B;
A<1>;
C<2>;

I wouldn't expect B to show up at <1> but at <2> - though I wouldn't get any proposals in either case because of the default reg filter in the completion engine.

[msujew]

I see, thanks for the example. I have created a separate issue, see #1368 👍