Why is SKIP_TOKEN not terminal?

[larsgw]

I was getting a "Maximum call stack exceeded" error by putting a SKIP_TOKEN in a MANY with a GATE (the latter evaded the infinite loop detection). I think this is because SKIP_TOKEN is not a Terminal unlike CONSUME. Is this correct behavior? Both are consuming a token internally, the latter just checks whether it matches the expected type or not (and the return values are different, but that's I think mainly for EmbeddedActionsParser).

[msujew]

Is this correct behavior?

I believe this isn't correct, but not in the way you assume. Even with a GATE, the validation should prevent you from creating the parser. Keep in mind that the error validations of Chevrotain aren't just recommendations. Every ignored validation error will lead to a parser error somewhere (self analysis or runtime). Evading them is never a good idea.

To explain this behavior a bit more in-depth would require some knowledge about how LL(k) parsers work. In short: SKIP_TOKEN never specifies which token type it will skip. Therefore, it cannot be respected during the lookahead, unlike CONSUME which specifies which token it will accept.

On another note, I believe you're allowed to have SKIP_TOKEN in a normal for-loop. Only actual parser functions need to be wrapped inside of MANY. So your parser rule is allowed to contain this:

if (!this.IS_RECORDING) {
  while (...) {
    this.SKIP_TOKEN();
  }
}

[larsgw]

Evading them is never a good idea.

Well I didn't mean to :)

In short: SKIP_TOKEN never specifies which token type it will skip. Therefore, it cannot be respected during the lookahead, unlike CONSUME which specifies which token it will accept.

But couldn't I add a category to every token and CONSUME that category, which would do the same thing?

On another note, I believe you're allowed to have SKIP_TOKEN in a normal for-loop. Only actual parser functions need to be wrapped inside of MANY. So your parser rule is allowed to contain this

Then I get a different error:

EarlyExitException: Expecting: expecting at least one iteration which starts with one of these possible Token sequences::
    <[line break]>
but found: '1'

In this case the iteration contains (1) a SUBRULE with the while loop you propose, followed by (2) a CONSUME of a line break. It seems to me that chevrotain doesn't consider that SUBRULE does anything?

[msujew]

But couldn't I add a category to every token and CONSUME that category, which would do the same thing?

Yeah, that should work.

In this case the iteration contains (1) a SUBRULE with the while loop you propose, followed by (2) a CONSUME of a line break. It seems to me that chevrotain doesn't consider that SUBRULE does anything?

When performing a lookahead somewhere in your grammar, the parser encountered your SUBRULE (which apparently starts with SKIP_TOKEN?) whose only real token is line break. Since the token stream actually contains 1 at that position, it will not try to go into that subrule, since it doesn't expect to parse 1. You have to make sure that the parser is able to actually decide where it needs to go.

Anyway, using CONSUME with a "catch-all" category should have the desired effect (assuming that you order everything correctly).

[bd82]

On another note, I believe you're allowed to have SKIP_TOKEN in a normal for-loop. Only actual parser functions need to be wrapped inside of MANY. So your parser rule is allowed to contain this:
if (!this.IS_RECORDING) {
while (...) {
this.SKIP_TOKEN();
}
}

FYI: There is a built wrapper for this.IS_RECORDING

• https://chevrotain.io/docs/guide/internals.html#the-solution-1