Writing lexer and parser of ANTLR grammar by hand

[KvanTTT]

Maybe it's sounds a bit strange but I suggest writing grammar lexer and parser by hand. It provides the following benefits:

• Removing dependency on any version of ANTLR. ANTLR 4 relies on ANTLR 3, it's very strange and ANTLR 3 is very outdated. The faster we get rid of dependencies the faster we can switch to WASM. The same related to StringTemplate and other dependencies.
• Clarifying what we need from parser. I beleive when we write parser manually we better understand what it should look like at, what tree structure we need, how to handle parsing errors.

I started work at this direction some time ago:

• Firstly, I totally removed LeftRecursiveRuleWalker.g without affecting functionality but with code simplification, see MR to my ANTLR fork: Left recursion fixes KvanTTT/antlr4#6. It's worth mention that this MR removes more lines than adds (-200).
• Secondly, I tried to write a grammar parser by hand in Kotlin and I succeed: https://github.com/KvanTTT/yantlr The resulting parser can handle simple grammars.

[kaby76]

When writing a compiler from scratch, one often bootstraps a bit of the code. Once the compiler is working at a basic level, the bootstrap code is replaced with code written in the language targeted by the compiler. We do that because we want to "think" in that language.

Antlr4 is written in Antlr3, but the bootstrap layer (Antlr3) was never rewritten in Antlr4 because there wasn't a tool to convert the Antlr3 grammar into Antlr4. It wasn't a priority.

As a "consolation prize," the antlr4 grammar was written. It works for third-party tooling, e.g., formatting, linting, grammar coverage, and Antlr4 grammar editor plugins. Along with the grammars in gramamrs-v4, these tools are as important as the parser generator itself.

While we assume that the Antlr4 grammar is correct, time and again the assumption has been proven to be wrong. antlr/grammars-v4#3815.

The Antlr5 parser could be written by hand, but we will need "an official Antlr5 grammar" for tooling. It allows one to reason about the parse tree. Grammars concisely describes that structure; a verbose, natural language description along with a hand-written parser in any language alone cannot suffice.

[KvanTTT]

The Antlr5 parser could be written by hand, but we will need "an official Antlr5 grammar" for tooling

Sure, we can write formal grammar together with writing of parser or later.