Honest question regarding why

[svallory]

Hi!

I'm just starting to learn more about parsers, compilers, etc. I've been designing a language for the last 4 years.

About 3 years ago I started to implement it with PegJS, which was easy, but only gave me the parser, I would have to do all the rest by hand.

So I moved to Xtext because it seemed to provide a lot more just from the grammar. That actually made me give up for a long time. Xtext ecosystem is too complex and cumbersome for someone that is not familiar with the Java ecosystem.

Near the end of last year, I started looking into it again and found Langium which is what I'm using now.

But Langium is a new project and I've been looking around. I intend to contribute (as soon as I have a bit of free time) to whatever project I'm using.

So here's my honest question: why port ANTLR4 instead of supporting Langium (which is also open source) or picking up antlr4ts where it was left off?

[mike-lischke]

Hi @svallory, I took the freedom to enable Discussions on this repository and transferred your contribution to it. I hope that's ok for you.

I've been active in the ANTLR community for a long time. I have written a parser in Java to convert some resource files for UIs. I have used previous ANTLR versions over many years (also in commercial products like MySQL Workbench and MySQL Shell for VS Code) and even have written the C++ runtime for ANTL4 (supported by Dan McLaughlin and David Sisson). So, I have quite a deep knowledge about the tool.

On the other hand I never heard about Langium before. Your question is the first time I came across this tool. From the description it seems to be a very powerful tool and is certainly good for many people needing the feature set it provides. Though, as you can imagine, I have no reason to switch, given that ANTLR4 is very active and has such a huge community.

Regarding antlr4ts: this is a bit of a special relationship. I'm using this runtime for years already and it has served me very well over that time. But it is kind of special. It does not live in the main ANTLR4 eco system, it's not updated anymore and in alpha state for a long time already. It contains some very special optimisations which are not in the official ANTLR4 runtime and require extra work to make that runtime compatible with ATNs created by the official ANTLR4 tool (antlr4ts comes with an own tool build). Additionally, there seems to be no interest to develop antlr4ts further (I asked) and I am interested in converting ANTLR4 entirely to Typescript, out of curiosity. That would allow me to generate parsers in my VS Code extension w/o requiring Java to be installed.

[svallory]

I don't mind you moving the discussion at all!

Firstly, thank you for taking the time to write back, and with so many details. I really appreciate it!

The main reason I asked is that I saw that ANTLR generates the lexer, parser, listener, and visitor base classes. I don't think with all that it would take much to implement an LSP for semantic highlighting, type-checking, and auto-complete suggestions, am I wrong?

If so, maybe I should switch now to the ANTLR4 grammar, and infrastructure, rather than later. The language I'm designing is of a higher order than programming languages and it tries to be as close to English as possible without ceasing to be a context-free, formal language.

To give you an idea, it is so flexible that you can actually write poems in it and turn them into code.

poem.disl	poem.ts

Not that it should be encouraged in a system specification! We are already using disl to specify projects where I work and it looks like this:

An entire system was specified here last week to put a project from a group of mentees in order (so don't mind the mess, they are still learning)

The reason I'm showing you all this is to give you a sense of what I'm up against and humbly ask for advice on tools or pitfalls ahead since you have so much experience with parsers. I've been designing this language for five years and have the rules in my head but I only started actually implementing it a month or so ago.

Also, I would love to hear what you think about the idea and ergonomics of the languages.

[mike-lischke]

You can use ANTLR4 definitely for a language server. I haven't written one so far, but created language support libraries in different tools (2 mentioned above, another is my VS Code extension for ANTLR4, which provides language support for ANTLR4 grammars, including code completion, syntax highlighting, code hints etc.).

When it comes to designing a new programming language you should have a good idea for whom this might be interesting. It's not like there aren't many programming languages already in the world and a new one needs to have something really compelling, to be of interest. The idea of a programming language, which sounds like a natural language has been around for many decades already, without really taking shape. This might have to do with the fact that a programming language must be really clear and straightforward. Natural languages tend to be the opposite.

[svallory]

When it comes to designing a new programming language you should have a good idea for whom this might be interesting.

Agreed! And disl 1 has a clear focus: project managers and team leaders. And its first goal isn't to be a programming language, but, even though the second example goes to a very concrete level, the goal is to be a specification language.

It's not like there aren't many programming languages already in the world and a new one needs to have something really compelling, to be of interest.

I realize that. That's why I'm focusing on an area where the competition is low: project specification. The options I know to tackle that problem are English and some diagrams (not all are easily understood by clients). Which leads to a lot of misunderstanding from the client regarding the scope and features.

Then there's ContextMapper which is completely focused on programmers and I think that, even being a programmer, if you don't know DDD well, you'll have a hard time understanding it.

I imagine the following use case:

• Bring your IDE to the meeting and gather the requirements in Disl (auto-complete, warnings for unhandled events or unused Concepts (that's a new data structure in Disl, similar to classes)
• Generate diagrams from Disl and discuss details with your team
• If you chose to write the doc closer to code than English, you can convert (even using a tool like Prettier) the statement:
  this.invitees.any(i => i.code == invitation)
  to
  
  before sending it to the client
• formal spec your client understands, leaves no margin for multiple interpretations and your team can implement?

I guess that's basically it... project scope, documentation, feature specification, and diagrams all from a single source of truth. That's the mountain I'm trying to climb LOL

This might have to do with the fact that a programming language must be really clear and straightforward. Natural languages tend to be the opposite

I think you may be right about the reason. I'm avoiding that complexity mainly by having sensible synonyms for operators so you can tense and pluralize them (is/are/to be) and treating natural language in statements as decoration only. The syntax highlighter has some issues, but those words are colored and treated like comments. Other tricks are spaces in class names, special char sequences and function calls not needing ()

---

Thank you again! I'll check your extension. But back to a4tstool... I tried to find documentation regarding the relationship between ANTLR and its runtimes, as well as what exactly each does, but couldn't find a good source.

As far as I get it, you are creating a TS runtime so you can use it to use data generated by ANTLR to parse a file in the language you generated the data for and provide language features. Is that it?

You can use ANTLR4 definitely for a language server. I haven't written one so far, but created language support libraries in different tools (2 mentioned above, another is my VS Code extension for ANTLR4, which provides language support for ANTLR4 grammars, including code completion, syntax highlighting, code hints etc.).

Looks like your extension does most of what an LSP does. I imagine you are using separate VS code APIs for each feature. Did chose not to implement an LSP for any reason or was it just how things happened to be?

Finally, thank you for all the time. The project looks really interesting. Is there anything I can help you with?

[mike-lischke]

Thank you again! I'll check your extension. But back to a4tstool... I tried to find documentation regarding the relationship between ANTLR and its runtimes, as well as what exactly each does, but couldn't find a good source.

Most of the syntactic analysis, including prediction, ATN walk etc. is done in the ANTLR4 runtime. For each supported target an own runtime exists to execute the generated parsers in this target language.

Sometimes there are multiple runtimes to choose from. For example there are 2 Python runtimes and also runtimes for Javascript and Typescript. The JS target could easily be used also in Typescript projects. Unfortunately, target authors often choose only to implement a minimalistic set of features, just enough to get their projects going (JS target) or decide to go own ways for optimisations (TS target), which is why I started my own TS target, which will be a (nearly) 100% translation of the base Java target.

As far as I get it, you are creating a TS runtime so you can use it to use data generated by ANTLR to parse a file in the language you generated the data for and provide language features. Is that it?

The goal of the new target is to get away from the problems with the other targets, be a good citizen in the ANTLR4 community and include all features.

Looks like your extension does most of what an LSP does. I imagine you are using separate VS code APIs for each feature. Did chose not to implement an LSP for any reason or was it just how things happened to be?

Tbh. I'm not sure why I started doing it like that. I though several times in the past about converting the code to use a language server. But so far I had no compelling reason. The speed of the extension is totally acceptable.

I would rather convert the TS runtime to WASM (e.g. via AssemblyScript) to increase its speed, before I start writing a language server, which has also to do with the idea to use the extension in a pure web environment (e.g. an ANTLR4 playground).

Finally, thank you for all the time. The project looks really interesting. Is there anything I can help you with?

It's my pleasure! If you really want to help then just grab the repository and use the runtime with a sample parsing project. That would require to clean up more of the generated files. I'm a bit slow on that because I develop my java2typescript project at the same time and often regenerate the TS sources which have not been cleaned up.

[svallory]

If you really want to help then just grab the repository and use the runtime with a sample parsing project.

Awesome! I'll do that this weekend. I'll get a better idea of what I get from ANTLR.

I really liked the easiness of working with Langium to create LSPs since the entire stack becomes Typescript-based. Their parser is currently based on Chevrotain which has limitations compared to ANTLR. I'm going to ask them if they would be willing to change it to ANTLR using a4tstool once it's mature enough. I would love the combination! Maybe they even offer some help :)

[svallory]

Hey, @mike-lischke! Do you mind adding to the README an overview of how to use the runtime? I was looking for a quick tutorial but didn't find a good one targeted at using a runtime.

E.g. I'm still unsure if I'll need Java and the original ANTLR to compile the parser and then run it in the runtime or if a4tstool will take care of all the commands. Also, does it install a binary? What's it called? what are the options?

[mike-lischke]

As an early adopter you unfortunately have all the burden to figure out what needs to be done on your own, which makes this part so difficult.

I mentioned it already, but let me make it as clear as possible: the runtime doesn't work currently, nor does it compile. It was machine translated and needs cleanup to even be accepted by tsc. I'm in the middle of that process, but there are quite a few files, which still need this cleanup.

The next stumbling stone is the generation of a parser. The main stream ANTLR4 tool includes, since the last release, a Typescript target, so it can be used to generate parser and lexer for a language.

And then you have to replace the references to the "TS" runtime (which is actually an incomplete JS runtime) by the runtime in this repo and should finally be able to parse something.

I have not gone through this process yet, as I'm still in the runtime generation phase.

To avoid confusion: the a4tstool repo in its entirety is like the mainstream ANTLR4 repository, but stripped down to the absolute minimum for the TS work. At some point it will contain the same generation code like the mainstream ANTLR4 tool, but converted to TS. The TS runtime is just a part of that (just like any other runtime in the mainstream repo) and will be released separately (as node module), once it is mature enough.

[svallory]

Thanks, @mike-lischke ! Things are clearer now! My main question was regarding this:

At some point it will contain the same generation code like the mainstream ANTLR4 tool, but converted to TS

I'm interested in having a completely TS-based process. It's fine if for now, we need the original ANTLR4 to generate the parser, as long as the goal is to ditch any JVM dependency in the long run.

I wouldn't mind helping finish the conversion. But I believed that would need to happen in turns so we don't step on each other's code changes. Do you by any chance work on a fixed schedule? Or do you see another way of splitting the work?

[mike-lischke]

Yes, good points. It's tricky to schedule in a certain manner as it often happens that you have to jump on a different file and fix that first before you can go back and finish the original one. Sometimes this even goes in loops.

So I wonder if you would mind to work on the JREE unit tests instead. They are in a different repository. You need both repos currently anyway, because you have to link the lib folder in the a4tstool repo to the lib folder in the other repo (or copy it, but that is not a good idea).

[mike-lischke]

@svallory Looks like you didn't find time to pick a task yet. Meanwhile I separated the JRE classes into an own project and released it as an own Node.js package (jree). However, the test coverage dropped significantly as I added lots of new classes and interfaces there and I really could need a helping hand to write tests for them.

[svallory]

Hi @mike-lischke, I haven't yet because the learning curve looks pretty steep right now since I'm new to ANTLR. I would need to study it, then study your code just to get something running.

Do you think you have time this weekend for an introductory call to help me set up an a4tstool dev env and give me some pointers?

I need to get some basic type-checking and cross-reference working on my language. We'll use it to define a second project and "Go to definition" and "Show references" are the features in VS Code we are missing the most.

Btw, I've created grammar files for DISL in XText and Langium, so I'm not totally new to automated parser creation.

[mike-lischke]

The good news is for the JREE you don't need any ANTLR4 knowledge. It's just Typescript classes to test (if you want that). For the a4tstool there's still clean up to do. Nothing compiles yet. I plan to improve the readme files for both project in the next couple days to help newcomers to get up and running quickly. If that's not enough we can either chat or do a call (depending on the timezones).

[svallory]

Hi, @mike-lischke! I was feeling under the weather this weekend so I spent most of it resting. I think a readme depicting the main moving parts should be enough. Let me know when it is done, I'll take a look and let you know if I still have any questions I can answer looking through the code after that

[mike-lischke]

I updated the readme last weekend, so I hope it's now more useful.