What will the target architecture look like ?

[ericvergnaud]

browser first:

• tool in browser, no backend required
• use LSP for IDE integration

unified wasm runtime:

• reduces each runtime to wrappers -> simpler and more consistent targets
• requires machinery to generate the wasm lexer and parser

other evolutions:

• targets as plugins

[ericvergnaud]

Here is how the runtime architecture will look like:

Only representing 3 target languages for clarity.
Also there are 2 generated wrappers per target: 1 for the parser and 1 for the lexer (arrows omitted for clarity)

[ftomassetti]

• If I look at this diagram, it seems to me that the Java, TS, and C# applications have access exclusively to the generated listeners and visitors and not also to the wrappers. Maybe additional arrows can be added
• What does the wrapper expose? The lexer and parser classes? Anything else?
• The generated WASM lexer and parser are generated by the ANTLR 5 tool, right? But who is generating the wrappers? Is that also the tool? Or is it a plugin for the tool? Or is that something that separate tools can do? I am not that familiar with WASM, so I am not sure if there are tools that can take a WASM file and generate Java wrappers

[ericvergnaud]

You are correct that some arrows are missing. The app can access listener, visitor and wrappers.
The wrapper can in theory expose anything, I guess that will grow over time and feature requests.
The wrappers are generated by target-specific add-ons (plugins).

[ericvergnaud]

Updated the diagram

[jimidle]

Well, the only thing here is that I think a lot of potential targets will never be able to use cllas in to WASM from a wrapper. I very much doubt that Go will evere be able to use that in anger for instance. They are just not really compatible. I am very skeptical of this approach I am sorry to say.

[ericvergnaud]

Let's see how it goes, I hope to prove you wrong soon enough 🤣

[ericvergnaud]

Here is how the tool architecture will look like:

Only representing 1 target add-on for clarity.

A critical feature is the direct contract between the tool WASM and the target language WASM. This ensures that the add-on can run in any environment (browser, LSP, CI/CD...)

[ericvergnaud]

As discussed here, the tool will not initially be able to compile the generated parser and lexer in a browser. That said, the above architecture remains valid for a number of reasons:

• the tool can run the interpreter in a browser
• sticking to wasm for add-ons ensures minimal migration in the future (once we have a usable compiler that runs in the browser)

[ericvergnaud]

What's in the tool ? What's in the wrapper ?

Although the WebAssembly committee is actively working on WASI (a spec for calling the host from within the WASM) it will likely take years for it to become broadly available. In the meantime (and maybe forever) we want ANTLR WASM components to only deal with text (and text streams).
This is where the wrappers come into play: they not only serve as a friendly-api provider for the WASM components, but they also deal with the file system, the network etc.. This stuff is mundane, and doesn't even need to be identical from one language to another.

As an example, the runtime will not accept a file as an input, but rather a character stream as an input, which fetches the characters through a callback to the wrapper. Similarly, the runtime will log warnings and errors through a stream, and will output a serialized ParseTree. The wrapper will know how to deserialize the ParseTree into 'something' that can be traversed by the ParseTreeWalker (how much data will be available locally vs fetched via calls to the WASM parser is a technical detail).

Similarly, the tool wrapper will take care of reading options, loading target add-ons, etc... It could provide everything required for lexer/parser generation in the form of a "provider", that the core tool would invoke to get options, main, imported and included grammars, etc...

[kaby76]

Question: How do I automate testing of grammars in a Github Actions Bash command-line shell if WASI is not available? (Still reading WebAssembly specs.)

[ericvergnaud]

We're not looking to require WASI, rather the wrapper around the WASM engine will take care of IO.

[kaby76]

(Sorry for the naive question, as I still am reading the tools and specs.)

Yes, IO support already exists, but it depends on the toolchain. E.g., in the Emscripten compiler for C++ with Node.js, I can use stdio.

12/30-06:44:23 ~/wasm
$ cat > hello.cpp <<EOF
#include <string>
#include <iostream>

int main() {
	std::string s = "Hello World";
	std::cout << s << std::endl;
        return 0;
}
EOF
12/30-06:44:44 ~/wasm
$ emcc hello.cpp
12/30-06:44:51 ~/wasm
$ node a.out.js
Hello World
12/30-06:44:57 ~/wasm
$

But, I cannot use open because it is a posix concept. test-demo.sh.txt

$ bash test-demo.sh.txt
usage: C:/msys64/home/Kenne/wasm/a.out.js <from> <to>
program exited (with status: 1), but keepRuntimeAlive() is set (counter=0) due to an async operation, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)
12/30-07:18:19 ~/wasm
$

However, it is supported with WASI, with some prototyped tools to support C. https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-tutorial.md#running-common-languages-with-wasi

Without a file system, how do we write command-line tools to write a file?

[kaby76]

Sorry, my "open()" example had a bug. argv[] does not seem to be supported, or something in the "glue" .js isn't right. Attached is an updated Bash script that shows open() does not work. test-demo.sh.txt. As far as I can tell, Emscripten likely defines open() as a function that always returns -1. So, the question still stands: how do we do anything without a file system (e.g., for ".tokens", for ".dot"-generated files, etc).

[ericvergnaud]

I guess it would look something like the following:

writer = new FileWriter()
writeCallback = (stuff) -> writer.write(stuff);
wasm.runFunction("xyz",  { writeCallback: writeCallback })

Practically, the wasm knows nothing about files, it's just provided with a callback function.

[ericvergnaud]

I'm leaving maven and LSP aside for now, until the above is discussed and reaches consensus.

[KvanTTT]

The topic contains high-level/external architecture description (API, wrappers), but it doesn't cover low-level/internal architecture like processing stages (ParseTree -> ATN (also its lowerings) -> target-language (codegen)), general algorithms (ALL, is it possible to simplify or even remove DFA in runtime?). I'm going to create a new topic or extend the existing one after a while.

[jimidle]

Ha :). Ok mate. Go for it!

…

On Thu, Feb 15, 2024 at 15:16 ericvergnaud ***@***.***> wrote: Let's see how it goes, I hope to prove you wrong soon enough 🤣 — Reply to this email directly, view it on GitHub <#6 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMCFCS762OLDZIPGAG3YTZ3KFAVCNFSM6AAAAABAWYEF7GVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIOBVGYYTI> . You are receiving this because you commented.Message ID: ***@***.***>