From cbe1886b47b1afedced267e6f533e8a16c262ad7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicol=C3=B2=20Ribaudo?= Date: Fri, 21 Jun 2024 18:09:24 +0200 Subject: [PATCH 1/3] Create 0000-format-preserving-code-printer.md --- rfcs/0000-format-preserving-code-printer.md | 208 ++++++++++++++++++++ 1 file changed, 208 insertions(+) create mode 100644 rfcs/0000-format-preserving-code-printer.md diff --git a/rfcs/0000-format-preserving-code-printer.md b/rfcs/0000-format-preserving-code-printer.md new file mode 100644 index 0000000..1639277 --- /dev/null +++ b/rfcs/0000-format-preserving-code-printer.md @@ -0,0 +1,208 @@ +# Format-preserving code printer + +- Repo: babel/babel, or starting as its own separate repo +- Start Date: 2024-05-02 +- RFC PR: +- Related Issues: +- Authors: Nicolò Ribaudo +- Champion: Nicolò Ribaudo +- Implementors: + +# Summary + +This RFC proposes introducing a new mode in `@babel/generator`, to support printing code by presevring the original format. `@babel/generator` already has a `retainLines` option to preserve line numbers, and this RFC is taking it one step further. + +# Basic example + +Consider a Babel configuration that transforms the following input code down to ES5: + +```js +let x = () => { + return ( + 1 + 2 + ) * 3; +}; + +console.log(x); + +for (var i = 0; i<3; i+=1) + console.log(i); +``` + +Currently, Babel generates the following code: +```js +var x = function () { + return (1 + 2) * 3; +}; +console.log(x); +for (var i = 0; i < 3; i += 1) console.log(i); +``` + +Under this new mode, Babel would try to preserve the original formatting as much as possible: +```js +var x = function () { + return ( + 1 + 2 + ) * 3; +}; + +console.log(x); + +for (var i = 0; i<3; i+=1) + console.log(i); +``` + +# Motivation + +While we think of Babel's output as basically "bytecode for the browser", and not meant to be read/modified by humans, that is in practice not the reality. + +There are two main reasons why as a user you may want Babel to better preserve as much code style information as possible when transforming the code: + +## Code migration tools + +Babel can be used as a tool to write "codemods": you write a pugin to transform your code, for example to migrate across versions of your library, and run it on your _source_ file overwriting them. See, for example: +- https://github.com/kentcdodds/babel-codemod-example +- https://assets.ctfassets.net/nn534z2fqr9f/3bytbtRUOWLkWugqGgn174/18fa7bf319ac168a5aa7644747a30bfb/Babel__A_refactoring_tool.pdf +- https://github.com/codemod-js/codemod + +These tools however face smoe challanges when it comes to preserve the output format: +1. they can assume that the input code was formatted with Prettier, and re-format the output code with Prettier; +2. they can use Babel together with [recast](https://github.com/benjamn/recast), but this only works when Recast is up-to-date with whatever Babel AST nodes are being used. +3. use Babel to find the locations where to change the code, and then do manual string slicing rather than AST transforms + +## Debugging the compiled code + +The more similar the compiled code is to the originally authored one, the easier it is to debug it. Other than step-by-step debugging, having accurate line _and column_ numbers ensures that stack traces point to the appropriate locations. + +One alternative solution to this is to use source maps, rather than preserving the code formatting, but this also has its drawbacks: +- the quality of source maps is not always optimal (including those generated by Babel), leasing to potentially confusing +- given that source maps can hide the code that is being executed, some companies' security policies recommend not using them and instead always checking the code that is actually running + +# Detailed design + +## API + +We currently have two options to preserve some formatting in the output code: +- `retainLines`, that preserves line numbers +- `retainFunctionParens`, that preserves parentheses around function expressions (because some engines use them as a hint to eagerly compile the function body) + +To implement this RFC, we need two new options: +- `retainParens`, to preserve all parentheses from the input code. +- `retainColumns`, to preserve the _column_ of the tokens other than their lines. + +## Implementation + +A guiding principle is that this new option should not require all the transforms, and most of the code printing logic, to be aware of it. Instead, it should be able to automatically match the locations of the generated code to those in the original code. It's possible that some plugins would need some ad-hoc handling, similarly to how sometimes we need to explicitly change some locations to improve source maps, but that must be the exception and not the rule. + +While `@babel/generator` doesn't internally build a list of tokens, it concatenates string and every piece is de facto one JavaScript token. Those can be seen in the various [`.word("...")`](https://github.com/babel/babel/blob/8a8785984fddeed53934b5e23d0990050a196523/packages/babel-generator/src/generators/statements.ts#L262) and [`.token("...")`](https://github.com/babel/babel/blob/8a8785984fddeed53934b5e23d0990050a196523/packages/babel-generator/src/generators/statements.ts#L322) calls through the code. + +The [core of the code generator](https://github.com/babel/babel/blob/c36fa6a9299df95b801695736f78ac87211f009f/packages/babel-generator/src/printer.ts) maintains a stack of nodes that are being printed, so whenever one of those `.word`/`.token` calls happen, the [core of the code generator](https://github.com/babel/babel/blob/c36fa6a9299df95b801695736f78ac87211f009f/packages/babel-generator/src/printer.ts) happens we know which code we are printing. We can thus traverse the list of input tokens, and if there is any token in the input list that: +1. matches the token that we are printing +2. is a _direct descendant_ of the node we are printing + +we can use its location. + +This alrogithm will probably require some caching to avoid traversing the list of tokens multiple times, as well as some optimized search based on the fact that most tokens are going to be printed after the same token that preceeded them in the input stream. + +The (2) requirement is needed so that when, for example, printing the outer `BinaryExpression` in `(1 + 2) + 3`, we correctly associate the `+` token to print with the second `+` token in the input and not with the first one. + +This strategy works because most nodes directly contain at most one occurrence of the same token. There are a few exceptions, that will probably need special handling: +- `,` in lists (function arguments, array literals, object literals, import attributes) +- `;` in the head of `for` loops + +## Testing + +We should have two types of tests: both with and withuot transformations applied. + +For the first case, we can just re-use all the `input.js` files in the generators test and verify that `generate(parse(input)) === input`. + +For transformations-related tests, such as testing what happens when converting `const` to `var`, we should probably keep the tests in the transform plugin packages. + +# Drawbacks + +This change can have a significant performance impact, since for every part of the output code `@babel/generator` has to find the original corresponding token in the tokens list to match its position. + +We must make sure that the logic is self-contained enough to not affect performance of the classic code printer. + +# Alternatives + +It would be possible to implement this as a separate code generator, that users can specify through the [`generatorOverride`](https://github.com/babel/babel/blob/8a8785984fddeed53934b5e23d0990050a196523/packages/babel-core/src/config/plugin.ts#L13) option. However, having it as a separate fork will have a much higher maintainance cost. + +# Adoption strategy + +This is opt-in, and it will never replace the default behavior. Users that need it can simply turn on the new mode. + +# How we teach this + +This isn't a particularly complex concept to understand, so simply documenting the new option would be enough. + +Once that is done, it'd be good to have examples of how it can be used to refactor code. + +# Open questions + + + +**What is the exact interaction between `retainLines` and `retainColumns`?** + +It would be useful to be able to use `retainColumns` without also having to use `retainLines`, but with it still trying to preserve relative column changes. + +For example, given this input code: +```js +let a = + + 1 + 2; +``` +and a transform that simply adds `"use strict";` at the beginning, there are two possible output. One that preserves line numbers and column numbers as much as possible: +```js +"use strict";let a= + + 1 + 2; +``` +and one that preserves _formatting_ as much as possible, even if the line numbers don't match anymore: +```js +"use strict"; +let a = + + 1 + 2; +``` + +Maybe instead of `retainLines`/`retainColumns` we should have `retainLines`/`preserveFormat`, where for now `preserveFormat` requires `retainLines` (giving the first output) but in the future could work independently (giving the second output). `preserveFormat` would also imply `retainParens`. + +What approach is the best? + +**What to do about `` comments?** + +Given that `` are interpreted as comments in some environments, Babel always enforces a space between them: ``. Here the choice is between preserving the exact code formatting, or preserving the code meaning (i.e. making sure that they are not interpreted as comments). + +I'd erro towards the side of safety, but if we decide to go for the other behavior I'd want the safe behavior to still be presend behind some option/plugin. + +## Frequently Asked Questions + + + +## Related Discussions + +This is a common feature request: + +- [#16301](https://github.com/babel/babel/issues/16301) — _[Bug]: The source code is indented to 4 Spaces, babel translated into 2_ +- [#10674](https://github.com/babel/babel/issues/10674) — _@babel/generator output code different with input code +- [#8974](https://github.com/babel/babel/issues/8974) — _@babel/generator is removing the parentheses_ +- [#497](https://github.com/babel/babel/issues/497) — _Retaining whitespace/pretty printing_ + From c3ab697aa68ebd1600d8bb4a60ee96d390c86b32 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicol=C3=B2=20Ribaudo?= Date: Fri, 21 Jun 2024 18:12:25 +0200 Subject: [PATCH 2/3] Update 0000-format-preserving-code-printer.md --- rfcs/0000-format-preserving-code-printer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0000-format-preserving-code-printer.md b/rfcs/0000-format-preserving-code-printer.md index 1639277..6475fc5 100644 --- a/rfcs/0000-format-preserving-code-printer.md +++ b/rfcs/0000-format-preserving-code-printer.md @@ -1,6 +1,6 @@ # Format-preserving code printer -- Repo: babel/babel, or starting as its own separate repo +- Repo: babel/babel - Start Date: 2024-05-02 - RFC PR: - Related Issues: From 38bdd84a5a66e4f9be415f29a22438e617acdfed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicol=C3=B2=20Ribaudo?= Date: Sun, 30 Jun 2024 20:41:52 +0200 Subject: [PATCH 3/3] Update 0000-format-preserving-code-printer.md --- rfcs/0000-format-preserving-code-printer.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/rfcs/0000-format-preserving-code-printer.md b/rfcs/0000-format-preserving-code-printer.md index 6475fc5..0206072 100644 --- a/rfcs/0000-format-preserving-code-printer.md +++ b/rfcs/0000-format-preserving-code-printer.md @@ -86,9 +86,12 @@ We currently have two options to preserve some formatting in the output code: - `retainLines`, that preserves line numbers - `retainFunctionParens`, that preserves parentheses around function expressions (because some engines use them as a hint to eagerly compile the function body) -To implement this RFC, we need two new options: -- `retainParens`, to preserve all parentheses from the input code. -- `retainColumns`, to preserve the _column_ of the tokens other than their lines. +To implement this RFC, we would introduce one new option: +- `preserveFormat`, to preserve line-by-line the format of the original code (where there are no new nodes injected). + +`preserveFormat` can be used together with `retainLines`, to make sure that even when there are new nodes injected: +- all other nodes are in their original location +- all lines not affected by the new/transformed nodes preserve their formatting ## Implementation @@ -153,9 +156,9 @@ Once that is done, it'd be good to have examples of how it can be used to refact If you plan to implement this on your own, what help would you need from the team? --> -**What is the exact interaction between `retainLines` and `retainColumns`?** +**How to format new injected nodes?** -It would be useful to be able to use `retainColumns` without also having to use `retainLines`, but with it still trying to preserve relative column changes. +It would be useful to be able to use `preserveFormat` without also having to use `retainLines`, but with it still trying to preserve relative column changes. For example, given this input code: ```js @@ -177,9 +180,7 @@ let a = 1 + 2; ``` -Maybe instead of `retainLines`/`retainColumns` we should have `retainLines`/`preserveFormat`, where for now `preserveFormat` requires `retainLines` (giving the first output) but in the future could work independently (giving the second output). `preserveFormat` would also imply `retainParens`. - -What approach is the best? +I am not sure yet about how such algorithm could work. **What to do about `` comments?**