fix: Correct error message for invalid bytes in multiline strings and comments

[WillLillis]

This change provides a correct error message when an invalid byte is found inside of a multiline string line, comments, and doc comments.

Inside multiline strings

Taking the example from #20900:

const foo =
    \\const S = struct {
    \\<TAB>// hello
    \\}
;

Before:

src/main.zig:4:1: error: expected ';' after statement
    \\ // hello
^

After:

src/main.zig:4:5: error: expected 'a string literal', found invalid bytes
    \\ // hello
    ^~~~~~~~~~~
src/main.zig:4:7: note: invalid byte: '\t'
    \\ // hello
      ^~~~~~~~~~~

Doc comments

/// Some <TAB>comment

Before:

src/main.zig:1:1: error: expected type expression, found 'invalid token'
/// Some  comment
^~~~~~~~~~~~~~~~~

After:

src/main.zig:1:1: error: expected 'a document comment', found invalid bytes
/// Some  comment
^~~~~~~~~~~~~~~~~
src/main.zig:1:10: note: invalid byte: '\t'
/// Some  comment
         ^~~~~~~~~~~~~~~~~

Comments

// Some <TAB>comment

Before:

src/main.zig:1:1: error: expected type expression, found 'invalid token'
// Some  comment
^~~~~~~~~~~~~~~~~

After:

src/main.zig:1:1: error: expected 'a comment', found invalid bytes
// Some  comment
^~~~~~~~~~~~~~~~~
src/main.zig:1:9: note: invalid byte: '\t'
// Some  comment
        ^~~~~~~~~~~~~~~~~

I'm not 100% confident in the set of invalid bytes I added in lowerAstErrors, as it's just based off of those listed in the multiline_string_literal_line case of Tokenizer.next(). I think this part of the PR in particular could benefit from a close second check. :)

I initially thought that the similar problem with invalid bytes inside comments (mentioned here), but there isn't a .comment variant (only .doc_comment and .container_doc_comment) of Token.Tag to report the expected token as. Another variant to Ast.Error.Tag could be added to handle this case, but I wanted to check before doing so. See followup comment.

One minor nit/ bikeshed question: The rendered error message for this case reads error: expected 'a string literal', found invalid bytes. Since this error is triggered by the first single invalid byte encountered by the tokenizer, should this message use "byte" instead?

• Closes unhelpful error message on invalid bytes inside multi line string literal #20900
• Closes The error message for TAB in comments is very unclear and misleading #22160

[WillLillis]

Looking at this again while rebasing, I realized handling invalid bytes inside comments was pretty straightforward. I put the associated changes in a separate commit so it can be reverted in case I took the wrong approach.

[Vexu]

This doesn't need to be split into an error and a note like the previous implementation was, "comment/string contains invalid byte '<byte>'" would be more readable IMO.

[WillLillis]

This doesn't need to be split into an error and a note like the previous implementation was, "comment/string contains invalid byte '<byte>'" would be more readable IMO.

I thought the error/note implementation was helpful because the note shows exactly where the (potentially invisible) invalid byte is in the token. Do you think the gain in readability is worth the tradeoff in losing this information? If so I'm happy to try to rework this soon (hopefully later today/tomorrow).

[Vexu]

You could have the error point at the invalid byte. I don't think the start of the token (so //, \\ or ") is that necessary since it's always on the same line.

[Vexu]

A further improvement would be adding an unterminated string/char literal error for string and char literals where the invalid byte is EOF/NL.

[WillLillis]

A further improvement would be adding an unterminated string/char literal error for string and char literals where the invalid byte is EOF/NL.

That's a good idea. I implemented this locally, would it be a better to push those changes to this branch or open a followup PR once this is merged?

[Vexu]

I think a follow up would be better.