string.utf_codepoint returns an error for valid codepoints U+FFFE and U+FFFF

[mooreryan]

The string.utf_codepoint returns an error for valid codepoints U+FFFE (65,534) and U+FFFF (65,535).

The line of code is here.

These two codepoints are two of the 66 so-called "noncharacter code points". Here are some excerpts from 23.7.1 Noncharacters: U+FFFE, U+FFFF, and Others of the Unicode core spec:

...they are not illegal in interchange, nor does their presence cause Unicode text to be ill-formed....They are not prohibited from occurring in valid Unicode strings which happen to be interchanged. This distinction, which might be seen as too finely drawn, ensures that noncharacters are correctly preserved when “interchanged” internally, as when used in strings in APIs, in other interprocess protocols, or when stored....It is not recommended to simply delete noncharacter code points from such text, because of the potential security issues caused by deleting uninterpreted characters

If the goal is for string.utf_codepoint to return Error(Nil) for noncharacter codepoints, then there are many missing noncharacter codepoints that do not return an error, e.g., U+1FFFF.

However, I think that the correct behavior would be for the string.utf_codepoint function to return Ok for U+FFFE and U+FFFF. (As examples, the Gleam accepts the literals "\u{FFFE}" and "\u{FFFF}" as valid unicode code points, and both Elixir and Rust also accept them as valid code points)

[lpil]

Thank you

[jooaf]

Hello, if it is possible may I pick this up? For building out a solution and testing locally, should I just run gleam build and gleam test. Sorry if this is a naive question.

[mooreryan]

Hey @jooaf, I apologize--I did not notice you commented on this issue asking to work on it before I went ahead and fixed it. If you want I could close my PR and you could take it (since it is a good first issue!)

[jooaf]

Hey @mooreryan, no worries at all! That's so kind of you! Since you have already done the work, I think you should go ahead with your PR. I will just be on the lookout for another good first issue :)

[mooreryan]

Completed in #781.