Update float syntax rules and formatting

[DaelonSuzuka]

Per @AlfishSoftware's feedback or #746, this PR removes the float literal rule I copied from the MagicPython grammar and (hopefully!) fixes the existing float rules.

[AlfishSoftware]

The best idea would be to take the Godot source as a reference, but sadly it's difficult to interpret the tokenizer code into a regex. But looking at the error messages, it disallows multiple adjacent _ (though it weirdly allows trailing _; probably an oversight on their end).

You should probably replace all [0-9][0-9_]* with \d+(?:_\d+)* -- or \d+(?:_\d+)*_? if you want to be super accurate and allow even a trailing underscore. Also I think line 270 might also match no mantissa digits, like just a . or .e1.

I tried to come up with one taking GDShader as a base and modifying it. I think these regexes work for float literals:

• \d+(?:_\d+)*_?[eE][-+]?\d+(?:_\d+)*_? (no dot, exponent mandatory)
• (?:(?:\d+(?:_\d+)*_?)?[.]\d+(?:_\d+)*_?|\d+(?:_\d+)*_?[.])(?:[eE][-+]?\d+(?:_\d+)*_?)?
  (with dot: digits are mandatory in at least one of both sides, exponent is optional)

You can inspect and test them here (both are joined with a |): https://regex101.com/r/6VUN0B/1

Note that I didn't include word boundaries, as I suppose you already have them around all identifiers and keywords.

---

If you use this, remember to do a similar replacement also on the integer literals (binary, decimal and hex) to disallow consecutive _ on them as well:

• Binary: 0b[01]+(?:_[01]+)*_?
• Decimal: \d+(?:_\d+)*_?
• Hex: 0x\h+(?:_\h+)*_? (\h is valid in Oniguruma regex, equivalent to [0-9a-fA-F])

[DaelonSuzuka]

@AlfishSoftware Great info, thanks!

Hex: 0x\h+(?:\h+)*? (\h is valid in Oniguruma regex, equivalent to [0-9a-fA-F])

Very cool, I've always tried to stick to very basic regex to hopefully avoid implementation specific details, but you've convinced me to actually read the Oniguruma docs.

Also I think line 270 might also match no mantissa digits, like just a . or .e1.

Good catch, it does!

The best idea would be to take the Godot source as a reference
<snip>
disallows multiple adjacent _

I'm conflicted about this because, imo, the goal of the textmate grammar is not to strictly enforce the language/interpreter rules. The exact language rules aren't formally specified, and that's a moving target, and I'm not abandoning support to Godot 3-specific syntax. Therefore, the grammar's target must be the union of all the valid syntax of all versions of Godot 3, all versions of Godot 4, plus some margin to allow for mistakes.

It's also intended to be used in conjunction with the Godot LSP, which will yell about consecutive underscores anyways:

[AlfishSoftware]

I'm not abandoning support to Godot 3-specific syntax.

Ah, you're completely right. I didn't test it in Godot 3 at all, just assumed it was the same. Plus, there's no harm in allowing adjacent underscores as it doesn't conflict with any other syntax. The _ in Godot 3 seems to be allowed even after ., e, b and x, so those regexes must be right (except for that dot-only case).

I've always tried to stick to very basic regex to hopefully avoid implementation specific details

You're right, that's actually a good idea, as there's stuff like porting to GitHub syntax, etc. IIRC, there's not that much difference in Oniguruma; what stands out to me is mostly \h and \G (which it think can be useful in sub-patterns, if I understand correctly).

[AlfishSoftware]

There's no harm in allowing it (and the LSP will raise an error anyway), but just in case you missed it, I think it's also matching cases with just _ after e without a number. E.g.: .0e_, 0.e_, 0.0e_, 0e_