Consider using parser gem across all Ruby versions

[alisnic]

Problem

Solagraph uses RubyVM::AbstractSyntaxTree when it's available. When not, it's using the parser gem. This creates friction if a solargraph plugin wants to support different Ruby versions, because:

class SomeConvention < Solargraph::Convention::Base
  def local source_map
     # PROBLEM: depending on Ruby Version, source_map#node will be a different object
     # For ruby < 2.6, it will be an instance of AST::Node
     # For greater, it will be RubyVM::AbstractSyntaxTree::Node
  end
end

This leads to the fact that if I have code that uses source_map#node, like for extracting methods from db/schema.rb, I have to have 2 implementations depending on Ruby version https://github.com/alisnic/solar-rails/blob/master/lib/solar-rails/schema.rb#L66.

Solution

Proposed solution is to use parser gem across all ruby versions, especially since Ruby's own AST docs stipulate that it's not stable:

If you are looking for a stable API or an API working under multiple Ruby implementations, consider using the parser gem or Ripper. If you would like to make RubyVM::AbstractSyntaxTree stable, please join the discussion at bugs.ruby-lang.org/issues/14844.

https://docs.ruby-lang.org/en/master/RubyVM/AbstractSyntaxTree.html

[castwide]

I have to admit parser has practical benefits that make it a tempting choice. From an objective standpoint, it wins in terms of robustness, stability, and maturity. From a subjective standpoint, I prefer the syntax of its generated ASTs. The only major issue I have with it is speed.

Supporting two parsers is an unfortunate complication that was introduced in v0.39.0 for performance reasons. RubyVM is consistently faster than parser in all my benchmarks, sometimes by as much as an order of magnitude (e.g., ~14ms vs. ~144ms parsing lib/solargraph/api_map.rb). Given how often ASTs need to be built and rebuilt during active editing, RubyVM results in significantly better UX in Ruby 2.6+.

I continued supporting parser to avoid dropping support for Ruby 2.4+, at least in the short term. I'm not sure I want to go back to a parser-only solution and lose the performance enhancements that benefit, by my estimate, well over 95% of Solargraph users.

Side note: I've also considered the possibility of going RubyVM-only since RBS support would require Ruby 2.6+ anyway.

[alisnic]

Makes sense. I wasn't aware of the performance tradeoff that you were facing. I'll just manually call parser gem to keep the code simple, and think about transitioning to RubyVM AST code.

Should I close the issue?

[castwide]

Makes sense. I wasn't aware of the performance tradeoff that you were facing. I'll just manually call parser gem to keep the code simple, and think about transitioning to RubyVM AST code.

Should I close the issue?

Yeah, this should be okay to close for now. If you think of another approach that might allow us to maintain a single parser with minimal impact to performance, please feel free to open a new issue.

[eregon]

As a note, RubyVM::AbstractSyntaxTree is an experimental and unstable API, for instance the order of children is not guaranteed, see https://docs.ruby-lang.org/en/3.1/RubyVM/AbstractSyntaxTree.html for more details.
It will also only ever be available on CRuby, never on other Ruby implementations, since RubyVM is CRuby-specific APIs.

So IMHO it's a mistake for any gem to use RubyVM::AbstractSyntaxTree because it's very likely to break at some point and it's not nice to use anyway.

I do sympathize with the wish to parse faster.
I think we should have something similar to the parser gem but standard in all Rubies and built-in for maximum efficiency (there could also be a native extension for older versions).
Maybe it's something we can achieve with the upcoming parser from @kddnewton.

[kddnewton]

You could use https://github.com/ruby-syntax-tree/syntax_tree if you wanted for this. It's just one layer on top of ripper, so it gets the benefits of the fast parser without an experimental/unstable API. It also is supported on other Ruby runtimes like JRuby/TruffleRuby because they also ship a ripper implementation. I would be more than happy to pair on/help with any migration if you wanted to experiment with it.

[kddnewton]

With the new parser I'm writing, I plan on building Syntax Tree nodes directly from the parser, so it should end up being the fastest and most stable option around.