Interest: adding a test:eslint-compat script for upstream ESLint ecosystem tests?

[JoshuaKGoldberg]

👋 Coming over from eslint/eslint#19139 -> eslint/rfcs#127: we're working on an RFC in ESLint core to add "ecosystem tests". We'd like to test popular third-party plugins such as eslint-plugin-unicorn as part of ESLint's CI. The goal is to detect API breakages before versions of ESLint or plugins are published that are incompatible with each other.

The current version of the RFC suggests:

• Plugins would add a test:eslint-compat script to run rule tests
• ESLint would add a CI job that runs installation, build, and then the test:eslint-compat job for each plugin

Question: is this something that would be ok to add in this repo?

The RFC is also still very much open for comments if anybody is interested! 🙂

[fisker]

Personally I'm fine to add it.
What I can't understand is why it's necessary to add a special script.
Can't we try run "test" by default?
Or can't we have a configuration file to tell what to run for each plugin?

[sindresorhus]

I'm fine with this too.

[JoshuaKGoldberg]

What I can't understand is why it's necessary to add a special script.
Can't we try run "test" by default?
Or can't we have a configuration file to tell what to run for each plugin?

Some repos attach more than just ESLint/rule tests to the test script. Examples:

• This repo has both lint and test:*
• typescript-eslint runs tests for everything from rules to utility packages

The RFC's idea was that a specific test:eslint-compat script would let plugins choose exactly what they want tested for compatibility.

Are you thinking you'd want everything to be tested for compatibility? E.g. including lint?

[fisker]

No, I mean, the test should run test script by default, but run test:js for us. This should be done on ESLint side, not plugin side. Add a script that not used by us is strange, and we don't even know when ESLint will stop running tests for us. In my experience, most integration tests not
done on the client side.

• babel have scripts for every projects they
  tested https://github.com/babel/babel/tree/main/scripts/integration-tests
• Node.js has configuration file for different repos https://github.com/nodejs/citgm/blob/83ffabd74ef2ab309301678ec0e872d630870ba4/lib/lookup.json#L1