Automatic testing to confirm that improper usage fails Pyright type-check

[WasabiFan]

I am writing some library code which takes heavy advantage of type checking to assert correctness before runtime (a la typestate programming). Since appropriate behavior of my code includes catching certain mistakes at type-check-time, I'd like to integration-test the typing of my module.

Specifically, I'd like to be able to write a set of Python tests which, when run, will pass a sample through Pyright and snapshot-test the checker output. My tests should fail if the code sample passes type-check, or if the output from Pyright changes relative to past runs.

What I'm looking for is similar in spirit to trybuild in Rust, which runs the Rust compiler on samples to validate returned errors.

I'm confident there are tests of this sort in Pyright itself. I've glanced through the samples but haven't taken a close look at the infrastructure. I suspect that this existing infrastructure wouldn't be of great use in an otherwise Python project (I want pytest to be the singular test harness entry point) but I would love to hear any learnings from that.

Is anyone aware of an existing project or prior art in this area? If not, are there any particular suggestions on how one should go about doing this? It would be pretty straightforward to write a utility library to invoke the CLI on test files, but since Pyright is a Node.js library rather than native Python it'll be slightly ugly; I figure others might have thoughts on the least-bad way to do it.

[JelleZijlstra]

python/mypy#8655 asks for a similar feature and has some discussion of what's already possible with mypy.

Another place to look would be the test suites for existing mypy plugins, such as pydantic at https://github.com/samuelcolvin/pydantic/blob/5261fd05a0374b84ce2602d45990baf480fa2417/tests/mypy/test_mypy.py

I don't have anything to share regarding pyright. I suspect it's a bit more difficult to extend its internal test suite in a Python project because pyright is written in TypeScript, not Python.

[jakebailey]

If not, are there any particular suggestions on how one should go about doing this? It would be pretty straightforward to write a utility library to invoke the CLI on test files, but since Pyright is a Node.js library rather than native Python it'll be slightly ugly; I figure others might have thoughts on the least-bad way to do it.

I think you're going to hit this no matter what; pyright runs in node so you'll need node to run it.

If you're looking for something in CI, I maintain https://github.com/jakebailey/pyright-action for GitHub Actions, which runs pyright over whatever you want and then reports the errors. But, if you're looking to have a list of known errors and check diff, I think you'd need to be using --outputjson and reading that to compare.

[WasabiFan]

--outputjson is a good tip, thanks!

In my case, I have a separate CI pipeline (GitLab) running my tests and I want this new flavor to hook in as a standard Python test, so the action itself probably won't be a part of this flow, but I'll certainly check it out to get a sense of what automating the CLI looks like.

I think you're going to hit this no matter what; pyright runs in node so you'll need node to run it.

Agreed. The question is more about mechanism: options I see include running the CLI directly, or hooking into internals like what the other answer suggested based on mypy. Both involve Node, but the former would use purely the pre-packaged tool while the latter would launch a custom Node.js script which then invoked arbitrary Pyright logic. My inclination remains to go with the wholesale CLI route, but I'm interested in hearing if there are thoughts in the other direction (APIs of interest).

[jakebailey]

Pyright doesn't have any stable APIs or anything to this extent; the only thing we have is the CLI exectuables. I think that really depends on what you're looking for, and if it's straight up diagnostics, then I think --outputjson is going to do what you're looking for.

I'd think you could also diff the program outputs even without JSON to get a change; this is what TypeScript does for its own integration testing. For example, they typecheck pyright itself periodically to find any regressions and guide the default errors they have there, and diff the typechecker errors.

If you're looking for another "call pyright from Python" example, this is what we have in typeshed for manual testing: https://github.com/python/typeshed/blob/master/tests/pyright_test.py (though I've upgraded them to my action there for PR commenting stuff).

[WasabiFan]

Thanks! I think that's all I need. I'll probably go for the "diff the textual output" approach, assuming I indeed get around to implementing this. I'm sure if I do I'll throw it up on GitHub for others.

[WasabiFan]

Since it didn't seem like there was much prior art in this area, I've implemented some simple infrastructure for this myself.

I developed it initially in a private repo but I've re-posted it here for others' reference: https://github.com/WasabiFan/pyright-ui-test-poc

I'd love to hear thoughts/feedback on it and whether others think it would be useful. I have developed a spatial frame transformation library as part of an internal project which uses generics to denote the reference frame a measurement is in, and am now testing it with these utilities. So far, it's been going pretty well.

Thanks to everyone for the other answers and discussion in this thread.

[erictraut]

This is looking good. Thanks for exploring this.

A couple of thoughts and suggestions.

1. It's a bit unfortunate that the test code snippets need to live within string literals. This prevents you from opening the code in an editor with pyright or pylance installed and seeing at a glance the same diagnostics that the command-line version of pyright generates. I guess each test snippet could be moved to its own ".py" file, but that has its own downsides. For reference, we use both approaches in the pyright test suite. For core type checking functionality, we place code in their own ".py" files (see here, and for language server functionality, we place the test code snippets within comments (see here). Both approaches have their pros and cons.

2. When you invoke the pyright CLI, you might consider using the --outputjson option and store that JSON in your snapshots. The JSON output is harder for a human to read but easier for tools to parse. So I guess it depends on how you expect the snapshots to be used. I just wanted to point out the --outputjson option in case you weren't familiar with it.

3. We periodically improve the readability of pyright's diagnostics based on feedback from users. Sometimes even small changes in wording can make a big improvement. But wording changes will be flagged as differences when using your technique. I don't have a good suggestion here, but I wanted to highlight this as a potential challenge when using a snapshot approach. When you upgrade to a newer version of pyright, it may be necessary to recreate the snapshots.

[JelleZijlstra]

You could put the code in a function and extract it, so it doesn't need to be in a string literal. That's what I'm doing in my typechecker's tests (example: https://github.com/quora/pyanalyze/blob/master/pyanalyze/test_stacked_scopes.py#L141 implementation: https://github.com/quora/pyanalyze/blob/master/pyanalyze/test_node_visitor.py#L154), but it does have some issues; for example import * is illegal within a function.

[WasabiFan]

Thanks for the thoughts @erictraut!

It's a bit unfortunate that the test code snippets need to live within string literals. [...]

Agreed. Thanks for the context on that, and it's worth pointing out. So far, since my tests have been many small (~3-5 line) snippets, I've been reasonably accepting of the limitations of string literals, but I'll be keeping this in mind as I try it out more. Moving them to dedicated files is something I'm considering but would require appropriate configuration to omit them from my project's normal typecheck flow while potentially retaining them in vscode/Pylance. So I haven't really formed an opinion yet but I'll watch it to see how much editing of these test snippets I need to do.

If I were really invested in this, I'd also consider a vscode extension with a new file type that understood contextually which strings were code snippets and flagged those strings as "embedded language" blocks. That would at least get syntax highlighting, and maybe (?) some error checking with a bit of fiddling, but not e.g. formatter support. It's an interesting question.

When you invoke the pyright CLI, you might consider using the --outputjson option and store that JSON in your snapshots. [...]

Thanks for the note. I'll keep this in mind.

I must admit, the non-breaking spaces partially used to indent the textual output gave me around twenty minutes of extreme confusion 😆 Using the JSON format would at least normalize some of that formatting and make it easier to diff the addition/omission of whole errors.

We periodically improve the readability of pyright's diagnostics based on feedback from users. [...]

Fair point. Right now, I have a fairly significant hole in my design, in that it uses a global Pyright installation unconditionally -- that means if users have differing versions of the tool, they'll get differing results. As I start to come across cases of discrepancies I'll look into options here, but I suspect that a package.json which installs Pyright locally and a lookup to grab that copy would be a pretty reasonable solution.

It still doesn't solve the existence of differences as you mentioned, though. I'll watch for failures as I upgrade my CI to new versions and update the expected outputs as needed. The good news here is that since I'm using auto-generated snapshots the upgrade process is fairly straightforward (poetry run update-snapshots and then review the diff). Snapshots have their own downsides, but overall I'm willing to eat the overhead in keeping up with Pyright errors.

[WasabiFan]

You could put the code in a function and extract it, so it doesn't need to be in a string literal. [...]

Ooh, that's quite interesting. I'll have to keep that in mind, and explore your sample -- it seems like that might be a more attractive option. Although seeing the errors in the editor might make it hard to distinguish between test code errors and expected errors. Thanks for the reference!

[RobertCraigie]

I needed a tool like this for https://github.com/RobertCraigie/prisma-client-py

Didn't find this discussion until now so I ended up implementing my own solution: https://github.com/RobertCraigie/pytest-pyright

My solution works slightly differently:

• Tests are written as python files in a single directory
• The pytest plugin discovers them and runs pyright on each file individually, parsing the output using the --outputjson flag
• Diagnostics are then matched to the corresponding line in the source file and the diagnostic message is checked against a comment

A benefit to an implementation like this is that pyright configuration can be tested by writing tests in subdirectories.

There are a couple downsides to my solution, however they could be fixed in the future:

• Multi-line errors are not supported yet
• Currently, diagnostics must be updated by hand

Example:

from typing import Optional

def main(string: Optional[str]) -> None:
    print(string.split('.'))  # E: "split" is not a known member of "None"

Tests written for my own project can be found here: https://github.com/RobertCraigie/prisma-client-py/tree/main/typesafety/pyright