Any Narrowing Question

[GBeauregard]

Hi, while looking into a really complicated printing function in the rich codebase I finally narrowed down some pyright errors as originating from this behavior; the subsequent branches are getting narrowed to being Any | type because of a really buried isinstance check for type in a previous branch.

We've discussed it for quite a while now in #python irc, but I think we need some clarification.

from dataclasses import is_dataclass
from typing import Any

def analyze(obj: Any):
    if is_dataclass(obj) and not isinstance(obj, type):
        reveal_type(obj)  # revealed as Any
    else:
        reveal_type(obj)  # revealed as Any | type

def analyze_false(obj: Any):
    if is_dataclass(obj) and not isinstance(obj, type) and False:
        reveal_type(obj)
    else:
        reveal_type(obj)  # revealed as Any | type

Is this sort of narrowing to Any | type really needed? I can understand narrowing to unions with Any in general, but I don't understand why it should be considered appropriate here in the case of an and-buried isinstance check. Can you provide any examples of why the behavior is like this?

[erictraut]

This type evaluation result is a natural application of the normal rules for "or" and "and" expressions and isinstance type narrowing.

In general, Any is never narrowed (or widened), but there are three important exceptions to this rule: isinstance, issubclass, and a user-defined type guard function. These three cases are allowed to narrow Any.

Consider this example.

def analyze(obj: Any):
    if isinstance(obj, int) or isinstance(obj, str):
        reveal_type(obj)

I think you'd agree that the revealed type should be int | str in this case.

Now, let's replace the first isinstance call with a call to some other random function that doesn't imply any type narrowing.

def analyze(obj: Any):
    if random_function(obj) or isinstance(obj, str):
        reveal_type(obj)

Does it make sense that the revealed type would be Any | str in this case?

Now, turning our attention back to your example. The type of obj in the else class is effectively equivalent to:

def analyze(obj: Any):
    if not is_dataclass(obj) or isinstance(obj, type):
        reveal_type(obj) # Any | type

The not is_dataclass(obj) subexpression doesn't provide any narrowing, but the isinstance call does. The resulting type is Any | type.

You might think "isn't Any | type the equivalent of just Any"? Actually, it's not. We'd be discarding type information in that case. If we've gained the insight that obj might be a type along some code path, it's useful to retain that information so we can verify that if it is subsequently used in a manner that's not appropriate for a type object, it's reported as an error.

[GBeauregard]

Thanks a lot. This is a really great answer; I'm a bit embarrassed to have majored in math and not thought about the negation properly!

The way this actually showed up in the rich codebase obscured this in a really really tricky way you might find interesting:

_BRACES: Dict[type, Callable[[Any], Tuple[str, str, str]]] = {
    os._Environ: lambda _object: ("environ({", "})", "environ({})"),
    array: _get_braces_for_array,
    defaultdict: _get_braces_for_defaultdict,
    Counter: lambda _object: ("Counter({", "})", "Counter()"),
    deque: lambda _object: ("deque([", "])", "deque()"),
    dict: lambda _object: ("{", "}", "{}"),
    UserDict: lambda _object: ("{", "}", "{}"),
    frozenset: lambda _object: ("frozenset({", "})", "frozenset()"),
    list: lambda _object: ("[", "]", "[]"),
    UserList: lambda _object: ("[", "]", "[]"),
    set: lambda _object: ("{", "}", "set()"),
    tuple: lambda _object: ("(", ")", "()"),
    MappingProxyType: lambda _object: ("mappingproxy({", "})", "mappingproxy({})"),
}
_CONTAINERS = tuple(_BRACES.keys())
def analyze_false(obj: Any):
    if is_dataclass(obj) and not isinstance(obj, type) and False:
        reveal_type(obj)
    elif isinstance(obj, _CONTAINERS):
        reveal_type(obj)  # revealed as Any | type

I was expecting the _CONTAINERS isinstance to narrow it down, but it can't because the _BRACES: Dict[type is basically the same as _BRACES: Dict[Any, whereas if it had been something narrower like Sized then the _CONTAINERS isinstance would have actually narrowed it down further.

And all of this was buried in a ton of ifs with really long statements.