Remove type checker-specific symbols from builtins.pyi and typing.pyi

[JelleZijlstra]

I'm working to get rid of the type checker-specific symbols that currently are in our core stubs but don't exist at runtime. I'll use this issue to track the work needed. This involves changes both to typeshed and to type checkers.

Affected names:

• typing._TypedDict. Suggesting to rename to _typeshed.TypedDictFallback. (mypy and pyright)
• typing._promote -> _typeshed._promote (mypy only)
• typing.AwaitableGenerator -> _typeshed.AwaitableGenerator (mypy and pyright)
• builtins.function (mypy and pyright). Need to look more into why we can't just use types.FunctionType.
  • Working on this in mypy but it requires updating a ton of test cases
• builtins.ellipsis (mypy and pyright). Should add a new name to _typeshed, similar to NoneType.

builtins.module was also mentioned in the past but it's been fixed already (mypy PR: python/mypy#3107).

Linked issues and PRs:

• Remove function from builtins #2999
• class ellipsis is public but shouldn't #3556
• Remove typing.AwaitableGenerator #3932
• update ellipsis type mypy#11564
• Don't use builtins.function and others anymore mypy#8240
• Mypy crashes if there is no 'function' or 'module' in builtins mypy#2765
• Remove typing.AwaitableGenerator from typeshed mypy#8699

[AlexWaygood]

• Callable has no attribute __kwdefaults__ mypy#5958

contains some insight into why builtins.function and types.FunctionType used to be different.

But changes that we made to builtins.function in typeshed (to reduce false-positives) mean that they're not so different anymore, e.g.:

• Add missing function attributes to builtins.function #6804

[srittau]

#13816 is a first step towards solving it, by making the symbols available in the module _typeshed._tc. We can then ask type checker authors to switch to that new module and remove the symbols from the wrong places at some later time.

I've left out builtins.ellipsis, since it can be replaced by types.EllipsisType after support for Python 3.9 is dropped. (Added as to do item to #13782).

[srittau]

For reference, I've linked this issue and the PR on discuss.python.org: https://discuss.python.org/t/removing-type-checker-internals-from-typeshed/87960

[srittau]

See the initial comment of #13816 for the current proposal. I'll try to keep it updated according to the consensus of the discussion.

[erictraut]

Thanks for doing this work. Transitioning to the new types will be a little tricky to coordinate between typeshed and all the type checkers. Would it make sense to provide aliases in typing.pyi that shadow (i.e. import and re-export) the new type definitions in _typeshed.pyi for a period of time? You could set expectations that they're going away completely in, say, six months. That would help give type checker maintainers a chance to make the required changes without a hard cut-over. It would also ease the transition for users who download their own versions of typeshed rather than using the version that's bundled with their type checker.

[srittau]

Would it make sense to provide aliases in typing.pyi that shadow (i.e. import and re-export) the new type definitions in _typeshed.pyi for a period of time?

The idea is to keep the existing definitions for the time being, and only remove them when we're reasonably sure that type checkers have adopted the new names.

[JelleZijlstra]

Instead of adding _typeshed._tc, should we instead encourage type checkers to stop relying on typeshed for these completely?

Looking at the current PR:

• promote is something mypy-specific, it should ideally be removed from typeshed completely
• AwaitableGenerator is mypy implementation details for a mostly obsolete feature. Do other type checkers use this?
• TypedDictFallback and NamedTupleFallback are potentially useful for others (and it appears at least pyright uses them), but the comments for TypedDictFallback are full of discussion about how the signature must be a certain way for the mypy plugin to work correctly. So wouldn't it be better for mypy to host those stubs?

(Also I removed my assignment; I intended to work on this a few years ago but never had the time, sorry!)

[AlexWaygood]

AwaitableGenerator is mypy implementation details for a mostly obsolete feature. Do other type checkers use this?

According to @erictraut in python/mypy#8240 (comment), pyright uses it (but that was three years ago!)

[AlexWaygood]

TypedDictFallback and NamedTupleFallback are potentially useful for others (and it appears at least pyright uses them), but the comments for TypedDictFallback are full of discussion about how the signature must be a certain way for the mypy plugin to work correctly. So wouldn't it be better for mypy to host those stubs?

From red-knot's perspective, I'd prefer for these fallback stubs to continue to be hosted at typeshed, I think. #13646 is a great example of a bug in these stubs that was caught by typeshed's CI, but which might have taken much longer to spot if red-knot had to maintain its own version of a stub for typing._TypedDict rather than sharing one as a community project, and the bug had been in red-knot's version of typing._TypedDict. Having these fallback stubs in typeshed means that any bugfixes to these stubs benefits all type checkers, and all type checkers benefit from the checks typeshed runs in CI to ensure the consistency and quality of its stubs.

[srittau]

Instead of adding _typeshed._tc, should we instead encourage type checkers to stop relying on typeshed for these completely?

Ideally yes. But this is probably a fairly easy change for type checkers for now, and – in my opinion – a considerable improvement for us. Stopping to rely on typeshed might need bigger changes.

[rchen152]

+1 to copying these symbols out of builtins and typing (and eventually removing them from there). Echoing @AlexWaygood , I'd prefer for TypedDictFallback and NamedTupleFallback to remain in _typeshed, as they're potentially useful for all type checkers. (Pyrefly is already using typing.NamedTuple.)