Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

isort does not recognize _collections_abc as part of the standard library #2250

Open
LLyaudet opened this issue Mar 20, 2024 · 5 comments · May be fixed by #2295
Open

isort does not recognize _collections_abc as part of the standard library #2250

LLyaudet opened this issue Mar 20, 2024 · 5 comments · May be fixed by #2295

Comments

@LLyaudet
Copy link

Hello,

In my repository:
https://github.com/LLyaudet/python-repeatable-iterable/blob/main/src/python_repeatable_iterable/__init__.py
I have the following imports:

from typing import Iterable, Iterator, TypeVar, cast
from _collections_abc import dict_items, dict_keys, dict_values

from python_none_objects import NoneIterable

But isort does not recognize that

from _collections_abc import dict_items, dict_keys, dict_values

is part of the standard library and regroup it like that:

from typing import Iterable, Iterator, TypeVar, cast

from _collections_abc import dict_items, dict_keys, dict_values
from python_none_objects import NoneIterable

Best regards,
Laurent Lyaudet
@devdanzin
Copy link

Hello, we would like to fix this issue. Is there agreement that _collections_abc should be added to the list of standard library modules?

@mtrd3v mtrd3v mentioned this issue Jul 18, 2024
Open
@LLyaudet
Copy link
Author

Hello @devdanzin :),
Thanks for looking at this problem.
Sorry for my late answer, I have been away from my laptop for some time.
There is indeed this agreement.
_collections_abc is somewhat a private standard library modules, as is shown by this URL :

https://github.com/python/cpython/blob/main/Lib/_collections_abc.py

I would suggest that you add all other "private standard library modules" that are listed here:
https://github.com/python/cpython/tree/main/Lib

Thank you very much, best regards,
Laurent Lyaudet

@LLyaudet
Copy link
Author

Hello @devdanzin, @mtr-d3v, :),
https://docs.python.org/3/library/index.html
states that built-in modules are stdlib modules that are written in C contrary to other Python stdlib modules that are written in Python.

The library contains built-in modules (written in C) that provide access to system functionality such as file I/O that would otherwise be inaccessible to Python programmers, as well as modules written in Python that provide standardized solutions for many problems that occur in everyday programming.

Thus, there may be confusion with builtin functions, exceptions, and constants that can be used in Python code without import, but the builtin module sys for example still requires an import sys to be used.

A complete list can be given by sys.stdlib_module_names.

laurent@laurent-GL73-8SD:~$ python3
Python 3.12.3 (main, Jul 31 2024, 17:43:48) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.builtin_module_names
('_abc', '_ast', '_bisect', '_blake2', '_codecs', '_collections', '_csv', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_opcode', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha2', '_sha3', '_signal', '_socket', '_sre', '_stat', '_statistics', '_string', '_struct', '_symtable', '_thread', '_tokenize', '_tracemalloc', '_typing', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'cmath', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'zlib')
>>> sys.stdlib_module_names
frozenset({'_osx_support', 'tracemalloc', 'doctest', 'cmd', 'codecs', '_codecs_jp', 'telnetlib', 'warnings', '_lzma', '_codecs_hk', '_contextvars', '_collections_abc', '_sqlite3', 'code', '_threading_local', 'gettext', '_csv', 'mailcap', 'queue', 'selectors', '_overlapped', 'pipes', 'zlib', 'dbm', 'chunk', '_tracemalloc', 'cgi', 'array', 'pdb', 'turtle', 'copyreg', 'cgitb', 'colorsys', 'signal', '_multiprocessing', '_ast', 'curses', 'grp', 'mailbox', 'ipaddress', 'fnmatch', 'wave', 'tty', 'imghdr', '_bz2', 'sre_parse', 'gc', '_compression', 'html', 'pydoc_data', 'typing', '_locale', 'io', 'contextlib', '__future__', 'filecmp', 'runpy', 'string', 'unicodedata', 'pathlib', 'turtledemo', 'ntpath', 'sqlite3', 'pty', '_msi', 'concurrent', '_asyncio', 'weakref', '_struct', '_thread', '_zoneinfo', 'msilib', '_uuid', '_curses_panel', 'pstats', 'nt', 'sched', '_weakref', '_codecs_kr', '_codecs_tw', 'sre_constants', 'linecache', 'pyclbr', 'atexit', '_sha3', 'idlelib', 'termios', 'sysconfig', 'modulefinder', 'tokenize', 'codeop', 'unittest', 'sre_compile', 'smtplib', '_markupbase', 'base64', 'encodings', 'fractions', '_io', 'multiprocessing', 'getpass', 'subprocess', 'pyexpat', 'importlib', '_abc', 'wsgiref', 'hashlib', 'select', 'rlcompleter', 'audioop', 'this', 'antigravity', 'pkgutil', '_decimal', '_gdbm', 'lzma', 'fcntl', '_signal', 'numbers', 'json', 'platform', 'ctypes', 'getopt', '_sha2', 'genericpath', 'plistlib', 'dataclasses', 'pwd', 'tempfile', 'resource', '_dbm', 'winsound', 'zoneinfo', '_random', 'reprlib', 'itertools', '_statistics', 'math', '_hashlib', 'decimal', 'zipfile', 'nturl2path', '_bisect', '_posixshmem', '_scproxy', 'venv', 'operator', 'hmac', 'errno', 'textwrap', 'readline', 'asyncio', '_multibytecodec', 'posixpath', 'zipimport', '_tokenize', 'sys', 'keyword', 'binascii', 'configparser', 'shutil', '_lsprof', 'stringprep', 'ast', 'contextvars', 'shelve', 'crypt', 'nis', 'profile', 'stat', '_string', '_opcode', 'spwd', '_pylong', 'winreg', '_aix_support', '_socket', '_sha1', '_pyio', 'cProfile', '_py_abc', 'logging', '_codecs_iso2022', 'marshal', 'struct', 'symtable', '_codecs', '_codecs_cn', 'os', 'graphlib', '_sre', 'xml', '_posixsubprocess', 'urllib', '_operator', 'nntplib', 'enum', '_curses', 'builtins', 'csv', '_tkinter', 'tkinter', 'quopri', 'tarfile', 'mmap', 'ossaudiodev', 'pickletools', 'sndhdr', '_queue', 'secrets', '_blake2', 'xmlrpc', '_compat_pickle', '_collections', 'threading', 'webbrowser', 'pickle', '_frozen_importlib', 'copy', '_md5', '_functools', '_elementtree', 'gzip', 're', 'timeit', 'statistics', 'shlex', 'zipapp', '_ctypes', 'poplib', '_symtable', 'py_compile', 'ssl', 'ensurepip', 'imaplib', 'bz2', 'bisect', 'opcode', '_strptime', 'posix', 'functools', 'calendar', 'socket', 'socketserver', 'xdrlib', 'random', 'msvcrt', 'netrc', 'argparse', 'optparse', 'http', 'mimetypes', '_stat', '_json', 'syslog', 'inspect', 'datetime', '_crypt', 'dis', 'token', 'collections', 'faulthandler', '_pydecimal', 'uu', 'uuid', 'tabnanny', 'pydoc', '_frozen_importlib_external', '_pydatetime', '_heapq', 'email', 'sunau', '_weakrefset', 'difflib', 'compileall', 'trace', 'types', 'glob', '_ssl', '_warnings', 'ftplib', '_sitebuiltins', 'fileinput', 'aifc', 'time', 'cmath', 'abc', 'heapq', 'lib2to3', 'site', '_pickle', 'tomllib', 'locale', '_typing', '_winapi', 'bdb', 'traceback', '_imp', 'pprint', '_datetime'})

Note that sys.stdlib_module_names has been added in Python 3.10.
python/cpython#87121
Before that only sys.builtin_module_names was available.

But unfortunately, isort code use the documentation to know the modules list:
https://www.sphinx-doc.org/en/master/usage/domains/python.html
py:module
https://github.com/PyCQA/isort/blob/main/scripts/mkstdlibs.py

As of now I don't have yet a way to fetch all stdlib modules for a specific Python version using sphinx.
The only solution I see right now is to launch a docker image using Python images on Dockerhub.
https://hub.docker.com/_/python
It would work with sys.stdlib_module_names for Python 3.10 and after.
And it would work partially for Python 3.9 and before.

Maybe a sphinx specialist would be able to find a way?

Best regards,
Laurent Lyaudet

@devdanzin devdanzin linked a pull request Oct 8, 2024 that will close this issue
Open
@devdanzin
Copy link

Hi @LLyaudet, sorry for my late reply, I've also been away for a while.

I've created #2295 implementing your idea of using sys.stdlib_module_names for Python 3.10 and later, falling back to sys.builtin_module_names in 3.8 and 3.9. It still downloads the module list using Sphinx and uses the information from sys to complement the fetched list, which is important for these older versions.

This fixes your original report and many other omissions, but adds a lot of new entries to the known modules lists so isort developers should assess whether there is any downside to this approach.

Thanks for your analysis and pointers!

@LLyaudet
Copy link
Author

LLyaudet commented Oct 8, 2024

Hello @devdanzin :),
Nice, thank you :).
Best regards,
Laurent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use sys.stdlib_module_names to complement known stdlib modules devdanzin/isort
2 participants
@LLyaudet @devdanzin