pdm is sensitive to section order in pyproject.toml with unhelpful error messages

[qris]

Describe the bug

A simple pyproject.toml can cause pdm add to fail with:

[NonExistentKey]: 'Key "pdm" does not exist.'

To reproduce

Run pdm init, then replace the contents of pyproject.toml with:

[project]
name = "pdmtest"
version = "0.1.0"
description = "Default template for PDM package"
authors = []
dependencies = []
requires-python = "==3.12.*"
readme = "README.md"
license = {text = "MIT"}

[tool.pdm.scripts]

[tool.pdm]
distribution = false

[dependency-groups]

[tool.ruff]

Then run pdm add -e . --dev.

Expected Behavior

INFO: Adding group dev to lockfile
Adding packages to dev dev-dependencies: -e file:///${PROJECT_ROOT}/#egg=pdmtest
/ 0:00:00 Resolve for environment (==3.12.*) None
...

What actually happens is:

INFO: Adding group dev to lockfile
Adding packages to dev dev-dependencies: -e file:///${PROJECT_ROOT}/#egg=pdmtest
[NonExistentKey]: 'Key "pdm" does not exist.'
WARNING: Add '-v' to see the detailed traceback

Making random changes to pyproject.toml, such as reordering sections or removing the [tool.ruff] section, makes it work.

Environment Information

PDM version:
2.20.1
Python Interpreter:
C:\Users...\Documents\pdmtest.venv\Scripts\python.exe (3.12)
Project Root:
C:/Users/.../Documents/pdmtest
Local Packages:

{
"implementation_name": "cpython",
"implementation_version": "3.12.3",
"os_name": "nt",
"platform_machine": "AMD64",
"platform_release": "10",
"platform_system": "Windows",
"platform_version": "10.0.19045",
"python_full_version": "3.12.3",
"platform_python_implementation": "CPython",
"python_version": "3.12",
"sys_platform": "win32"
}

pdm -v output

Usage: pdm [-h] [-V] [-c CONFIG] [-v | -q] [--no-cache] [-I] [--pep582 [SHELL]] [-n] ...

    ____  ____  __  ___
   / __ \/ __ \/  |/  /
  / /_/ / / / / /|_/ /
 / ____/ /_/ / /  / /
/_/   /_____/_/  /_/

Options:
  -h, --help            Show this help message and exit.
  -V, --version         Show the version and exit
  -c CONFIG, --config CONFIG
                        Specify another config file path [env var: PDM_CONFIG_FILE]
  -v, --verbose         Use `-v` for detailed output and `-vv` for more detailed
  -q, --quiet           Suppress output
  --no-cache            Disable the cache for the current command. [env var: PDM_NO_CACHE]
  -I, --ignore-python   Ignore the Python path saved in .pdm-python. [env var: PDM_IGNORE_SAVED_PYTHON]
  --pep582 [SHELL]      Print the command line to be eval'd by the shell for PEP 582
  -n, --non-interactive
                        Don't show interactive prompts but use defaults. [env var: PDM_NON_INTERACTIVE]

Commands:
  add                   Add package(s) to pyproject.toml and install them
  build                 Build artifacts for distribution
  cache                 Control the caches of PDM
  completion            Generate completion scripts for the given shell
  config                Display the current configuration
  export                Export the locked packages set to other formats
  fix                   Fix the project problems according to the latest version of PDM
  import                Import project metadata from other formats
  info                  Show the project information
  init                  Initialize a pyproject.toml for PDM. Built-in templates: - default: `pdm init`, A simple template with
                        a basic structure. - minimal: `pdm init minimal`, A minimal template with only `pyproject.toml`.
  install               Install dependencies from lock file
  list                  List packages installed in the current working set
  lock                  Resolve and lock dependencies
  outdated              Check for outdated packages and list the latest versions on indexes.
  publish               Build and publish the project to PyPI
  python (py)           Manage installed Python interpreters
  remove                Remove packages from pyproject.toml
  run                   Run commands or scripts with local packages loaded
  search                Search for PyPI packages
  self (plugin)         Manage the PDM program itself (previously known as plugin)
  show                  Show the package information
  sync                  Synchronize the current working set with lock file
  update                Update package(s) in pyproject.toml
  use                   Use the given python version or path as base interpreter. If not found, PDM will try to install one.
  venv                  Virtualenv management

Additional Context

No response

Are you willing to submit a PR to fix this bug?

• Yes, I would like to submit a PR.

[2e0byo]

Adding an empty [tool.pdm] entry appears to fix it (have no idea why). Presumably something in those keys triggers a codepath which then looks for that key.

[jiangbaihe]

我也遇到了同样的报错，经过一段时间的测试，发现了问题所在。

引用您的文件内容：

[tool.pdm.scripts]

[tool.pdm]
distribution = false

按顺序处理文件内容，创建tool.pdm.scripts的时候，tool.pdm并没有创建，所以tool.pdm这个键是不存在的。
只需要把这两条的顺序调换，先创建tool.pdm，再创建其余的子键即可

例如：

[tool.pdm]  # this frist
distribution = false

[tool.pdm.scripts]  # and than add other sub keys

that's all, enjoy it!!!

[qris]

Thanks @jiangbaihe ! Google Translate says:

I also encountered the same error. After a period of testing, I found the problem. Quote your file content:
Process the file contents in order. When tool.pdm.scripts is created, tool.pdm is not created, so the tool.pdm key does not exist. Just swap the order of these two items, create tool.pdm first, and then create the rest of the subkeys.

For example:

[tool.pdm]  # this first
distribution = false

[tool.pdm.scripts]  # and than add other sub keys

I still think the error message is very unhelpful.

[noirbizarre]

I would say it comes from 1 of the 2 underlying pdm libraries, tomli (for Python < 3.11 only, the builtin tomllib starting 3.11) and tomlkit for writing.
Having the detailed traceback using -v would help to identify which one is involved.

But I think this is a general issue with this library and so this bug must be reported upstream (it should be easy to craft a minimal reproducible test case).

Given the TOML specs says:

Defining dotted keys out-of-order is discouraged.

It's probable this case is just not tested or supported upstream.

[qris]

Here is the stack trace:

INFO: Adding group dev to lockfile
Adding packages to dev dev-dependencies: -e file:///${PROJECT_ROOT}/#egg=pdmtest
Traceback (most recent call last):
  File "/home/chris/pdm/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/core.py", line 392, in main
    return core.main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/core.py", line 270, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/core.py", line 265, in main
    self.handle(project, options)
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/core.py", line 195, in handle
    command.handle(project, options)
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/cli/commands/add.py", line 71, in handle
    self.do_add(
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/cli/commands/add.py", line 146, in do_add
    group_deps = project.add_dependencies(requirements, group, selection.dev or False, write=False)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/project/core.py", line 735, in add_dependencies
    setter(deps)
  File "/home/chris/pdm/lib/python3.12/site-packages/pdm/project/core.py", line 667, in update_dev_dependencies
    del self.pyproject._data["tool"]["pdm"]
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/chris/pdm/lib/python3.12/site-packages/tomlkit/container.py", line 868, in __delitem__
    del table[key]
        ~~~~~^^^^^
  File "/home/chris/pdm/lib/python3.12/site-packages/tomlkit/items.py", line 1502, in __delitem__
    self.remove(key)
  File "/home/chris/pdm/lib/python3.12/site-packages/tomlkit/items.py", line 1472, in remove
    self._value.remove(key)
  File "/home/chris/pdm/lib/python3.12/site-packages/tomlkit/container.py", line 352, in remove
    raise NonExistentKey(key)
tomlkit.exceptions.NonExistentKey: 'Key "pdm" does not exist.'

[qris]

I think it's this tomlkit bug. Although it would be nice if pdm produced a more helpful error message, at least with -v we can see that it fails while trying to rewrite pyproject.toml using tomlkit, which is helpful.