[BUG] Making project-url as Dynamic causes PyPI to block the wheel starting from 75.8.0

[justinchuby]

setuptools version

setuptools==75.8.0

Python version

Python 3.13

OS

Linux

Additional environment information

No response

Description

After upgrading to setuptools==75.8.0 in https://github.com/microsoft/onnxscript, the PKD-INFO generated became Metadata-Version: 2.2, with a new line Dynamic: project-url added to the metadata file. This causes PyPI to reject the wheel with message

packaging Dynamic: project-url' is invalid for 'requires-dist'

{"code":null,"details":{"errors:":"BadRequest, 'packaging Dynamic: project-url' is invalid for 'requires-dist'. See https://packaging.python.org/specifications/core-metadata for more information., {\"message\": \"The server could not comply with the request since it is either malformed or otherwise incorrect.\\n\\n\\n'packaging Dynamic: project-url' is invalid for 'requires-dist'. See https://packaging.python.org/specifications/core-metadata for more information.\\n\\n\", \"code\": \"400 'packaging Dynamic: project-url' is invalid for 'requires-dist'. See https://packaging.python.org/specifications/core-metadata for more information.\", \"title\": \"Bad Request\"}"},"innerError":null}

Expected behavior

I would expect the wheel to be valid for PyPI even when the project url is dynamic. The URLs field needs to be dynamic for us because we want to generate an url based on the git commit.

How to Reproduce

1. git clone https://github.com/microsoft/onnxscript.git
2. python -m build

Output

N/A

[justinchuby]

cc @gramalingam @titaiwangms @xadupre

[abravalheri]

Thank you @justinchuby

I think this is related to #4797.
We are waiting for a confirmation regarding the interpretation of the standard.
It is not immediate to me why Project-URL is rejected, though. It does not seem to have been deprecated...

Could you please confirm you are using the project_urls config in setup.py, rather than the download_url or url (the later 2 correspond to Home-page and Download-URL which are now deprecated, so PyPI might deal with them differently).

[abravalheri]

I see... is invalid for requires-dist is a bit puzzling...

@justinchuby, could you please add the generated METADATA file to the thread?

[abravalheri]

I tried to create a minimal reproducer (one that does not need to go to all the code in onnxscript with the following script:

mkdir -p /tmp/test-onnxscript
cd /tmp/test-onnxscript

wget https://github.com/microsoft/onnxscript/raw/refs/heads/main/pyproject.toml
wget https://github.com/microsoft/onnxscript/raw/refs/heads/main/README.md
wget https://github.com/microsoft/onnxscript/raw/refs/heads/main/MANIFEST.in
wget https://github.com/microsoft/onnxscript/raw/refs/heads/main/LICENSE
wget https://github.com/microsoft/onnxscript/raw/refs/heads/main/VERSION
wget https://github.com/microsoft/onnxscript/raw/refs/heads/main/setup.py
mkdir onnxscript
touch onnxscript/__init__.py
sed -i 's/"setuptools>=.*"/"setuptools>=75.8.0"/' pyproject.toml

python3.12 -m venv .venv
.venv/bin/python -m pip install -U pip build packaging
.venv/bin/python -m build

unzip dist/onnxscript-0.1.0.dev20250114-py3-none-any.whl onnxscript-0.1.0.dev20250114.dist-info/METADATA
# bat -A onnxscript-0.1.0.dev20250114.dist-info/METADATA  # Looks Ok

.venv/bin/python - <<EOF
from packaging.metadata import Metadata, parse_email


# Validate generated files using 'packaging' for benchmark
for example in ("onnxscript.egg-info/PKG-INFO", "onnxscript-0.1.0.dev20250114.dist-info/METADATA"):
    with open(example, "rb") as f:
        raw, unparsed = parse_email(f.read())
        print("\n", example)
        print("Unparsed, should be empty:", repr(unparsed))  # seems fine
        raw.pop("license_files", None)  # See on-going https://github.com/pypa/setuptools/issues/4759
        print("Should have no exceptions:", Metadata.from_raw(raw, validate=True)) # seems fine, no exceptions
EOF

I cannot detect why the file is being malformed and why the parser in PyPI thinks that 'packaging Dynamic: project-url' is part of requires-dist.

@justinchuby could you please share the METADATA file with us (preferably by attaching it to the comment, so that copy/paste does not change the line endings).?

[justinchuby]

This is the file generated, the only change I saw when diffing is the metadata version and the dynamic: line (line 52)

https://gist.github.com/justinchuby/cdf2588573fdde2729cd37a55fcd6f1b

I can attach the original file later

[justinchuby]

PKG-INFO.txt

[justinchuby]

Could you please confirm you are using the project_urls config in setup.py, rather than the download_url or url (the later 2 correspond to Home-page and Download-URL which are now deprecated, so PyPI might deal with them differently).

I think so. It can be seen in https://github.com/microsoft/onnxscript/blob/347aa066181af888f649fea963d6ab2c3c9044f4/setup.py#L34

[justinchuby]

I see what you are saying now... The error message seems to suggest that the Dynamic: project-url line is part of the Requires-Dist: line. This is puzzling.

[justinchuby]

METADATA.zip file from the wheel if it's different

[abravalheri]

I copied the 2 files you provided and tested again in my little benchmark script:

.venv/bin/python - <<EOF
from packaging.metadata import Metadata, parse_email


# Validate generated files using 'packaging' for benchmark
for example in ("METADATA", "PKG-INFO.txt"):
    with open(example, "rb") as f:
        raw, unparsed = parse_email(f.read())
        print("\n", example)
        print("Unparsed, should be empty:", repr(unparsed))  # seems fine
        raw.pop("license_files", None)  # See on-going https://github.com/pypa/setuptools/issues/4759
        print("Should have no exceptions:", Metadata.from_raw(raw, validate=True)) # seems fine, no exceptions
EOF

Both seem to be fine:

 METADATA
Unparsed, should be empty: {}
Should have no exceptions: <packaging.metadata.Metadata object at 0x7f444342a180>

 PKG-INFO.txt
Unparsed, should be empty: {}
Should have no exceptions: <packaging.metadata.Metadata object at 0x7f44433c7a10>

When I see the files with bat -A I can also see that the line endings seem to be OK:

Any chance there was a problem during parsing and/or a the file was corrupted during the upload?

[justinchuby]

Thanks for checking the file @abravalheri ! I will go back and do some more checks

[justinchuby]

It looks like another internal issue we need to resolve. Sorry for the false signal, and thanks again for your help!

[justinchuby]

Sorry I have to reopen. After resolving internal issues, we continue to see this error when release to PyPI.

[abravalheri]

Hi @justinchuby, to continue investigating the issue we need a minimal reproducer that does not depend on PyPI.

Can this error be verified in an isolated state prior to be uploaded? This eliminates potential errors with corrupted uploads or specific parsers employed by external systems that can themselves contain bugs sometimes.

Ideally we want to be able to verify a malformed METADATA file built locally, that does not follow the core metadata spec. What is puzzling is that they seem to be following the spec and pass the validation using the packaging library (which is widely accepted by the community as a reference implementation for parsing), so I think we reached a block of what we can investigate with the current reproducer.