invalid_url check seems to fail with 403

[lorepirri]

The linter recently failed to detect a redirect and is giving an error 403 unauthorized (instead of a redirect 302).

It might happen in other cases, but so far happens on redirects like https://xgboost.readthedocs.io -> https://xgboost.readthedocs.io/en/stable/.

Not much has changed recently in the check:

anaconda-linter/anaconda_linter/utils.py

Line 118 in 7f7d728

response = requests.head(url, allow_redirects=False, timeout=HTTP_TIMEOUT)

On a local execution, of the linter on the same feedstocks listed below, there is no 403 error.

This leads me to think that maybe Prefect might have been blacklisted. Or some other changes in the code somewhere else (cache?).

Other tests

This locally works with an error code 302:

import requests
from typing import Final

HTTP_TIMEOUT: Final[int] = 120
r = requests.head(
    "https://xgboost.readthedocs.io", allow_redirects=False, timeout=HTTP_TIMEOUT
)
print(r.status_code, r.headers["Location"])

Examples

1. PR: lightgbm 4.5.0  AnacondaRecipes/lightgbm-feedstock#9

   clone/recipe/meta.yaml:90: invalid_url: https://lightgbm.readthedocs.io : Not reachable: 403
   

2. PR: xgboost 2.1.1 fix AnacondaRecipes/xgboost-feedstock#20

   clone/recipe/meta.yaml:125: invalid_url: https://xgboost.readthedocs.io/ : Not reachable: 403