Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use alternative index url when installing build dependencies #68

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

ryanking13
Copy link
Member

@ryanking13 ryanking13 commented Dec 2, 2024

Close #43

This PR changes the index url for installing build dependency. The concept of this PR is explained in #43.

  • pyodide-build will try to install the build dependencies from https://pypi.anaconda.org/pyodide-build/.

    • This repository contains patched numpy, scipy + common build dependencies mirrored from PyPI.
    • https://github.com/ryanking13/pyodide-build-deps is pushing packages to this package index (will transfer to Pyodide org) after this PR is approved.
  • If downloading the package is not available, it will fallback to PyPI.

  • When building recipes, recipes with cross-build-env / cross-build-files will keep install the native packages to hostsitepackages

    • hostsitepackages takes precedence to the sitepackages, so the build system will grab the packages from hostsitepackages not from site-packages
    • so no breaking changes in in-tree build.

Advantages

Pyodide xbuildenv no longer need to build the numpy and scipy or hold it in the zip file. Also upgrading the numpy and scipy will be decoupled from Pyodide release.

Drawbacks

After this PR, we need to manage a separate repository pyodide-build-deps. This repository contains a few recipes (numpy, scipy) and we need to update them when there are breaking changes in numpy or scipy.

We also need to manage other common build dependencies. These packages can be simply mirrored from PyPI, so no complex setup is needed. But it can be annoying.

@ryanking13 ryanking13 marked this pull request as ready for review January 7, 2025 08:48
@ryanking13
Copy link
Member Author

This PR is ready to be reviewed now. I'll open a PR in pyodide/pyodide to test this.

@ryanking13 ryanking13 changed the title [DRAFT] Use alternative index url when installing build dependencies Use alternative index url when installing build dependencies Jan 8, 2025
@hoodmane
Copy link
Member

I guess I owe you a review on this one too.

@ryanking13
Copy link
Member Author

Yeah, a review would be helpful, thanks! I think it is almost ready, but I am still trying to make pyodide/pyodide#5309 pass the test :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Publish and install build dependencies to/from alternative package index
2 participants