build: Migrate from `hatch` to `uv` #3723

dangotbanned · 2024-12-29T17:42:54Z

Will close #3577

Description

This PR transitions us from hatch to uv with a solution for the issue I raised in (#3577 (comment)).

Note

Don't be alarmed by the +3k additions, it is mostly in uv.lock
More info

Using `uv`

Install uv for the first time or run:

uv self update

Install python:

uv python install 3.12

Go to the root of the repo:

cd altair/

Initialize a new virtual env:

uv venv -p 3.12

Install/update dependencies:

uv sync --all-extras

Also see CONTRIBUTING

Task Runner

I've been following astral-sh/uv#5903 pretty closely and after experimenting rolling my own runner - I settled on https://github.com/taskipy/taskipy.

Usage

Works in a similar way to hatch run ..., with commands defined in [tool.taskipy.tasks].

Changes

Composition

Composing tasks uses "... && ...", rather than ["...", "..."]

Support a task that runs a list of tasks taskipy/taskipy#89

Invoking

Invoking a task:

hatch run generate-schema-wrapper

uv run task generate-schema-wrapper

Alternatives

These were also mentioned in astral-sh/uv#5903.

https://github.com/wntrblm/nox
- Pretty extensive for testing
- uv integration is new
- Define tasks in a noxfile.py
https://github.com/casey/just
- Similar to make
- Really powerful
- Maybe a bit of a learning curve for the DSL
https://github.com/omnilib/thx
- Define tasks in pyproject.toml
- The most similar to hatch
- Very new
- AFAICT, uses venv and no support for uv

I chose taskipy as:

has enough functionality for what we're doing
easy to understand
easiest to replace later

The last point is quite apparent when looking at the changes in (00f353a) vs (https://github.com/vega/altair/blob/ba836abb77b8e1eb8b5d194ed9442916ccdbf2a9/tasks.toml)

Tasks

https://ss64.com/bash/mkdir.html We would want to see an error if `doc/` didn't exist. The docs wouldn't build as all the content is in that directory. Working on migrating to a cross-platform shell that doesn't require this option anyway https://www.nushell.sh/commands/docs/mkdir.html

Decouples from `bash`, simplifies (#3577 (comment))

Provides 2 interfaces: - `tasks.py`: to define using decorators - `tasks.toml`: an example of defining in a similar way to `hatch` scripts #3577 (comment)

Steps to reproduce: - `uv self update` or install `uv` for the [first time](https://docs.astral.sh/uv/getting-started/installation/) - `uv python install 3.12` - `cd $ALTAIR_REPO_ROOT` - `uv venv -p 3.12 --seed` - `uv sync --all-extras`

This is the non-`hatch` equivalent of `tool.hatch.envs.hatch-test.default-args` https://hatch.pypa.io/latest/config/internal/testing/#default-arguments

Bug revealed after changing the `pytest` task to only contain `"pytest"`

https://github.com/vega/altair/actions/runs/12535437009/job/34957434641

- Add docs - Handle semicolon joining in one place - align `mkdir_cmd` with `mkdir -p` (https://ss64.com/bash/mkdir.html)

`*_cmd` functions and these variables are unrelated Planning to namespace the functions, using `cmd`

All of these tools are now accessible under a single namespace, with discoverable docs

You can now safely run: ```py import tools tools.tasks.app.run(["lint", "format"]) ```

Terms like "clean", "build", "publish" have another meaning when talking about the distribution https://docs.astral.sh/uv/concepts/projects/build/

Covers part of step 8, step 9 in https://github.com/vega/altair/blob/7c2d97ffe80749a7cd1b24703547e40f4918172d/RELEASING.md

- Extra prep for string escaping while in a subprocess - Use `--force` in `git add` to override (https://github.com/vega/altair/blob/7c2d97ffe80749a7cd1b24703547e40f4918172d/doc/.gitignore#L1) - Raise a more informative error when a subprocess fails

All previous functionality is possible without this now. Did some tidying up of what is left using (https://github.com/vega/vega-datasets/blob/369b462f7505e4ef3454668793e001e3620861ff/taplo.toml) ## Remaining sections - `build`, `metadata`, `version` tables are used by `hatchling` - (https://hatch.pypa.io/latest/why/#build-backend) - The `doc` environment is retained **only** due to how `_HatchRunner` is implemented - Otherwise we can remove that - (6e792a1) - The test matrix functionality isn't something I've tried to reproduce, so left that in - Unsure how often others use it, but we could swap that out for `nox` - https://github.com/narwhals-dev/narwhals/blob/024d5d2294d1dcdd2e2f043d59eb03a3238c9e87/noxfile.py#L13

Didn't realise this was part of the Public API, as it wasn't listed in https://github.com/python-poetry/tomlkit/blob/635831f1be9b0e107047e74af8ebecc7c0e4b7bf/tomlkit/__init__.py#L31-L59 https://tomlkit.readthedocs.io/en/latest/api/#module-tomlkit.toml_file

dangotbanned · 2025-01-01T15:34:15Z

tools/sync_website.py

+    if no_commit:
+        print(f"Unused commit message:\n  {commit_message!r}")
+    elif branch != "main":
+        # FIXME: Unsure how to reproduce the RELEASING.md steps from this PR
+        # https://github.com/vega/altair/blob/main/RELEASING.md
+        print(f"Unable to push from {branch!r}.\n" "Must be on 'main'.")
+    else:
+        add_commit_push_github(commit_message)


FIXME

@mattijn I was wondering if you might be able to help me with this?
Looking at altair-viz.github.io(contributors), it seems like you've run these steps the most.

All the steps up this sequence are working:

altair/doc/sync_website.sh

Lines 24 to 27 in 48e976e

# add commit, and push to github

git add . --all

git commit -m "$MSG"

git push origin master

These are the equivalent (with --force (404bb26)):

altair/tools/sync_website.py

Lines 33 to 35 in aaca9bc

CMD_ADD = "git add . --all --force"

CMD_COMMIT = "git commit -m"

CMD_PUSH = "git push origin master"

Which are run in:

add_commit_push_github()

altair/tools/sync_website.py

Lines 106 to 113 in aaca9bc

def add_commit_push_github(msg: str, /) -> None:

os.chdir(DOC_REPO_DIR)

print("Pushing ...")

# NOTE: Ensures the message uses cross-platform escaping

cmd_commit = *CMD_COMMIT.split(" "), msg

commands = (CMD_ADD, cmd_commit, CMD_PUSH)

for command in commands:

run_check(command)

Problem

Delete L146-L149 and it produces an error that I'm unsure how to solve.
AFAIK, it didn't seem to be related to permissions on https://github.com/altair-viz/altair-viz.github.io - but that is a blocker I'm expecting

RELEASING.md

altair/RELEASING.md

Lines 37 to 54 in 48e976e

7. Merge release branch into main, make sure that all required checks pass

8. On main, build source & wheel distributions. If you work on a fork, replace `origin` with `upstream`:

git switch main

git pull origin main

hatch clean # clean old builds & distributions

hatch build # create a source distribution and universal wheel

9. publish to PyPI (Requires correct PyPI owner permissions):

hatch publish

10. build and publish docs (Requires write-access to altair-viz/altair-viz.github.io):

hatch run doc:publish-clean-build

11. On main, tag the release. If you work on a fork, replace `origin` with `upstream`:

I don't know if this can be resolved prior to merging to main.

100% sure it will need an assist from one of you fine people 🙂

- Allows `uv run tools/tasks.py test` to be replaced with `uv run task test` - Handles subprocess better for `pytest` and `sphinx-build` Suggested in astral-sh/uv#5903 (comment)

pyproject.toml

Currently blocking `generate-schema-wrapper e43e464

Left markdown comments on sections that still need work

CONTRIBUTING.md

Switching directories is handled by https://github.com/taskipy/taskipy?tab=readme-ov-file#working-directory

Resolves #3723 (comment)

#3723 (comment)

Related #3764

Reverts a good chuck of new stuff added in this PR, prior to 00f353a Next commit will refactor some of what is left - to simplify `[tool.taskipy.tasks]`

- Has parts of prior `sync_website.py` - Utilizes for more concises commands in `[tool.taskipy.tasks]`

Easier to maintain and understand if this is in one place

- Add missing entries to `__all__` - Add missing `copytree` doc - Remove incorrect `dir_exists` doc

dangotbanned · 2025-01-09T19:12:36Z

Important

I did a bit of a drive-by fix on README.md

The current instructions send you along this route to get to how to contribute:

I remember this confusing me when I first contributed and it happened again while I was updating the docs.
I hope that change makes sense to others.
I felt I needed to include it here to avoid having pip mess with uv

dangotbanned added 14 commits December 24, 2024 12:25

ci: replace sync_website.sh with .py

ddc8358

Decouples from `bash`, simplifies (#3577 (comment))

ci(DRAFT): migrate hatch scripts to hybrid tasks.py runner

6ebc0ee

Provides 2 interfaces: - `tasks.py`: to define using decorators - `tasks.toml`: an example of defining in a similar way to `hatch` scripts #3577 (comment)

ci: switch Tasks runner to "uv"

fbc5a0f

build(uv): Generate uv.lock

f614e4f

Steps to reproduce: - `uv self update` or install `uv` for the [first time](https://docs.astral.sh/uv/getting-started/installation/) - `uv python install 3.12` - `cd $ALTAIR_REPO_ROOT` - `uv venv -p 3.12 --seed` - `uv sync --all-extras`

refactor: Move PYTEST_ARGS -> tool.pytest.ini_options.addopts

0fa8ecc

This is the non-`hatch` equivalent of `tool.hatch.envs.hatch-test.default-args` https://hatch.pypa.io/latest/config/internal/testing/#default-arguments

fix: Add recursive guard during Tasks._expand

dca3f0d

Bug revealed after changing the `pytest` task to only contain `"pytest"`

ci: add export-tasks and update tasks.toml

8bf774f

ci: try fixing anywidget import error

3513259

https://github.com/vega/altair/actions/runs/12535437009/job/34957434641

feat: Improve *_cmd functions

c160c4c

- Add docs - Handle semicolon joining in one place - align `mkdir_cmd` with `mkdir -p` (https://ss64.com/bash/mkdir.html)

refactor: reduce overloading of term "cmd"

53bc298

`*_cmd` functions and these variables are unrelated Planning to namespace the functions, using `cmd`

feat: adds cmd helper class

8323881

All of these tools are now accessible under a single namespace, with discoverable docs

chore: re-enable tasks.py export

1fd5790

You can now safely run: ```py import tools tools.tasks.app.run(["lint", "format"]) ```

docs: fix incorrect import path

cd9ccf9

dangotbanned added the maintenance label Dec 29, 2024

dangotbanned added 11 commits December 29, 2024 22:01

Merge branch 'main' into migrate-hatch-scripts

ee426d9

ci: Prefix doc-related tasks

74c10bb

Terms like "clean", "build", "publish" have another meaning when talking about the distribution https://docs.astral.sh/uv/concepts/projects/build/

ci: Adds clean, build, publish tasks

9900aa1

Covers part of step 8, step 9 in https://github.com/vega/altair/blob/7c2d97ffe80749a7cd1b24703547e40f4918172d/RELEASING.md

ci: disable hard-coded no-op for tools/sync_website.py

ba836ab

ci: Add different no-op in sync_website.py

9285871

fix: Resolve typo in hatch extras check

6e792a1

ci: Support updating tasks table in-place

fc6496f

Merge branch 'main' into migrate-hatch-scripts

aaca9bc

dangotbanned commented Jan 1, 2025

View reviewed changes

dangotbanned added 2 commits January 2, 2025 13:37

Merge remote-tracking branch 'upstream/main' into migrate-hatch-scripts

9150935

Merge remote-tracking branch 'upstream/main' into migrate-hatch-scripts

f240884

dangotbanned mentioned this pull request Jan 6, 2025

refactor: Centralize Vega project versioning #3720

Merged

5 tasks

dangotbanned changed the title ~~build(DRAFT): Migrate from hatch to uv~~ build: Migrate from hatch to uv Jan 6, 2025

ci: Port to taskipy

00f353a

- Allows `uv run tools/tasks.py test` to be replaced with `uv run task test` - Handles subprocess better for `pytest` and `sphinx-build` Suggested in astral-sh/uv#5903 (comment)

dangotbanned commented Jan 7, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

dangotbanned added a commit that referenced this pull request Jan 8, 2025

chore(typing): Side-port mypy fix from (#3723)

d94e0b6

Currently blocking `generate-schema-wrapper e43e464

dangotbanned added 2 commits January 8, 2025 14:16

Merge remote-tracking branch 'upstream/main' into migrate-hatch-scripts

d79f621

docs(DRAFT): Update some of CONTRIBUTING.md

0e099de

Left markdown comments on sections that still need work

dangotbanned commented Jan 8, 2025

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

dangotbanned commented Jan 8, 2025

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

dangotbanned added 4 commits January 8, 2025 15:30

docs: Update NOTES_FOR_MAINTAINERS.md

b116d29

Switching directories is handled by https://github.com/taskipy/taskipy?tab=readme-ov-file#working-directory

docs: Update README.md

a9bcba1

docs: Update RELEASING.md

6d89e85

docs: Update CONTRIBUTING.md#Setting Up Your Environment

d9b0de2

Resolves #3723 (comment)

dangotbanned mentioned this pull request Jan 8, 2025

pillow version pin causing build failures #3762

Closed

dangotbanned added 7 commits January 8, 2025 19:16

docs: Update test matrix guidance

8c1f143

#3723 (comment)

Merge remote-tracking branch 'upstream/main' into migrate-hatch-scripts

361df23

ci(uv): run uv sync --all-extras

872d337

Related #3764

ci(uv): Switch lint.yml to uv sync

20f2894

ci(uv): Switch docbuild.yml to uv sync

449aaac

revert: Remove _tasks.py dependents

b9e6eb9

Reverts a good chuck of new stuff added in this PR, prior to 00f353a Next commit will refactor some of what is left - to simplify `[tool.taskipy.tasks]`

refactor: Remove _tasks.py, work into fs.py

2ed212a

- Has parts of prior `sync_website.py` - Utilizes for more concises commands in `[tool.taskipy.tasks]`

dangotbanned mentioned this pull request Jan 9, 2025

Upgrade to ruff==0.9.0 #3765

Open

dangotbanned added 2 commits January 9, 2025 18:42

docs: Deduplicate dev instructions

2b962ea

Easier to maintain and understand if this is in one place

chore: misc cleaning up tools.fs

0ed58e5

- Add missing entries to `__all__` - Add missing `copytree` doc - Remove incorrect `dir_exists` doc

dangotbanned marked this pull request as ready for review January 9, 2025 18:59

dangotbanned requested review from joelostblom, mattijn, jonmmease, binste and ChristopherDavisUCI January 9, 2025 19:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: Migrate from `hatch` to `uv` #3723

build: Migrate from `hatch` to `uv` #3723

dangotbanned commented Dec 29, 2024 •

edited

Loading

dangotbanned Jan 1, 2025

dangotbanned Jan 9, 2025

dangotbanned commented Jan 9, 2025

	# add commit, and push to github
	git add . --all
	git commit -m "$MSG"
	git push origin master

	CMD_ADD = "git add . --all --force"
	CMD_COMMIT = "git commit -m"
	CMD_PUSH = "git push origin master"

	def add_commit_push_github(msg: str, /) -> None:
	os.chdir(DOC_REPO_DIR)
	print("Pushing ...")
	# NOTE: Ensures the message uses cross-platform escaping
	cmd_commit = *CMD_COMMIT.split(" "), msg
	commands = (CMD_ADD, cmd_commit, CMD_PUSH)
	for command in commands:
	run_check(command)

	7. Merge release branch into main, make sure that all required checks pass

	8. On main, build source & wheel distributions. If you work on a fork, replace `origin` with `upstream`:

	git switch main
	git pull origin main
	hatch clean # clean old builds & distributions
	hatch build # create a source distribution and universal wheel

	9. publish to PyPI (Requires correct PyPI owner permissions):

	hatch publish

	10. build and publish docs (Requires write-access to altair-viz/altair-viz.github.io):

	hatch run doc:publish-clean-build

	11. On main, tag the release. If you work on a fork, replace `origin` with `upstream`:

build: Migrate from hatch to uv #3723

Are you sure you want to change the base?

build: Migrate from hatch to uv #3723

Conversation

dangotbanned commented Dec 29, 2024 • edited Loading

Description

Using uv

Task Runner

Usage

Changes

Composition

Invoking

Alternatives

Tasks

dangotbanned Jan 1, 2025

Choose a reason for hiding this comment

FIXME

Problem

dangotbanned Jan 9, 2025

Choose a reason for hiding this comment

dangotbanned commented Jan 9, 2025

build: Migrate from `hatch` to `uv` #3723

build: Migrate from `hatch` to `uv` #3723

dangotbanned commented Dec 29, 2024 •

edited

Loading

Using `uv`