python3Packages.torcheval: init at 0.0.7

[bengsparks]

A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[osbm]

Fixes: #376031

[osbm]

i think torchtnt should be declared as on its own library so that nixpkgs users have access to both libraries

[bengsparks]

@osbm That version of torchtnt is unlikely to be of great to use outside of this package.
I've updated the inclusion of torchtnt with a comment explaining how this came to be.

[bengsparks]

Alternatively, I could patch / correct a codepath that makes use of the breaking change in torchtnt, then nixpkgs' torchtnt can be used.
The diff in question is a

--- a/torcheval/tools/module_summary.py
+++ b/torcheval/tools/module_summary.py
@@ -29,7 +29,7 @@ from torch.utils._pytree import PyTree, tree_flatten
 from torch.utils.hooks import RemovableHandle
 
 from torchtnt.utils import Timer
-from torchtnt.utils.version import is_torch_version_geq_1_13
+from torchtnt.utils.version import is_torch_version_geq
 from typing_extensions import Literal
 
 _ATTRIB_TO_COL_HEADER = {
@@ -242,7 +242,7 @@ def _get_module_flops_and_activation_sizes(
     module_kwargs = module_kwargs or {}
     flops_forward = None
     flops_backward = None
-    if not is_torch_version_geq_1_13():
+    if not is_torch_version_geq("1.13.0"):
         warnings.warn(
             "Please install PyTorch 1.13 or higher to compute FLOPs: https://pytorch.org/get-started/locally/"
         )

[osbm]

It would actually be awesome to open pull requests on these upstream repositories first, if they get added quickly nixpkgs would never have to deal with these. Can you open these PRs or should i? I am talking about the both patches of torcheval and your patch of torchtnt.

[bengsparks]

@osbm unfortunately, while this patch does let the tests run, it reveals further breaking changes down the line, including the removal of torchtnt.utils.timer.py'sTimer.start and Timer.stop functions, with the only remaining alternative being the @contextmanager-annotated function Timer.action.

The usage of Timer.start and Timer.stop is complex enough (tools/module_summary.py:210) that I don't see how to replace the calls to Timer.start / Timer.stop with Timer.action, as _register_hooks makes use of register_module_forward_pre_hook to start and stop the timers.

forward_elapsed_time_handles = _register_hooks(
    module,
    [
        (_forward_time_pre_hook(forward_timer_mapping), _HookType.FORWARD_PRE_HOOK), # call to Timer.start
        (
            _forward_time_hook(forward_timer_mapping, forward_elapsed_times_sec), # call to Timer.stop
            _HookType.FORWARD_HOOK,
        ),
    ],
)

...

for hook_handle in forward_elapsed_time_handles:
    hook_handle.remove()

At that point it would be better to wait for torcheval to be tagged with 0.0.7 on GitHub, and bump this package accordingly (which I would gladly do as maintainer).
This PR, which made it into the GitHub repository 10 days before torcheval 0.0.7 was released on Pypi, removed torchtnt as a runtime dependency, which should make this easier.

Until then, I think I'll submit this package.
If you have a suitable codebase, could you import and call some functions of this package to see if things work as expected?

[osbm]

I have some scripts i can test it. But for that you need to add this package to the top-level/python-packages.nix list.

If you can also add me to the maintainer list to this package i can help update/maintain it in the future.

[bengsparks]

@osbm I've added the relevant entry to top-level/python-packages.nix. Let me know if something breaks

[bengsparks]

The newest push stems from an updated comment that documents why the tests disabled in 27fb1ae cannot work, even if they are granted filesystem access.
The resulting derivation remains identical.

[bengsparks]

Using torchtnt-nightly directly as input causes tests to fail immediately

{ torchtnt-nightly }:
buildPythonPackage {
  # ...
  dependencies = [
    torchtnt-nightly
    typing-extensions
  ];
}

> =========================== short test summary info ============================
> ERROR tests/tools/test_flops.py
> ERROR tests/tools/test_module_summary.py
> !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
> ================== 1 deselected, 1 warning, 2 errors in 1.63s ==================
error: build of '/nix/store/1jx7cmyhhrpx52ryq93463m718sdnki6-python3.12-torcheval-0.0.6.drv' on 'ssh-ng://rosetta-builder' failed: builder for '/nix/store/1jx7cmyhhrpx52ryq93463m718sdnki6-python3.12-torcheval-0.0.6.drv' failed with exit code 2;
       last 25 log lines:
       > E   ImportError: cannot import name 'is_torch_version_geq_1_13' from 'torchtnt.utils.version' (/nix/store/mhzrnmh20f4kild55i233yzg71ywy24d-python3.12-torchtnt-nightly-2024.8.1/lib/python3.12/site-packages/torchtnt/utils/version.py)

When overriden to use the version I had picked out, the tests pass.

{ torchtnt-nightly }:
let
  torchtnt = torchtnt-nightly.overridePythonAttrs rec {
    pname = "torchtnt-nightly";
    version = "2023.1.25";
    src = fetchPypi {
      inherit pname version;
      hash = "sha256-eFouZgdVQaqrXIKLY7geLV8GEVPGu6IHKIPK6aLJH8w=";
    };
  };
in
buildPythonPackage {
  # ...
  dependencies = [
    torchtnt
    typing-extensions
  ];
}

$ nix-build -A python3Packages.torcheval --system aarch64-linux                 
/nix/store/sxcy8l7isk1ynxhg406zvkymv58d1lx6-python3.12-torcheval-0.0.6

[GaetanLepage]

@bengsparks have you tried reaching upstream?
This packaging endeavor would be made significantly easier if the versions of the packages were matching.
Maybe you can open issues to ask torchtnt to make a new release?

[bengsparks]

@GaetanLepage I think waiting for torcheval 0.0.7 on GitHub is the correct move then. (I have, since my work here, come to understand that Nix cannot fix package conflicts in Python).
This version of torcheval uses torchtnt as a dev dependency, which would make this process trivial. The only dependency would be typing extensions.

Alternatively I could package torcheval 0.0.7 from PyPI, but there are no tests there.
pytorch/torcheval#215 has seen no activity, and the repository doesn't seem particularly active.

[GaetanLepage]

@GaetanLepage I think waiting for torcheval 0.0.7 on GitHub is the correct move then. (I have, since my work here, come to understand that Nix cannot fix package conflicts in Python). This version of torcheval uses torchtnt as a dev dependency, which would make this process trivial. The only dependency would be typing extensions.

Alternatively I could package torcheval 0.0.7 from PyPI, but there are no tests there. pytorch/torcheval#215 has seen no activity, and the repository doesn't seem particularly active.

Well, I think you could wait for a long time.
Instead, I suggest to use src.rev to manually fetch this commit.
Set the version to 0.0.7 and we should be good to go.

[GaetanLepage]

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 376129

---

x86_64-linux

❌ 2 packages failed to build:

• python312Packages.torcheval
• python312Packages.torcheval.dist

---

aarch64-linux

❌ 2 packages failed to build:

• python312Packages.torcheval
• python312Packages.torcheval.dist

[GaetanLepage]

Tests are missing the skimage dependency (it's scikit-image I think).

[bengsparks]

nixpkgs-review is failing on my system due to reasons unrelated to this PR, so I'm currently unable to test locally.

[GaetanLepage]

nixpkgs-review is failing on my system due to reasons unrelated to this PR, so I'm currently unable to test locally.

As this is a leaf package, you don't really need nixpkgs-review. Simply ensure that you can build python31[2-3]Packages.torcheval on the systems you have access to :)

[GaetanLepage]

LGTM, thanks!

[GaetanLepage]

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 376129

---

x86_64-linux

✅ 4 packages built:

• python312Packages.torcheval
• python312Packages.torcheval.dist
• python313Packages.torcheval
• python313Packages.torcheval.dist

---

aarch64-linux

✅ 4 packages built:

• python312Packages.torcheval
• python312Packages.torcheval.dist
• python313Packages.torcheval
• python313Packages.torcheval.dist