Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add OCI image with Loom on Python3 #6

Merged
merged 5 commits into from
May 2, 2024
Merged

feat: add OCI image with Loom on Python3 #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e93abd1
feat: add OCI image with Loom
ships Apr 19, 2024
704b5e3
refactor: explode inline function call
ships Apr 24, 2024
89e5693
refactor: use buildLayeredImage
ships Apr 24, 2024
adb18b1
fix: Include loom itself as output after refactor
ships May 1, 2024
9e19987
feat: rename inferenceql.loom -> gensql.loom
ships May 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/nix.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ jobs:
- run: |
nix build \
-L \
-o ./loom \
'.#loom'
-o ./imgLoom \
'.#ociImgLoom'

- name: Login to Docker Hub
# only run this when running on main, because
Expand All @@ -44,3 +44,8 @@ jobs:
run: |
docker load -i ./imgGensqlQuery
docker push --all-tags probcomp/gensql.query
- name: push loom
if: github.ref == 'refs/heads/main'
run: |
docker load -i ./imgLoom
docker push --all-tags probcomp/inferenceql.loom
31 changes: 11 additions & 20 deletions flake.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# 1. Add foo to inputs
# 2. Add foo as a parameter to the outputs function
# 3. Add here: foo.flakeModule
./lib/devtools
./lib
inputs.flake-parts.flakeModules.easyOverlay
];
systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin" ];
Expand All @@ -36,35 +36,26 @@

toolkit = self.lib.basicTools pkgs;

callPackage = pkgs.newScope (
pkgs
// { inherit callPackage; }
// packages
);
callPy3Package = pkgs.newScope (
pkgs
// pkgs.python3Packages
// { inherit callPackage; }
// packages
);
scopes = (self.lib.mkScopes pkgs);
loom = scopes.callPy3Package ./pkgs/loom { };

packages = {
ociImgLoom = pkgs.callPackage ./pkgs/oci/gensql.loom {
inherit nixpkgs ociImgBase;
};

packages = loom.more_packages // {
inherit
loom
ociImgBase
ociImgGensqlQuery
ociImgLoom
;

goftests = callPackage ./pkgs/goftests { };
parsable = callPackage ./pkgs/parsable { };
pymetis = callPackage ./pkgs/pymetis { };
distributions = callPackage ./pkgs/distributions { };
loom = callPy3Package ./pkgs/loom { };
};
in {
devShells.default = pkgs.mkShell {
buildInputs = [] ++ toolkit;
};

ships marked this conversation as resolved.
Show resolved Hide resolved
inherit packages;
};

Expand Down
6 changes: 6 additions & 0 deletions lib/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
_: {
flake.lib = {
basicTools = import ./devtools;
mkScopes = import ./mkScopes;
};
}
24 changes: 10 additions & 14 deletions lib/devtools/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
_: {
flake = {
lib.basicTools = pkgs: with pkgs; [
coreutils
curl
file
gawk
git
gnugrep
gnused
which
];
};
}
pkgs: with pkgs; [
coreutils
curl
file
gawk
git
gnugrep
gnused
which
]
13 changes: 13 additions & 0 deletions lib/mkScopes/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
pkgs: let
callPackage = pkgs.newScope (
pkgs
// { inherit callPackage; }
);
callPy3Package = pkgs.newScope (
pkgs
// pkgs.python3Packages
// { inherit callPackage; }
);
in {
inherit callPackage callPy3Package ;
}
2 changes: 1 addition & 1 deletion pkgs/distributions/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ python3Packages.buildPythonPackage {

env.DISTRIBUTIONS_USE_PROTOBUF = 1;

# https://github.com/numba/numba/issues/8698#issuecomment-1584888063
# https://github.com/numba/numba/issues/8698#issuecomment-1584888063
env.NUMPY_EXPERIMENTAL_DTYPE_API = 1;

pythonImportsCheck = [
Expand Down
20 changes: 15 additions & 5 deletions pkgs/loom/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,18 @@
, wheel
, cpplint
, cython_0
, goftests
, imageio
, matplotlib
, mock
, nose
, numpy
, pandas
, parsable
, pep8
, pkgs
, protobuf3_20
, pyflakes
, python3Packages
, pytest
, pymetis
, scikit-learn
, scipy
, simplejson
Expand All @@ -29,10 +26,14 @@
, stdenv
, zlib
, eigen
, distributions
, gperftools
}:
let
goftests = callPackage ./../goftests { };
parsable = callPackage ./../parsable { };
pymetis = callPackage ./../pymetis { };
distributions = callPackage ./../distributions {inherit goftests parsable;};
Comment on lines +32 to +35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're gonna use a custom scoped callPackage it seems like it'd be a bit more idiomatic to have a attrset of private packages merged into the callPackage scope. That way packages within the flake can use the args injection to reference them from other packages without having to deal with the paths, but they won't be exposed directly under the flake packages field.


protobuf = protobuf3_20;

version = "0.2.10";
Expand Down Expand Up @@ -118,7 +119,7 @@ buildPythonPackage {
pyflakes
];

# https://github.com/numba/numba/issues/8698#issuecomment-1584888063
# https://github.com/numba/numba/issues/8698#issuecomment-1584888063
env.NUMPY_EXPERIMENTAL_DTYPE_API = 1;

env.DISTRIBUTIONS_USE_PROTOBUF = 1;
Expand Down Expand Up @@ -179,6 +180,15 @@ buildPythonPackage {

passthru.loom-cpp = loom-cpp;

passthru.more_packages = {
inherit
goftests
distributions
pymetis
parsable
;
};

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
passthru.ociImage = callPackage ./ociImage.nix { };

So you'd end up with nix build .#loom.ociImage rather than a separate sibling package.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. I like the inheritance here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am weighing this abstraction against like .#oci.loom ...

passthru.tests.run = callPackage ./test.nix { inherit src; };

passthru.test-shell = callPackage ({
Expand Down
25 changes: 25 additions & 0 deletions pkgs/oci/gensql.loom/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{ pkgs,
nixpkgs,
system,
ociImgBase,
}: let
# in OCI context, whatever our host platform we want to build same arch but linux
systemWithLinux = builtins.replaceStrings [ "darwin" ] [ "linux" ] system;
crossPkgsLinux = nixpkgs.legacyPackages.${systemWithLinux};
python = crossPkgsLinux.python3;

scopes = (import ./../../../lib/mkScopes) crossPkgsLinux;

loom = scopes.callPy3Package ./../../loom { };
in pkgs.dockerTools.buildLayeredImage {
name = "probcomp/gensql.loom";
tag = systemWithLinux;
fromImage = ociImgBase;
contents = [ loom python ];
config = {
Cmd = [ "${python}/bin/python" "-m" "loom.tasks" ];
Env = [
"LOOM_STORE=/loom/store"
];
};
}
Loading