Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update readme descriptions #16

Closed
wants to merge 54 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
2416308
feat(bayes3d): init bayes3d @ ba4234a4
srounce Apr 5, 2024
82b2700
WIP: try using pypi wheel
srounce Apr 26, 2024
eea7adf
WIP: building opencv from source failing with py.typed not found
srounce Jun 26, 2024
7c0daf3
WIP: everything builds but has package conflicts due to genjax being …
srounce Jun 27, 2024
4c3872e
WIP: genjax still broken, jaxlib FOD is non-reproducible
srounce Jun 28, 2024
d343d10
WIP: bayes3d building with overrides
srounce Jul 9, 2024
84416b9
WIP: add bayes3d pythonImportsCheck
srounce Jul 9, 2024
66ff185
WIP: Fettling to get bayes3d demo ipynb working
srounce Jul 9, 2024
9c31e28
feat: First working version of Bayes3D in Ipynb
srounce Jul 9, 2024
d36d542
chore(flake): expose packages as checks
zimbatm Jul 9, 2024
4d45705
chore(flake): use final/prev
zimbatm Jul 9, 2024
90e17a9
chore(flake): add the devshell python to the checks
zimbatm Jul 9, 2024
ff6060b
fix: Correct invalid packages attributes
srounce Jul 9, 2024
1449164
chore(flake): Remove unused poetry2nix
srounce Jul 9, 2024
eee9ba4
fix(distributions): Patch source to use GNU Sed on Darwin
srounce Jul 9, 2024
47517c5
fixup! chore(flake): Remove unused poetry2nix
zimbatm Jul 10, 2024
b49dbdc
python3Packages.sppl: fix build
zimbatm Jul 10, 2024
a093b28
chore(flake): disable aarch64-linux
zimbatm Jul 10, 2024
0560a06
chore(flake): switch tensorflow and jaxlib versions based on support
zimbatm Jul 10, 2024
3fae381
chore(nix): remove last uses of python3Packages
zimbatm Jul 10, 2024
488b76c
fix(distributions): add missing pyflakes for macOS
zimbatm Jul 10, 2024
2c39803
chore: remove unused packages
zimbatm Jul 10, 2024
dec6612
chore: run deadnix on all the files
zimbatm Jul 10, 2024
6d41acb
chore: run nixfmt on all the files
zimbatm Jul 10, 2024
06ef7e9
feat: re-export the loom docker images as .#loomOCI
zimbatm Jul 10, 2024
17acdad
fix(open3d): Enable pythonImportsCheck and add missing deps
srounce Jul 10, 2024
88b3d11
feat: Add .envrc.local template for providing GitHub token
srounce Jul 10, 2024
4a6b9c9
fix(tensorflow-probability): Per-system fetchAttrs hashes for Bazel b…
srounce Jul 10, 2024
68bd503
fix(bayes3d): Only include CUDA flavoured packages on Linux
srounce Jul 10, 2024
396f7b9
chore(devshell): Move devshell to its own flake-parts module
srounce Jul 10, 2024
4514a86
chore(flake): Move legacyPackages.python3Packages definition and over…
srounce Jul 10, 2024
2bf4768
chore(flake): Remove redundant FIXMEs
srounce Jul 10, 2024
4d9ae85
fix(opencv-python): Add aarch64-darwin hash for Python3.11 wheel
srounce Jul 10, 2024
0bf3ba3
fix(opencv-python): Remove breakpointHook and commented lines
srounce Jul 10, 2024
af65293
fix(flake): Python packages can access flake inputs
srounce Jul 11, 2024
e59fe69
chore(flake): Cleanup moved loadPackages function
srounce Jul 11, 2024
7a850bd
python3Packages.dm-tree: use wheel
zimbatm Jul 11, 2024
4fc0f17
fix(dm-tree): Use fetchPypi and enable Darwin platforms
srounce Jul 12, 2024
8496854
chore(.envrc): Disable entering devshell upon loading direnv environment
srounce Jul 12, 2024
7343a8b
fix(devshell): Execute CUDA-specific shellhook code only if CUDA is s…
srounce Jul 12, 2024
8ccf614
fix(bayes3d): Toggle CUDA inclusion based on nixpkgs.cudaSupport setting
srounce Jul 12, 2024
a48f9bf
fix(open3d): Toggle CUDA inclusion based on nixpkgs.cudaSupport setting
srounce Jul 12, 2024
9d507f3
chore(direnv): restore nix shell
zimbatm Jul 29, 2024
5738d57
chore(demo): fix demo notebook
zimbatm Jul 29, 2024
73c006d
Merge pull request #1 from numtide/feat/bayes3d
zimbatm Jul 29, 2024
380ed0a
chore: restructure project
zimbatm Jul 29, 2024
0a18c72
chore: remove unused tcmalloc package
zimbatm Jul 29, 2024
5fa8b47
chore: rename ociImgBase to baseOCI
zimbatm Jul 29, 2024
3942402
chore(README): list all the packages
zimbatm Jul 29, 2024
fc9bb4f
chore: format the nix code with nixfmt-rfc-style
zimbatm Jul 29, 2024
56149b7
chore: remove dead code with deadnix
zimbatm Jul 29, 2024
07ae56f
Merge pull request #12 from numtide/restructure
zimbatm Jul 29, 2024
ecd6acf
feat: add jupyter-bayes3d package (#13)
zimbatm Jul 29, 2024
adb8294
chore: Add package descriptions and links in README.md
srounce Jul 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .envrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
# ^ added for shellcheck and file-type detection

# Reload if any of these files change
watch_file .envrc.local

if [[ $(type -t use_flake) != function ]]; then
echo "ERROR: use_flake function missing."
echo "Please update direnv to v2.30.0 or later."
exit 1
fi

if [ -f .envrc.local ]; then
source_env .envrc.local

export NIX_CONFIG="access-tokens = github.com=$NIX_GITHUB_TOKEN"
else
echo "
WARNING: .envrc.local not found, please run \`cp .envrc.local.template .envrc.local\`
and fill in the missing values."
fi

use_flake
8 changes: 8 additions & 0 deletions .envrc.local.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/usr/bin/env bash

# This value is a GitHub PAT, it can be generated on the following page:
# https://github.com/settings/tokens
#
# Ensure the "repo" scope is enabled
export NIX_GITHUB_TOKEN=<TODO>

2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
result
result-*
.ipynb_checkpoints/
.envrc.local
136 changes: 120 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,39 @@ To access the `lib` code exported by this flake, declare this repo as a flake in

## Packages

### `.#sppl`
List of Nix packages available in this repo.

### `.#baseOCI`

### `.#loomOCI`

A Loom container image is also provided. It can be built and loaded into your local Docker registry with the following command:

```console
$ docker load -i $(nix build '.#loomOCI' --no-link --print-out-paths)
```

### `.#jupyter-bayes3d`

A jupyter environment with bayes libraries available.

Example:
```console
$ nix run .#jupyter-bayes3d notebook ./notebooks/demo.ipynb
```

## Python Packages

Here are all the python packages this flake provides, on top of all the ones
available in nixpkgs.

All the packages are compiled against Python 3.11.

### `.#bayes3d`

Python [library by ProbSys](https://github.com/probsys/sppl) packaged for python3.9 .
Bayes3D is a 3D scene perception system based on probabilistic inverse graphics.

* [GitHub](https://github.com/probcomp/bayes3d)

### `.#loom`

Expand All @@ -52,27 +82,101 @@ platform-dependent `distributions`.

Your options are:

```bash
nix build '.#packages.x86_64-linux.loom' # same as `.#loom` if that is your OS/arch
nix build './envs-flake#packages.x86_64-darwin.ociImgLoom'
```console
$ nix build '.#packages.x86_64-linux.loom' # same as `.#loom` if that is your OS/arch
$ nix build './envs-flake#packages.x86_64-darwin.ociImgLoom'
```

If you are running on Mac silicon (`aarch64-darwin`), that OCI image will run but behavior is not defined or supported.

#### `.#loom.morePackages.distributions`
### `.#distinctipy`

distinctipy is a lightweight python package providing functions to generate colours that are visually distinct from one another.

* [GitHub](https://github.com/alan-turing-institute/distinctipy)
* [PyPi](https://pypi.org/project/distinctipy)

### `.#distributions`

Native library for probability distributions in python used by Loom. NOTE: this ONLY builds for `x86_64` architectures and only runs on linux.

#### `.#loom.morePackages.parsable`
#### `.#loom.morePackages.pymetis`
#### `.#loom.morePackages.goftests`
#### `.#loom.morePackages.tcmalloc`
* [GitHub](https://github.com/posterior/distributions)

Other upstream python packages required by Distributions and/or Loom.
### `.#dm-tree`

#### `.#loom.ociImg`
Tree is a library for working with nested data structures. In a way, tree generalizes the builtin map function which only supports flat sequences, and allows to apply a function to each "leaf" preserving the overall structure.

A Loom container image is also provided as a passthru attribute of `loom`. It can be built and loaded into your local Docker registry with the following command:
```sh
docker load -i $(nix build 'github:OpenGen/nix/loom-oci-img-attribute#loom.ociImg' --no-link --print-out-paths)
```
* [GitHub](https://github.com/deepmind/tree)
* [PyPi](https://pypi.org/project/dm-tree)

### `.#genjax`

GenJAX is an implementation of Gen on top of JAX - exposing the ability to programmatically construct and manipulate generative functions, as well as JIT compile + auto-batch inference computations using generative functions onto GPU devices.

* [GitHub](https://github.com/probcomp/genjax)

### `.#goftests`

Goftests is intended for unit testing random samplers that generate arbitrary plain-old-data, and focuses on robustness rather than statistical efficiency. In contrast to scipy.stats and statsmodels, goftests does not make assumptions on the distribution being tested, and requires only a simple (sample, prob) interface provided by MCMC samplers.

* [GitHub](https://github.com/posterior/goftests)

### `.#open3d`

Open3D is an open-source library that supports rapid development of software that deals with 3D data.

* [GitHub](https://github.com/isl-org/Open3D)
* [PyPi](https://pypi.org/project/open3d)

### `.#opencv-python`

Wrapper package for OpenCV python bindings.

* [GitHub](https://github.com/opencv/opencv-python)
* [PyPi](https://pypi.org/project/opencv-python)

### `.#oryx`

Oryx is a library for probabilistic programming and deep learning built on top of Jax.

* [GitHub](https://github.com/jax-ml/oryx)
* [PyPi](https://pypi.org/project/oryx)

### `.#parsable`

Parsable is a lightweight decorator-based command line parser library. Parsable was written to be simpler than argparse, optparse, and argh.

### `.#plum-dispatch`

Multiple dispatch in Python.

* [GitHub](https://github.com/beartype/plum)
* [PyPi](https://pypi.org/project/plum-dispatch)

### `.#pymetis`

PyMetis is a Python wrapper for the Metis graph partititioning software.

* [GitHub](https://github.com/inducer/pymetis)
* [PyPi](https://pypi.org/project/PyMetis)

### `.#pyransac3d`

pyRANSAC-3D is an open source implementation of Random sample consensus (RANSAC) method. It fits primitive shapes such as planes, cuboids and cylinder in a point cloud to many aplications: 3D slam, 3D reconstruction, object tracking and many others.

* [GitHub](https://github.com/leomariga/pyRANSAC-3D)
* [PyPi](https://pypi.org/project/pyransac3d)

### `.#sppl`

Probabilistic programming system for fast and exact symbolic inference.

* [GitHub](https://github.com/probsys/sppl)
* [PyPi](https://pypi.org/project/sppl)

### `.#tensorflow-probability`

TensorFlow Probability is a library for probabilistic reasoning and statistical analysis in TensorFlow.

* [GitHub](https://github.com/tensorflow/probability)
* [PyPi](https://pypi.org/project/tensorflow-probability)
62 changes: 30 additions & 32 deletions envs-flake/flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,14 @@
gensqlquery.url = "github:OpenGen/GenSQL.query";
};

outputs = inputs@{ self, nixpkgs, flake-parts, opengen, gensqlquery, ... }:
outputs =
inputs@{
nixpkgs,
flake-parts,
opengen,
gensqlquery,
...
}:
flake-parts.lib.mkFlake { inherit inputs; } {
imports = [
# To import a flake module
Expand All @@ -20,46 +27,37 @@
# 3. Add here: foo.flakeModule
inputs.flake-parts.flakeModules.easyOverlay
];
systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin" ];

systems = [
"x86_64-linux"
"aarch64-linux"
"aarch64-darwin"
"x86_64-darwin"
];

# NOTE: This property is consumed by flake-parts.mkFlake to specify outputs of
# the flake that are replicated for each supported system. Typically packages,
# apps, and devshells are per system.
perSystem = { config, self', inputs', pkgs, system, ... }:
let
toolkit = opengen.lib.basicTools pkgs;
sppl = opengen.packages.${system}.sppl;
perSystem =
{ pkgs, system, ... }:
let
toolkit = opengen.lib.basicTools pkgs;
sppl = opengen.packages.${system}.sppl;

ociImgGensqlQuery = pkgs.callPackage ./images/gensql.query {
inherit nixpkgs opengen gensqlquery;
};
ociImgGensqlQuery = pkgs.callPackage ./images/gensql.query { inherit nixpkgs opengen gensqlquery; };

ociImgLoom = pkgs.callPackage ./images/gensql.loom {
inherit nixpkgs opengen;
};
ociImgLoom = pkgs.callPackage ./images/gensql.loom { inherit nixpkgs opengen; };

packages = {
inherit
ociImgGensqlQuery
ociImgLoom
;
};
in {
devShells.default = pkgs.mkShell {
packages = [] ++ toolkit;
};
packages = {
inherit ociImgGensqlQuery ociImgLoom;
};
in
{
devShells.default = pkgs.mkShell { packages = [ ] ++ toolkit; };

devShells.sppl = pkgs.mkShell {
packages
= [sppl]
++ sppl.checkInputs
++ toolkit
;
};
devShells.sppl = pkgs.mkShell { packages = [ sppl ] ++ sppl.checkInputs ++ toolkit; };

inherit packages;
};
inherit packages;
};

# NOTE: this property is consumed by flake-parts.mkFlake to define fields
# of the flake that are NOT per system, such as generic `lib` code or other
Expand Down
24 changes: 16 additions & 8 deletions envs-flake/images/gensql.loom/default.nix
Original file line number Diff line number Diff line change
@@ -1,25 +1,33 @@
{ pkgs,
{
pkgs,
nixpkgs,
opengen,
system,
}: let
}:
let
# in OCI context, whatever our host platform we want to build same arch but linux
systemWithLinux = builtins.replaceStrings [ "darwin" ] [ "linux" ] system;
crossPkgsLinux = nixpkgs.legacyPackages.${systemWithLinux};
python = crossPkgsLinux.python3;

base = opengen.packages.${system}.ociImgBase;
base = opengen.packages.${system}.baseOCI;

loom = opengen.packages.${systemWithLinux}.loom;
in pkgs.dockerTools.buildLayeredImage {
in
pkgs.dockerTools.buildLayeredImage {
name = "probcomp/gensql.loom";
tag = systemWithLinux;
fromImage = base;
contents = [ loom python ];
contents = [
loom
python
];
config = {
Cmd = [ "${python}/bin/python" "-m" "loom.tasks" ];
Env = [
"LOOM_STORE=/loom/store"
Cmd = [
"${python}/bin/python"
"-m"
"loom.tasks"
];
Env = [ "LOOM_STORE=/loom/store" ];
};
}
13 changes: 7 additions & 6 deletions envs-flake/images/gensql.query/default.nix
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
{ pkgs,
nixpkgs,
{
pkgs,
system,
opengen,
gensqlquery,
}: let
}:
let
# in OCI context, whatever our host platform we want to build same arch but linux
systemWithLinux = builtins.replaceStrings [ "darwin" ] [ "linux" ] system;
crossPkgsLinux = nixpkgs.legacyPackages.${systemWithLinux};

base = opengen.packages.${system}.ociImgBase;
base = opengen.packages.${system}.baseOCI;
ociBin = gensqlquery.packages.${systemWithLinux}.bin;
in pkgs.dockerTools.buildImage {
in
pkgs.dockerTools.buildImage {
name = "probcomp/gensql.query";
tag = systemWithLinux;
fromImage = base;
Expand Down
Loading
Loading