dependency management

[tfiers]

context: I'm wondering what package manager to use for installing jupyterbook (which is a PyPI-package) and its dependencies.

I then also wondered which install option we will generally recommend for the main project.

Poetry seems nice (lock files are good), though its install instructions (legacy | recommended) seem a bit daunting to adopt it as general recommended pkg manager.
I have to admit I was a bit loathe to install it as it's yet another directory in my homedir :p (same with .conda, .jupyter, ..).

I tried the install instructions: had to change python3 to python for it to work, and manually add %APPDATA$\Python\Scripts to my PATH (first did it wrong and added C:\Users\lpxtf3\AppData\Roaming\Python\Scripts %AppData%\Roaming\Python\Scripts to my path, but should've followed the manual literally).
Also, apparently you can move .poetry/ away from your homedir using POETRY_HOME, which is nice.
poetry install in the repo root then worked nicely.

Conda's dependency resolver is slow (though it can be sped up with the right settings). Conda is more widespread in the compneuro community than Poetry I think, and unlike pip or Poetry, it can also define non-PyPI dependencies (including the Python version itself).
This might become especially important when GPU-training our SNNs (e.g.)
It by default has no lock files. But if using conda we should use conda-lock, which provides them (and is bound to eventually become part of base conda).

An idea is maybe to combine Conda and Poetry:
conda and its environment.yml installs only Python and Poetry (and in the future potential other non PyPI software if needed),
and Poetry install PyPI packages (and our own python package).
Then we don't have to worry about Poetry's install -- miniconda install is an executable on windows and mac, so it's a bit easier. It also adds to path for you.
We then don't have duplication of packages between pyproject.toml > [tool.poetry.dependencies] and environment.yml.

What do you guys think? (@synek)

Then one thing left to decide is what to do with requirements.txt.
How will we keep it in sync with pyproject.toml > [tool.poetry.dependencies]? Or shall we simply delete it.
There's a trade-off here.
Allowing a simple pip install -f requirements.txt oneliner makes getting started as easy as possible for new contributors. They then don't have to install conda and wait for it to resolve the environment and install Poetry and an exact Python version, and then run poetry install.
The 'danger' is that the dependency lists get out of sync, leading to frustration when things don't work.

[tfiers]

The described conda+Poetry normalized combo is what's described here: https://stackoverflow.com/a/71110028/2611913, apparently used with success

[tfiers]

a bonus nice thing about conda lock: no package resolution is needed ("conda-lock acts as an external presolve for conda"), so should speed up cond install command

[thesamovar]

I would be in favour of keeping it light and easy to understand, either conda or pip. Slow resolve of conda isn't an issue if we try to minimise the number of external packages we bring in. My personal vote would be for conda because it's what I already use and find that the environments are nicely handled.

[tfiers]

Yes, definitely also in favour of "light and easy to understand" 👍

[tfiers]

The fastest onboarding is also without lock files: then users can just use already installed package versions, without having to download and install exactly pinned versions.

So I would propose for the final solution to have two install options:

• fully reproducible (using lockfile(s))
• as fast and light as possible

[thesamovar]

Yeah although if people aren't using the lockfiles then it's not really fully reproducible. How about we worry about that if it becomes a problem? I think even having a standard environment and getting people to use that would be a huge improvement on what people usually do!

[rorybyrne]

The lockfile is used automatically, under the hood. So if you run poetry install it looks for a lockfile and if it finds one then that's the dependency tree that it installs. If not, it fetches the latest versions of everything.

[rorybyrne]

The same is presumably true for the conda lock system, or any others.

[tfiers]

true. in general, YAGNI, bottom-up, needs-based, 'organic' organization is the way to go in my experience

[thesamovar]

Ooh, I hadn't heard that YAGNI thing - it's excellent!

[tfiers]

Together with some of the maxims from the Zen of Python it's one of those programming principles that keeps coming up and showing its usefulness

[tfiers]

OTOH.
For my own jupyterbook, one day, my website suddenly wouldn't build anymore. Turned out there was an update in one of the dependencies (I had specified my dependencies only loosely, i.e. not pinned).

I then had to dive into the log files of an older github actions run that had still completed succesfully to track down the exact versions of jupyterbook's dependencies, to then pin them in the requirements.txt file, so that the build would work again.

[tfiers]

Agree. I'll do pinning for the jupyterbook (there is not really such an onboarding requirement there).
and then for the main project I agree we should have only pip or conda and no lock files atm
What do you think @synek, @ghoshm ?

[tfiers]

I think we can marry reproducibility with loose version requirements with a regularly run CI that alerts us when a new dependency version breaks some of our code

[rorybyrne]

Maybe we could put just the conda or pip instructions at the top of the README.md, and have a smaller "alternative installation" section lower down for the others? That might simplify things.

Also, we could (and probably should) add a Makefile so people can just run make install-pip or make install-conda and it'll install all the dependencies.

[tfiers]

What would those makefiles roughly contain? (I am not very experienced with Make)

[rorybyrne]

A Makefile might look like this:

install:
        conda install --env environment.yml

clean:
        rm -rf .venv
        rm -rf build/
        rm -rf figures/

install-pip:
        pip install -r requirements.txt

Then you just run make clean and it executes the corresponding commands in the Makefile. You can also run make clean install-pip and it'll run the clean commands followed by the install-pip commands.

It gets more complex (fast), but that's the basics.

[rorybyrne]

There are a few options that I personally like:

1. We can just support all 3 dependency managers, maintaining pyproject.toml, requirements.txt, and environment.yml side-by-side.
2. We can focus on pip and the requirements.txt install path.
3. We can add a Makefile with a make install command, and ask people to run that. We could also add other commands like make virtualenv, make activate-venv, etc..
4. We can give people a one-liner for installing poetry and then the dependencies: curl -sSL https://install.python-poetry.org | python3 - && poetry install

Personally, I lean towards 1 in the short-term or maybe 4. I really dislike conda so I'd like if we could at least keep Poetry as an option.

[rorybyrne]

I definitely don't want us to remove conda or pip. If the poetry export route seems like too much hassle, then I'm happy to just focus on conda or pip and I'll keep my poetry files locally.

[rorybyrne]

(though there's something to be said for encouraging people to improve their coding literacy by stepping outside the conda + windows paradigm)

[ghoshm]

My vote would be for conda!

People are familar with it, and I agree with @thesamovar that for now the most important thing is that people get onboard with the project.

[rorybyrne]

Okay I'll on a PR to settle things on conda.

I'll also look into synchronising the 3x systems by exporting from pyproject.toml to requirements.txt and environment.yml automatically via Github Actions. If it works elegantly, then I'll open another PR some time in the future so we can see how it works and consider using it.

[rorybyrne]

Okay, done. Not proud of it, but it's done!

#15

[tfiers]

(cross-posting from #15 as it's generally useful):

Highly recommend checking out miniforge/mambaforge if you hadn't seen it before: https://github.com/conda-forge/miniforge
It's how I recommend installing conda.

Conda-forge is a great project & channel (the official/default conda channel is not very good)

Having conda-forge as the default channel (and not just as an extra channel besides Anaconda Inc's) also greatly speeds up dependency resolution (version solving).
It's the nr 1 most important way to speed up conda. More: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/conda-performance.html#improving-conda-performance
(Speed has been a major focus of the dev team in the past years – they're well aware of the problem and have already made good improvements).