Update pre-commit, linting and dev requirements

1. Disable autofix from pre-commit.ci

2. Add `nbstripout` in pre-commit which is a tool for stripping output from jupyter notebooks.
This is useful for keeping notebook size small and notebook diffs (semi-)sane.

3. Replace `prospector` with `ruff`. `ruff` is way faster and works great as a pre-commit
 hook. Furthermore is under very active development and it is quite easy to enable more
checks.

4. Remove deptry.

5. Require poetry 1.2+ for development since we make use of dependency groups.
For example in the sample notebooks we use use `matplotlib` for
visualization and `scipy` for filling missing IOC data. These are **not** `searvey`
dependencies. They are just something we chose to use in our examples. In this sense it
doesn't make sense to have them as "extras", because installing them will not add any
functionality to `searvey`.  Still we want them to be present in the dev environment in order
to be able to develop/use the notebooks. Dependency groups is an elegant way to
both allow installing them and to keep them separated from "core" and "dev" dependencies.

Note: I removed `black` and `ruff` from the dev-dependencies because we can run them
through pre-commit. The benefit of this approach is that these tools run into their own
dedicated environments and thus we have fewer dependencies in the searvey environment.
Running them should be done with `make style` and `make lint` respectively.

.gitignore

@@ -143,3 +143,5 @@ cython_debug/
 
 Notebooks/
 conda.recipe/
+
+ignored/

.pre-commit-config.yaml

@@ -1,9 +1,10 @@
 ---
+ci:
+  skip: ["mypy", "docs"]
+  autofix_prs: false
 default_language_version:
   python: "python3"
 fail_fast: false
-ci:
-  skip: ["poetry-lock", "lint", "mypy", "docs"]
 
 repos:
   - repo: "https://github.com/pre-commit/pre-commit-hooks"
@@ -41,35 +42,37 @@ repos:
           - "--py38-plus"
 
   - repo: "https://github.com/psf/black"
-    rev: "22.12.0"
+    rev: "23.1.0"
     hooks:
       - id: "black"
 
-  # - repo: "https://github.com/fpgmaas/deptry"
-  #   rev: "0.7.1"
-  #   hooks:
-  #     - id: "deptry"
-  #       args: ["--ignore-notebooks"]
+  - repo: "https://github.com/charliermarsh/ruff-pre-commit"
+    # Ruff version.
+    rev: 'v0.0.253'
+    hooks:
+      - id: "ruff"
+
+  - repo: "https://github.com/kynan/nbstripout"
+    rev: "0.6.1"
+    hooks:
+      - id: "nbstripout"
 
   - repo: "https://github.com/python-poetry/poetry"
-    rev: "1.3.0"
+    rev: "1.4.0"
     hooks:
       - id: "poetry-check"
       - id: "poetry-lock"
+        args: ["--check"]
       - id: "poetry-export"
-        args: ["--without-hashes", "-f", "requirements.txt", "-o", "requirements/requirements.txt"]
+        name: 'poetry export main'
+        args: ["--without-hashes", "--only", "main", "-f", "requirements.txt", "-o", "requirements/requirements.txt"]
       - id: "poetry-export"
-        args: ["--without-hashes", "-f", "requirements.txt", "--dev", "-o", "requirements/requirements-dev.txt"]
+        name: 'poetry export dev'
+        args: ["--without-hashes", "--with", "dev", "--with", "docs", "--with", "jupyter", "-f", "requirements.txt", "-o", "requirements/requirements-dev.txt"]
 
   - repo: "local"
     hooks:
 
-      - id: "lint"
-        name: "lint"
-        entry: "make lint"
-        language: "system"
-        types: ["python"]
-
       - id: "mypy"
         name: "mypy"
         entry: "make mypy"

Makefile

@@ -1,18 +1,31 @@
 .PHONY: list docs
 
 list:
-	@LC_ALL=C $(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$'
+	@LC_ALL=C $(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | grep -E -v -e '^[^[:alnum:]]' -e '^$@$$'
+
+init:
+	poetry install --with dev --with docs --with jupyter --sync
+	pre-commit install
+
+style:
+	pre-commit run black -a
 
 lint:
-	prospector --absolute-paths --no-external-config --profile-path .prospector.yaml -w profile-validator searvey
+	pre-commit run ruff -a
 
 mypy:
 	dmypy run searvey
 
+test:
+	python -m pytest -vlx
+
+cov:
+	coverage erase
+	python -m pytest --cov=searvey -n auto --durations=10
+
 docs:
 	make -C docs html
 
 deps:
-	deptry --ignore-notebooks ./
-	poetry export --without-hashes -f requirements.txt -o requirements/requirements.txt
-	poetry export --without-hashes -f requirements.txt --with dev -o requirements/requirements-dev.txt
+	pre-commit run poetry-lock -a
+	pre-commit run poetry-export -a

README.md

@@ -7,10 +7,9 @@
 
 Searvey aims to provide the following functionality:
 
-- Unified catalogue of observational data including near real time.
+- Unified catalogue of observational data including near real time (WIP).
 
-- Real time data analysis/clean up to facilitate comparison with numerical
-  models.
+- Real time data analysis/clean up to facilitate comparison with numerical models (WIP).
 
 - On demand data retrieval from multiple sources that currently include:
 
@@ -20,22 +19,66 @@ Searvey aims to provide the following functionality:
 
 ## Installation
 
-The package can be installed with `conda`:
+The package can be installed with `pip`:
+
+```
+pip install searvey
+```
+
+and conda`:
+
+```
+conda install -c conda-forge searvey
+```
 
-`conda install -c conda-forge searvey`
 
 ## Development
 
+In order to develop `searvey` you will need:
+
+- Python 3.8+
+- GNU Make
+- [poetry](https://python-poetry.org/) >= 1.2 (you can install it with [pipx](https://github.com/pypa/pipx): `pipx install poetry`).
+- [poetry-dynamic-versioning](https://github.com/mtkennerly/poetry-dynamic-versioning) which is a poetry plugin.
+  Take note that this needs to be installed in the same (virtual) environment as poetry, not in the `searvey` one!
+  If you used `pipx` for installing `poetry`, then you can inject it in the proper env with `pipx inject poetry poetry-dynamic-versioning`.
+- [pre-commit](https://pre-commit.com/). You can also install this one with `pipx`: `pipx install pre-commit`
+
+In order to setup the dev environment you can use:
+
 ```
 python3 -mvenv .venv
 source .venv/bin/activate
-poetry install
-pre-commit install
+make init
+```
+
+which will:
+
+1. create and activate a virtual environment,
+2. install the full set of dependencies
+3. Setup the pre-commit hooks
+
+After that you should run the tests with:
+
+```
+make test
 ```
 
+If you execute `make` without arguments, you should see more subcommands. E.g.
+
+```
+make mypy
+make lint
+make docs
+make deps
+```
+
+Check them out!
+
+### Jupyter
+
 If you wish to use jupyterlab to test searvey, then, assuming you have an
-existing jupyterlab
-installation, you should be able to add a kernel to it with:
+existing jupyterlab installation, you should be able to add a kernel to it with:
 
 ```bash
 python -m ipykernel install --user --name searvey

docs/source/conf.py

@@ -14,9 +14,7 @@
 from os import PathLike
 from pathlib import Path
 
-import toml
 from dunamai import Version
-from setuptools import config
 
 
 def repository_root(path: PathLike = None) -> Path:
@@ -42,14 +40,12 @@ def repository_root(path: PathLike = None) -> Path:
 )
 
 # -- Project information -----------------------------------------------------
-metadata = toml.load("../../pyproject.toml")["tool"]["poetry"]
-
-project = metadata["name"]
+project = "searvey"
 copyright = f"{datetime.date.today().year}, https://github.com/oceanmodeling"
 
 # The full version, including alpha/beta/rc tags
 try:
-    release = Version.from_any_vcs().serialize()
+    release = Version.from_any_vcs().serialize(dirty=True)
 except RuntimeError:
     release = os.environ.get("VERSION", "0.0.0")