From 91e35e7f29eb3a67f9c1ebcd049e792484cd908a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Sarah=20H=C3=BClsen?= <49907095+sarah-hlsn@users.noreply.github.com> Date: Mon, 20 Nov 2023 09:32:51 +0100 Subject: [PATCH] adapt testing and continuous integration guides --- ....ipynb => Guide_CLIMADA_conventions.ipynb} | 23 +- doc/guide/Guide_Configuration.ipynb | 17 -- doc/guide/Guide_Euler.ipynb | 16 -- doc/guide/Guide_Git_Development.ipynb | 20 -- doc/guide/Guide_Py_Performance.ipynb | 18 -- doc/guide/Guide_PythonDos-n-Donts.ipynb | 19 -- ..._and_Testing.ipynb => Guide_Testing.ipynb} | 227 ++---------------- ...ontinuous_integration_GitHub_actions.ipynb | 137 +++++++++++ doc/guide/github-actions.rst | 29 --- doc/index.rst | 6 +- 10 files changed, 163 insertions(+), 349 deletions(-) rename doc/guide/{Guide_Miscellaneous.ipynb => Guide_CLIMADA_conventions.ipynb} (95%) rename doc/guide/{Guide_Continuous_Integration_and_Testing.ipynb => Guide_Testing.ipynb} (68%) create mode 100644 doc/guide/Guide_continuous_integration_GitHub_actions.ipynb delete mode 100644 doc/guide/github-actions.rst diff --git a/doc/guide/Guide_Miscellaneous.ipynb b/doc/guide/Guide_CLIMADA_conventions.ipynb similarity index 95% rename from doc/guide/Guide_Miscellaneous.ipynb rename to doc/guide/Guide_CLIMADA_conventions.ipynb index c408c4b50c..41d24d8621 100644 --- a/doc/guide/Guide_Miscellaneous.ipynb +++ b/doc/guide/Guide_CLIMADA_conventions.ipynb @@ -8,28 +8,7 @@ } }, "source": [ - "# Miscellaneous CLIMADA conventions" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - }, - "tags": [], - "toc": true - }, - "source": [ - "## Contents\n", - "\n", - "- [Dependencies (python packages)](#Dependencies-(python-packages))\n", - "- [Class inheritance](#Class-inheritance)\n", - "- [Paper repository](#Paper-repository)\n", - "- [Utility function](#Utility-function)\n", - "- [Impact function renaming - if to impf](#Impact-function-renaming---if-to-impf)\n", - "- [Data dependencies](#Data-dependencies)\n", - "- [Side Note on Parameters](#Side-Note-on-Parameters)" + "# CLIMADA coding conventions" ] }, { diff --git a/doc/guide/Guide_Configuration.ipynb b/doc/guide/Guide_Configuration.ipynb index 76b7a05501..a8141623ad 100644 --- a/doc/guide/Guide_Configuration.ipynb +++ b/doc/guide/Guide_Configuration.ipynb @@ -7,23 +7,6 @@ "# Constants and Configuration" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Content\n", - "\n", - "- [Constants](#Constants)\n", - " - [Hard Coded](#Hard-Coded)\n", - " - [Configurable](#Configurable)\n", - " - [Where to put constants?](#Where-to-put-constants?)\n", - "- [Configuration](#Configurable)\n", - " - [Configuration files](#Configuration-files)\n", - " - [Accessing configuration values](#Accessing-configuration-values)\n", - " - [Default Configuration](#Default-Configuration)\n", - " - [Test Configuration](#Test-Configuration)\n" - ] - }, { "cell_type": "markdown", "metadata": { diff --git a/doc/guide/Guide_Euler.ipynb b/doc/guide/Guide_Euler.ipynb index ee6ddccd94..3265123ce2 100644 --- a/doc/guide/Guide_Euler.ipynb +++ b/doc/guide/Guide_Euler.ipynb @@ -7,22 +7,6 @@ "# Using Climada on the Euler Cluster (ETH internal)" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## Content\n", - "\n", - "- [Access to Euler](#Access-to-Euler)\n", - "- [Installation and working directories](#Installation-and-working-directories)\n", - "- [Pre-installed version of Climada](#Pre-installed-version-of-Climada)\n", - "- [Working with Git branches](#Working-with-Git-branches)\n", - "- [Fallback: Conda installation](#Fallback:-Conda)\n", - "- [Run a Jupyter Notebook on Euler](#Run-Jupyter-Notebook-on-Euler)\n", - "- [Trouble shooting](#Trouble-shooting)" - ] - }, { "cell_type": "markdown", "metadata": {}, diff --git a/doc/guide/Guide_Git_Development.ipynb b/doc/guide/Guide_Git_Development.ipynb index 614ff3949b..f5f38fe656 100644 --- a/doc/guide/Guide_Git_Development.ipynb +++ b/doc/guide/Guide_Git_Development.ipynb @@ -12,26 +12,6 @@ "Chris Fairless" ] }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "## Contents\n", - "\n", - "- [Git and GitHub](#Git-and-GitHub)\n", - "- [Gitflow](#Gitflow)\n", - "- [Installing CLIMADA for development](#Installing-CLIMADA-for-development)\n", - "- [Does it belong in CLIMADA?](#Does-it-belong-in-CLIMADA?)\n", - "- [Features and branches](#Features-and-branches)\n", - "- [Pull requests](#Pull-requests) \n", - "- [General tips and tricks](#General-tips-and-tricks)\n", - " " - ] - }, { "cell_type": "markdown", "metadata": { diff --git a/doc/guide/Guide_Py_Performance.ipynb b/doc/guide/Guide_Py_Performance.ipynb index 6581c45bee..9b9a477eb9 100644 --- a/doc/guide/Guide_Py_Performance.ipynb +++ b/doc/guide/Guide_Py_Performance.ipynb @@ -20,24 +20,6 @@ "⚠️ **Don't over-optimize** at the expense of readability and usability." ] }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "skip" - }, - "toc": true - }, - "source": [ - "## Contents \n", - "\n", - "- [Profiling](#Profiling)\n", - "- [General considerations](#General-considerations)\n", - "- [NumPy-realted tips and best practice](#NumPy-related-tips-and-best-practice)\n", - "- [Miscellaneous](#Miscellaneous)\n", - "- [Take-home message](#Take-home-messages)" - ] - }, { "cell_type": "markdown", "metadata": { diff --git a/doc/guide/Guide_PythonDos-n-Donts.ipynb b/doc/guide/Guide_PythonDos-n-Donts.ipynb index 99c0f49716..15d90ae83b 100644 --- a/doc/guide/Guide_PythonDos-n-Donts.ipynb +++ b/doc/guide/Guide_PythonDos-n-Donts.ipynb @@ -11,25 +11,6 @@ "# Coding in Python: Dos and Don’ts" ] }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "## Content\n", - "\n", - "- [To Code or Not to Code?](#To-Code-or-Not-to-Code?)\n", - "- [Clean Code](#Clean-Code) \n", - "- [Commenting & Documenting](#Commenting-&-Documenting) \n", - "- [Exception Handling and Logging](#Exception-Handling-and-Logging)\n", - "- [Importing](#Importing)\n", - "- [How to structure a method or function](#How-to-structure-a-method-or-function)\n", - "- [Debugging](#Debugging)" - ] - }, { "cell_type": "markdown", "metadata": { diff --git a/doc/guide/Guide_Continuous_Integration_and_Testing.ipynb b/doc/guide/Guide_Testing.ipynb similarity index 68% rename from doc/guide/Guide_Continuous_Integration_and_Testing.ipynb rename to doc/guide/Guide_Testing.ipynb index ce1800d507..6475baacff 100644 --- a/doc/guide/Guide_Continuous_Integration_and_Testing.ipynb +++ b/doc/guide/Guide_Testing.ipynb @@ -5,56 +5,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Testing and Continuous Integration" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## Content\n", - "\n", - "- [Testing CLIMADA](#Testing-CLIMADA)\n", - "- [Notes on Testing](#Notes-on-Testing)\n", - "- [Continuous Integration](#Continuous-Integration)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Testing CLIMADA\n", - "\n", - "Executing the entire test suite requires you to install the additional requirements for testing.\n", - "See the [installation instructions](install.rst) for [developer dependencies](install-dev) for further information.\n", - "\n", - "\n", - "### Installation Test\n", - "\n", - "From the installation directory run\n", - "```\n", - "make install_test\n", - "```\n", - "It lasts about 45 seconds. If it succeeds, CLIMADA is properly installed and ready to use.\n", - "\n", - "### Unit Tests\n", - "\n", - "From the installation directory run\n", - "```\n", - "make unit_test\n", - "```\n", - "It lasts about 5 minutes and runs unit tests for all modules.\n", - "\n", - "### Integration Tests\n", - "\n", - "From the installation directory run\n", - "```\n", - "make integ_test\n", - "```\n", - "It lasts about 15 minutes and runs extensive integration tests, during which also data from external resources is read. An open internet connection is required for a successful test run." + "# Testing" ] }, { @@ -274,172 +225,38 @@ ] }, { - "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ + "## Testing CLIMADA\n", "\n", - "## Continuous Integration\n", - "\n", - "The CLIMADA Jenkins server used for continuous integration is at [(https://ied-wcr-jenkins.ethz.ch) ](https://ied-wcr-jenkins.ethz.ch)." - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Automated Tests\n", - "\n", - "On Jenkins tests are executed and analyzed automatically, in an unbiased environment. The results are stored and can be compared with previous test runs.\\\n", - "Jenkins has a GUI for monitoring individual tests, full test runs and test result trands.\\\n", - "Developers are requested to watch it. At first when they push commits to the code repository, but also later on, when other changes in data or sources may make it necessary to review and refactor code that once passed all tests.\n", - "\n", - "##### Developer guidelines:\n", - "\n", - "- All tests must pass before submitting a pull request.\n", - "- Integration tests don't run on feature branches in Jenkins, therefore developers are requested to run them locally.\n", - "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed.\n", - "\n", - "#### GitHub Actions\n", - "\n", - "We adopted test automation via GitHub Actions in an experimental state.\n", - "See [GitHub Actions CI](github-actions.rst) for details." - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Test Coverage\n", - "\n", - "Jenkins also has an interface for exploring code coverage analysis result.\\\n", - "This shows which part of the code has never been run in any test, by module, by function/method and even by single line of code.\n", - "\n", - "__Ultimately every single line of code should be tested.__\n", - "\n", - "##### Developer guidelines:\n", - "\n", - "- Make sure the coverage of novel code is at 100% before submitting a pull request.\n", - "\n", - "Be aware that the having a code coverage alone does not grant that all required tests have been written!\\\n", - "The following artificial example would have a 100% coverage and still obviously misses a test for `y(False)`" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - ".." - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "been here\n", - "been there\n", - "been everywhere\n", - "been here\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "----------------------------------------------------------------------\n", - "Ran 2 tests in 0.003s\n", - "\n", - "OK\n" - ] - } - ], - "source": [ - "def x(b:bool):\n", - " if b:\n", - " print('been here')\n", - " return 4\n", - " else:\n", - " print('been there')\n", - " return 0\n", - "\n", - "def y(b:bool):\n", - " print('been everywhere')\n", - " return 1/x(b)\n", - "\n", - "\n", - "import unittest\n", - "class TestXY(unittest.TestCase):\n", - " def test_x(self):\n", - " self.assertEqual(x(True), 4)\n", - " self.assertEqual(x(False), 0)\n", + "Executing the entire test suite requires you to install the additional requirements for testing.\n", + "See the [installation instructions](install.rst) for [developer dependencies](install-dev) for further information.\n", "\n", - " def test_y(self):\n", - " self.assertEqual(y(True), 0.25)\n", "\n", - "unittest.TextTestRunner().run(unittest.TestLoader().loadTestsFromTestCase(TestXY));" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Static Code Analysis\n", - "\n", - "At last Jenkins provides an elaborate GUI for pylint findings which is especially useful when working in feature branches.\n", + "### Installation Test\n", "\n", - "_Observe it!_\n", + "From the installation directory run\n", + "```\n", + "make install_test\n", + "```\n", + "It lasts about 45 seconds. If it succeeds, CLIMADA is properly installed and ready to use.\n", "\n", - "##### Developer guidelines:\n", + "### Unit Tests\n", "\n", - "- _High Priority Warnings_ are as severe as test failures and must be addressed at once.\n", - "- Do not introduce new _Medium Priority Warnings_.\n", - "- Try to avoid introducing _Low Priority Warnings_, in any case their total number should not increase." - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ + "From the installation directory run\n", + "```\n", + "make unit_test\n", + "```\n", + "It lasts about 5 minutes and runs unit tests for all modules.\n", "\n", - "### Jenkins Projects Overview\n", + "### Integration Tests\n", "\n", - "* #### [climada_install_env](https://ied-wcr-jenkins.ethz.ch/job/climada_install_env/)\n", - " Branch: __develop__ \\\n", - " Runs every day at 1:30AM CET\n", - " - creates conda environment from scratch\n", - " - runs core functionality system test (`make install_test`)\n", - " \n", - "* #### [climada_ci_night](https://ied-wcr-jenkins.ethz.ch/job/climada_ci_night/)\n", - " Branch: __develop__ \\\n", - " Runs when climada_install_env has finished successfully\n", - " - runs all test modules\n", - " - runs static code analysis\n", - "\n", - "* #### [climada_branches](https://ied-wcr-jenkins.ethz.ch/job/climada_branches/)\n", - " Branch: __any__ \\\n", - " Runs when a commit is pushed to the repository\n", - " - runs all test modules _outside of climada.test_\n", - " - runs static code analysis\n", - "\n", - "* #### [climada_data_api](https://ied-wcr-jenkins.ethz.ch/job/climada_data_api/)\n", - " Branch: __develop__ \\\n", - " Runs every day at 0:20AM CET\n", - " - tests availability of external data APIs\n", - " \n", - "* #### [climada_data_api](https://ied-wcr-jenkins.ethz.ch/job/climada_data_api/)\n", - " Branch: __develop__ \\\n", - " No automated running\n", - " - tests executability of CLIMADA tutorial notebooks." + "From the installation directory run\n", + "```\n", + "make integ_test\n", + "```\n", + "It lasts about 15 minutes and runs extensive integration tests, during which also data from external resources is read. An open internet connection is required for a successful test run." ] } ], diff --git a/doc/guide/Guide_continuous_integration_GitHub_actions.ipynb b/doc/guide/Guide_continuous_integration_GitHub_actions.ipynb new file mode 100644 index 0000000000..8e7f08bad3 --- /dev/null +++ b/doc/guide/Guide_continuous_integration_GitHub_actions.ipynb @@ -0,0 +1,137 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Continuous Integration and GitHub Actions" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Automated Tests\n", + "\n", + "On Jenkins tests are executed and analyzed automatically, in an unbiased environment. The results are stored and can be compared with previous test runs.\\\n", + "Jenkins has a GUI for monitoring individual tests, full test runs and test result trands.\\\n", + "Developers are requested to watch it. At first when they push commits to the code repository, but also later on, when other changes in data or sources may make it necessary to review and refactor code that once passed all tests.\n", + "The CLIMADA Jenkins server used for continuous integration is at [(https://ied-wcr-jenkins.ethz.ch) ](https://ied-wcr-jenkins.ethz.ch).\n", + "\n", + "##### Developer guidelines:\n", + "\n", + "- All tests must pass before submitting a pull request.\n", + "- Integration tests don't run on feature branches in Jenkins, therefore developers are requested to run them locally.\n", + "- After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Test Coverage\n", + "\n", + "Jenkins also has an interface for exploring code coverage analysis result.\\\n", + "This shows which part of the code has never been run in any test, by module, by function/method and even by single line of code.\n", + "\n", + "__Ultimately every single line of code should be tested.__\n", + "\n", + "##### Developer guidelines:\n", + "\n", + "- Make sure the coverage of novel code is at 100% before submitting a pull request.\n", + "\n", + "Be aware that the having a code coverage alone does not grant that all required tests have been written!\\\n", + "The following artificial example would have a 100% coverage and still obviously misses a test for `y(False)`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Static Code Analysis\n", + "\n", + "At last Jenkins provides an elaborate GUI for pylint findings which is especially useful when working in feature branches.\n", + "\n", + "_Observe it!_\n", + "\n", + "##### Developer guidelines:\n", + "\n", + "- _High Priority Warnings_ are as severe as test failures and must be addressed at once.\n", + "- Do not introduce new _Medium Priority Warnings_.\n", + "- Try to avoid introducing _Low Priority Warnings_, in any case their total number should not increase." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Jenkins Projects Overview\n", + "\n", + "* #### [climada_install_env](https://ied-wcr-jenkins.ethz.ch/job/climada_install_env/)\n", + " Branch: __develop__ \\\n", + " Runs every day at 1:30AM CET\n", + " - creates conda environment from scratch\n", + " - runs core functionality system test (`make install_test`)\n", + " \n", + "* #### [climada_ci_night](https://ied-wcr-jenkins.ethz.ch/job/climada_ci_night/)\n", + " Branch: __develop__ \\\n", + " Runs when climada_install_env has finished successfully\n", + " - runs all test modules\n", + " - runs static code analysis\n", + "\n", + "* #### [climada_branches](https://ied-wcr-jenkins.ethz.ch/job/climada_branches/)\n", + " Branch: __any__ \\\n", + " Runs when a commit is pushed to the repository\n", + " - runs all test modules _outside of climada.test_\n", + " - runs static code analysis\n", + "\n", + "* #### [climada_data_api](https://ied-wcr-jenkins.ethz.ch/job/climada_data_api/)\n", + " Branch: __develop__ \\\n", + " Runs every day at 0:20AM CET\n", + " - tests availability of external data APIs\n", + " \n", + "* #### [climada_data_api](https://ied-wcr-jenkins.ethz.ch/job/climada_data_api/)\n", + " Branch: __develop__ \\\n", + " No automated running\n", + " - tests executability of CLIMADA tutorial notebooks." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### GitHub Actions\n", + "\n", + "CLIMADA has been using a private Jenkins instance for automated testing (Continuous Integration, CI).\n", + "We recently adopted [GitHub Actions](https://docs.github.com/en/actions) for automated unit testing.\n", + "GitHub Actions is a service provided by GitHub, which lets you configure CI/CD pipelines based on YAML configuration files.\n", + "GitHub provides servers which ample computational resources to create software environments, install software, test it, and deploy it.\n", + "See the [GitHub Actions Overview](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) for a technical introduction, and the [Workflow Syntax](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions) for a reference of the pipeline definitions.\n", + "\n", + "The CI results for each pull request can be inspected in the \"Checks\" tab.\n", + "For GitHub Actions, users can inspect the logs of every step for every job.\n", + "\n", + "##### Note\n", + "As of CLIMADA v4.0, the default CI technology remains Jenkins.\n", + "GitHub Actions CI is currently considered experimental for CLIMADA development.\n", + "\n", + "##### Unit Testing Guideline\n", + "This pipeline is defined by the ``.github/workflows/ci.yml`` file.\n", + "It contains a single job which will create a CLIMADA environment with Mamba for multiple Python versions, install CLIMADA, run the unit tests, and report the test coverage as well as the simplified test results.\n", + "The job has a [strategy](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategy) which runs it for multiple times for different Python versions.\n", + "This way, we make sure that CLIMADA is compatible with all currently supported versions of Python.\n", + "\n", + "The coverage reports in HTML format will be uploaded as job artifacts and can be downloaded as ZIP files.\n", + "The test results are simple testing summaries that will appear as individual checks/jobs after the respective job completed." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/doc/guide/github-actions.rst b/doc/guide/github-actions.rst deleted file mode 100644 index efaddc2766..0000000000 --- a/doc/guide/github-actions.rst +++ /dev/null @@ -1,29 +0,0 @@ -================= -GitHub Actions CI -================= - -CLIMADA has been using a private Jenkins instance for automated testing (Continuous Integration, CI), see :doc:`Guide_Continuous_Integration_and_Testing`. -We recently adopted `GitHub Actions `_ for automated unit testing. -GitHub Actions is a service provided by GitHub, which lets you configure CI/CD pipelines based on YAML configuration files. -GitHub provides servers which ample computational resources to create software environments, install software, test it, and deploy it. -See the `GitHub Actions Overview `_ for a technical introduction, and the `Workflow Syntax `_ for a reference of the pipeline definitions. - -The CI results for each pull request can be inspected in the "Checks" tab. -For GitHub Actions, users can inspect the logs of every step for every job. - -.. note:: - - As of CLIMADA v4.0, the default CI technology remains Jenkins. - GitHub Actions CI is currently considered experimental for CLIMADA development. - ---------------------- -Unit Testing Pipeline ---------------------- - -This pipeline is defined by the ``.github/workflows/ci.yml`` file. -It contains a single job which will create a CLIMADA environment with Mamba for multiple Python versions, install CLIMADA, run the unit tests, and report the test coverage as well as the simplified test results. -The job has a `strategy `_ which runs it for multiple times for different Python versions. -This way, we make sure that CLIMADA is compatible with all currently supported versions of Python. - -The coverage reports in HTML format will be uploaded as job artifacts and can be downloaded as ZIP files. -The test results are simple testing summaries that will appear as individual checks/jobs after the respective job completed. diff --git a/doc/index.rst b/doc/index.rst index 2ff9c9bc89..4bc892165f 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -97,14 +97,14 @@ Jump right in: Development with Git guide/Guide_CLIMADA_Tutorial guide/Guide_Configuration - guide/Guide_Continuous_Integration_and_Testing + guide/Guide_Testing + guide/Guide_continuous_integration_GitHub_actions guide/Guide_Review guide/Guide_PythonDos-n-Donts guide/Exception_Logging Performance and Best Practices - Coding Conventions + CLIMADA Coding Conventions Building the Documentation - guide/github-actions .. toctree::