Merge pull request #79 from aremazeilles/docker_generation

Docker generation workflow

modules/ROOT/nav.adoc

@@ -1,5 +1,5 @@
 * xref:index.adoc[Welcome]
+* xref:template.adoc[Protocol template]
+* xref:pi_spec.adoc[Performance indicator]
 * xref:data_format.adoc[Input data format]
 * xref:model.adoc[Human model]
-* xref:pi_spec.adoc[Performance indicator]
-* xref:template.adoc[Protocol template]

modules/ROOT/pages/pi_spec.adoc

@@ -26,6 +26,7 @@ There exists:
 ** Scenario _Overcoming obstacles_: maximum height of an obstacle the subject can walk over, ...
 ** Scenario _Picking and carrying objects_: Weight of the object which can be picked up and carried, Walking speed while carrying an object, ...
 
+[[sec:pi_algo_general]]
 === A bit of context: the PI Manager
 
 A key component of the Benchmarking Software is the `PI Manager`.
@@ -80,6 +81,7 @@ That way, a future comparison of this experiment with another only consists in c
 
 The preparation of the Docker image containing the processing algorithm will be more extensively detailed later on.
 
+[[sec:pi_repo_structure]]
 == PI repository structure
 
 To reach that structure, a PI code repository must have the following structure:
@@ -245,3 +247,48 @@ WARNING: The scoring is _ideally_ performed **per run**.
          The score aggregation across the N runs (like how to extract an experiment `step_time` score given all the `step_time` vectors obtained in the successive runs) is specified in the template.
          See the <<template.adoc#inter_run_aggregation, inter-run aggregation definition>>.
          It will be automatically computed by the PI Manager
+
+
+## Code deployment methodology
+
+### Generalities
+
+As mentioned <<pi_algo_general, previously>>, the benchmarking process is handled by the `PI manager`, that is in charge of launching the computation of the metrics, given the data collected in an experimentation.
+
+Based on the protocol the experiment is associated to, the `PI Manager` can deduce the algorithm(s) to be executed.
+
+For encapsulation matters, it has been decided to deploy the processing algorithm using Docker.
+This way, the `PI manager only handle a set of docker images`, independently of the language used for implementing the metric.
+
+As described in <<sec:pi_repo_structure, PI Algo repository structure >>, each metric code repository contains a `Dockerfile` that defines how to generate the Docker image.
+This file is used in the CI process to generate automatically the Docker image.
+
+Now the `PI manager` needs to know where to find that image.
+We propose using a common environment for hosting the generated images, currently named the https://hub.docker.com/orgs/eurobenchtest[Eurobenchtest hub].
+
+NOTE: The Docker location associated to an algorithm is supposed to be indicated in the Protocol Definition sheet, in the page related to <<template.adoc#sec:template_pi_algo, PI Algo specification>>.
+
+### Docker hub update methodology
+
+We propose the following methodology:
+
+1. A PI code repository ready to be uploaded will be either migrated or forked within the https://github.com/eurobench[Eurobench Github organization].
+It is named Eurobench PI repository in the rest of the text.
+2. Even if public code is encouraged, the Eurobench PI repository may be set as private on explicit request of the code owner.
+3. The Eurobench PI repository will at least contain two branches:
+   * a development branch into which some developments may be included
+   * a release branch in which only validated evolution from the development branch can be inserted.
+4. The deployment of the docker image will be manually triggered by defining a tag, that should be associated to a commit on the release branch.
+5. The deployment to the docker hub is done then automatically through the CI/CD process.
+
+In the particular case that the code requires an update, and / or the code owner suggests a code evolution:
+
+1. The update request can be triggered through a merge request, or a direct communication with the Eurobench maintainers.
+2. It has to be shown that the code update goes through the regression tests associated to the code.
+3. The update is brought to the development branch and tested.
+4. Once validated, the commit can be merged to the release branch, and a tag is created to trigger the docker image deployment to the Eurobench Docker Hub.
+
+NOTE: The code evolution may require some discussion, as such update may also require an update of the experiments already scored with the previous version of the docker image.
+      It could be automatically checked whether the input/output data used for the regression testing has changed in between the previous and the current version.
+
+WARNING: Only the Eurobench team should be authorized to enable the deployment of an updated docker image (step 4).

modules/ROOT/pages/template.adoc

@@ -213,6 +213,7 @@ That score could be presented to the user on the front-end, to summarize the sco
 The inter-run aggregation is thus using the following pattern: `[ [number, operator] ]`.
 It has to be filled for all output format, even scalar (since we still have to aggregate the scalar value of each run).
 
+[[sec:template_pi_algo]]
 == PI Algo Sheet
 
 .Performance Indicator sheet
@@ -241,7 +242,7 @@ It has to be filled for all output format, even scalar (since we still have to a
 
 === Input command
 
-At this moment it contains the entry point and the parameters the entry point is expecting. 
+At this moment it contains the entry point and the parameters the entry point is expecting.
 The format for the string is the following one.
 
 ```