add documentation

README.md

@@ -1,19 +1,24 @@
 ![banner](images/banner.png)
 
-**[Trawl Watch](https://twitter.com/TrawlWatch)** is an initiative launched by the **[Bloom Association](https://www.bloomassociation.org/en/)** to track and expose the most destructive fishing vessels. Inspired by L’[Avion de Bernard](https://www.instagram.com/laviondebernard/), which monitors the movements of private jets, Trawl Watch aims to make visible the impact of these massive trawlers on our oceans. These vessels, often referred to as mégachalutiers, deploy gigantic nets that can engulf marine life from the surface down to the ocean floor. The consequences are both ecological—as they devastate crucial nursery and breeding areas for marine animals—and social, as they deprive artisanal fishermen of a healthy marine ecosystem. The solution proposed by Bloom is to dismantle these industrial fishing ships and redistribute their quotas to small-scale fishers. A petition has been launched, and Bloom continues to track these megatrawlers while awaiting action from European institutions12.
+## What is Trawl Watch
+
+**[Trawl Watch](https://twitter.com/TrawlWatch)** is an initiative launched by the **[Bloom Association](https://www.bloomassociation.org/en/)** to track and expose the most destructive fishing vessels. Inspired by L’[Avion de Bernard](https://www.instagram.com/laviondebernard/), which monitors the movements of private jets, **Trawl Watch** aims to make visible the impact of these massive trawlers on our oceans. These vessels, often referred to as _mégachalutiers_, deploy gigantic nets that can engulf marine life from the surface down to the ocean floor. The consequences are both ecological—as they devastate crucial nursery and breeding areas for marine animals—and social, as they deprive artisanal fishermen of a healthy marine ecosystem. The solution proposed by **Bloom** is to dismantle these industrial fishing ships and redistribute their quotas to small-scale fishers. A petition has been launched, and **Bloom** continues to track these megatrawlers while awaiting action from European institutions.
 
 **Did you know that, in Europe, the largest fishing vessels, which represent 1% of the fleet, catch half of the fish?** These factory-vessels can measure up to 144 meters in length and catch 400,000 kilos of fish per day! This is as much as 1,000 small-scale fishing vessels in one day at sea.
 
 **These veritable sea monsters are devastating Europe’s biodiversity and coastlines.** It is important to measure the scale of the damage: about 20 of these factory-vessels can obliterate hundreds of thousands of marine animals and biodiversity treasures in one day, including in the so-called ‘Marine Protected Areas’ of French territorial waters, which are not protected at all.
 
-In addition, more and more ‘mega trawlers’ – vessels over 25 meters in length – come right up to the coast to fish, so close to the beaches that they can be observed from the shore. These industrial monsters were not designed to fish in coastal waters, where small artisanal fishers operate.
+## What is Bloom Association
+
+**BLOOM** is a non-profit organization founded in 2005 that works to preserve the marine environment and species from unnecessary destruction and to increase social benefits in the fishing sector. **BLOOM** wages awareness and advocacy campaigns in order to accelerate the adoption of concrete solutions for the ocean, humans and the climate. **BLOOM** carries out scientific research projects, independent studies and evaluations that highlight crucial and unaddressed issues such as the financing mechanisms of the fishing sector. **BLOOM**’s actions are meant for the general public as well as policy-makers and economic stakeholders.
 
-## Installing a Trawl Watch with `venv` and `poetry`
+## Installing Trawl Watch with `poetry`
 
 ### Prerequisites:
 
 1. Python (≥ `3.10`) installed on your system.
-2. Ensure you have `venv` and [`poetry`](https://python-poetry.org/) installed. If not, you can install them using `pip`.
+2. Ensure [Docker](https://docs.docker.com/get-docker/) is installed.
+3. Ensure you have `poetry` installed. If not, you can install them using `pip`.
 
 ```bash
 pip install poetry
@@ -26,42 +31,26 @@ pip install poetry
    Clone the GitHub repository you want to install locally using the `git clone` command.
 
    ```bash
-   $ git clone https://github.com/dataforgoodfr/12_bloom.git
+   git clone https://github.com/dataforgoodfr/12_bloom.git
    ```
 
 2. **Navigate to the Repository Directory:**
 
    Use the `cd` command to navigate into the repository directory.
 
    ```bash
-   $ cd 12_bloom/
+   cd 12_bloom/
    ```
 
-3. **Set Up a Virtual Environment using `venv`:**
-
-Create a virtual environment using `venv` to isolate the dependencies for the project.
-
-```bash
-$ python -m venv venv
-```
-
-4. **Activate the Virtual Environment:**
+3. **Configure `poetry` to create a Virtual Environment inside the project:**
 
-   Activate the virtual environment to work within its isolated environment.
-
-   On Windows:
+   Ensure that poetry will create a `.venv` directory into the project with the command:
 
    ```bash
-   venv\Scripts\activate
+   poetry config virtualenvs.in-project true
    ```
 
-   On Unix or MacOS:
-
-   ```bash
-   source venv/bin/activate
-   ```
-
-5. **Install Project Dependencies using `poetry`:**
+4. **Install Project Dependencies using `poetry`:**
 
    Use `poetry` to install the project dependencies.
 
@@ -71,86 +60,65 @@ $ python -m venv venv
 
    This will read the `pyproject.toml` file in the repository and install all the dependencies specified.
 
-6. **(Optional) Install Development Dependencies:**
-
-   If there are separate dependencies for development, you can install them using:
+5. **Make sure everything is all right using `poetry env info`:**
 
    ```bash
-   poetry install --dev
+   poetry env info
    ```
 
-7. **Run the Project:**
-
-   After installing the dependencies, you can run the project using the appropriate commands specified in the repository's documentation or README file.
+   It should looks something likes:
 
    ```bash
-   poetry run <command_to_run_project>
+   Virtualenv
+   Python:         3.11.2
+   Implementation: CPython
+   Path:           /home/guillaume/12_bloom/.venv
+   Executable:     /home/guillaume/12_bloom/.venv/bin/python
+   Valid:          True
+
+   System
+   Platform:   linux
+   OS:         posix
+   Python:     3.11.2
+   Path:       /usr
+   Executable: /usr/bin/python3.11
    ```
 
-8. **Deactivate the Virtual Environment:**
+6. **Activate the Virtual Environment:**
+
+   Activate the virtual environment to work within its isolated environment.
 
-   Once you're done working with the project, deactivate the virtual environment.
+   On Unix or MacOS:
 
    ```bash
-   deactivate
+   poetry shell
    ```
 
-#### Additional Notes:
+7.
 
-- Ensure you are using the correct Python version specified by the repository, especially if it's mentioned in the `pyproject.toml` file.
-- Always refer to the repository's documentation or README file for any specific instructions or configurations required for setting up and running the project.
+### Once you're done working with the project, deactivate the virtual environment.
 
-This documentation provides a general guideline for setting up a project from a GitHub repository using `venv` and `poetry`. Adjustments may be needed based on the specific requirements and configurations of the repository you are working with.
-
-### Requirements
-
-### FAQ
-
-Suivre les trajectoires de milliers de bateaux de pêche en quasi temps réel afin de pouvoir analyser leurs pratiques de pêche dans des zones maritimes protégées (AMP) à partir de données GPS récupérées (via antennes satellites et/ou terrestres)
-
-- https://www.un.org/sustainabledevelopment/economic-growth/
-- https://www.un.org/sustainabledevelopment/climate-change/
-- https://www.un.org/sustainabledevelopment/oceans/
-
-- https://petitions.bloomassociation.org/en/stop-mega-trawlers/
-
-## Setting up your local development environment
-
-### FAQ
-
-## bloom data source
-
-## [Automatic identification system](https://en.wikipedia.org/wiki/Automatic_identification_system)
-
-limits:
-
-- out of range
-- intense traffic area
-- expensive
-
-- https://spire.com/maritime/
-
-## [Marine Protected Areas]
-
-- https://www.cell.com/one-earth/fulltext/S2590-3322(20)30150-0
+```bash
+deactivate
+```
 
-## [Shom](https://www.shom.fr/)
+## Documentation
 
-pour visualiser les droits miles nautique et les droits historiques
+[Dendron](https://marketplace.visualstudio.com/items?itemName=dendron.dendron) is a powerful Visual Studio Code, or [VSCodium](https://vscodium.com/), extension designed to streamline and enhance the documentation process. With **Dendron**, documenting projects becomes intuitive and efficient, thanks to its hierarchical note-taking system. Users can organize their documentation into a tree-like structure, making it easy to navigate and manage. The extension offers robust features such as bidirectional linking, which allows for seamless navigation between related notes, and support for Markdown formatting, enabling users to create rich and visually appealing documentation. Additionally, **Dendron** provides powerful search functionality, enabling users to quickly locate specific information within their documentation vault. Overall, **Dendron** empowers developers, writers, and teams to create comprehensive and well-organized documentation, facilitating better knowledge management and collaboration. The documentation is locaed inside the `./docs/notes` directory.
 
-# Bloom Project
+Here's some basic Dendron shortcuts:
 
-This project is maintained by D4G in order to gather vessels data.
-A cron job is launched every 15min, does calls to API and save the data in a Postgresql database.
+| Shortcut       | Description  |
+| -------------- | ------------ |
+| `Cmd/Ctrl + L` | Lookup notes |
 
-## About directory architecture
+## More information can be found there
 
-The domain directory ...
-The infra directory ...
+1. [Database initialisation and versioning](./docs/database.initialisation.md)
+2. [Development environment](./docs/development.environment.md)
+3. [Useful SQL examples](./docs/sql.examples.md)
+4. [Data models](#todo)
 
-More information can be found there :
+## FAQ
 
-1. [Database initialisation and versioning](./documentation/database_init.md)
-2. [Development environment](./documentation/dev_env.md)
-3. [Useful SQL examples](./documentation/sql_examples.md)
-4. [Data models](#todo)
+#todo

docs/notes/ais.md

@@ -0,0 +1,21 @@
+---
+id: g1i0o9ctmib83n4mydtsqfo
+title: Automatic identification system
+desc: ""
+updated: 1708461719224
+created: 1708461557185
+---
+
+The automatic identification system (AIS) is an automatic tracking system that uses transceivers on ships and is used by vessel traffic services (VTS). When satellites are used to receive AIS signatures, the term Satellite-AIS (S-AIS) is used. AIS information supplements marine radar, which continues to be the primary method of collision avoidance for water transport. Although technically and operationally distinct, the ADS-B system is analogous to AIS and performs a similar function for aircraft.[^1]
+
+## Limits
+
+- out of range
+- intense traffic area
+- expensive
+
+## Data source
+
+https://spire.com/maritime/
+
+[^1]: https://en.wikipedia.org/wiki/Automatic_identification_system

documentation/database_init.md → docs/notes/database.initialisation.md

@@ -1,16 +1,18 @@
-# Database initialisation and versioning
-
-
-## Database initialisation
+---
+id: 8vo1vo09y37hwnrkmcy7w77
+title: Database initialisation
+desc: ""
+updated: 1708410470701
+created: 1708410417004
+---
 
 First you need to run scripts which are in alembic/init_script :
+
 - the load_vessels_data.py script will load vessels metadata from the data/chalutier_pelagique.csv file.
 - the load_geometry_data.py file will load shape data from the Nonterrestrial_WDPA_Jan2023.shp file. This file is not included in this github project but you can ask for it. It's only used for the alerting part.
 
 The second step is to load the [distance-from-port-v20201104.tiff](https://globalfishingwatch.org/data-download/datasets/public-distance-from-port-v1) and [distance-from-shore.tif](https://globalfishingwatch.org/data-download/datasets/public-distance-from-shore-v1) files. They are only used for the alerting part.
+
 - install psql and raster2pgsql.
 - install raster type in db with postgis-raster using `create extension postgis_raster`
-- adapt this command for each file :  `raster2pgsql -t auto -I -C -M /PATH_TO/distance-from-shore.tif public.distance_shore | PGPASSWORD='POSTGRES_PASSWORD' psql -h POSTGRES_HOSTNAME -d POSTGRES_DB -U POSTGRES_USER -p POSTGRES_PORT`
-
-## Database versioning
-The command ` alembic upgrade head` can be used in the root of the project in order to update the database schema to the last version.
+- adapt this command for each file : `raster2pgsql -t auto -I -C -M /PATH_TO/distance-from-shore.tif public.distance_shore | PGPASSWORD='POSTGRES_PASSWORD' psql -h POSTGRES_HOSTNAME -d POSTGRES_DB -U POSTGRES_USER -p POSTGRES_PORT`

docs/notes/database.versioning.md

@@ -0,0 +1,9 @@
+---
+id: kofmnl4h6i8e9uacvtd2wkr
+title: Database versioning
+desc: ""
+updated: 1708410507271
+created: 1708410490074
+---
+
+The command ` alembic upgrade head` can be used in the root of the project in order to update the database schema to the last version.

documentation/dev_env.md → docs/notes/development.environment.md

@@ -1,32 +1,39 @@
-# Development environment
+---
+id: 6gumcsjv53j1xsyandigv97
+title: Development environment
+desc: ""
+updated: 1708410800063
+created: 1708410548545
+---
 
 You can develop locally using Poetry and your own database but you can also use the Makefile to :
-1) launch a local postgresql dockerized
-2) launch a python environment dockerized
 
+1. launch a local postgresql dockerized
+2. launch a python environment dockerized
 
 If you work with Mac m1, the containers may not work as expected
 
-## Local Database 
+## Local Database
 
 You will need to source the .env.template file but before you should modify it according to your local configuration, for example using POSTGRES_HOSTNAME=localhost
 
 ## Development Database
+
 First, you need to create an .env.test file, you can use the .env.template file :
 ` cp .env.template .env.test`
 
 Next, you have to set the SPIRE_TOKEN variable. You can ask for it.
 
 Launch the following command :
-` make launch-dev-db `
+`make launch-dev-db`
 
 Now, a postgresql db is available in your localhost, port 5480. And a pgadmin is available port 5080
 
 You can remove it thanks to this command:
 ` make rm-db`
 
 TIPS : you can use the following command to launch the psql client :
-` docker exec -ti postgres_bloom psql -d bloom_db -U bloom_user `
+`docker exec -ti postgres_bloom psql -d bloom_db -U bloom_user`
 
 ## Development environment
 
@@ -44,11 +51,9 @@ To delete the container:
 A second option is to launch directly the app.py command thanks to this command : (the container is automatically removed after)
 ` make launch-app`
 
-
 ## tests & precommit hook
+
 Please install the [precommit hook](https://pre-commit.com/) tool locally to avoid any issue with the CI/CD.
 
 You may also want to launch tests :
 ` tox -vv`
-..
-

docs/notes/development.poetry.md

@@ -0,0 +1,9 @@
+---
+id: iuuwulp9e75lg0jd18l0wyp
+title: Poetry
+desc: ''
+updated: 1708410825991
+created: 1708410825991
+---
+
+[Poetry](https://python-poetry.org/)

docs/notes/mpa.md

@@ -0,0 +1,9 @@
+---
+id: t40gr1j4y9jqm2ba4yh7xuf
+title: Marine Protected Areas
+desc: ""
+updated: 1708461758182
+created: 1708461744075
+---
+
+- https://www.cell.com/one-earth/fulltext/S2590-3322(20)30150-0

docs/notes/shom.md

@@ -0,0 +1,11 @@
+---
+id: dxdppizit5jtxu6cuwl6kz6
+title: Shom
+desc: ""
+updated: 1708461781678
+created: 1708461768767
+---
+
+- https://www.shom.fr/
+
+pour visualiser les droits miles nautique et les droits historiques

docs/notes/sql.examples.md

@@ -0,0 +1,31 @@
+---
+id: dgw0omk9y2u92olzvdxgddp
+title: Useful SQL examples
+desc: ""
+updated: 1708410750577
+created: 1708410619351
+---
+
+## List of mmi scrapped since last update
+
+```sql
+SELECT mmsi as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi
+```
+
+## More recent timestamp scrapped since last update
+
+```sql
+SELECT tm,array_agg(mmsi) FROM (SELECT mmsi,max(timestamp) as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi) as foo GROUP BY tm
+```
+
+## mmsi not scrapped by Spire
+
+```sql
+SELECT mmsi FROM vessels WHERE mmsi IS NOT NULL AND mmsi NOT IN (SELECT mmsi as tm FROM spire_vessel_positions WHERE timestamp > '2023-06-17' GROUP BY mmsi)
+```
+
+## number of distincte position per boat which has more than 1000 positions gathered
+
+```sql
+SELECT * FROM (SELECT COUNT(*) as sum_position,mmsi FROM (SELECT DISTINCT position,mmsi FROM spire_vessel_positions) as foo GROUP BY mmsi) as bar WHERE sum_position > 1000
+```