make prod and dev more similar, config more DRY (#78)

* make prod and dev more similar, config more DRY

* (+x on entrypoint)

* fix test (maybe?)

* fix test (maybe?)

* persist caddy data

* improvements as per review

* fix tests (hopefully ?)

* cleanup gitignore

* fix bad merge ?

* (add .editorconfig and fix line endings, thanks windows)

* remove browse directive in caddyfile

* Improve .env reading in orchestrator

* Improve silk intercept fuction

* Improve readme

* Ignore .env and add .env.example

* Update readme

Co-authored-by: Mario Baranzini <[email protected]>

.editorconfig

@@ -0,0 +1,11 @@
+root = true
+
+[*]
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+end_of_line = lf
+
+[*.py]
+indent_style = space
+indent_size = 4

.env.example

@@ -0,0 +1,13 @@
+QFIELDCLOUD_HOST=localhost
+
+SECRET_KEY=change_me
+
+AWS_ACCESS_KEY_ID=KEY
+AWS_SECRET_ACCESS_KEY=KEY
+AWS_STORAGE_BUCKET_NAME=BUCKET_NAME
+AWS_S3_REGION_NAME=ch-dk-2
+AWS_S3_ENDPOINT_URL=https://sos-ch-dk-2.exo.io
+
+POSTGRES_USER=qfieldcloud_db_admin
+POSTGRES_PASSWORD=3shJDd2r7Twwkehb
+POSTGRES_DB=qfieldcloud_db

.github/workflows/test.yml

@@ -22,33 +22,22 @@ jobs:
       - name: Check code with Flake8
         run: |
           cd orchestrator
-          pipenv run python -m flake8 --ignore=E501,F401,W503 ../django-app/qfieldcloud/
-      - name: Create .env.db file from secret
+          pipenv run python -m flake8 --ignore=E501,F401,W503 ../docker-app/qfieldcloud/
+      - name: Create .env file from secret
         uses: SpicyPizza/create-envfile@v1
         with:
+          envkey_DEBUG: 1
+          envkey_SECRET_KEY: ${{ secrets.SECRET_KEY }}
           envkey_POSTGRES_USER: "qfieldcloud_test_db_admin"
           envkey_POSTGRES_PASSWORD: ${{ secrets.SQL_PASSWORD }}
           envkey_POSTGRES_DB: "qfieldcloud_test_db"
-          file_name: conf/.env.db
-      - name: Create .env.app file from secret
-        uses: SpicyPizza/create-envfile@v1
-        with:
-          envkey_DEBUG: 1
-          envkey_SECRET_KEY: ${{ secrets.SECRET_KEY }}
           envkey_DJANGO_ALLOWED_HOSTS: "localhost 127.0.0.1 [::1]"
-          envkey_SQL_ENGINE: "django.db.backends.postgresql"
-          envkey_SQL_DATABASE: "qfieldcloud_test_db"
-          envkey_SQL_USER: "qfieldcloud_test_db_admin"
-          envkey_SQL_PASSWORD: ${{ secrets.SQL_PASSWORD }}
-          envkey_SQL_HOST: "db"
-          envkey_SQL_PORT: 5432
-          envkey_DATABASE: "postgres"
           envkey_AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           envkey_AWS_SECRET_ACCESS_KEY: ${{ secrets.SECRET_ACCESS_KEY }}
           envkey_AWS_STORAGE_BUCKET_NAME: "qfieldcloud-test"
           envkey_AWS_S3_REGION_NAME: "zrh1"
           envkey_AWS_S3_ENDPOINT_URL: "https://os.zrh1.flow.swiss"
-          file_name: conf/.env.app
+          file_name: .env
       - name: Build and run docker containers
         run: |
           docker-compose up -d --build

.gitignore

@@ -1,13 +1,7 @@
 __pycache__/
-/django-app/staticfiles/
-/django-app/user_projects_files/
-/user_projects_files/
 *.log
 .htmlcov/
 .coverage
-/conf/Caddyfile
-/conf/.env.app
-/conf/.env.db
 /conf/supervisord.conf
 .vscode
-/postgres_data/
+.env

README.org

@@ -18,29 +18,26 @@
    Permissions documentation is [[https://github.com/opengisch/qfieldcloud/blob/master/docs/permissions.org][here]]
 ** Development
 *** Launch a local instance
-    Copy the example configuration files and update them to your desire
+    Copy the =.env.example= into =.env= file and configure it to your
+    desire with a good editor
     #+begin_src sh
-      cd conf
-      cp .env.app.example .env.app
-      cp .env.db.example .env.db
-      emacs .env.app
-      emacs .env.db
+      cp .env.example .env
+      emacs .env
     #+end_src
-    For a local development instance you don't need =Caddyfile= and
+    For a local development instance you don't need to configure
     =supervisord.conf= configuration files.
 
     To build development images and run the containers:
     #+begin_src sh
       docker-compose up -d --build
     #+end_src
 
-    It will read =docker-compose.yml= and start a django built-in
-    server at =http://localhost:8000=
+    It will read =docker-compose.yml= and =docker-compose.override.yml=
+    and start a django built-in server at =http://localhost:8000=
 
-    To run django commands on the docker, enter in the containers bash
-    shell or place =docker-compose run app= before the command, e.g.:
+    Run the django database migrations
     #+begin_src sh
-      docker-compose exec app python manage.py ...
+      docker-compose exec app python manage.py migrate
     #+end_src
 
     To lauch the RQ (Redis Queue) workers that runs the QGIS containers jobs:
@@ -73,33 +70,40 @@
     - https://app.qfield.cloud/ This is the production instance. At
       the moment the deploy is done manually.
 
-    On the servers, the =docker-compose.prod.yml= is used. There are
-    no mounted folders. To apply changes, the docker image must be
-    re-built.
+    On the servers, we need only the =docker-compose.yml= and not the
+    "override" one. There are no mounted folders. To apply changes,
+    the docker image must be re-built.
 *** Launch a server instance
-    Copy the example configuration files and update them to your desire
+    Copy the =.env.example= into =.env= file and configure it to your
+    desire with a good editor
     #+begin_src sh
+      cp .env.example .env
+      emacs .env
+    #+end_src
+
+    Configure the =.env= file to your desire with a good editor
+    #+begin_src emacs-lisp
       cd conf
-      cp .env.app.example .env.app
-      cp .env.db.example .env.db
-      cp Caddyfile.example Caddyfile
       cp supervisord.conf.example supervisord.conf
-      emacs .env.app
-      emacs .env.db
-      emacs Caddyfile
       emacs supervisord.conf
     #+end_src
 
-    Run and build the docker containers
+    Run and build the docker containers (*not the
+    =docker-compose.override.yml= file*)
+    #+begin_src sh
+      docker-compose -f docker-compose.yml up -d --build
+    #+end_src
+
+    Run the django database migrations
     #+begin_src sh
-      docker-compose -f docker-compose.prod.yml up -d --build
+      docker-compose exec app python manage.py migrate
     #+end_src
 
     Prepare and run supervisord with the workers
     #+begin_src sh
       apt install python3-pip  # If needed
       pip3 install pipenv  # If needed
-      # If needed you should upgrade your python version on your host to 3.8.3 or superior
+      # You should upgrade your python version on your host to 3.8.3 or superior
       cd orchestrator
       pipenv --python /usr/bin/python3.8 install
       pipenv run supervisord -c ../conf/supervisord.conf

client/client.py

@@ -8,8 +8,8 @@
 from glob import glob
 from pathlib import Path
 
-BASE_URL = 'http://dev.qfield.cloud/api/v1/'
-# BASE_URL = 'http://localhost:8000/api/v1/'
+# BASE_URL = 'http://dev.qfield.cloud/api/v1/'
+BASE_URL = 'http://localhost/api/v1/'
 
 
 @click.group()