Merge pull request #501 from ARGOeu/devel

Version 3.4.5

CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## [3.4.5] - 2022-11-03
+
+### Added
+
+* ARGO-3971 Store default ports in POEM
+* ARGO-3980 Support bulk delete of service types
+* ARGO-3981 Introduce bulk add service types view
+* ARGO-3983 Add service types pagination
+* ARGO-4009 Enable Hot Module Replacement and Django internal web server
+* ARGO-4014 Define min and max width of name column in Service Types list
+
+### Changed
+
+* ARGO-3711 Reduce Metric model to what is strictly necessary
+* ARGO-3982 Remove service types sync and related DB tables
+* ARGO-4004 Ensure that only service type names without whitespace can created from the UI
+* ARGO-4008 Have red border around changed description field
+* ARGO-4012 One common Save button instead of one in every row
+* ARGO-4031 Switch API key page to react-hook-form library
+* ARGO-4032 Switch groups page to react-hook-form library
+* ARGO-4039 Switch package page to react-hook-form library
+* ARGO-4051 Ensure POEM wheel essential dependencies installed automatically
+
+### Fixed
+
+* ARGO-4005 Marking field tuple on filtered view ends with description fields populated from neighboring tuple
+* ARGO-4006 Case insensitive search on bulk delete and placeholder missing
+* ARGO-4028 Adding of new metric profile is broken
+* ARGO-4094 Metric parameter overrides not handling space in parameter value
+* ARGO-4095 Clean requirements.txt
+
 ## [3.4.4] - 2022-09-01
 
 ### Added

README.md

@@ -28,7 +28,7 @@ Web application is served with Apache web server and uses PostgreSQL as database
 
 Backend: `Django`, `dj-rest-auth`, `django-tenants`, `django-webpack-loader`, `djangorestframework`, `djangorestframework-api-key`, `djangosaml2`, `psycopg2-binary`
 
-Frontend: `ReactJS`, `formik`, `react-autosuggest`, `react-diff-viewer`, `react-dom`, `react-fontawesome`, `react-helmet`, `react-notifications`, `react-popup`, `react-router`, `react-select`, `react-table`, `reactstrap`, `webpack`, `yup`
+Frontend: `ReactJS`, `formik`, `react-hook-form`, `react-autosuggest`, `react-diff-viewer`, `react-dom`, `react-fontawesome`, `react-helmet`, `react-notifications`, `react-popup`, `react-router`, `react-select`, `react-table`, `reactstrap`, `webpack`, `yup`
 
 ## User documentation and instances
 
@@ -246,6 +246,9 @@ Part of the REST API is protected by token so for tenants that consume those API
     ReportsTopologyTags = https://api.devel.argo.grnet.gr/api/v2/topology/tags
     ReportsTopologyGroups = https://api.devel.argo.grnet.gr/api/v2/topology/groups
     ReportsTopologyEndpoints = https://api.devel.argo.grnet.gr/api/v2/topology/endpoints
+    ServiceTypes = https://api.devel.argo.grnet.gr/api/v2/topology/service-types
+    Metrics = https://api.devel.argo.grnet.gr/api/v4/admin/metrics
+
 
 This section lists WEB-API methods for the resources that are not stored in
 POEM's PostgreSQL DB, but instead are consumed from ARGO WEB-API services. POEM
@@ -435,13 +438,49 @@ Starting of multi-container application:
 docker/ $ docker-compose up
 ```
 
+### Web server
+
+In development enviroment, application can be served via Apache web server or internal Django web server coupled with Webpack's dev server for the Hot Module Reload functionality (HMR). Helper make target rules are provided in [poem/Poem/Makefile](poem/Poem/Makefile).
+
+#### Apache
+
+For the Apache web serving, bundle created by the Webpack need to be manually planted as Django's staticfile everytime bundle is recreated. Webpack can monitor the changes in the React's code and recreate the bundle on the fly.
+
+Start Webpack's watch mode:
+```
+make devel-watch
+```
+
+After changes are done and developer wants to see how they are reflected, he needs to place newly created bundle as Django staticfile and restart the Apache within container enviroment. For that purpose, make target rule is prepared:
+```
+make place-new-bundle
+```
+
+#### Django web server
+
+Advantage of using Django's web server is that backend code can be easily debugged as developer can use debugger and breakpoints and trace the execution of code. Django web server is automatically reloaded for every change of the backend code. Webpack's bundles are automatically placed as they are picked up from `webpack-dev-server` running at `localhost:3000`. Moreover, developer can use HMR functionality as `webpack-dev-server` is able to trigger browser reload for every change in the frontend code.
+
+Start Django web server at `0.0.0.0:8000`:
+```
+make devel-django-server
+```
+
+Start `webpack-dev-server` as `localhost:3000`:
+```
+make devel-webpack-server
+```
+Or use HMR:
+```
+make devel-webpack-server-hmr
+```
+
 ### Packaging
 
 Deployment of new versions is done with wheel packages that contain both backend Python and frontend Javascript code. Packages are build using setuptools and helper make target rules are provided in [Makefile](Makefile) and in [poem/Poem/Makefile](poem/Poem/Makefile). Latter is used to create a devel or production bundle of frontend Javascript code and place it as Django staticfiles, while the former is used to create Python wheel package.
 
 * frontend `Makefile` package targets:
-  - `make devel` - create a development Webpack bundle
-  - `make prod` - create a production Webpack bundle
+  - `make devel-bundle` - create a development Webpack bundle
+  - `make prod-bundle` - create a production Webpack bundle
   - `make place-new-bundle` - place created bundle as Django staticfile
 * backend `Makefile` package targets:
   - `make wheel-devel` - create date-tagged wheel package

bin/poem-tenant

@@ -69,8 +69,6 @@ su -m -s /bin/sh $RUNASUSER -c \
 
 if [[ $name != 'all' ]]
 then
-    su -m -s /bin/sh $RUNASUSER -c \
-    "poem-manage tenant_command loaddata initial_data.json --schema=$schema"
     su -m -s /bin/sh $RUNASUSER -c \
     "poem-manage tenant_command import_internal_metrics --schema=$schema"
 fi

docker/Dockerfile

@@ -51,8 +51,7 @@ WORKDIR poem-2-devel
 RUN source /etc/profile.d/venv_poem.sh; \
     workon poem; \
     make wheel-devel; \
-    pip3 install dist/*; \
-    pip3 install -r requirements.txt
+    pip3 install dist/*;
 ADD https://cacerts.digicert.com/TERENAeScienceSSLCA3.crt.pem /etc/pki/tls/certs/
 RUN yum -y install ca-policy-egi-core; \
     rm -rf /etc/pki/ca-trust/source/anchors/*; \

docker/docker-compose.yml

@@ -9,6 +9,8 @@ services:
     hostname: poem-react
     tty: true
     stdin_open: true
+    ports:
+      - "8000:8000"
     depends_on:
       - db-poem
     volumes:
@@ -19,6 +21,7 @@ services:
       - ../docker/syncsite.sh:/home/user/syncsite.sh
       - ../docker/safety.sh:/home/user/safety.sh
       - ../docker/collectstatic.sh:/home/user/collectstatic.sh
+      - ../docker/run-django-server.sh:/home/user/run-django-server.sh
       - ../docker/restarthttpd.sh:/home/user/restarthttpd.sh
       - ../:/home/user/poem-source
       - ../poem/Poem/:$VENV/lib/python3.6/site-packages/Poem
@@ -31,10 +34,8 @@ services:
       - ../bin/poem-clearsessions:$VENV/bin/poem-clearsessions
       - ../bin/poem-manage:$VENV/bin/poem-manage
       - ../bin/poem-genseckey:$VENV/bin/poem-genseckey
-      - ../bin/poem-syncservtype:$VENV/bin/poem-syncservtype
       - ../bin/poem-tenant:$VENV/bin/poem-tenant
       - ../bin/poem-token:$VENV/bin/poem-token
-      - ../bin/poem-importservices:$VENV/bin/poem-importservices
     networks:
       app_net:
         ipv4_address: 172.19.0.2

docker/poem.conf

@@ -27,7 +27,8 @@ Reports = https://api.devel.argo.grnet.gr/api/v2/reports
 ReportsTopologyTags = https://api.devel.argo.grnet.gr/api/v2/topology/tags
 ReportsTopologyGroups = https://api.devel.argo.grnet.gr/api/v2/topology/groups
 ReportsTopologyEndpoints = https://api.devel.argo.grnet.gr/api/v2/topology/endpoints
-ReportsCrud = True
+ServiceTypes = https://api.devel.argo.grnet.gr/api/v2/topology/service-types
+Metrics = https://api.devel.argo.grnet.gr/api/v2/admin/metrics
 
 
 [GENERAL_ALL]

docker/run-django-server.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+RUNASROOT="su -m -s /bin/bash root -c"
+PIDS=$(pgrep -f runserver | tr '\n' ' ')
+$RUNASROOT "kill -9 $PIDS 2>/dev/null"
+
+$RUNASROOT ". /home/pyvenv/poem/bin/activate && poem-manage runserver --settings=Poem.settings-devserver 0.0.0.0:8000"

docker/setup.sh

@@ -10,4 +10,3 @@ cp -f poem.conf $VIRTUAL_ENV/etc/poem/poem.conf
 chown -R apache:apache $VIRTUAL_ENV
 ln -f -s $VIRTUAL_ENV/etc/httpd/conf.d/poem.conf /opt/rh/httpd24/root/etc/httpd/conf.d/
 ln -f -s $VIRTUAL_ENV/etc/cron.d/poem-clearsessions /etc/cron.d/
-ln -f -s $VIRTUAL_ENV/etc/cron.d/poem-sync /etc/cron.d/

etc/poem.conf.template

@@ -27,6 +27,7 @@ Reports = https://api.devel.argo.grnet.gr/api/v2/reports
 ReportsTopologyTags = https://api.devel.argo.grnet.gr/api/v2/topology/tags
 ReportsTopologyGroups = https://api.devel.argo.grnet.gr/api/v2/topology/groups
 ReportsTopologyEndpoints = https://api.devel.argo.grnet.gr/api/v2/topology/endpoints
+ServiceTypes = https://api.devel.argo.grnet.gr/api/v2/topology/service-types
 Metrics = https://api.devel.argo.grnet.gr/api/v2/admin/metrics
 
 [GENERAL_ALL]

helpers/metric_json_helper.py

@@ -0,0 +1,62 @@
+#!/usr/bin/python3
+
+import json
+import argparse
+import os
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        "Convert old metric format to new for all the files in the given "
+        "directory; newly created files will have suffix '-new'"
+    )
+    parser.add_argument(
+        "-d", "--directory", dest="directory", required=True, type=str,
+        help="directory with the dumped json files"
+    )
+    args = parser.parse_args()
+
+    files = [
+        os.path.join(args.directory, f) for f in os.listdir(args.directory) if
+        os.path.isfile(os.path.join(args.directory, f))
+    ]
+
+    for file in files:
+        with open(file, "r") as f:
+            data = json.load(f)
+
+        changed = False
+        new_data = [item for item in data if item["model"] != "poem.metrictype"]
+        for item in new_data:
+            if item["model"] == "poem.metric":
+                if item["fields"]["probekey"]:
+                    item["fields"]["probeversion"] = \
+                        f"{item['fields']['probekey'][0]} " \
+                        f"({item['fields']['probekey'][1]})"
+
+                else:
+                    item["fields"]["probeversion"] = None
+
+                item["fields"].pop("mtype")
+                item["fields"].pop("probekey")
+                item["fields"].pop("description")
+                item["fields"].pop("parent")
+                item["fields"].pop("probeexecutable")
+                item["fields"].pop("attribute")
+                item["fields"].pop("dependancy")
+                item["fields"].pop("flags")
+                item["fields"].pop("files")
+                item["fields"].pop("parameter")
+                item["fields"].pop("fileparameter")
+
+                changed = True
+
+        if changed:
+            extension = file.split(".")[-1]
+            new_name = file.split(f".{extension}")[0] + f"-new.{extension}"
+
+            with open(new_name, "w") as f:
+                json.dump(new_data, f, indent=4)
+
+
+main()

poem/Poem/Makefile

@@ -9,11 +9,20 @@ devel-watch:
 	rm -rf frontend/bundles/reactbundle/* ; \
 	node_modules/.bin/webpack --config webpack.config.js --progress --mode development --watch
 
-devel:
+devel-django-server:
+	docker exec -ti $(container_name) ./run-django-server.sh
+
+devel-webpack-server:
+	node_modules/.bin/webpack-dev-server --config webpack.devserver.config.js --progress
+
+devel-webpack-server-hmr:
+	node_modules/.bin/webpack-dev-server --config webpack.devserver.config.js --progress --hot
+
+devel-bundle:
 	rm -rf frontend/bundles/reactbundle/* ; \
 		node_modules/.bin/webpack --config webpack.config.js --progress --mode development
 
-prod:
+prod-bundle:
 	rm -rf frontend/bundles/reactbundle/* ; \
 		node_modules/.bin/webpack --config webpack.config.js --progress --mode production
 

poem/Poem/api/internal_views/app.py

@@ -107,6 +107,7 @@ def get(self, request):
         options.update(webapiaggregation=settings.WEBAPI_AGGREGATION)
         options.update(webapithresholds=settings.WEBAPI_THRESHOLDS)
         options.update(webapioperations=settings.WEBAPI_OPERATIONS)
+        options.update(webapiservicetypes=settings.WEBAPI_SERVICETYPES)
         options.update(version=version)
         options.update(webapireports=dict(
             main=settings.WEBAPI_REPORTS,

poem/Poem/api/internal_views/metricprofiles.py

@@ -13,22 +13,6 @@
 from .utils import error_response
 
 
-class ListAllServiceFlavours(APIView):
-    authentication_classes = (SessionAuthentication,)
-
-    def get(self, request):
-        service_flavours = poem_models.ServiceFlavour.objects.all()
-        serializer = serializers.ServiceFlavourSerializer(
-            service_flavours, many=True
-        )
-        data = []
-        for item in serializer.data:
-            data.append(
-                {'name': item['name'], 'description': item['description']}
-            )
-        return Response(sorted(data, key=lambda k: k['name'].lower()))
-
-
 class ListMetricProfiles(APIView):
     authentication_classes = (SessionAuthentication,)