Skip to content

Core DC

Jump to bottom
Dimitris Lioprasitis edited this page Feb 9, 2021 · 2 revisions

Prerequisites

On the Core DC machine we need Docker, Python3.7, Grafana, Influx DB installed

Docker

docker build –t 5g_anomaly_detection –f Dockerfile .

Python

  • Tested with Python 3.7.0 and pip 10.0.1
  • Required Python packages: pandas, numpy, scikit-learn, jsonschema, matplotlib, keras (2.3.1), tensorflow (2.2.1), tensorflow-cpu (2.2.1), h5py, influxdb
  • All required packages with tested versions can be installed from requirements.txt file using the command (from directory, where requirements.txt is located):

pip install –r requirements.txt

Grafana

Grafana can be installed either locally (https://grafana.com/docs/grafana/latest/installation/) or within a Docker container (https://hub.docker.com/r/grafana/grafana).
After installation, two data sources must be set: a Prometheus data source and an InfluxDB data source. After that the created dashboard can be imported from a json file (ui/threat-detection-ui.json).

Influx DB


In addition, there are certain columns in some measurements, which require specific values:

  • node_cpu_seconds_total:
    • cpu: must has values 0 and 1. These two are used for collecting and aggregating metrics.
  • node_network_receive_bytes_total/ node_network_transmit_bytes_total:
    • device: enp1s0, enp0s20u1 and ppp0 are used for collecting metrics and aggregating received and transmitted bytes and their rate.
  • ul_bitrate/dl_bitrate:
    • cellId: There are two values for cellId: 1 and 2. cellId 1 is used for LTE connections and cellId 2 is used for 5G connections. The algorithms collects metrics only from cellId 2, which are 5G related.

Execution

Docker

  • Run the docker container (in the background) with the following command:
    • docker run –d –env ${ENV_PARAM_NAME_1}=${ENV_PARAM_VALUE_1} –env … -p ${HOST_PORT}:1234 5g_anomaly_detection
  • Docker ENV params:
    • TRAIN_MODEL: Train Model before starting it for predicting in real time. (Default: True)
    • EVAL_MODEL: Evaluate Model in order to suggest some missing thresholds, that are not user defined. If all thresholds are set as ENV params this function can be avoid, because it will not override user-defined values. If EVAL_MODEL is set to False and user has not set some thresholds they will be set with a default value equal to 0.1. (Default: True)
    • CPU_TH: RMSE anomaly threshold for predicted CPU percentage rate in user mode. (Default: 0.1)
    • MEM_TH: RMSE anomaly threshold for predicted RAM percentage rate. (Default: 0.1)
    • CPU_RX_TH: RMSE anomaly threshold for predicted RX CPU percentage rate. (Default: 0.1)
    • CPU_TX_TH: RMSE anomaly threshold for predicted TX CPU percentage rate. (Default: 0.1)
    • NET_UP_TH: RMSE anomaly threshold for predicted bytes transmitted rate for selected interfaces. (Default: 0.1)
    • NET_DOWN_TH: RMSE anomaly threshold for predicted bytes received rate for selected interfaces. (Default: 0.1)
    • NET_5G_UP_TH: RMSE anomaly threshold for predicted bytes transmitted rate for 5G cell. (Default: 0.1)
    • NET_5G_DOWN_TH: RMSE anomaly threshold for predicted bytes received rate for 5G cell. (Default: 0.1)
    • OVERALL_TH: RMSE anomaly threshold for all predicted features aggregated. (Default: 0.1)
    • INFLUX_HOST: IP of machine where InfluxDB is running. (Default: localhost)
    • INFLUX_PORT: Port, where InfluxDB is listening. (Default: 8086)
    • INFLUX_USER: Username for connecting in InfluxDB. (Default: admin)
    • INFLUX_PASS: Password for connecting in InfluxDB. (Default: admin)
    • INFLUX_DB: Database, where all collected metrics and detected anomalies are stored. (Default: metrics_db)
    • INFLUX_ANOMALIES_MEASUREMENT: Measurement in ${INFLUX_DB} where all detected anomalies will be saved. (Default: detected_anomalies)
  • Docker ports:
    • Port 1234: A simple python http server is running in order to provide docs for reading

If there is no change in training process or in the training data, TRAIN_MODEL param can be set to False, as the root folder contains a trained and evaluated model in the given data

Python (optional)

If the algorithm is executed from Python command line in testing mode, the file thresholds.json in data folder must exists. If there is no such file in the data folder, it must be created manually. Its structure is the following:

{"cpu_threshold": 0.1, "mem_threshold": 0.1, ...}

The required keys for thresholds.json file are the following: cpu_threshold, mem_threshold, cpu_tx_threshold, cpu_rx_threshold, net_up_threshold, net_down_threshold, net_5g_up_threshold, net_5g_down_threshold, overall_threshold.

  • Training Mode (from root folder, where find_anomalies.py is located):
python find_anomalies.py \  
            --mode train \
            --model model/5g_autoencoder.h5 \
            --evaluate false \
            --thresholds_file data/thresholds.json
  • Training Mode with evaluation (from root folder, where find_anomalies.py is located):
python find_anomalies.py \
            --mode train \
            --model model/5g_autoencoder.h5 \
            --evaluate true \
            --thresholds_file data/thresholds.json
  • Testing Mode (from root folder, where find_anomalies.py is located). The following parameters must be set either as variables in shell or directly in the following command to execute the algorithm in test mode:
    • ${INFLUX_HOST}: IP of machine where InfluxDB is running.
    • ${INFLUX_PORT}: Port, where InfluxDB is listening.
    • ${INFLUX_USER}: Username for connecting in InfluxDB.
    • ${INFLUX_PASS}: Password for connecting in InfluxDB.
    • ${INFLUX_DB}: Database, where all collected metrics and detected anomalies are stored.
    • ${INFLUX_ANOMALIES_MEASUREMENT}: Measurement in ${INFLUX_DB} where all detected anomalies will be saved.
python find_anomalies.py \
    --mode test \
    --model model/5g_autoencoder.h5 \
    --thresholds_file data/thresholds.json \
    --influx_host ${INFLUX_HOST} \
    --influx_port ${INFLUX_PORT} \
    --influx_user ${INFLUX_USER} \
    --influx_pass ${INFLUX_PASS} \
    --influx_db ${INFLUX_DB} \
    --influx_measurement ${INFLUX_ANOMALIES_MEASUREMENT}

Grafana

After configuring data sources and importing the custom dashboard we get the following image:

Contents

Home
Overview
Edge Cloud
Autoencoder
Core DC

Clone this wiki locally