demo/builtin/dien/DIEN_DEMO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aeea794e",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/builtin/dien/DIEN_DEMO.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0261374c",
   "metadata": {},
   "source": [
    "\n",
    "# DIEN DEMO\n",
    "\n",
    "Online ads display CTR predication, evolved from DIN(Alibaba) which uses sequence model to simulate user interest evolving process.\n",
    "\n",
    "* original source\n",
    "    * Source repo: https://github.com/alibaba/ai-matrix\n",
    "\n",
    "\n",
    "# Content\n",
    "* [Overview](#Overview)\n",
    "    * [Model Architecture](#Model-Architecture)\n",
    "    * [Optimizations](#Optimizations)\n",
    "    * [Performance](#Performance)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Environment Setup](#1.-Environment-Setup)\n",
    "    * [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "    * [3. Data Prepare](#3.-Data-Prepare)\n",
    "    * [4. Train](#4.-Train)\n",
    "\n",
    "------\n",
    "\n",
    "# Overview\n",
    "\n",
    "## Model Architecture\n",
    "\n",
    "<div><img src=\"./img/dien-arch.png\" alt=\"DIEN Model Architecture\" width=\"600\"></div>\n",
    "\n",
    "* DIEN (from bottom to up)\n",
    "    * Behavior layer: convert sequence behaviors to embedding vector  \n",
    "    * Interest Extractor Layer: extracts interest sequence based on behavior sequence\n",
    "    * Interest evolving layer: AUGRU models interest evolving process that is relative to target item.\n",
    "\n",
    "\n",
    "## Optimizations\n",
    "\n",
    "* Motivation\n",
    "    * Original ETL was implemented with pure python\n",
    "    * Original Training on single node CPU showed 2.55x gap to GPU\n",
    "    * Inference on CPU nodes can be run with 8X parallism\n",
    "    \n",
    "* Data Process\n",
    "    * speeding up by 15x\n",
    "    * Data Ingestion: Rewrite Data Ingestion wit spark and directly load for preprocessing, 35x speed up\n",
    "    * PreProcessing: re-implement DataProcessing with RecDP spark, 12.27x speed up\n",
    "\n",
    "<div><img src=\"./img/dien-dataprocessing.png\" alt=\"DIEN Training\" width=\"900\"></div>  \n",
    "\n",
    "* Training\n",
    "    * Speeding up by 8.12x\n",
    "    * Tensorflow optimization: Switch to use Intel optimized Tensorflow2\n",
    "    * Optimized DataLoader: complete data categorify in ETL\n",
    "    * Scaling out: Scaling out training from single node to 4 CLX-8535 nodes\n",
    "\n",
    "<div><img src=\"./img/dien-scaling.png\" alt=\"DIEN Inference\" width=\"900\"></div>\n",
    "    \n",
    "* Inference\n",
    "    * Improved 882x\n",
    "    * inference scaling out from single process on one node to 64 processes on 4 nodes\n",
    "    * Optimized DataLoader: complete data categorify in ETL\n",
    "    * Multi instance inference\n",
    "\n",
    "\n",
    "## Performance\n",
    "\n",
    "* For Training\n",
    "    * Our optimized DIEN end to end training time on CPU vs. on AWS P4D A100 shows gap as 2.14x(single CLX node), after scaling out to 4 CLX nodes, gap is reduced to 1.05x\n",
    "<div><img src=\"./img/dien-training-perf.png\" alt=\"DIEN Training\" width=\"500\"></div>\n",
    "\n",
    "* For Inference\n",
    "    * Our optimized DIEN inference throughput on CPU vs on AWS P4D A100 shows 2.24x better on single CLX, which can be linear scaling out to multiple nodes.\n",
    "<div><img src=\"./img/dien-infer-perf.png\" alt=\"DIEN Inference\" width=\"500\"></div>\n",
    "\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8e06878",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "\n",
    "* [1. Environment Setup](#1.-Environment-Setup)\n",
    "* [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "* [3. Data Prepare](#3.-Data-Prepare)\n",
    "* [4. Train](#4.-Train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc2cc4cc",
   "metadata": {},
   "source": [
    "## 1. Environment Setup"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2411d204",
   "metadata": {},
   "source": [
    "### (Option 1) Use Pip Install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7136b14",
   "metadata": {},
   "outputs": [],
   "source": [
    "! apt install -y git\n",
    "! pip install pyrecdp\n",
    "! pip install intel-tensorflow==2.10 tqdm psutil pandas sigopt\n",
    "! pip install e2eaiok-sda --pre"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "acb73441",
   "metadata": {},
   "source": [
    "### (Option 2) Use Docker\n",
    "``` bash\n",
    "# 1. git clone codes\n",
    "git clone https://github.com/intel/e2eAIOK.git\n",
    "cd e2eAIOK\n",
    "git submodule update --init --recursive\n",
    "\n",
    "# 2. build docker image\n",
    "python3 scripts/start_e2eaiok_docker.py -b tensorflow -w ${host0} ${host1} ${host2} ${host3} --proxy \"http://addr:ip\"\n",
    "\n",
    "# 3. Enter Docker\n",
    "sshpass -p docker ssh ${host0} -p 12344\n",
    "\n",
    "# 4. start jupyter notebook\n",
    "nohup jupyter notebook --notebook-dir=/home/vmagent/app/e2eaiok --ip=${hostname} --port=8899 --allow-root &\n",
    "Now you can visit demso in http://${hostname}:8899/.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46089e8c",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e3aeaaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "! sh workflow_prepare_dien.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc3898db",
   "metadata": {},
   "source": [
    "## 3. Data Prepare"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39fa4c1a",
   "metadata": {},
   "source": [
    "we are using amazon product review - book dataset. using wget to download original zip file from stanford repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ccb9e86",
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget wget https://zenodo.org/record/3463683/files/data.tar.gz\n",
    "! wget http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Books.json.gz\n",
    "! wget http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/meta_Books.json.gz\n",
    "! tar -jxvf data.tar.gz\n",
    "! gunzip reviews_Books.json.gz\n",
    "! gunzip meta_Books.json.gz\n",
    "\n",
    "! mkdir -p data/raw_data\n",
    "! mkdir -p data/train\n",
    "! mkdir -p data/valid\n",
    "\n",
    "! mv *json data/raw_data/; cp data/local_test_splitByUser data/raw_data/\n",
    "! mv data/local_test_splitByUser data/valid/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed83c56",
   "metadata": {},
   "source": [
    "lauch data process to downloaded files.\n",
    "original data is two json files, below script will do Data Ingestion, Data Process, and Feature Engineering to prepare for ready-to-train data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fc990eca",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/pyrecdp\n",
      "sr414\n",
      "Will assign 48 cores and 308502 M memory for spark\n",
      "23/03/21 03:45:18 WARN Utils: Your hostname, sr414 resolves to a loopback address: 127.0.1.1; using 10.1.2.14 instead (on interface enp134s0f1)\n",
      "23/03/21 03:45:18 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "23/03/21 03:45:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "recdp-scala-extension is enabled\n",
      "per core memory size is 6.276 GB and shuffle_disk maximum capacity is 1200.000 GB\n",
      "start spark process took 4.007592746987939 secs\n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//reviews-info\n",
      "parse reviews-info with spark took 15.613626144011505 secs                      \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//item-info\n",
      "parse item-info with spark took 11.843996569979936 secs                         \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//dicts/reviewer_id\n",
      "save_to_uid_voc took 48.98802922107279 secs                                     \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//dicts/asin\n",
      "save_to_mid_voc took 14.307553627993912 secs                                    \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//dicts/category\n",
      "save_to_cat_voc took 0.9827255790587515 secs\n",
      "root\n",
      " |-- asin: string (nullable = true)\n",
      " |-- asin_id: integer (nullable = true)\n",
      " |-- count: long (nullable = true)\n",
      " |-- category: string (nullable = true)\n",
      "\n",
      "root                                                                            \n",
      " |-- category: string (nullable = true)\n",
      " |-- asin: string (nullable = true)\n",
      " |-- asin_id: integer (nullable = true)\n",
      " |-- count: long (nullable = true)\n",
      " |-- cat_id: integer (nullable = true)\n",
      " |-- count: long (nullable = true)\n",
      "\n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//local_train_splitByUser.parquet\n",
      "categorify took 17.91746578400489 secs                                          \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//collapsed\n",
      "collapse hist took 12.911809311946854 secs                                      \n",
      "/usr/local/lib/python3.10/dist-packages/pyspark/sql/context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/pyspark/sql/dataframe.py:127: UserWarning: DataFrame constructor is internal. Do not directly use it.\n",
      "  warnings.warn(\"DataFrame constructor is internal. Do not directly use it.\")\n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//records_with_negative_sample\n",
      "add_negative_sample took 20.835241795983166 secs                                \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//records_with_negative_hists\n",
      "add_negative_hist_cols took 9.729390932945535 secs                              \n",
      "save_to_local_splitByUser took 14.818636714015156 secs                          \n",
      "Total process time is 180.96805579098873 secs\n"
     ]
    }
   ],
   "source": [
    "# Data Processing for train\n",
    "! python -u e2eaiok/modelzoo/dien/feature_engineering/preprocessing.py --train --dataset_path `pwd`/data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd79efca",
   "metadata": {},
   "source": [
    "lauch data process to downloaded files.\n",
    "original test data is csv file, below script will do Data Process and Feature Engineering to prepare for ready-to-inference data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1c353654",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/pyrecdp\n",
      "sr414\n",
      "Will assign 48 cores and 308502 M memory for spark\n",
      "23/03/21 03:48:41 WARN Utils: Your hostname, sr414 resolves to a loopback address: 127.0.1.1; using 10.1.2.14 instead (on interface enp134s0f1)\n",
      "23/03/21 03:48:41 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "23/03/21 03:48:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "recdp-scala-extension is enabled\n",
      "per core memory size is 6.276 GB and shuffle_disk maximum capacity is 1200.000 GB\n",
      "start spark process took 4.669647359987721 secs\n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//reviews-info\n",
      "parse reviews-info with spark took 15.729018479934894 secs                      \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//item-info\n",
      "parse item-info with spark took 9.373082284000702 secs                          \n",
      "load_uid_voc took 100.2910239250632 secs\n",
      "load_mid_voc took 28.93930171697866 secs\n",
      "load_cat_voc took 0.08822984201833606 secs\n",
      "/usr/local/lib/python3.10/dist-packages/pyspark/sql/context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/pyspark/sql/dataframe.py:127: UserWarning: DataFrame constructor is internal. Do not directly use it.\n",
      "  warnings.warn(\"DataFrame constructor is internal. Do not directly use it.\")\n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//test_records_with_negative_hists\n",
      "add_negative_hist_cols took 22.786608777940273 secs                             \n",
      "save data to file:////home/vmagent/app/e2eAIOK/demo/builtin/dien/data//output//local_test_splitByUser.parquet\n",
      "categorify took 91.15475151699502 secs                                          \n",
      "save_to_local_splitByUser took 2.655800345935859 secs                           \n",
      "Total process time is 257.0932072789874 secs\n"
     ]
    }
   ],
   "source": [
    "# Data Processing for test\n",
    "! python -u e2eaiok/modelzoo/dien/feature_engineering/preprocessing.py --test --dataset_path `pwd`/data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79dfcdcc",
   "metadata": {},
   "source": [
    "## 4. Train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9bff878",
   "metadata": {},
   "source": [
    "SDA is an autoHPO component, we use SDA to trigger DIEN training and validation.\n",
    "Noticed: set enable_sigopt to True, SDA will explore HyperParameter for this model. In our demo, we will use our searched best parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ea46731",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "587cb9c9",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-03-21 04:02:38,341 - E2EAIOK.SDA - INFO - ### Ready to submit current task  ###\n",
      "2023-03-21 04:02:38,349 - E2EAIOK.SDA - INFO - Model Advisor created\n",
      "2023-03-21 04:02:38,350 - E2EAIOK.SDA - INFO - model parameter initialized\n",
      "2023-03-21 04:02:38,351 - E2EAIOK.SDA - INFO - start to launch training\n",
      "2023-03-21 04:02:38,353 - sigopt - INFO - training launch command: /usr/bin//python -u e2eaiok/modelzoo/dien/train/ai-matrix/script/train.py --train_path data/train/local_train_splitByUser --test_path data/valid/local_test_splitByUser --meta_path data/meta.yaml --saved_path /home/vmagent/app/e2eaiok/result/DIEN/20230321_040238/74ee8e1d3e5b4458a4b60da27e1b4540e0503a691670915b44a6d643f933ed2f --num-intra-threads 48 --num-inter-threads 2 --mode train --embedding_device cpu --model DIEN --slice_id 0 --advanced true --seed 3 --data_type FP32 --batch_size 256\n",
      "2023-03-21 04:02:38.508460: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "non-resource variables are not supported in the long term\n",
      "2023-03-21 04:02:40.000878: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:544: UserWarning: `tf.nn.rnn_cell.GRUCell` is deprecated and will be removed in a future version. This class is equivalent as `tf.keras.layers.GRUCell`, and will be replaced by that in Tensorflow 2.0.\n",
      "  rnn_outputs, _ = dynamic_rnn(tf.compat.v1.nn.rnn_cell.GRUCell(HIDDEN_SIZE), inputs=self.item_his_eb,\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/keras/layers/rnn/legacy_cells.py:588: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/keras/layers/rnn/legacy_cells.py:602: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:240: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "  bn1 = tf.compat.v1.layers.batch_normalization(inputs=in_, name='bn1' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:241: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn1 = tf.compat.v1.layers.dense(bn1, 100, activation=None, name='f1' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:243: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn2 = tf.compat.v1.layers.dense(dnn1, 50, activation=None, name='f2' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:245: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn3 = tf.compat.v1.layers.dense(dnn2, 2, activation=None, name='f3' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Advanced train\n",
      "{'uid_voc': '/home/vmagent/app/e2eAIOK/demo/builtin/dien/data//uid_voc.pkl', 'mid_voc': '/home/vmagent/app/e2eAIOK/demo/builtin/dien/data//mid_voc.pkl', 'cat_voc': '/home/vmagent/app/e2eAIOK/demo/builtin/dien/data//cat_voc.pkl', 'train_file': 'data/train/local_train_splitByUser', 'test_file': 'data/valid/local_test_splitByUser', 'model_type': 'DIEN', 'seed': 3, 'batch_size': 256, 'data_type': 'FP32'}\n",
      "batch_size:  256\n",
      "model:  DIEN\n",
      "embedding_device cpu\n",
      "best model will be saved to /home/vmagent/app/e2eaiok/result/DIEN/20230403_163755/74ee8e1d3e5b4458a4b60da27e1b4540e0503a691670915b44a6d643f933ed2f/dnn_best_model\n",
      "/home/vmagent/app/e2eaiok/result/DIEN/20230403_163755/74ee8e1d3e5b4458a4b60da27e1b4540e0503a691670915b44a6d643f933ed2f/dnn_best_model/ckpt_noshuffDIEN3\n",
      "Number of uid = 8026324, mid = 2330066, cat = 2752\n",
      "embedding on cpu\n",
      "-----------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:381: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  query = tf.compat.v1.layers.dense(query, facts_size, activation=None, name='f1' + stag)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:391: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  d_layer_1_all = tf.compat.v1.layers.dense(din_all, 80, activation=tf.nn.sigmoid, name='f1_att' + stag)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:395: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  d_layer_2_all = tf.compat.v1.layers.dense(d_layer_1_all, 40, activation=tf.nn.sigmoid, name='f2_att' + stag)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:398: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  d_layer_3_all = tf.compat.v1.layers.dense(d_layer_2_all, 1, activation=None, name='f3_att' + stag)\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:177: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "  bn1 = tf.compat.v1.layers.batch_normalization(inputs=inp, name='bn1')\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:178: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn1 = tf.compat.v1.layers.dense(bn1, 200, activation=None, name='f1')\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:184: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn2 = tf.compat.v1.layers.dense(dnn1, 80, activation=None, name='f2')\n",
      "/home/vmagent/app/e2eAIOK/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:189: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn3 = tf.compat.v1.layers.dense(dnn2, 2, activation=None, name='f3')\n",
      "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/indexed_slices.py:436: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 144473832 elements. This may consume a large amount of memory.\n",
      "  warnings.warn(\n",
      "2023-03-21 04:02:41.637716: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start to load Data from disk\n",
      "Loading Data from disk is completed with 114.22454190254211 secs, start to train\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.\n",
      "OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info\n",
      "OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-95\n",
      "OMP: Info #156: KMP_AFFINITY: 96 available OS procs\n",
      "OMP: Info #157: KMP_AFFINITY: Uniform topology\n",
      "OMP: Info #179: KMP_AFFINITY: 2 packages x 24 cores/pkg x 2 threads/core (48 total cores)\n",
      "OMP: Info #213: KMP_AFFINITY: OS proc to physical thread map:\n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 48 maps to package 0 core 0 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 49 maps to package 0 core 1 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 50 maps to package 0 core 2 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 51 maps to package 0 core 3 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 52 maps to package 0 core 4 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 53 maps to package 0 core 5 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 6 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 54 maps to package 0 core 6 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 8 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 55 maps to package 0 core 8 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 9 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 56 maps to package 0 core 9 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 10 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 57 maps to package 0 core 10 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 11 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 58 maps to package 0 core 11 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 12 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 59 maps to package 0 core 12 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 0 core 13 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 60 maps to package 0 core 13 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 0 core 16 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 61 maps to package 0 core 16 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 0 core 17 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 62 maps to package 0 core 17 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 0 core 18 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 63 maps to package 0 core 18 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to package 0 core 19 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 64 maps to package 0 core 19 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to package 0 core 20 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 65 maps to package 0 core 20 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to package 0 core 21 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 66 maps to package 0 core 21 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to package 0 core 25 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 67 maps to package 0 core 25 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to package 0 core 26 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 68 maps to package 0 core 26 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to package 0 core 27 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 69 maps to package 0 core 27 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to package 0 core 28 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 70 maps to package 0 core 28 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to package 0 core 29 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 71 maps to package 0 core 29 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 24 maps to package 1 core 0 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 72 maps to package 1 core 0 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 25 maps to package 1 core 1 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 73 maps to package 1 core 1 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 26 maps to package 1 core 2 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 74 maps to package 1 core 2 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 27 maps to package 1 core 3 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 75 maps to package 1 core 3 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 28 maps to package 1 core 4 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 76 maps to package 1 core 4 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 29 maps to package 1 core 5 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 77 maps to package 1 core 5 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 30 maps to package 1 core 6 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 78 maps to package 1 core 6 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 31 maps to package 1 core 8 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 79 maps to package 1 core 8 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 32 maps to package 1 core 9 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 80 maps to package 1 core 9 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 33 maps to package 1 core 10 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 81 maps to package 1 core 10 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 34 maps to package 1 core 11 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 82 maps to package 1 core 11 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 35 maps to package 1 core 12 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 83 maps to package 1 core 12 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 36 maps to package 1 core 13 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 84 maps to package 1 core 13 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 37 maps to package 1 core 16 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 85 maps to package 1 core 16 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 38 maps to package 1 core 17 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 86 maps to package 1 core 17 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 39 maps to package 1 core 18 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 87 maps to package 1 core 18 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 40 maps to package 1 core 19 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 88 maps to package 1 core 19 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 41 maps to package 1 core 20 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 89 maps to package 1 core 20 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 42 maps to package 1 core 21 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 90 maps to package 1 core 21 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 43 maps to package 1 core 25 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 91 maps to package 1 core 25 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 44 maps to package 1 core 26 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 92 maps to package 1 core 26 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 45 maps to package 1 core 27 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 93 maps to package 1 core 27 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 46 maps to package 1 core 28 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 94 maps to package 1 core 28 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 47 maps to package 1 core 29 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 95 maps to package 1 core 29 thread 1 \n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52588 thread 0 bound to OS proc set 0\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52589 thread 1 bound to OS proc set 1\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52588 thread 2 bound to OS proc set 2\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52591 thread 3 bound to OS proc set 3\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52592 thread 4 bound to OS proc set 4\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52593 thread 5 bound to OS proc set 5\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52594 thread 6 bound to OS proc set 6\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52595 thread 7 bound to OS proc set 7\n",
      "OMP: Info #249: KMP_AFFINITY: pid 52411 tid 52596 thread 8 bound to OS proc set 8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter: 500 ----> train_loss: 1.7206 ---- train_accuracy: 0.5585 ---- train_aux_loss: 1.3809 ---- train_time: 50.919\n",
      " test_auc: 0.6161 ----test_loss: 1.7136 ---- test_accuracy: 0.5743 ---- test_aux_loss: 1.3869 ---- eval_time: 14.828 ---- num_iters: 474\n",
      "current auc is 0.6160514105711412, target auc is 0.82\n",
      "iter: 1000 ----> train_loss: 1.7038 ---- train_accuracy: 0.5858 ---- train_aux_loss: 1.3713 ---- train_time: 45.815\n",
      " test_auc: 0.6471 ----test_loss: 1.6312 ---- test_accuracy: 0.6043 ---- test_aux_loss: 1.3854 ---- eval_time: 13.945 ---- num_iters: 474\n",
      "current auc is 0.6470734103433915, target auc is 0.82\n",
      "iter: 1500 ----> train_loss: 1.6985 ---- train_accuracy: 0.6034 ---- train_aux_loss: 1.3719 ---- train_time: 46.629\n",
      " test_auc: 0.6736 ----test_loss: 1.6197 ---- test_accuracy: 0.6221 ---- test_aux_loss: 1.3804 ---- eval_time: 13.999 ---- num_iters: 474\n",
      "current auc is 0.6736113423878108, target auc is 0.82\n",
      "iter: 2000 ----> train_loss: 1.6944 ---- train_accuracy: 0.6122 ---- train_aux_loss: 1.3711 ---- train_time: 47.559\n",
      " test_auc: 0.6855 ----test_loss: 1.6686 ---- test_accuracy: 0.6355 ---- test_aux_loss: 1.3753 ---- eval_time: 13.795 ---- num_iters: 474\n",
      "current auc is 0.6855331806124442, target auc is 0.82\n",
      "iter: 2500 ----> train_loss: 1.6816 ---- train_accuracy: 0.6235 ---- train_aux_loss: 1.3629 ---- train_time: 48.232\n",
      " test_auc: 0.7129 ----test_loss: 1.5630 ---- test_accuracy: 0.6551 ---- test_aux_loss: 1.3303 ---- eval_time: 14.196 ---- num_iters: 474\n",
      "current auc is 0.7128996761887074, target auc is 0.82\n",
      "iter: 3000 ----> train_loss: 1.6259 ---- train_accuracy: 0.6320 ---- train_aux_loss: 1.3119 ---- train_time: 47.909\n",
      " test_auc: 0.7312 ----test_loss: 1.4966 ---- test_accuracy: 0.6701 ---- test_aux_loss: 1.3044 ---- eval_time: 13.850 ---- num_iters: 474\n",
      "current auc is 0.7311879528055277, target auc is 0.82\n",
      "iter: 3500 ----> train_loss: 1.5934 ---- train_accuracy: 0.6439 ---- train_aux_loss: 1.2835 ---- train_time: 48.849\n",
      "current auc is 0.7312 and test auc is 0.7290\n",
      " test_auc: 0.7290 ----test_loss: 1.5380 ---- test_accuracy: 0.6737 ---- test_aux_loss: 1.2665 ---- eval_time: 13.640 ---- num_iters: 474\n",
      "current auc is 0.7311879528055277, target auc is 0.82\n",
      "iter: 4000 ----> train_loss: 1.5732 ---- train_accuracy: 0.6543 ---- train_aux_loss: 1.2686 ---- train_time: 48.175\n",
      " test_auc: 0.7509 ----test_loss: 1.4811 ---- test_accuracy: 0.6895 ---- test_aux_loss: 1.2338 ---- eval_time: 13.702 ---- num_iters: 474\n",
      "current auc is 0.7508536108345111, target auc is 0.82\n",
      "iter: 4500 ----> train_loss: 1.5594 ---- train_accuracy: 0.6575 ---- train_aux_loss: 1.2571 ---- train_time: 47.171\n",
      "current auc is 0.7509 and test auc is 0.7401\n",
      " test_auc: 0.7401 ----test_loss: 1.4859 ---- test_accuracy: 0.6844 ---- test_aux_loss: 1.2112 ---- eval_time: 13.739 ---- num_iters: 474\n",
      "current auc is 0.7508536108345111, target auc is 0.82\n",
      "iter: 5000 ----> train_loss: 1.5420 ---- train_accuracy: 0.6632 ---- train_aux_loss: 1.2392 ---- train_time: 47.399\n",
      " test_auc: 0.7531 ----test_loss: 1.5399 ---- test_accuracy: 0.6966 ---- test_aux_loss: 1.2032 ---- eval_time: 14.080 ---- num_iters: 474\n",
      "current auc is 0.7530704396369798, target auc is 0.82\n",
      "iter: 5500 ----> train_loss: 1.5225 ---- train_accuracy: 0.6660 ---- train_aux_loss: 1.2251 ---- train_time: 47.575\n",
      " test_auc: 0.7615 ----test_loss: 1.4699 ---- test_accuracy: 0.7031 ---- test_aux_loss: 1.1731 ---- eval_time: 13.750 ---- num_iters: 474\n",
      "current auc is 0.7615231669552388, target auc is 0.82\n",
      "iter: 6000 ----> train_loss: 1.5071 ---- train_accuracy: 0.6736 ---- train_aux_loss: 1.2141 ---- train_time: 47.576\n",
      " test_auc: 0.7706 ----test_loss: 1.4138 ---- test_accuracy: 0.7110 ---- test_aux_loss: 1.1572 ---- eval_time: 13.644 ---- num_iters: 474\n",
      "current auc is 0.7705835669313894, target auc is 0.82\n",
      "iter: 6500 ----> train_loss: 1.4989 ---- train_accuracy: 0.6808 ---- train_aux_loss: 1.2088 ---- train_time: 48.097\n",
      " test_auc: 0.7719 ----test_loss: 1.4366 ---- test_accuracy: 0.7136 ---- test_aux_loss: 1.1581 ---- eval_time: 13.799 ---- num_iters: 474\n",
      "current auc is 0.7718854868344301, target auc is 0.82\n",
      "iter: 7000 ----> train_loss: 1.4845 ---- train_accuracy: 0.6849 ---- train_aux_loss: 1.1962 ---- train_time: 48.480\n",
      " test_auc: 0.7859 ----test_loss: 1.3848 ---- test_accuracy: 0.7241 ---- test_aux_loss: 1.1393 ---- eval_time: 14.104 ---- num_iters: 474\n",
      "current auc is 0.7859296785402612, target auc is 0.82\n",
      "iter: 7500 ----> train_loss: 1.4770 ---- train_accuracy: 0.6868 ---- train_aux_loss: 1.1896 ---- train_time: 48.094\n",
      " test_auc: 0.7922 ----test_loss: 1.3289 ---- test_accuracy: 0.7275 ---- test_aux_loss: 1.1303 ---- eval_time: 13.763 ---- num_iters: 474\n",
      "current auc is 0.7922211650426841, target auc is 0.82\n",
      "iter: 8000 ----> train_loss: 1.4720 ---- train_accuracy: 0.6889 ---- train_aux_loss: 1.1842 ---- train_time: 48.501\n",
      "current auc is 0.7922 and test auc is 0.7894\n",
      " test_auc: 0.7894 ----test_loss: 1.4068 ---- test_accuracy: 0.7300 ---- test_aux_loss: 1.1255 ---- eval_time: 13.885 ---- num_iters: 474\n",
      "current auc is 0.7922211650426841, target auc is 0.82\n",
      "iter: 8500 ----> train_loss: 1.4573 ---- train_accuracy: 0.6925 ---- train_aux_loss: 1.1733 ---- train_time: 48.998\n",
      " test_auc: 0.8006 ----test_loss: 1.3152 ---- test_accuracy: 0.7366 ---- test_aux_loss: 1.1288 ---- eval_time: 13.788 ---- num_iters: 474\n",
      "current auc is 0.800563502051076, target auc is 0.82\n",
      "iter: 9000 ----> train_loss: 1.4527 ---- train_accuracy: 0.6939 ---- train_aux_loss: 1.1694 ---- train_time: 48.878\n",
      "current auc is 0.8006 and test auc is 0.7978\n",
      " test_auc: 0.7978 ----test_loss: 1.3360 ---- test_accuracy: 0.7378 ---- test_aux_loss: 1.1028 ---- eval_time: 14.024 ---- num_iters: 474\n",
      "current auc is 0.800563502051076, target auc is 0.82\n",
      "iter: 9500 ----> train_loss: 1.4442 ---- train_accuracy: 0.7010 ---- train_aux_loss: 1.1644 ---- train_time: 48.336\n",
      " test_auc: 0.8060 ----test_loss: 1.2937 ---- test_accuracy: 0.7408 ---- test_aux_loss: 1.0855 ---- eval_time: 13.806 ---- num_iters: 474\n",
      "current auc is 0.8060179482436844, target auc is 0.82\n",
      "iter: 10000 ----> train_loss: 1.4295 ---- train_accuracy: 0.7017 ---- train_aux_loss: 1.1508 ---- train_time: 48.321\n",
      " test_auc: 0.8064 ----test_loss: 1.3057 ---- test_accuracy: 0.7454 ---- test_aux_loss: 1.0861 ---- eval_time: 13.829 ---- num_iters: 474\n",
      "current auc is 0.8063544719464357, target auc is 0.82\n",
      "iter: 10500 ----> train_loss: 1.4152 ---- train_accuracy: 0.7050 ---- train_aux_loss: 1.1388 ---- train_time: 49.178\n",
      " test_auc: 0.8229 ----test_loss: 1.2464 ---- test_accuracy: 0.7554 ---- test_aux_loss: 1.1185 ---- eval_time: 13.829 ---- num_iters: 474\n",
      "current auc is 0.8229381580220977, target auc is 0.82\n",
      "current AUC 0.8229381580220977 is greater than target 0.82, stop training\n",
      "iteration:  474\n",
      "iter: 10500\n",
      "Total recommendations: 5120000\n",
      "process time breakdown in seconds are session_init 2.881, train_prepare 112.141, train 1032.286, test_prepare 13.586, test 290.646, model save 0.000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-04-03 17:04:15,700 - sigopt - INFO - Training completed based in sigopt suggestion, took 1032.2858581542969 secs\n",
      "2023-04-03 17:04:15,701 - E2EAIOK.SDA - INFO - training script completed\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===============================================\n",
      "***    Best Trained Model    ***\n",
      "===============================================\n",
      "  Model Type: DIEN\n",
      "  Model Saved Path: /home/vmagent/app/e2eaiok/result/DIEN/20230403_163755/74ee8e1d3e5b4458a4b60da27e1b4540e0503a691670915b44a6d643f933ed2f\n",
      "  Sigopt Experiment id is None\n",
      "  === Result Metrics ===\n",
      "    AUC: 0.8229381580220977\n",
      "    training_time: 1032.2858581542969\n",
      "===============================================\n"
     ]
    }
   ],
   "source": [
    "from e2eAIOK.SDA.SDA import SDA\n",
    "import os\n",
    "\n",
    "python_path = !which python\n",
    "python_dir_path = str(os.path.dirname(python_path[0]))\n",
    "settings = dict()\n",
    "settings[\"data_path\"] = \"data/\"\n",
    "settings[\"enable_sigopt\"] = False\n",
    "settings[\"python_path\"] = python_dir_path\n",
    "settings[\"train_script\"] = \"e2eaiok/modelzoo/dien/train/ai-matrix/script/train.py\"\n",
    "\n",
    "sda = SDA(model=\"DIEN\", settings=settings) # default settings\n",
    "sda.launch()\n",
    "\n",
    "hydro_model = sda.snapshot()\n",
    "hydro_model.explain()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84a64b63",
   "metadata": {},
   "source": [
    "# 5. Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d7790c81",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best_train_model_path is /home/vmagent/app/e2eaiok/result/DIEN/20230403_163755/74ee8e1d3e5b4458a4b60da27e1b4540e0503a691670915b44a6d643f933ed2f/dnn_best_model\n",
      "data_path is /home/vmagent/app/recdp/demo/builtin/dien/data/\n",
      "2023-04-03 17:09:31.350327: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "non-resource variables are not supported in the long term\n",
      "Advanced train\n",
      "{'uid_voc': '/home/vmagent/app/recdp/demo/builtin/dien/data//uid_voc.pkl', 'mid_voc': '/home/vmagent/app/recdp/demo/builtin/dien/data//mid_voc.pkl', 'cat_voc': '/home/vmagent/app/recdp/demo/builtin/dien/data//cat_voc.pkl', 'train_file': '/home/vmagent/app/recdp/demo/builtin/dien/data//train/local_train_splitByUser', 'test_file': '/home/vmagent/app/recdp/demo/builtin/dien/data//valid/local_test_splitByUser', 'model_type': 'DIEN', 'seed': 3, 'batch_size': 128, 'data_type': 'FP32'}\n",
      "batch_size:  128\n",
      "model:  DIEN\n",
      "dnn_best_model_trained/ckpt_noshuffDIEN3\n",
      "2023-04-03 17:09:32.806013: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-04-03 17:09:32.810979: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 24. Tune using inter_op_parallelism_threads for best performance.\n",
      "embedding on cpu\n",
      "-----------------------------------\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:544: UserWarning: `tf.nn.rnn_cell.GRUCell` is deprecated and will be removed in a future version. This class is equivalent as `tf.keras.layers.GRUCell`, and will be replaced by that in Tensorflow 2.0.\n",
      "  rnn_outputs, _ = dynamic_rnn(tf.compat.v1.nn.rnn_cell.GRUCell(HIDDEN_SIZE), inputs=self.item_his_eb,\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/keras/layers/rnn/legacy_cells.py:588: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/keras/layers/rnn/legacy_cells.py:602: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:240: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "  bn1 = tf.compat.v1.layers.batch_normalization(inputs=in_, name='bn1' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:241: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn1 = tf.compat.v1.layers.dense(bn1, 100, activation=None, name='f1' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:243: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn2 = tf.compat.v1.layers.dense(dnn1, 50, activation=None, name='f2' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:245: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn3 = tf.compat.v1.layers.dense(dnn2, 2, activation=None, name='f3' + stag, reuse=tf.compat.v1.AUTO_REUSE)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:381: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  query = tf.compat.v1.layers.dense(query, facts_size, activation=None, name='f1' + stag)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:391: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  d_layer_1_all = tf.compat.v1.layers.dense(din_all, 80, activation=tf.nn.sigmoid, name='f1_att' + stag)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:395: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  d_layer_2_all = tf.compat.v1.layers.dense(d_layer_1_all, 40, activation=tf.nn.sigmoid, name='f2_att' + stag)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/utils.py:398: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  d_layer_3_all = tf.compat.v1.layers.dense(d_layer_2_all, 1, activation=None, name='f3_att' + stag)\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:177: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "  bn1 = tf.compat.v1.layers.batch_normalization(inputs=inp, name='bn1')\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:178: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn1 = tf.compat.v1.layers.dense(bn1, 200, activation=None, name='f1')\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:184: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn2 = tf.compat.v1.layers.dense(dnn1, 80, activation=None, name='f2')\n",
      "/home/vmagent/app/recdp/demo/builtin/dien/e2eaiok/modelzoo/dien/train/ai-matrix/script/model.py:189: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.\n",
      "  dnn3 = tf.compat.v1.layers.dense(dnn2, 2, activation=None, name='f3')\n",
      "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/indexed_slices.py:436: UserWarning: Converting sparse IndexedSlices to a dense Tensor with 144473832 elements. This may consume a large amount of memory.\n",
      "  warnings.warn(\n",
      "2023-04-03 17:09:34.403551: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled\n",
      "model restored from dnn_best_model_trained/ckpt_noshuffDIEN3\n",
      "OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.\n",
      "OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info\n",
      "OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-95\n",
      "OMP: Info #156: KMP_AFFINITY: 96 available OS procs\n",
      "OMP: Info #157: KMP_AFFINITY: Uniform topology\n",
      "OMP: Info #179: KMP_AFFINITY: 2 packages x 24 cores/pkg x 2 threads/core (48 total cores)\n",
      "OMP: Info #213: KMP_AFFINITY: OS proc to physical thread map:\n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 48 maps to package 0 core 0 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 49 maps to package 0 core 1 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 50 maps to package 0 core 2 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 51 maps to package 0 core 3 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 52 maps to package 0 core 4 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 53 maps to package 0 core 5 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 6 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 54 maps to package 0 core 6 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 8 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 55 maps to package 0 core 8 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 9 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 56 maps to package 0 core 9 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 10 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 57 maps to package 0 core 10 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 11 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 58 maps to package 0 core 11 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 12 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 59 maps to package 0 core 12 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 0 core 13 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 60 maps to package 0 core 13 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 0 core 16 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 61 maps to package 0 core 16 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 0 core 17 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 62 maps to package 0 core 17 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 0 core 18 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 63 maps to package 0 core 18 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to package 0 core 19 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 64 maps to package 0 core 19 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to package 0 core 20 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 65 maps to package 0 core 20 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to package 0 core 21 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 66 maps to package 0 core 21 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to package 0 core 25 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 67 maps to package 0 core 25 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to package 0 core 26 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 68 maps to package 0 core 26 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to package 0 core 27 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 69 maps to package 0 core 27 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to package 0 core 28 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 70 maps to package 0 core 28 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to package 0 core 29 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 71 maps to package 0 core 29 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 24 maps to package 1 core 0 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 72 maps to package 1 core 0 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 25 maps to package 1 core 1 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 73 maps to package 1 core 1 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 26 maps to package 1 core 2 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 74 maps to package 1 core 2 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 27 maps to package 1 core 3 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 75 maps to package 1 core 3 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 28 maps to package 1 core 4 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 76 maps to package 1 core 4 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 29 maps to package 1 core 5 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 77 maps to package 1 core 5 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 30 maps to package 1 core 6 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 78 maps to package 1 core 6 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 31 maps to package 1 core 8 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 79 maps to package 1 core 8 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 32 maps to package 1 core 9 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 80 maps to package 1 core 9 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 33 maps to package 1 core 10 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 81 maps to package 1 core 10 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 34 maps to package 1 core 11 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 82 maps to package 1 core 11 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 35 maps to package 1 core 12 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 83 maps to package 1 core 12 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 36 maps to package 1 core 13 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 84 maps to package 1 core 13 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 37 maps to package 1 core 16 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 85 maps to package 1 core 16 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 38 maps to package 1 core 17 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 86 maps to package 1 core 17 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 39 maps to package 1 core 18 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 87 maps to package 1 core 18 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 40 maps to package 1 core 19 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 88 maps to package 1 core 19 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 41 maps to package 1 core 20 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 89 maps to package 1 core 20 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 42 maps to package 1 core 21 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 90 maps to package 1 core 21 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 43 maps to package 1 core 25 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 91 maps to package 1 core 25 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 44 maps to package 1 core 26 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 92 maps to package 1 core 26 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 45 maps to package 1 core 27 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 93 maps to package 1 core 27 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 46 maps to package 1 core 28 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 94 maps to package 1 core 28 thread 1 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 47 maps to package 1 core 29 thread 0 \n",
      "OMP: Info #171: KMP_AFFINITY: OS proc 95 maps to package 1 core 29 thread 1 \n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30148 thread 0 bound to OS proc set 0\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30151 thread 1 bound to OS proc set 1\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30148 thread 2 bound to OS proc set 2\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30171 thread 3 bound to OS proc set 3\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30172 thread 4 bound to OS proc set 4\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30173 thread 5 bound to OS proc set 5\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30153 thread 6 bound to OS proc set 6\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30174 thread 7 bound to OS proc set 7\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30175 thread 8 bound to OS proc set 8\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30176 thread 9 bound to OS proc set 9\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30177 thread 10 bound to OS proc set 10\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30178 thread 11 bound to OS proc set 11\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30179 thread 12 bound to OS proc set 12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30152 thread 13 bound to OS proc set 13\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30180 thread 14 bound to OS proc set 14\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30181 thread 15 bound to OS proc set 15\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30182 thread 16 bound to OS proc set 16\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30142 thread 17 bound to OS proc set 17\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30183 thread 18 bound to OS proc set 18\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30184 thread 19 bound to OS proc set 19\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30185 thread 20 bound to OS proc set 20\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30156 thread 21 bound to OS proc set 21\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30186 thread 22 bound to OS proc set 22\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30187 thread 23 bound to OS proc set 23\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30188 thread 24 bound to OS proc set 24\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30149 thread 25 bound to OS proc set 25\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30159 thread 26 bound to OS proc set 26\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30189 thread 27 bound to OS proc set 27\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30190 thread 28 bound to OS proc set 28\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30191 thread 29 bound to OS proc set 29\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30139 thread 30 bound to OS proc set 30\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30192 thread 31 bound to OS proc set 31\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30193 thread 32 bound to OS proc set 32\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30194 thread 33 bound to OS proc set 33\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30138 thread 34 bound to OS proc set 34\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30195 thread 35 bound to OS proc set 35\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30196 thread 36 bound to OS proc set 36\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30197 thread 37 bound to OS proc set 37\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30198 thread 38 bound to OS proc set 38\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30199 thread 39 bound to OS proc set 39\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30200 thread 40 bound to OS proc set 40\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30157 thread 41 bound to OS proc set 41\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30201 thread 42 bound to OS proc set 42\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30202 thread 43 bound to OS proc set 43\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30203 thread 44 bound to OS proc set 44\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30143 thread 45 bound to OS proc set 45\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30204 thread 46 bound to OS proc set 46\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30205 thread 47 bound to OS proc set 47\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30206 thread 48 bound to OS proc set 48\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30154 thread 49 bound to OS proc set 49\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30207 thread 50 bound to OS proc set 50\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30208 thread 51 bound to OS proc set 51\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30209 thread 52 bound to OS proc set 52\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30144 thread 53 bound to OS proc set 53\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30210 thread 54 bound to OS proc set 54\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30211 thread 55 bound to OS proc set 55\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30212 thread 56 bound to OS proc set 56\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30155 thread 57 bound to OS proc set 57\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30146 thread 58 bound to OS proc set 58\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30213 thread 59 bound to OS proc set 59\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30214 thread 60 bound to OS proc set 60\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30215 thread 61 bound to OS proc set 61\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30216 thread 62 bound to OS proc set 62\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30217 thread 63 bound to OS proc set 63\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30218 thread 64 bound to OS proc set 64\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30147 thread 65 bound to OS proc set 65\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30219 thread 66 bound to OS proc set 66\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30220 thread 67 bound to OS proc set 67\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30221 thread 68 bound to OS proc set 68\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30145 thread 69 bound to OS proc set 69\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30222 thread 70 bound to OS proc set 70\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30223 thread 71 bound to OS proc set 71\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30224 thread 72 bound to OS proc set 72\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30161 thread 73 bound to OS proc set 73\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30225 thread 74 bound to OS proc set 74\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30226 thread 75 bound to OS proc set 75\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30227 thread 76 bound to OS proc set 76\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30160 thread 77 bound to OS proc set 77\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30228 thread 78 bound to OS proc set 78\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30229 thread 79 bound to OS proc set 79\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30230 thread 80 bound to OS proc set 80\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30140 thread 81 bound to OS proc set 81\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30231 thread 82 bound to OS proc set 82\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30232 thread 83 bound to OS proc set 83\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30233 thread 84 bound to OS proc set 84\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30141 thread 85 bound to OS proc set 85\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30234 thread 86 bound to OS proc set 86\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30235 thread 87 bound to OS proc set 87\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30236 thread 88 bound to OS proc set 88\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30158 thread 89 bound to OS proc set 89\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30237 thread 90 bound to OS proc set 90\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30238 thread 91 bound to OS proc set 91\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30239 thread 92 bound to OS proc set 92\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30150 thread 93 bound to OS proc set 93\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30240 thread 94 bound to OS proc set 94\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30241 thread 95 bound to OS proc set 95\n",
      "OMP: Info #249: KMP_AFFINITY: pid 29913 tid 30242 thread 96 bound to OS proc set 0\n",
      "test_auc: 0.8259 ----test_loss: 1.2329 ---- test_accuracy: 0.756410045 ---- test_aux_loss: 1.0762 ---- eval_time: 27.635 ---- prepare_time: 14.703\n",
      "Total recommendations: 121216\n",
      "Approximate accelerator time in seconds is 27.635\n",
      "Approximate accelerator performance in recommendations/second is 4386.245\n",
      "Process time breakdown, prepare data took 14.703 and test took 27.635, avg is prepare 2.941, test 5.527\n"
     ]
    }
   ],
   "source": [
    "# cp best trained model to dien path\n",
    "# path can be found from above output\n",
    "best_train_model_path = ! find /home/vmagent/app/e2eaiok/result/DIEN -name dnn_best_model | sort -u | tail -1\n",
    "best_train_model_path = best_train_model_path[0]\n",
    "print(f\"best_train_model_path is {best_train_model_path}\")\n",
    "data_path = ! echo `pwd`\"/data/\"\n",
    "data_path = data_path[0]\n",
    "print(f\"data_path is {data_path}\")\n",
    "\n",
    "! cd e2eaiok/modelzoo/dien/; rm -rf ./dnn_best_model_trained; cp -r {best_train_model_path} ./dnn_best_model_trained\n",
    "! cd e2eaiok/modelzoo/dien/; python train/ai-matrix/script/train.py --mode=test --advanced --slice_id=0 --batch_size=128 --num-inter-threads=1 --num-intra-threads=1 --train_path {data_path}/train/local_train_splitByUser --test_path {data_path}/valid/local_test_splitByUser --meta_path {data_path}/meta.yaml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}