WeightWatcher-VGG-ww2x.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# WeightWatcher - VGG  ww2x\n",
    "\n",
    "https://towardsdatascience.com/the-w3h-of-alexnet-vggnet-resnet-and-inception-7baaaecccc96\n",
    "\n",
    "\n",
    "VGG is one of the first, large scale modern architectures based on the classic convolutional model of LeCun ,, resembling a larger scale version of LeNet5.  While there are earlier models (AlexNet, etc), VGG is usefukl because:\n",
    "\n",
    "-  There are several variants, with 11, 13, 16, and 19 layers, and with and wthout Batch Normalization\n",
    "-  It is widely available in pyTorch and other frameworks\n",
    "-  There are versions trained on all of ImageNet, ImageNet-1K (a smaller data set), etc.\n",
    "-  Although large,  is still used in Transfer Learning.\n",
    "\n",
    "The general VGG series architecture consists of \n",
    "\n",
    "-  Several sets of Conv2D+ReLU layers (followed by Max Pooling), with feature maps increasing in size\n",
    "-  3 final Fully Connected (Dense/Linear) layers\n",
    "\n",
    "Below we show the VGG16 architecture, consisting of 16 layers.\n",
    "\n",
    "The VGG series are considered very large models, with an enormous number of parameters comparexd to later models like ResNet and DenseNet series.  Most notably, compared to later models,\n",
    "\n",
    "-  VGG models contain large, FC layers at the end\n",
    "-  VGG does not contain residual connections\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='VGG16.1.png',width=800, height=400)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Image(filename='VGG16.2.png',width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is noted that the VGGNet model (16?) has ov er 2X the number of parameters as one of the largest ResNet models, with far less top5 accuracy.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='CV-models.png',width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of results\n",
    "\n",
    "\n",
    "We first compute the Average Alpha $\\langle\\alpha\\rangle$ for all models, and compare to the Test Accuracy accross the models.  In contrast to expectations, and other models like ResNet, on average, $\\langle\\alpha\\rangle$ is increasing with Test Accuracy instead of decreasing.  In fact, $\\langle\\alpha\\rangle$ is strongly *negatively correlated* with reported Test Accuracy.  \n",
    "\n",
    "#### Why is this ?\n",
    "\n",
    "If we look at $\\alpha$ vs Layer Id, we see that  $\\alpha$ is increasing with Layer Id. That is, as information flows thorugh the network, the layers are less and less correlated.  THis suggests (to me) that the VGG networks are fairly bad at funneling information through the network.\n",
    "\n",
    "Instead, we need to use the Weighted Alpha metric $\\hat{\\alpha}$ , which is positively correlated with the Test Accuracy.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculation of Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T23:51:16.544327Z",
     "start_time": "2019-03-05T23:51:16.531457Z"
    }
   },
   "outputs": [],
   "source": [
    "# Suppress the powerlaw package warnings\n",
    "# \"powerlaw.py:700: RuntimeWarning: divide by zero encountered in true_divide\"\n",
    "# \"powerlaw.py:700: RuntimeWarning: invalid value encountered in true_divide\"\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=RuntimeWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from tqdm import tqdm\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-06T00:16:43.249725Z",
     "start_time": "2019-03-06T00:16:43.240611Z"
    }
   },
   "source": [
    "### Import WeightWatcher\n",
    "\n",
    "set custom Logging at WARN Level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.5'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import logging\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "logger = logging.getLogger(__name__)\n",
    "logger.setLevel(logging.INFO)\n",
    "\n",
    "import weightwatcher as ww\n",
    "import torchvision.models as models\n",
    "\n",
    "ww.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Helper Notebook to make Plots\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "importing Jupyter notebook from ModelPlots.ipynb\n"
     ]
    }
   ],
   "source": [
    "import import_ipynb\n",
    "import ModelPlots as MP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create all models now\n",
    "\n",
    "Pick colors from https://matplotlib.org/3.1.0/gallery/color/named_colors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "series_name = 'VGG'\n",
    "all_names = [ 'vgg11', 'vgg11_bn', 'vgg13', 'vgg13_bn', 'vgg16', 'vgg16_bn', 'vgg19', 'vgg19_bn']\n",
    "colors =    ['indigo', 'blue',    'purple',  'cyan',   'darkgreen','goldenrod','darkorange','red']\n",
    "\n",
    "\n",
    "all_models = []\n",
    "all_models.append(models.vgg11(pretrained=True))\n",
    "all_models.append(models.vgg11_bn(pretrained=True))\n",
    "\n",
    "all_models.append(models.vgg13(pretrained=True))\n",
    "all_models.append(models.vgg13_bn(pretrained=True))\n",
    "\n",
    "all_models.append(models.vgg16(pretrained=True))\n",
    "all_models.append(models.vgg16_bn(pretrained=True))\n",
    "\n",
    "all_models.append(models.vgg19(pretrained=True))\n",
    "all_models.append(models.vgg19_bn(pretrained=True))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "series_name = 'VGG'\n",
    "all_names = [ 'vgg11', 'vgg13',  'vgg16',  'vgg19', ]\n",
    "colors =    ['indigo', 'purple','darkgreen', 'red']\n",
    "\n",
    "\n",
    "all_models = []\n",
    "all_models.append(models.vgg11(pretrained=True))\n",
    "\n",
    "all_models.append(models.vgg13(pretrained=True))\n",
    "\n",
    "all_models.append(models.vgg16(pretrained=True))\n",
    "\n",
    "all_models.append(models.vgg19(pretrained=True))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get reported accuracies from pytorch website\n",
    "\n",
    "https://pytorch.org/docs/stable/torchvision/models.html\n",
    "\n",
    "<pre>\n",
    "<table class=\"docutils align-default\">\n",
    "<colgroup>\n",
    "<col style=\"width: 55%\" />\n",
    "<col style=\"width: 22%\" />\n",
    "<col style=\"width: 22%\" />\n",
    "</colgroup>\n",
    "<thead>\n",
    "<thead>\n",
    "<tr class=\"row-odd\"><th class=\"head\"><p>Network</p></th>\n",
    "<th class=\"head\"><p>Top-1 error</p></th>\n",
    "<th class=\"head\"><p>Top-5 error</p></th>\n",
    "</tr>\n",
    "</thead>\n",
    "<tbody>\n",
    "\n",
    "<tr class=\"row-odd\"><td><p>VGG-11</p></td>\n",
    "<td><p>30.98</p></td>\n",
    "<td><p>11.37</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>VGG-13</p></td>\n",
    "<td><p>30.07</p></td>\n",
    "<td><p>10.75</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>VGG-16</p></td>\n",
    "<td><p>28.41</p></td>\n",
    "<td><p>9.62</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>VGG-19</p></td>\n",
    "<td><p>27.62</p></td>\n",
    "<td><p>9.12</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>VGG-11 with batch normalization</p></td>\n",
    "<td><p>29.62</p></td>\n",
    "<td><p>10.19</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>VGG-13 with batch normalization</p></td>\n",
    "<td><p>28.45</p></td>\n",
    "<td><p>9.63</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>VGG-16 with batch normalization</p></td>\n",
    "<td><p>26.63</p></td>\n",
    "<td><p>8.50</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>VGG-19 with batch normalization</p></td>\n",
    "<td><p>25.76</p></td>\n",
    "<td><p>8.15</p></td>\n",
    "</tr>\n",
    "</tbody>\n",
    "</table>\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "top1_errors= {\n",
    "    \n",
    "    \"vgg11\": 30.98,\n",
    "    \"vgg11_bn\": 29.62,\n",
    "    \"vgg13\": 30.07,\n",
    "    \"vgg13_bn\": 28.45,\n",
    "    \"vgg16\": 28.41,\n",
    "    \"vgg16_bn\": 26.63,\n",
    "    \"vgg19\": 27.62,\n",
    "    \"vgg19_bn\": 25.76,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "top5_errors= {\n",
    "    \n",
    "    \"vgg11\": 11.37,\n",
    "    \"vgg11_bn\": 10.19,\n",
    "    \"vgg13\": 10.75,\n",
    "    \"vgg13_bn\": 9.63,\n",
    "    \"vgg16\": 9.62,\n",
    "    \"vgg16_bn\": 8.50,\n",
    "    \"vgg19\": 9.12,\n",
    "    \"vgg19_bn\": 8.15,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run WeightWatcher, collect summary and details (as dataframes) for all models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                                                                    | 0/8 [00:00<?, ?it/s]INFO:weightwatcher:\n",
      "\n",
      "python      version 3.9.5 (default, May 18 2021, 12:31:01) \n",
      "[Clang 10.0.0 ]\n",
      "numpy       version 1.19.5\n",
      "tensforflow version 2.5.0\n",
      "keras       version 2.5.0\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': False, 'savedir': 'ww-img', 'savefig': 'ww-img', 'rescale': True, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': True, 'min_evals': 50, 'max_evals': None, 'plot': False, 'mp_fit': True, 'layers': []}\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:Using weightwatcher 0.2x style layer and slice iterator\n",
      "Assuming nested distributions\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 vgg11\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      " 12%|█████████████▍                                                                                             | 1/8 [03:52<27:09, 232.82s/it]INFO:weightwatcher:\n",
      "\n",
      "python      version 3.9.5 (default, May 18 2021, 12:31:01) \n",
      "[Clang 10.0.0 ]\n",
      "numpy       version 1.19.5\n",
      "tensforflow version 2.5.0\n",
      "keras       version 2.5.0\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': False, 'savedir': 'ww-img', 'savefig': 'ww-img', 'rescale': True, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': True, 'min_evals': 50, 'max_evals': None, 'plot': False, 'mp_fit': True, 'layers': []}\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:Using weightwatcher 0.2x style layer and slice iterator\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 vgg11_bn\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      " 25%|██████████████████████████▊                                                                                | 2/8 [10:06<31:34, 315.67s/it]INFO:weightwatcher:\n",
      "\n",
      "python      version 3.9.5 (default, May 18 2021, 12:31:01) \n",
      "[Clang 10.0.0 ]\n",
      "numpy       version 1.19.5\n",
      "tensforflow version 2.5.0\n",
      "keras       version 2.5.0\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': False, 'savedir': 'ww-img', 'savefig': 'ww-img', 'rescale': True, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': True, 'min_evals': 50, 'max_evals': None, 'plot': False, 'mp_fit': True, 'layers': []}\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:Using weightwatcher 0.2x style layer and slice iterator\n",
      "Assuming nested distributions\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 vgg13\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      " 38%|████████████████████████████████████████▏                                                                  | 3/8 [14:59<25:27, 305.53s/it]INFO:weightwatcher:\n",
      "\n",
      "python      version 3.9.5 (default, May 18 2021, 12:31:01) \n",
      "[Clang 10.0.0 ]\n",
      "numpy       version 1.19.5\n",
      "tensforflow version 2.5.0\n",
      "keras       version 2.5.0\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': False, 'savedir': 'ww-img', 'savefig': 'ww-img', 'rescale': True, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': True, 'min_evals': 50, 'max_evals': None, 'plot': False, 'mp_fit': True, 'layers': []}\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:Using weightwatcher 0.2x style layer and slice iterator\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 vgg13_bn\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "WARNING:weightwatcher:pytorch layer: BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  type LAYER_TYPE.UNKNOWN not found \n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      " 50%|█████████████████████████████████████████████████████▌                                                     | 4/8 [19:25<19:19, 289.88s/it]INFO:weightwatcher:\n",
      "\n",
      "python      version 3.9.5 (default, May 18 2021, 12:31:01) \n",
      "[Clang 10.0.0 ]\n",
      "numpy       version 1.19.5\n",
      "tensforflow version 2.5.0\n",
      "keras       version 2.5.0\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:params {'glorot_fix': False, 'normalize': False, 'conv2d_norm': True, 'randomize': False, 'savedir': 'ww-img', 'savefig': 'ww-img', 'rescale': True, 'deltaEs': False, 'intra': False, 'channels': None, 'conv2d_fft': False, 'ww2x': True, 'min_evals': 50, 'max_evals': None, 'plot': False, 'mp_fit': True, 'layers': []}\n",
      "INFO:weightwatcher:Saving all images to ww-img\n",
      "INFO:weightwatcher:Using weightwatcher 0.2x style layer and slice iterator\n",
      "Assuming nested distributions\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 vgg16\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n",
      "Assuming nested distributions\n",
      "'nan' in fit cumulative distribution values.\n",
      "Likely underflow or overflow error: the optimal fit for this distribution gives values that are so extreme that we lack the numerical precision to calculate them.\n"
     ]
    }
   ],
   "source": [
    "all_summaries, all_details = [], []\n",
    "for im, name in enumerate(tqdm(all_names)):\n",
    "    print(im, name)\n",
    "    watcher = ww.WeightWatcher(model=all_models[im])\n",
    "    details = watcher.analyze(plot=False, mp_fit=True, ww2x=True, min_evals = 50)\n",
    "    all_details.append(details)\n",
    "    all_summaries.append(watcher.get_summary(details))\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_summaries[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.rcParams.update({'font.size': 20})\n",
    "from pylab import rcParams\n",
    "rcParams['figure.figsize'] = 5,5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'legend.fontsize': 'x-large',\n",
    "          'figure.figsize': (10,10),\n",
    "         'axes.labelsize': 'x-large',\n",
    "         'axes.titlesize':'xx-large',\n",
    "         'xtick.labelsize':'xx-large',\n",
    "         'ytick.labelsize':'xx-large'}\n",
    "plt.rcParams.update(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_details[0].columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_summaries[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "first_n_last_ids = [0, len(all_details)-2]\n",
    "first_n_last_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "MP.plot_all_metrics_vs_test_accuracies( \n",
    "    series_name, all_names, colors, all_summaries, all_details, top1_errors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_summaries[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "MP.plot_all_metric_histograms( \\\n",
    "    series_name, all_names, colors, all_summaries, all_details, first_n_last_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "MP.plot_all_metric_vs_depth( \\\n",
    "    series_name, all_names, colors, all_summaries, all_details, first_n_last_ids)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "all_details[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k, v in all_summaries[0].items():\n",
    "    print('{}: {:f}'.format(k,v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_summaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}