examples/1. Overview deepSI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bcb46811-238a-426a-a41f-0ee0c9922434",
   "metadata": {},
   "source": [
    "# Demonstration notebook `deepSI`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56e82acc-057a-4196-a197-56075a9b85bf",
   "metadata": {},
   "source": [
    "## 1.1 Creating/obtaining train, val and test data in the form `Input_Output_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a951858e-6605-4f2b-80b8-89e5ebbdf41b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train = Input_output_data u.shape=(8000,) y.shape=(8000,) sampling_time=Discrete time\n",
      "val = Input_output_data u.shape=(1000,) y.shape=(1000,) sampling_time=Discrete time\n",
      "test = Input_output_data u.shape=(1000,) y.shape=(1000,) sampling_time=Discrete time\n",
      "train_val = Input_output_data \"train Cascaded_Tanks\" u.shape=(1024,) y.shape=(1024,) sampling_time=4.0\n",
      "test = Input_output_data \"test Cascaded_Tanks\" u.shape=(1024,) y.shape=(1024,) sampling_time=4.0 state_initialization_window_length=5\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "import numpy as np\n",
    "\n",
    "## Generate dataset ## \n",
    "np.random.seed(0)\n",
    "ulist = np.random.randn(10_000) #input sequence\n",
    "x = [0, 0] #initial state\n",
    "ylist = [] #output sequence\n",
    "for uk in ulist:\n",
    "    ylist.append(x[1]*x[0]*0.1 + x[0] + np.random.randn()*1e-3)  #compute output\n",
    "    x = x[0]/(1.2+x[1]**2) + x[1]*0.4, \\\n",
    "        x[1]/(1.2+x[0]**2) + x[0]*0.4 + uk*(1+x[0]**2/10) #advance state\n",
    "\n",
    "# Put the inputs and outputs in a Input_output_data format\n",
    "data = dsi.Input_output_data(u=ulist, y=np.array(ylist)) \n",
    "\n",
    "# Split dataset\n",
    "train, val, test  = data[:8000], data[8000:9000], data[9000:]\n",
    "\n",
    "print(f'{train = }')\n",
    "print(f'{val = }')\n",
    "print(f'{test = }')\n",
    "\n",
    "### load dataset ###\n",
    "from nonlinear_benchmarks import Cascaded_Tanks\n",
    "train_val, test = Cascaded_Tanks()\n",
    "train, val = train_val[:850], train_val[850:]\n",
    "print(f'{train_val = }')\n",
    "print(f'{test = }')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d0907caf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing: you can ignore this if you are just learning deepSI and not testing the module.\n",
    "import hashlib, torch\n",
    "hash_array = lambda ar: hashlib.sha256((ar.detach().numpy() if isinstance(ar,torch.Tensor) else ar).data.tobytes()).hexdigest()[:8]\n",
    "assert hash_array(train.u)=='d2963ee6' and hash_array(test.y)=='fb132046'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c75f28-bc11-4df6-8b42-b93cdcb92fc3",
   "metadata": {},
   "source": [
    "## 1.2 Creating a SUBNET structure \n",
    "\n",
    "Creating a model starts by characterizing by which kind of data it will be applied on. Example: number of inputs (`nu=3` for 3 inputs) or scalar inputs `nu='scalar'`. Futhermore, the range of the outputs and inputs need to be characterized as to apply normalize and de-normalize the of `f`, `h` and `encoder`.\n",
    "\n",
    "If you want to use zero-mean, std one normalization (z-score) you can use `get_nu_ny_and_auto_norm` to characterize what kind of data you are working with.\n",
    "\n",
    "This can be used to initilize the SUBNET method (https://arxiv.org/abs/2012.07697, https://www.sciencedirect.com/science/article/pii/S0005109823003710) given by three realtionships; \n",
    "$$\n",
    "x_{k+1} = f_\\theta(x_k, u_k)\n",
    "$$\n",
    "$$\n",
    "\\hat{y}_k = h_\\theta(x_k)\n",
    "$$\n",
    "$$\n",
    "x_0 = \\psi_\\theta(u_\\text{past}, y_\\text{past})\n",
    "$$\n",
    "where $\\psi$ is the encoder that initializes the state. It is also possible to make $h_\\theta(x_k, u_k)$ by setting `feedthrough=True`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f3416b4f-d5ed-4428-b7c7-12317ea25913",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model.f(x, u) = tensor([[ 0.4626, -1.0500]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "import torch\n",
    "\n",
    "nx, nb, na = 2, 5, 5\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train_val) \n",
    "# Equivilent to \n",
    "# nu = 'scalar', ny = 'scalar', norm = dsi.Norm(umean=train.u.mean(0), ustd=train.u.std(0), ymean=train.y.mean(0), ystd=train.y.std(0))\n",
    "torch.manual_seed(1)\n",
    "model = dsi.SUBNET(nu, ny, norm, nx=nx, nb=nb, na=na)\n",
    "# this model has three components:\n",
    "# print(model.f)\n",
    "# print(model.h)\n",
    "# print(model.encoder)\n",
    "\n",
    "x = torch.randn(1,model.nx) # 1 batch size and nx state vector size \n",
    "u = torch.randn(1) #1 batch size and 'scalar' input size\n",
    "print(f'{model.f(x, u) = }') #the components can be evaluated in desired point\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1ccac259",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert str(model.f(x, u))=='tensor([[ 0.4626, -1.0500]], grad_fn=<AddBackward0>)'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdb0caef",
   "metadata": {},
   "source": [
    "## 1.3 Fitting is a function in deepSI\n",
    "\n",
    "Fitting can be done with the following function:\n",
    "\n",
    "```python\n",
    "def fit(model: nn.Module, train:Input_output_data, val:Input_output_data, n_its:int, T:int=50, \\\n",
    "        batch_size:int=256, stride:int=1, val_freq:int=250, optimizer:optim.Optimizer=None, \\\n",
    "            device=None, compile_mode=None, loss_fun=compute_NMSE, val_fun=compute_NMSE):\n",
    "```\n",
    "\n",
    "where `T` is the trunctation lenght of the subsequences, `val_freq` is the frequency of which the validation loop will run.\n",
    "\n",
    "You can also use `deepSI.fitting.fit_minimal_implementation` which is a much smaller function but is missing some features. This can be handy since this one does not implement learning rate schedulers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "41be12d3-00a6-4945-b1df-c1a40319108e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 816\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ae900c913954f7d848bce2d5229e411",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 1.28713!! nan samps/sec\n",
      "it     100 NRMS loss 0.59047 NRMS val 0.81747!! 3050.50 samps/sec\n",
      "it     200 NRMS loss 0.33494 NRMS val 0.55732!! 3237.22 samps/sec\n",
      "it     300 NRMS loss 0.21476 NRMS val 0.68364   3311.18 samps/sec\n",
      "it     400 NRMS loss 0.19359 NRMS val 0.50856!! 3305.96 samps/sec\n",
      "it     500 NRMS loss 0.17288 NRMS val 0.46514!! 3317.73 samps/sec\n",
      "it     600 NRMS loss 0.16436 NRMS val 0.41950!! 3309.43 samps/sec\n",
      "it     700 NRMS loss 0.15927 NRMS val 0.40306!! 3302.78 samps/sec\n",
      "it     800 NRMS loss 0.14528 NRMS val 0.38296!! 3306.76 samps/sec\n",
      "it     900 NRMS loss 0.14278 NRMS val 0.34408!! 3305.80 samps/sec\n",
      "it   1,000 NRMS loss 0.14716 NRMS val 0.37088   3309.38 samps/sec\n"
     ]
    }
   ],
   "source": [
    "train_dict = dsi.fit(model, train=train, val=val, n_its=1_000, T=30, batch_size=64, val_freq=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb45269f-47bb-4b82-9911-fa563580bd18",
   "metadata": {},
   "source": [
    "Here `train_dict` contains information about the training. More information will be given on this later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "63ee94c6-3c2f-47d5-bc11-4b4b753357b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbIAAADFCAYAAAAxI3fRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABb00lEQVR4nO2dd3hb1fnHP1fTe2/Hjp3p7L0TAiSQhLALFBpmKYUSSgMtZbTlV8oI0JbSUgpdrDLCXoEAISEJISF7D2d6xXtvWZbu749zJVvxtmVJts/nefRc+d4j6eha0ve+73mHoqqqikQikUgkfRSdtycgkUgkEklPkEImkUgkkj6NFDKJRCKR9GmkkEkkEomkTyOFTCKRSCR9GilkEolEIunTSCGTSCQSSZ/G4O0JnI3dbic3N5fg4GAURfH2dCQSiUTiJVRVpaqqioSEBHS6tu0unxOy3NxckpKSvD0NiUQikfgI2dnZDBo0qM3jPidkwcHBgJh4SEiIl2cjkUgkEm9RWVlJUlKSUxfawueEzOFODAkJkUImkUgkkg6XmWSwh0QikUj6NF0Wsk2bNnHJJZeQkJCAoih89NFHLsdVVeXhhx8mPj4ef39/Fi5cyPHjx901X4lEIpFIXOiykNXU1DBhwgSef/75Vo8//fTT/O1vf+PFF19k27ZtBAYGsmjRIurr63s8WYlEIpFIzqbLa2RLlixhyZIlrR5TVZVnn32W3/72t1x22WUAvPbaa8TGxvLRRx9x7bXX9my2AxlVhdpS0OlBbxI3nR5kioJkoFJfCY3iArmmoZG/rz9BSbWFO+YPZUh0kPiO+Id5d44Sj+DWYI/Tp0+Tn5/PwoULnftCQ0OZMWMGW7dubVXILBYLFovF+XdlZaU7p9Q/sNvgpcWQs911v6KDKbfAxc94Z14Sibc4+jm8vQxUOwCBwP2OYxnNxi19Bqbd6tm5STyOW4M98vPzAYiNjXXZHxsb6zx2NitXriQ0NNR5kzlkrVB5pqWIgfgS7/wvFKV7fk4SiTf5/h9OEWuXdY9AXXmvT0fiXbwetfjggw9SUVHhvGVnZ3t7Sr5HTZHYBifAb4vgwTNwfwYkThH7D7zrtalJJF6hulBsb/qU30/eQkr9m9w+dB0fX36YlPo3ma5/B3vEMKivgJPrvTtXSa/jViGLi4sDoKCgwGV/QUGB89jZmM1mZ86YzB1rg5oSsQ2KBoMJzEHgHw5Tfyz273nde3OTSLyBtQ6ARr0/n+zLBeDaaclcNC6e2BAzhTWNnAqdKcZmbvHWLCUewq1ClpqaSlxcHOvWrXPuq6ysZNu2bcyaNcudLzWwcFhkAVGu+0dfLrZVeU1iJ5EMBBqFkB0uaqC0poGwACPzhkdh1Ou4cVYKAO8WJ4uxO/4j3Yv9nC4LWXV1NXv37mXv3r2ACPDYu3cvWVlZKIrCihUreOyxx/jkk084cOAAN954IwkJCVx++eVunvoAorZYbAOjXfebgyBqhLgv3Sfdo6oA1j8Oz8+Ev0+XV+99Bc0i+y6zFoDzRsZg0Iufs2unJWEy6HirKEUbrML+d7wwSYmn6LKQ7dy5k0mTJjFp0iQA7r33XiZNmsTDDz8MwK9//Wt+/vOf89Of/pRp06ZRXV3NF198gZ+fn3tnPpBwWGSBUS2PpS0V21PfeG4+/QVrHfxjBmx6GoqOQHE6vH8b2DsRRCDxHqoKViFg35yqBmDBqBjn4cggM5dNSKCSID6Juk3sPPqpx6cp8RxdFrJzzz0XVVVb3F555RVA1MT6wx/+QH5+PvX19Xz99deMGDHC3fMeWDjchq0J2aDpYpu9TXzBJZ1n3yqoK3PdV5kDhz/yynQkncTW4IxYPFLciEGnMG+4q7fiptkpAPwld4zYkfGdyMOU9Eu8HrUo6QRtrZEBDJ4NxgAoOQF5ez06rT6NqjYFyYy8CH5fAfO1TKT1j3lvXpKO0awxgDpMTE0JJ9Tf6DJkbGIo01MiOG2PoSBgOKg2SF/j6ZlKPIQUsr5AW2tkICoXDJom7hcc9tiU+jwVOXBmJ6DAkqfFvll3gaKH0pNQdMyr05O0g1VU87ChpxEDs4e2coEH3DovFYD3a8UyCEdXe2R6Es8jhawvUOMQsta/sEQNF9sSWZy50zgSzGPHQpiWhO8XAkPmi/sbVnpnXpKO0SyyekwATB0c3uqwhaNiSYkM4BPLZLHj5HqwVHtkihLPIoXM11HVTgiZtgZZLIWs0xz8QGyHnue6//zfiW36GrA1enZOks6hRSzWqib0OoUJSWGtDtPrFG6Zk8pRNYlcXbyoy3jiaw9OVOIppJD5Og01zpyZVtfIoMkik0LWOVQVsjWLLO1i12PxE8AYKM55yQnPz03SMZqQ1akmRsUHE2huu2TsJRMSMOh0fNqgWWXSvdgvkULm6zjWxwz+YApsfUykJmSlp8Bm9cy8+jKVuVBTKNbD4sa5HtPpIVH70ZNX776J5lqsw8zENqwxBxGBJuaPiOZLm7aOfOwraGzo5QlKPI0UMl+nuVuxrZYtIYkictFuhbJMz82tr5K7R2xjRoMpoOXxUZeI7ZFPPDcnSedxWGSYGBEb3OHwSycmsEcdRokSDpYKaZX1Q6SQ+TodrY8B6HQQOVTclwEfHeMQsoSJrR8feZHY5uyAhtrWx0i8h+Zqt2BiWExQh8PPHRmDXqdnjVWztN+7xSmGkv6BFDJfp70csuY4Az5k2HiHOIVsUuvHQwdBUKxIus3a6rl5STpFQ30NAHWqmeExHVtkof5GZg2N5A1bU59ETspKOP0JKWS+Tm07VT2a41gnkwEK7aOqHQuZosCIReL+vlWemZek05SVVwDQqDcTFWTq1GMuGB3LEXUwnweIzvWsug4q88gormFHRil2u6yK05eRQubrWLSO2X6h7Y9z5EJV5vbufPo65ZlQVwo6I8SOaXtcmrZOlrfPM/OSdJqqKvGd0JkCUdpaNz6LhaNEs9/HyppZZc+kcdmfP+PqF7dyyys7qLfa3D5XiWeQQubrWKrE1tyBCyUkQWylkLXPmd1iGzsGDOa2x8WPF9uS43KdzMeoqRbfCb25lUCdNkgI82dcYii5aiTFYROc+/eZb2Od6ZcknXyTxz874va5SjyDFDJfp9NClii2lWd6dz59naOfiW1yB/3xguMgMEaskxUc6v15STpNba2ozmHyayMdpQ0uGC2sst+EPUW90tSNY6guj8eML/Po3jl8tX2/+yYq8RhSyHydrlpk9RWyDE975OwQ27SLOh7rqGF5TBab9SUa6sTn28+/a0J24RghZF8eLSWt7iWutD5Ko79r/dILP59HwaaX3TNRiceQQubrONbIzCHtjzMHN42pyuvdOfVVGi1QniXuR43sePworepHpoxc9CWs9cLV6x/YccRic0bGBjM5Ocz597xzF2O4/wQ8eAZ7yjzn/rD1v6Y0J90tc5V4Bilkvk4rFllmSQ0rVu3hsdWHqWtotkDtXCeT7sVWKT0NqGAKhqCYDocTp62TFRySvd58CLu2ZhkQ1DUhUxSFF2+Ywm3zUvnt0lHcvUCL9DUHobvpUyov/rf4kwYi/jMdtb7CrfOW9B5tFymT+AZnCVm91catr+7kRKFwr9Q0NLLySu0HNyQBio7KgI+2KD0ptpFD266S0pyoEaAziGoQFTlNkaESr6I01oICAV20yABigv34zdLRrTypQsjUa8hrsBH/1R0ApH/xImmX39/T6Uo8gLTIfJ2zhOydndlOEQN4d2cOhZWiP5O0yDrAkWPnqILSEQZTkwuy4GDvzEnSJeqtNkx2C9A9IeuI+NnXcTByMQD2w5+6/fklvYMUMl+nmZCpqsprW0UtxUcuHcPUweE02lXe3ZUjxjgjF6VF1iqlp8U2YkjnH+PINZNC5hOU1DTgp4iiv/5dDPboLNGXP0aDqmd0wwEKDsoKIH0BKWS+jK2xqa27OYT0gipOFFZjNui4cnIiV04eBMC6IwVijMwla58yTcjCUzv/mLixYntmj/vnI+kypdUN+CGETGmt4LMbiE0azvcB5wKQv+PjXnkNiXuRQubLNFQ13TcH8/VhIVjzhkcR7Gfk3JEidHhPdjmlNQ0yl6wjSk+JbVcssiHniu2Jr2WhWR+guMaCP8K1iNG/914oVXQK98/7vvdeQ+I2pJD5MvVa6L3BH/RG1h4pBGCBVm4nIcyf4TFBqCrszCiVFll7NDaIgA2AiK5YZONFwWabBQoO987cJJ2mtLoBf80iw9g7FhnAoEmilFWK5RjWepmX6etIIfNlmq2PldU0sC+7HIAFaU2h45OTwwHYm10OwfFiZ20JWOs9ONE+QEW2qNJhDBCV7TuLooiu0QB5e3tlapLOU1rTgJ/S+xZZypBRFBCBSbGRsW9jr72OxD1IIfNlmgnZnuwyAIZEBxIT0lReZ5KW4Lknqxz8w8GgHZNJ0a443IrhKZ0LvW+Oo+5ivixf5G1KaxvwQ+uC3otCptPryAicCEDFUSlkvo4UMl+mmZDtziwHmiwwBxM1IdufU45Npckqk0LmSnciFh3EjRNbWXPR61TXWZvWyAy9uEYGNAyaCUBg3vZefZ0OUVWwWcXnz9FoV+KC24XMZrPxu9/9jtTUVPz9/Rk6dCiPPvooqqyM0HWc5amC2ZUpLLKzhWx4TDCBJj01DTaRXybXyVrHGbGY0vXHRqeJbdExWeHDy9Rb6jEodvFHbwZ7AOGjzgUgtf6QWGP1BjUl8NJieDQKXpgNfxwKx9d6Zy4+jNuF7KmnnuKFF17g73//O0eOHOGpp57i6aef5rnnnnP3S/V/NIvMbgpmX045AJMHh7kM0esURieIGotH8yulRdYWTousC4EeDiKHNVX4KDnp3nlJukRDXU3TH70Y7AEwZNRkStUg/Gig9KQXrDK7Df53OWSfFTn5xlWw9y3Pz8eHcbuQbdmyhcsuu4ylS5eSkpLCVVddxYUXXsj27V42z/simpBV409tgw1/o77V1u7DY8W+4wXVEKIJWaUUMhcc5am6kkPmwGCGlLni/omv3TcnSZdpbBBCZlf0oDf26msFmI0cNoo8wrLDG3r1tVrl+380rcvGjYeZdzZ5FDY9LQO6muF2IZs9ezbr1q3j2LFjAOzbt4/NmzezZMmSVsdbLBYqKytdbhINTcjKGkUDyOGxQeh1LQMVhscEAXCsoErmkrWGtb7JkooZ1b3nGDxHbB1tYCReodEiCgTY9X5dD9rpBkURUwDQZXu4A0Lxcfj6EXF/zgq441tYvBJu3ySCukpPCTGTAL0gZA888ADXXnstaWlpGI1GJk2axIoVK1i2bFmr41euXEloaKjzlpQkC7M60YSs0KoJWSvWGMAIh0VWWC1di61RchxUm/gBcJyfrpIsFv45uV5UXJF4BbtDyAx+HYx0E4NnAxBbvke4+jzF+sfAboWY0TDvl037/UJh6Z/F/b1vyTVbDbcL2TvvvMMbb7zBm2++ye7du3n11Vf505/+xKuvvtrq+AcffJCKigrnLTs7291T6rtoQpZbJ5oUDI8NanWYwyLLLKnBEqDlSEnXYhOOwJew5O5fxSfPBnMo1JVCoUyM9haOFi69HbHoIH7kNKpVPwLsNZ77v+cfhMMfifuX/A38zupFOHIp6IxQldvUX2+A43Yhu++++5xW2bhx47jhhhu45557WLlyZavjzWYzISEhLjeJhha1mFMjhGxYdOtCFh1sJtTfiF2FTGuo2FmVB3a7R6bp81Tli21QXPefQ29oyifLlXUXvYXdUSaslyMWHYweFMEu+wgAao5/65HXZOdLYhuWDIlTWh43+jUVs5ZrtkAvCFltbS06nevT6vV67PJHtetoQpZdK4RscGTrUVqKojA0WlQCP1EbBCjCLVErc06AJiEL7kJFj9bQ3Ewc/axnzyPpFqqqOutdKh4SshA/I5l+Iv2iMnNv77+gqsK+VeL+4qdA18ZP9Pgfiu2W56R7kV4QsksuuYTHH3+czz77jIyMDD788EOeeeYZrrjiCne/VP9Hcy0WWU0ADApvO9w4KUIcy6qwNpVgkgEfgmo3WGQAaUvFNut7ae16gXqrHZPqqHzvGSEDsEVqPekKj/b+ix36EKxaisGgqW2Pm3KzcK+WnYai9N6fl4/jdiF77rnnuOqqq7jzzjsZNWoUv/rVr7j99tt59NFH3f1S/R+taHAN/kQFmfE36dscmqSJXHZpLYRqkYsVUsgAqNLa3AT3UMhiRosSYJaKppJXEo9RbWl0FgzWe8giAwhKEm680OqTPbZ+9mWXc8vL21n5+RFs9lae69QGsY2fAEExLY87MAVA0jRxP3Nzj+bUH3C7kAUHB/Pss8+SmZlJXV0dJ0+e5LHHHsNkMrn7pfo/9eUAlKtBJEW0/8VN1iyy7LI6Wd3jbBwRnD0VMr2xqVzVaVl/z9PUWBqdTTU95VoEGDRsPDZVIcBe3eSm7gZ1DTZue20n36QX8c9Np/jf1gzXAXZb05rXuQ91/ISDtdzGzC3dnlN/QdZa9FVUFepEWaoyNahdtyLAIE3ockprIUQ03KQyp1en2GdwCFlPXYvQlBi9u/UoXEnvUW1pdDbVxOih8HtgdHIMJ1VxcVh9eme3n+eLQ3kUVlmcf7/0XYarVbZvlVgO8Atr6oPXHilabmPGdwN+nUwKma9iqQK7yFcqJ4jYYHO7wx2uxZyyOuzB0iJz0lDbJGTdKU91NpNvFNu8/d6rvzdAqXERst4tT9WcUH8jJ00i4KP0WPetn03HRPDVzbNTCPEzkFVay/bTpeKgpQo+vlPcn36bi1A3NNoprGylikfiVDAFiTXgk+u6Pa/+gBQyX0WzxqyKCQsmIoPaF7L4UD8MOoUGm50Kk+gcLdfIaCoW7BcqEqLbIL+inn9uPMmWkx1EeoaniudChb2vu2+ekg6paWhyLeKphGiNikjRk045s6tbj1dVlc0nxGfrwtGxLB4rvAOfHdAuNr9/wTn2v7XzWLU9i8Kqej7ck8Pcp9Yz/Yl1rFi1B3tzC87oB6MvE/ezB3YJQIO3JyBpA03IqnUidywysP01RoNeR0KYP1mlteSpkYSDjFqEpqCMiKFtJkNX1Fq58h/fkVshrnpfvH4yi8e2UQFEUWDSDbD173DsS5j6496YtaQVqi02j/Qiaw1j8nQogKjKgyJita2w+DY4VlBNUZUFP6OOKSnhNNjsvLMzhy8O5vPIpWPR734NgE/sc3h0czVwoMVzfLQ3l3NGRHPl5EFNOx35ZAO8xZC0yHwVLdCjElF+KjKo42CZxDBtncwWJnbIpOimGovt9CH701fpThEDePzzIzQ0tnPexl8jtie/EW02JB5BuBYdvcg8a5EljJhMrWrG314jSp51kW+PFwEwPTUSs0HPnGFRhPobKa5u4Ni374kO5sCfrT9gbGII4weJwgbBfgbuWzSSexaKpOy/rTvu2hJrkBa5ePIbsFT34B32baSQ+Sp15QCU2cVaQEQHFhlAXKj4cmdaQgAFbA0yKdphkUUObfVwRa2V93aJoJj/3DiVqCAz2aV1fH2koO3njBsP0aPAZoEsGTHWGSpqraz8/AhPf3GUakv3alXWeCnYA2BsUgQHVLHGWn3y+w5Gt8ThVpw3LAoAo17H4jHCvWjbLVzUa21TqAlM5s3bZvLJXXPZ9/CF7PndBSw/bxi3nZNKoElPRkktu7PKmp540DThbbDWwNHVPXmLfRopZL6K5lostQkhi+pgjQwgNkR8uXOrbTIp2oHTtdi6RfbOzmzqrDbS4oJZMCqGa6YKt80Hu9uJ+FSUpmRVWQ2/U9y9ag//3HSKf2w4ya/e2det56huFn7vyWAPgGA/IxlmEfBRfrxrlfAtjTa2nRJBHXM0IQNYOj6ec3V7GVuxAYD/2pZw36KRhPiJ9jShAUYMevETHWAysEhbV/t4b7MgLkWB0ZeK+6c3dfl99RekkPkqmmuxTBVf2M64FuNChNgVVNbLpGgH7QiZza7y2vcZANw0OwVFUbhikjhvm44VU9Oe5ZA6X2z3vOFMXJe0zs6MUjYeK3L+/cWhfE4Wdd0NJiwybY3Mw65FAEvsZPHSebu79Lg9WeXUWW1EBZlIi2vqYDFraCS3mb4EYI1tGvUJs7h6StvdPy4eL9ZtvzpU4OpeTNZKpw3gfDIpZL6K5losV4PwM+oIMHUcl+OwyPIr6pslRQ9gIbPWNb3/iJauxXVHCsgurSPU38jlE4WADYsJYnBkAA02O9+daMctO3IJBEQJ1+3xr3pj9v2Gv64Ta0rXTkvi/DRRraJdi7cNRLCHwyLzbLAHQPiIWQBE1x4XaR2dZPNx8TmaMywKXbN+gsY9rzIHYZ2+H3ANz/1ossvxs5k9NIoAk578ynoOnKloOpA0HVBEhG4PErb7MlLIfBXNIqtQA4kM7NitCBCrrZEVVFogVLuyqxjAbXFKtdB7cygERLQ4/M9Nwlq7bnqys/yXoiicN1L82H6TXtj2c5uDYMK14n6Gh6qi90F2ZZbx7fFiDDqF5ecN47IJcfhhYd2Rds5tGzSv7OENi2x02igK1DD02LHmdL4DwrfaBdHcZm5F6sph9QoA7MMu4N8P3Oasl9oWfkY9544UqTVfHWq2husfBrGikzVZHm4A6iNIIfNVNIusgkCiOuFWBIgLcQhZPXZHdY+KAVzdo1h0KSeyZej9joxSdmWWYTLo+PGcFJdj52lWwzdHi1xdOGfj6Bp9aoOMDm0FVVX5y1rxP7hqUixJm37FZR+P5ajfLbxetozqU13LfXKJWvSCRTYkOoh0RVj2+cc6tzZaXtvAgZxyAOYObyZk6Wucd3WLn0TpZJ+8C0eLdbK1h88KRhosrEUypZBJfAkt2KNCDexUxCKIvmSKAo12lWo/rRzTQBYyRyPEmNEtDr2wQYTl/2DyIGJCXK/uZ6RG4G8ULpxDue2sf6XMBVMwlGVAuo+1djn8CXy0HJ6fCb8PFbdXLobqoo4f6yZW789j84lijHqFB/0+gL1vOI9FKZUEvXYBZG3r9PNVe3mNTFEUKkNFwEd11v5OPebrI4XYVUiLCyY+VBNfux22Pi/uz/slRA3r9BzOGxmDXqeQXlBFRnFN04FkTcgGaBStFDJfxeFaJLDDqh4OjHqdM7qxUOeo7iGFjFhXIUvPr2L90UIUBX56TssgED+j3nn1vP5oOy4wvxCYdqu4v/8dt0y5x6gqvHY5vHODqDxSdKTpWMa38M6NXVrf6S6r9+dyz9t7AbhtdjKhh98UB8ZexcexdzYN3L+q089Z0+CdElXNCYwXoqOWZXRq/JeHxJrVojHN6nxmboaCA+IiaMYdXXr90AAjM4cIN7mLVebolZd/EOorWnlk/0YKma+ifRjFGlnnOwc43Iu5qiZkVfkDtyZgQesW2T83Cmtsydg4UqMCW33oAs29uK69fDKA4ReIbc4O77sXVRW2/xtOfeO6v3mx5Kwt8ER8ryZyf34gj7vf2kOjXeWyiQn8MmY71JWCfwRc8U8KxvyEnzbcIwanrwFb5/LKaiy2ZuH3nrfIABIGC4ssuO5M+25nhCt0kxat6ShJBcDhj8V29KXtt2ppA4d78avDzQI7guMgchigwumBt2YrhcxXaRBug1r8OhV67yBWC8HPsQSA3gSoTY0lBxI1xVCqVfVwtF4Bcspq+XifyMO5Y37rSdKAM7puX05F6wVbHSRMFlfWVXmw66Wez7sn7HwJ1twn7qeeA7+vELdfpYvtwkeaxv5xSK9YZtmltfzq3X3YVfjh1CSeuTQF/YaV4uDce0BvYERsMBvsE6lQtPPWyYK3LtXvDZ5fIwMYPGoKdlUhiXxyMk+2O3bjsSIsjXYGRwY0hd1b60XKBsDYK7s1hwtGixzRnZllFFc3VdN3VswfgGH4Ush8Fa2ley3mTkctAkQHiyvVwmoLBGpXex5cF/EZHJGEMWMgsGmR/X9bM7HZVeYMi2T8oLA2Hx4T4scErUxQu+5FU4CoVg6w/T89nXX3sdbDZ/eK+0GxsOy9lmPmroDFTzb9/cpFbp/GvzadorbBxtTB4Txx5Tj0u16CmkIITYYpNwEwMi6YBoy82zhPPEirM9gR3qzs4cAvLJaTxuEA5OxrX4C/OCguIBePiWsK5jizExrrhJU8dEG35pAQ5s+4xFBUFVbva5YcnSDy3MjrXsJ5X0YKmS+iqk6LrE41EdEFiyxGa/dSWGVpclvUdD3Uuc/jqHKQeo5zl6XRxjs7RTrCzbM7bumyYJS48v26o1DxOXeDziDWo0rav0rvNZoXjb1lDRjauPiZ+TMYpVWCyN3j1qv3ugYb7+4S5/feC0eg1ymw+3/i4HkPaV0DhPs70KTn08aZ4lhWxyWf7HaVugYrfooj2MM7FhlAbaQo1Fud1bZgWBptzgugRc3dihlaN+eUuW0Wse4MV00RUclvbMtqcnEmTBTbvH3ed3N7GClkvkijBRAfzjrMRHXBInMkRRdW1jcJWXUH6zz9EYeQDZnv3PXFwXzKaq3Eh/pxnpaP0x4LRonzt/lEUfv1Af3DmxpueqveXbYmBsMWtllX0sk1r0HSDHH/nRuFoLmB704UU2+1kxjmz6whkaIfXtlpUHSQttQ5TlEUkiICSFeTUFFEUnkHXoOahkbMjohF8Er4vYPgwRMB8C892uY62ZYTJVRbGokNMTOxueXvKGmWPLNHc7hiciL+Rj3HC6vZkaHVXowaKaI5G6q6Vdi4LyOFzBexNq1d1GHu0hqZi0UWqP1YDzTXoqUaSk6I+44fbEQQAoirWUcNu/YYHR/CkOhA6q12Pt+f1/7gtIvF9qiXwvAPfSS2Q87reKyiwHWrRBJtTRG8/gO3fEbWHRUXTAtHxQhXmuNcJEwWEZ7NSI4IoB4zVQFa4r4jwrQNahuaVfUArwpZYpqoOD/EfppTzUPgm+FwKy4aE+daraMoXWwdCczdJMTPyOWTRPWe17/PFDv1Bq3KB6Ia/gBCCpkvogmZRTVgQ9/pPDKAGC3Yo7ByALsWHaHR/uHOih51DTZnvT+XUOh2UBTFWfvO4ZJsE4fFkb3dmQPoMcoyIGe7cG+Ou6pzjwmIEC7IsGSoLYFXlop1tm5it6vOah0Ol6zTZThySYvxyVoVi1xTithReKTFmOa4BHrojKDTd3uuPcUcL6JgE5RStqe3/Fw02uys1aJdFzf/rFmqmirtRI/s8TyWzRgMwJqDeU1BH46Aj+yO3bX9CSlkvogW6FGPiSCzAT9j57+0MVqwR1G1BXvgAHUtOqIVw1Ocu749XuR0e41JCGn9ca1w5eRE9DqFnZllru0zziYkQavnqHZqzcetONa5EiaLMOzO4hcCV78q3FHF6V3K6Tqbg7kVFFZZCDTpmaHlOZGvJQ3HT2wxPjlSCNkJksWODiwyl/JUXrTGAPAPp84g1vtOH2uZGL0jo4zSmgbCAoxMT21WGs1hjQXFtloyrauMTQxlQlIYVpvadKGVOEVsu9nJuq8ihcwXcQR6dNGtCBAVZEJRRGX3aoP2ZRlorsUzWnXyuPHOXV9pyaMXjontdDkgEGuOV2oV8Z/56lj7uUNDzxfbDU+C3da1OfcE57rLjPbHtUbiZDj3AXH/+xeE1dANHAEx84ZHYzboxWe4WFunaZb+4GBQuBCjA1atuHWnLDLvVfU4G1uESIyuyT6Aze76mXAkQV8wKtbVhe24wGml0kx3WTZdXAh84mjtkjBJbMuzRArKAEEKmS/iCL1XzV1yKwIY9DpnuH4J4qqRmgEmZDk7xVbrnttoszsTmzvrVmzO3QuGY9ApbD5RzJvbs9oeOPNnYpu3t8ka8QSOq+/Eqc5dlkYbn+3P447/7eJnr+9qvwDymCuFOBQdhTX3d2sKjvPrCJARSbmqsD6CY1uMd5Rr2lUnWpNQeERE67ZBjcWGv7POoveFzD9VXDSMtB5lTzNL3W5Xm8Lux571WXMEIDmS6N3ABaNj0SlwNL+K3PI6ERkaI6IqvbZe6wWkkPkiVmGR1Xcxh8yBM+DDpiVhDiQhszVCrmaRaUK2I6OMslor4QFGpg4O7/JTJkUEcM8FotX8bz48yI0vbeePXx5lzYE86q3NLK/IoRQlCKvs0Hef9ux9dBZrHaoWer/0g1oueGYj1/9nGzOfWMfyN3fzxaF81hzM55aXd/Dnr9JbtyjDB4vgD4D9b0N119ZUs0trOZRbiaI0FVzmiPb+x1zR6mMcFWj21Uai6owi0q6dTg01Xmyq2Rr6RJGzlabLclr7ANszSsmvrCfYz+BaJBigSrOaIoe7bR7hgSYmJYvPtPNixdGVoVl+XkFlPWfK69z2ur5GrwjZmTNnuP7664mMjMTf359x48axc+fO3nip/olLMnTXLDJoqu6R2xgkdtSXD5wyVYWHRbCMOQSihPg4XD0Lz3b1dIGfzR/KbfNE7tmmY0U8/81JfvbGbmatXMezXx9j++lSfvLqTl7IFG7Ikv1fdRwg4g5KTqDYGylTgzhUE8Lxwmo2nyimrNZKbIiZ2+cP4fqZwv303PoTPKNVo3c+vNrCn79K58F9UdREjgN7Y5dTCD7TokFnpkY2dTIv1PLaHDUAzyIswIjZoMOKgcbQwdpk2s7Bq7Y0YnZW9fC+RUaMKFU1Qsnhy4N5zgsER/fmi8bGCxdrc6o0wWvFQu0J5zfr1gDAuKvF9swusFRx8EwF5/5xA/Of/ob1R/vnennH3Rq7SFlZGXPmzOG8885jzZo1REdHc/z4ccLDu34lPGDRSgfVqaYur5FBU8BHTp0ZFD2oNhGZFhLv1mn6JI71osQpoNOhqqqzuOqF3XArOtDpFH6zdDTXTE3iuxPFHCusZsPRQnIr6nn26+M8+7VYDxqjF2HV03TpnPPZfpaMjSNYa13fG1TkHCUUOK3G8cilYxkcGUBhlYWk8ACmpYQ7hXtEbDAPf3yI59afINTfyE/mDeFofiW3vrLTeaUeYxjJPYYDsOlPMPF6MLT87FVbGnl/Vw5H8yuZOjiCpePjeXObcLdePEH7fNVXQP4Bcb/ZOmVzFEUhPtSPjJJaqgOSCC87IXLOaD19wKU7tLeDPQAih6MqOsKoobY0j+2nSxmTGMrq/ULILp2Y4Dq+saHJMxLU/c9ha8wfEc0fv0xn68lirDY7xpB4EY1angX73+avRyZSp3kO/vzVMc5Pc6+Q+gJuF7KnnnqKpKQkXn75Zee+1NSOqyhImqGF39dj7nTl++Y4QvALqhtEeabqAvElGhBC5ro+dii3kjPldQSY9Mw729XTDYbHBjM8VrhsbXaVzw/k8d/Np8kqrWXq4HDuu3Ae6v/+hH9NEUPqj/DR3rHcMHNwj1+3LdIP72E6UO4/mBtnDW4zkOXGWSlU1Tfyxy/TeeyzI2xIL2JnZin1VlELMDUqkP+kL2GZ/mtiKs/Ahiconvkg/9uayb6ccvSKgl6nsPVUCVX1Ijn8re3Z/OajA9Rb7UQFmbhCC4ohe7uw7CKGQETb3/3YECFkpeZEwqGpEWoruJSn8gWLzOiHEp4KpScZrsvhxY0nmTkkkqr6RoZEBYqE8Obk7ABU0VU8sONk/K4wOj6EyEATJTUN7MkqF5GSk2+E9Y9hO7yajccTnWMP5VaSW15HQpgPXAy4Ebe7Fj/55BOmTp3K1VdfTUxMDJMmTeLf//53m+MtFguVlZUutwFPD12LzjWyymZJ0QNlnczRTFOLlHMkQc8fEd2lNIbOoNcpXDIhgY+Wz2H37y7gXzdOZXhcCIqWlHyVfiPv7Ohd96IlX7zf0EFpHUZj3nnuUJafJ6p+bNaqcMwbHsXHy+fw8s3TWDJlOI9YbxSDN/+Fnz39H/667jgb0otYd7SQrw4XOH+ob5g5mIhAE/VWUQrpoYtGEWDSrouLjoptG9aYg3ito3m+XrvAKmtbyKpdKt/7yI9wtHAvjtKf4Zv0IlauEe/7jvlDXZOgoWnddvBs0Ln3Z1enU5itdZ/efFz7no9YDICSsYkE2xkGhfszNlGknbSbRtJHcbtFdurUKV544QXuvfdeHnroIXbs2MHdd9+NyWTipptuajF+5cqVPPLII6080wDG2lRnMbE7rkVHmaoqCwRrVshACcV1/BhGpNLQaOfdXaIf2yUTEtp5kJuZdD0ceId5+gPcd6aCnLJaBoW7P0Ch3mojvOYEKJA4tONKEYqicN+iNBaOiuX7U6UMjwni/LQY54/uk1eO4846K18c38pi/Q7e1T3EpYmruXpaMkadQm2DjZFxwcwaEolOp3D/kjTWHSkgNSrQtQCzwyrWfujbIk6LXMxUY5gDTYnsrVBjaSTAi92hWyUmDdI/48pBVfxX+9gtHBXjrIPogiMVoYNz0l3mDY/i0325bDpezL0XjhSVQ1Lmocv4lqv1GykaNQtVVTl4ppJdmWVcPN6D3wcP4HYhs9vtTJ06lSeeeAKASZMmcfDgQV588cVWhezBBx/k3nvvdf5dWVlJUlKSu6fVt9AssrqeRi1W1kOcwyIbANU9akvFWiBA2GC+OpxPUZWFmGCzs/WFR0icAijEKWVEU87awwXcMsf97vV9R44xQzmNHYXYced3+nGTksOdkW7NMeh1/P1Hk9m6+lrYK9YaP74uHiWqdddokNnAZRMTXXfa7XB8rbg/4sJ25xGnucCPN2gXW6UZIgS/FcuyuqGRCGcemY8ImWZxjmk8wnt3zKK0poHzml0YuOAomRblvojF5jjc5vtzyqmotRIaYESdeitKxrdco9/IsSGPUNxo5tWtmezLLu+VOXgTt7sW4+PjGT3aNeFv1KhRZGW1nn9jNpsJCQlxuQ101B4kREOTRVZUbUH1136wPF02yRtkbxfbyOHgF+KsQXfttCSM3YxW7BbmIGfE5BhdhjNq0t2cPCzyx0pMiShdqejRDiaDjvmX/wRCxcWk8vb17eZ3taA8U3gU9GaIm9DuUIdFdrA2HFBECL7jQuQsXMPvfWCNDJo6KxQdYWqMCCZq83PmsMgih/XKVOJD/RkWE4Rdha2nhPflaMgcctUIopUKptd8wwhtbfdkUU2HTUH7Gm7/ds+ZM4f09HSXfceOHWPw4N5b8O5vNNRpTTVVM+EBXReyaC1AxGpTqddK6QwIIXNUcR80jeMFVXx/qhS9TuG6Gcmen0u8+BEfr5xiV2ZZ+9Xzu0lJtqiGYQvrhWCqpX8W26IjXQvHd5Saihohiti2Q5y2RpZTZRclvqBN92KNr4XfgygzFdqJElv1FU0ekV4SMoC52jrZpuNCyDacqmJVo7DUjUc/JjUqEEWBijorJTX9Kx3H7UJ2zz338P333/PEE09w4sQJ3nzzTf71r3+xfPlyd79Uv6WhrlrcMfpjMnT9X2Qy6JwVQSoVLZdsIAiZI+Q7fjxvaCHhC9JinFUkPIoWNTnbfAqrTWXbqdYtje5SWtOAsUJYnCGJI9z63ACMWARTbxX3375euG07g6M8WEL71hg0JUUXVllQw7QL3TYiF6stzarf+8oaGUCs5n0qaEfIijW3YlBciy4A7uScEULI1h8pxGZXWXekgA/sc7EpBji9Eb+cLSRpa7UnC6t7bR7ewO1CNm3aND788EPeeustxo4dy6OPPsqzzz7LsmXL3P1S/RZrvbDI9ObuBwg41slK7YFiR115T6fl+2hloSxRY3hfC/K4YZaXPAFaO40JHEPBzrfH3Rtss+VkMYMV4bIMiO0FIQM4576m+1/+pnOPcUTnOWr+tUNUkAmdVhe0PlizbNqIXKyxNOLvaxYZNNVNLDjY9hhHJG0vrY85mDMsirAAI/mV9fxr0yl2ZpaRp8RSP/4GMWDr8wyNFr8HJ4tabz/TV+mVhYOLL76YAwcOUF9fz5EjR7jtttt642X6LTaL+JAZ/IK6/RyOdbLiRu3qtb9bZLWlzhJHmyrjqLI0khDqx5yhPc8d6xaxY8EYiL+9mmFKLt8ed2/6w3cniklRtCoNEUPc+txOQuJh1l3i/oF3ofRU++OtdZCj1X1MmNzh0xv0OqK1C65KU/udGnytRJUTrVQVx75su1C0o6VKB+kIPcVs0HPVZBEx+dQXIhVg4agYAmf+WAw4vYnhkeJ8n5AWmaS3cQR7GP0Cu/0cCY4cnTotd6qx+72m+gSOK+KwwXxwWFRwv2RCQusRZJ5Ab4BBoqXGNF06J4tq3FbrTlVVvj1WxODeFjKARY8LN6ndCulftD/2+FdgqYCQxFYr3reGw71Y0ahVP2mjJ1pV84RoXwn2ABi+CIyBUJ3fdoktRzpCG+W63MnPzx9Oopbs7G/Uc+8FWih+QBRYa5huEnM8USSFTNLbaOH35oDgbj9FkqNxYbVIWO33QpYn3IrWmLGsOyoW1j2aO9Ya0aMAmBQihHX7afesk2WW1GIpzyNAsaAqOlGOqDdxWBL15e2Pc/TbSp0P+s6V5Yp1CJnVccHVUuzrrTYaGu3NKnv40BqZweSsu0jBgZbHbY1NEYuxY3p9OqEBRj64czaPXT6WT38+l5FxwSIBW2u4OapGiGpGG52t+ypSyHwQRfsyBwR037Xo6PeUU6WF2TZaejwvn0YL9DimiEToIdGBXWqg2SuYxMVEcrCwCndmuMe9u7mZW1EJTWq1JqJbMWsXVB31KnNGLHY+Ms8RuVjaoP0UtWKROUpi+Vz4vYNBYj3UmT/XnMJDYLOAKQjCPLNeGxvix/UzBzMsptnvh9YrL6ZINGHNKaulodHukfl4AilkPoheE7Kg4NBuP4fDIsuu1Pz2/dkiU1XI2grAF6Uin+qyCYldaqDZK2hrOQmah9htQna8mBSdlpvWm25FB45Iu/p2ysfVljb9kA+e2+mndlhkJfVtW2SV9SIROkDxsYRoBw6XYWtriKe/FduUeW4vTdUlhoqyaYb8vUQYrdhV+lVbFylkPojRLj5gISE9EDItzPaM07XYjy2y4mNQnomqN/HKGVFpokX1cW+ghYlH+4mLiWOFVVTUWXv0lDa7ypaTxUxUtPWYyKE9er5OYdaEzNKOkO1/GxqqRRSflnrQGRxrZEX12kVHOxZZoM5HLTKHxdrQiruuWHO3xvduoEeHBMeDokdBZVS48NJklPQf96IUMh/ETxVf5rAetL6JCjLhb9RTr2prFf1ZyE6uByAvbApVdjPjEkNJjep+oIzb0ITMDwspkQGoKuztYXmgA2cqqKxvZIFhr9gxcknP5tgZOiNk2v+AiT/qkuXhcC0W1mlC1opFVqVZZH4Oi8yXohZBuA1BCPnZOHIb3dhMs1soCpjEd2JoqDjXmf1onUwKmY+hNlowIq5Aw8O6L2SKojAo3B8LmpCpNrHw3B/ROiR/Vy8qXFzq7SAPB44fXGsd47SiuodyK3r0lN+dKCaAeuLQAkc6EebeYwK1FIZTG6Ayr+XxhtqmROikmV16aodrMb+2bYussk5bI/PFPDJwCkQLiyxvv6g2ozNA6jzPz+tstM9jiuboySip9eJk3IvbiwZLekZVRQWOEIWoHjYjTYoIIKuwWSBAYz3oux9A4pPYbXB6EwAbyyJRlGYNHr1NMyEbkxTCp/tyOZTbszZF3x4vIlXRxCQgSpRJ6m0Gz4aASFEH8Zk0GHKe+PF2NDFttIiIRr9QiOu4Cn9zHBZZWYMOzLRrkZl9sbIHNAmZ5SyL7NAHYjvqEnBTLcweoQUfJWk/AZnStSjpLUrLRSkgC0b8/Hp25Tk0OpCG5tcq/dG9mL8fyjOx6vzYZB/HtJQI75Skag2nkNU6IygPnem+RVbb0MiuzDKGOoSslytFODH6w02fglm7lD/1jai/WF0gbvXlou/ddau6LDJBZgPBZgP1aBdc1raDPUyqr1pkmjI01rkmRTvyx4Z2vjNBr2IUgpsYKNbNM/uRRSaFzMcoKxNCVq/0/MuaFheCig6rw73YHyMXtdylgwylkiCubq0XlLdw/KhbqhiTIEQgo6TWaWF0le2nS7HaVCb4a1VCPCVkIHKgbvoEJvxIWF4ux8bB7Zu6nfAbG+pHsao9Z00xNLoWtK2qb0TBjlH1cYsMmiro2G1NRawTp3p+Tq2hWWRx/kLIsstqabT1jxD8fularK+r4eRflhDdkA2KgmnmbYRd+ECrfY58jarKcgAsup4vaI+ME9FU9RgwYm31arfPoy2mH2yIJ9jP4FsNAx0RhaWniNDXER/qR15FPUfyqkQ7+i6yWavXODWoCCrwfABBwkS44gWR7mBvBGttS1HrBnEhfpwoDMNqCMTYWCPqLUaPdB6vrLM2rY+B7wmZwSysHWsN/PGsKFJTkMt78SqahyDMYMVkCKSh0U5eRb0zVacv0y8tsuqKEsY07COGUmLUEsK2Pgnpa7w9rU5RVSlcT42Gnn+4hsUEodcplNi1Vbf+2FwzS9Sx22UfwZWTEvE36b08oWaEJEB4Kqh2yNrG6Hjxfzia3711ss0nhJCl4GHX4tkoiqjc4QYRA0fAh0KZv5Yw7Ciyq1FxtpD5Wh6ZosB5D7V+bPKNoPORz6RmOeqq8xmsiVd/CcHvl0IWEBTKzilP83+2H5Nh1zoDf/dX706qk9TWiB85u6Hn4eN+Rj0pkQHkqpFiR0VOj5/Tp2ioRc3bC8BOdSQ/muGDPe9S5oht5manhXw0v4MKGa1QVGXhaH4VCnZCakT7Fq+HdLuJeC3g44xB6wzvKOmkUVzdgJ+jO7Te5N3E4raYeSec+6Drvik3w8Lfe2M2rZOiJaof/oTBkeL3pb9ELvrgJ6LnBASFMvWS2/Gb/VNusD4AgHpmV59wrdVUlQOgmN2TB5UWH0Ie3ROyzw/k8cD7+323UvaZXSj2RvLUCGKThjuFwqdwVLnIaBKy9G4I2ZaTwho7J7YBpbEWdEYI90Hh7gbJmnVwwq5FmxYecTleWFXfVJ7K16wxBzodnPsA/L6i6XbJX4Xb0VcYdYnY5u1lZKgW8NFPcsn6pZA5CDQZyFZjKFDDUOzWplwXH6a2WlhkBj/3/ChPSgprssgqz3T6cXuzy7nzjd2s2pHNLa9sx2b3vdbo9kxRlmqnfQTX+aI1BjB4ltjm7WdUlIjMS8+v6nKreUc/s8VxmghGpHa6MK+vkxwphGxnvRaoo1nZDoqqLL5Z+b6vETrI6eqeohNBUtK12Ae4ZU4KoLDDrlWnztrizel0Ckut+KHqSeX75kwZHE6eJmRqFyyyN77PdN7PLq1j4zHfW18rTxd17A7oR3PxeB/JHTubsMEi38tuZUjNHox6hWpLIzllnfcOqKrqDPSYFqwlQvcTtyI0WWQbqkR5MYqPO+s6NjTaKau1+m4ydF9DS8weUSciKqVrsQ8Q7GdkZGwwO+xa1FDWNu9OqAPqrTZnUqV/kHsW0sckhFKkE5UZGkqyOvWYRpudtUcKtMeLAIX3d3XemvMI1noC87cDEJ42Hz+jjyyon42iwNgrATAcep+h0SLnqCvuxZNFNeRX1mMy6EhRc8VObwV69AKxIX6Y9DoK7KE0BiUAqrPTdEmNyH0M12k/uGYvdzTo66QIIYspEcnsWSW1Pult6Sr9WshA9OfZaddawWdvb7uLqw+QU1ZLgCJyvdxlkZkMOkJjRemmzroWt58upbzWSkSgiT9cJnoobTpW5FNtH8oOrcWs1pOrRrDgXB9JOG2LtKVie3ojabGakBV0Xsg2a92lpw4Ox1B2QuzsR0Km1ykMihBrXyXRM8TOI6sB4VYESPPT8rPCkjw+v36FFvBhLDpIuL6OBpud/Mq+n1/a74UsPMDIUTUZq95fdK89K7TXl8gqrSUY7crT5L5SUkmp4kfPbK0QdfE6YM1B0SLkglGxTEoKJyrITJWlkR0ZpW6bU0858/37AOwPmM2IOB+/Sk+aAXozVOUxI1T8IHclctGxPnbOiOimiL5+5FoEGBkrLtx2h2gXJYc+gIYaCiqFkI00at2wPdTTq98SkgChSSiqnXkhYrmgPwR89Hshm5YSgQ09p/WaVZLfShdXH+F4QTVJila1IdR9V57jhyVTqWrRXh2sk9ntKl8eEkK2eFwcOp3C+WnRAKw74hvrZDa7SlT+ZgCCJ1zs5dl0AqM/JInmi5NtopN1eidzyRoa7Ww9JdbF5icbmqxqX0mydRNp2sXI+oYxQqxqS+CzX5JZUsMoJZNFlq/EwKTOt4iRtIH22ZnsLy4O+sM6Wb8XMkdpoAOqlnHfWhdXH+F4YbWz8687GyZOHRxOhiqCIUoy9rU7dk92GYVVFoLNBuYMFWtr56eJXLx1Rwu6HG3XG2w9dJI4hOBPmbPYy7PpJEPmA5BcIdYmThbVYGns2M29K7OM2gYbUUEmRqraGmdoMviH9dZMvYIjNeFwQS1MuUns3PcWP1k3iTXmB/FT6yBqJIy+3HuT7C/ETwRgun0v0D+KB/d7IXNU137forUjP/alz7YzOZ1fSoIi3EhEpLrteYP9jBT4CyEvOL6r3bEf7xXBBAtGxWAyiI/HvOFRmPQ6MktqOVnk/Q994TcvAFBpjMYvuGcdAjxGqhAyvzNbCPXTYbOrnCzs+Fx+q62PzR0Wha7osNjZxQrzfQFHUFF6fhXVE29tUWi3MmQEXP+e71TJ6MuMvAiAodW7AbVfhOD3eyGLDREJid83pGL3CxfrZGd2enlWLVFVldqi0+gVFbvBH4Ji3fsC2o+fPa9t12pxtYV3dwrX41VTmlybgWYDM4eKEP51WjSjtyiorCeh+DsA1GEXeHUuXSJhMpiCUerKWBwpxCm9oGP3osv6WO5esTO2/wlZUkQAyREBNNpVvs+2wI/eof68R3jFtpjrGn5DxQ1rISzZ29PsH8SNA70Zc2MlqUp+v6iC3++FLMBkIDrYjB0d5fFauSBHN1sf4kx5HUmNwnWkRAxxe4Hj6KFTAIiqSm/1uM2usmLVXuqsNsYkhDBnWKTL8YWjYgBYd9S762Sbv/6EmTpR+SH0vLu9OpcuoTc4q8OfZz4KdBzwUVhVz0GtEefc1BDR2BIguWvNK/sK84YLV/aag/mgN7LKcBm/t95IYeR0kqLDvDu5/oTBJApAA5OU42SU1PjEkkFP6PdCBjA+UayTHfbX2imcWOfF2bTO8YJqrtCLAAalFxa0h0+cg01ViKOIM1mnWhx/Zm06m08U42/U85cfTkQ5S0jPTxNCtjOjlJwy71zBNVrqmHHgdwBUBQ2B6DSvzKPbaOtk4xr2Ah3nkn11qABVhQlJYcQcfhmqckXfr+RZvT1Tr3DlZFHZ44M9Ofzo39/zh9XClXrjrBQvzqqforWWmaQ/Sb3VTmFV3+5V2OtC9uSTT6IoCitWrOjtl2oTx0Lyd+p4sSN3N9T6Tig5QOW+T1ii1zrujrva7c8fGBJBllGsu2Xs2+BybOvJEp7/5iQAT/5gHCNiW+awDQoPYO6wKOwq/PXr4y2O9zqqSt7rtzNIFRGVpqv/1Sfa8riQeg4AceW7MdPA0bz2hewLLQ3i4lFhsOlPYufCR5x9pfobUwaHs3R8PKoKW06WYFfhmqmDuH6mDLl3O4OEh2aaUVzUZvTxEPxeFbIdO3bwz3/+k/Hjx/fmy3TIsBiRk7W7PACiR4m2Gqc3enVOLpSc5MKjvwXgZPzSpirVbqY8chIA1tNbnfvsdpXHPxdXvtdNT+KyiYltPn7FQpG79O6uHNYf9exambr6XpKyPwbg07SnMQ/ug2HYsWMhNAl9Yx1zdQfIr6ynorb1JptFVRZn2P2lQUfBUgkhg2DCdZ6cscf5yzUT+csPJ/DQRWl8fvc8nr5qAnpdH7tg6QtoFtkwewZmGvr8OlmvCVl1dTXLli3j3//+N+Hh3o0scwjZyaJqGLZA7Dz4fu+9YFkmZGwWPdDy9otGhNVtrC011KCuvgd/tY5t9jRqFv2l16blP1S4pCLL9mLXytJ8uj+Xg2cqCTIb+OWF7ecmTU2J4MdzhFX36/cOUF7b0O54t5F/EGXXS9hVhd/bfszsi2/2zOu6G0WBoecBcK6/uBJuqzfZqu1Z2Owqk5LDiM3RcqhGX+qbLUzciMmg44pJg/jpOUMZneDjie59mbBkCIzGgI0xSkaPIhdVVeXFjSf5/SeH2rww6216rUP08uXLWbp0KQsXLuSxxx5rc5zFYsFiafLPVlZ2r+lgezjq2xVXN1CR9kNCt/4djn4mosC0RU+3kLML3r0ZKs6uaagAKgTGtNrcUgFsqsJT9ht4a1CU++ZzFqkTz4MtMNJ+kv0Z+YxKjuHpL0Twxx3zhxAV1HHLiV8vHsmm40WcKKzmz18d49HLezGCTlXh4Puon92LAqyzT0adeiuRnZinz5I0A3a/xgyDKDWVXlDFjCGugTVWm503tonP0I+nRsK6z8WBUZd6dKqSfoyiwKBpkP45k3QneiRkH+/N5ck1IoCprLaBv147yV2z7DS9cnm3atUqdu/ezcqVKzscu3LlSkJDQ523pCT311ILNBsYFC4qWxxuTIQRS4R78V/zoSq/5QOK0uHkN/CXcfCHSPjbZHhhLuxbBVv/ATv+C5YqOLMLik/ANyvh5aXwn/NbETEALSKonQ7NTzQuwx4/CbOh9/JkzNFDqdSHY1Js7NnyFa9uyeBMeR2xIWZundtGAraqirJWdlFn0c+od9ZffHtHNoVVvVSnrboQVi2D929Fqa/gtD2WP+pu4e4Ffbw0U5KoJZjacAwjjRxpZZ3sta2Z5FfWExVk5qL8F4RbMWqE87ESiVtIFOtkE3UnyCjunmvRYY05+PxAnuc8Nc1wu0WWnZ3NL37xC9auXYufX8ctFx588EHuvfde59+VlZW9Imaj40PIKavjcF4lsxavhGNrxIE/a+60NqwlAEq1f9SHtzft++ze1scCv7feyAF7KgG6Ru64/HzmxCN6R+Xthy8ehIZmP16Kjn8Mfpb/Ho3i9iER3X+DnUFRqB28gJBT76E/+imPHBTW3y8vGIm/qRUBPfAevH+ruB8UB3NXwPTbmT00isnJYezOKuf1rZnc24FLskvUlcGXvxWu38amVifXNDzMzYtm9m1rDCByGPhHYKwrZaxymr3Zrv/zwsp6/vq1qAf66/MHoV+/Shy46I/93q0o8TCDtMhF3QlOF9dgt6vourgeeaq4hqP5VRj1MDsgj6CaTL7bHsLSc3tnnb8t3P7N2LVrF4WFhUyePBmDwYDBYGDjxo387W9/w2AwYLO5luUxm82EhIS43HoDh7/9UG6FqJqx9BnXAe1YS5l0otfV0PNZH3sL0+r/waaIHxA1+hy+tY3hp58Wc8w4EuInwOQb4KEcly6y6sOl/C9PhB07SkL1JrHTrgBgvm4fCnbmDoviqimDXAcVn4C3ftQkYgDV+fDFA/CHcNj6PD+ZJyy4N7Zl9agq/ob0Qhb/+Wv+8PA98PtQeCoF9r7uFLHtwQsZXf8S4bFJ/GSe+6qdeA1FcQbzLNFv50heJYVa9XG7XeWX7+6jsr6RsYkh/CAmDxrrRZCHVhlEInEbCZNRURikFBNoLSWztOtW2Xqt/uozER/zqvWXPG/6G0s3LIXH4qAy190zbhO3W2QLFizgwAHX6hG33HILaWlp3H///ej13ikx46i5ePCMSDBl2q0Qkghv/bBpUOIUsNajXvgo/8hK5k9rj9E8T/BH0wbxxCXDoCwDKvMgZY6wsiKHcrDcyK1/34wKPHfFOKYMDuf6/2xj2+lSlv1nGwtHxVJUVc/IuGDumD+UYD/R3fd0cQ15FfWY9DqmpfSyRQYoQ+ZjN4cw2FLIU+PyWfyDxa5XYaWn4bVLm4rT+oUK0c/fD9/9Vez78iEW/TCFmGAzhVUW1h0pYMm4rje2XL0/l9+/tZH3jP9Hiu6sKMiFj/Bv2yU8vuYoep3Cn6+e2KtuV48y7mo48gkXmffyROMy1h0t5Lrpyfxn8ym+PV6Mn1HHsz+ciH73E2J86ry+l2og8X38QlCi06DoCBN1JziaV0lqVGCXnuLrIwWkKVlcUvW264HGOnhmFNx3EgJ7/wLd7UIWHBzM2LGuAQCBgYFERka22O9JJiaFAaIwb0WtldAAI4xcLCyjZhzKreCRTw+z/bRw7/xwahLjk0L57UcHeXNHDvNGxLBk3BiIFetEJM/Ablf5zStbUFW4ZEICM7XF+xeun8IPXtjC6eIa3tou1s6+PlLIjtNlvP6TGZgMOjaki3JFkweHte7eczfmYHQTfwTbXuQa1oLfbWJ/4RH4/D7IEF2XCYyBq15ydpRl3FUi9Pvdm6HoKPp3b+ThIQ9z18FhvLUju8tC9um+XN585012mh917qvDzIMNt7LDfw6jTsbx9RGxgPzgkjTGDXJPo1GfYMh8UHQMsp0hgWJe3ZJBXIifc8H84YvHMCwqEA6LdAMZ5CHpNQZNcQrZkbzKLn2Py2sb2JlZxt367c59NyuPscT6NT80bBA7Cg7CkHPdO+dW6LWoRV8jOthMalQgp4tr2JVV6qzo7qCk2sKfvkpn1Y5sVBXMBh0PXzKaZTNEMuaZsjr+seEkD314gKkpEUQHN63VvLIlg33Z5QSZDfx26Sjn/ohAE5/+fC4f7M6huMqC2ajnxQ0n2Z5Ryj82nGDFwhGs3i/M70Vj4jxwFjSm/QS2/wuOfwkvzm3Z2iZ1vrDCooa57o8ZBbeuhVeWQv5+Lj7xMP9R/sC3xyG7tJakiPYTdYuqLHx5KJ+1hwsoOb6N1c1EjMuepyzlCo68soszBVWc0Wo63nnuUG6d2w9cis3xCxXWf84Ozjcd5vX8KG55RSTDXzVlENdNTxL9uCrPiLFnFdCVSNxG3ATgdUYqObzdQYL+2aw/WojdbuMm89cinu3ivxB0bCIP7E9hWFI8U8LrwOSeBsEd4REh27BhgydepkOmDg7ndHENW06UuAjZtlMlLH9zD8XVIg3gkgkJPLAkjcQwf+eYFQtHsCG9iMN5lTz4wX7+dcNUdDqFb9ILeeJzUfvv14tHEhviGuASZDa4lNhJjgjg52/t4flvTjA4MoDdWeUoCizthmuu20QNh2EXCCFrLmKRw2HxkyLXri1Xll8I3LIG/nsBFB7mbb/HmVP3LO/uyuHeC0a0+pBGm50/fXWMlzafpsFm5wb9V7xqfqVpwIL/g0nXkwB8fNccPtmbS05ZLeeMiGaqB9ytXmHIeZCzg9sHZfJWhoLNrnLJhASeuGIcimoXkbAAs+4CY8dBUxJJt4gW39lhSg5H8rqW+vTVoQJSlXzC1Eow+MHEZZxDIav35/Fo4w18dM2c3phxqwyoMChHvcAvD+c7i2Tuyy7nlld2UFxtYXhMEO/dMYvnrpvkImIgEjX/fM0EjHqFr48U8tP/7eR3Hx3kJ6/upNGucumEBG7oRCmdi8fHs3BULFabyj1vi95gl4xPICbEwz9WF/8FkpoVn532E7hrBwxf2PF6jDkIfvi6uKtauN3wKau2Z1HX0LK/Vr3Vxm2v7eTFjSex2aw8Ff4xjxpfaRpwzyGY1xQB6mfUc820JO69cGT/FTFwJuYnlW5hy33zWPfL+Tx33STROidnB5QcB3MIzLjDyxOV9GviRNWlVF0BlvJ8ijpZc7G2oZGNx4oYo2RozzMODGZn4ef9OeUeDcMfUEI2f2Q0fkYd2aV17Mgo40RhNTe/vJ3aBhtzh0Xx6c/ntvvjOSo+hD9dLUrmfH2kkP99n4nNrnLl5ET+ePX4FoV2W0NRFB67fCyRgSYAQvwM/Mqd4eudJTQRbv0S/q9crBMu/XPXAgoih8KV/wbgRsNajFU5vNAsnwSgrkGI2DfpRQQaFbYP+hs/rGu2KHzjxxB6VsTkQGHQNAiIgvoKYvM3OJP2qSmBlxaJ+0POFRawRNJbBERAjFjvn6g7wa7Msk497MtD+dRZbcwK0DrOx08Qm1B/RsQGYVfhuxMlvTLl1hgwa2QgWrpcMSmRt7Zn8+v39lFtsVFWa2XCoFBevGEKfsaOgy0um5jI0Ogg3t2ZTYNN5cIxsZw7IrpTIuYgLtSPNb+Yx+YTxcwcEknCWdafR+lJNNyYK2Dr3zHn7eN2w2p+vz4as0HH/BHRnCmv489fpXOsoJpzTOn8J/AFTMVaisP0n8KFj4Ghj+eE9QSdHkZfBjv/C29fLy4oAN67pWnMvLZzFSUStxE5FAoPkaQUsiuzlMVjO16vf2+XELC5QblQjtOyA5g3PJpjBdVsOlbE0vGeWTIZUEIGcPeC4azel0eGViQzLS6Yl2+ZTpC586dibGIoYxN7FkUXE+LnbFvRZ9EbhSC9egnXGTfyX9sS/vgl/PHLpp5nVwbs4xn7U+DIbb7mNfEDLoG59wghA/jTcGGlOYpZ/+C/kOD5Uj+SAUh4CgDJSiGfdsIiyyypYcvJEhTsJNZrnTDimwtZFP/dfJpvjxehqmqXLvK7y4ByLYIwfd++fRbXTkvivkUjefeOWURobj5JN0iZB8mzMKoNfBb+DDOS/IkKMjEsJohHxpcJEXNw9atSxJoTlgTxE8X9miJI12oqjlgMY3/gtWlJBhiakA1SijhwpoIaS2O7w1/+LgNVhR+nlKCvLxORiTGjncdnpEZiMujIragXhdo9wICzyEBU+XjyB95tLdNvUBS48l/wwhyCanN4u/YK0fzR0gDHtBw9YyBc95azsaSkGbeuhQ0rIXePKIU2/lqYf79MgJZ4jnARpDbUUIy1TuXb48Vtuhcr6qy8szMbgFvC9kI+MHKJyzKBv0nP9JQIvjtZzIEzFQyL6f0Q/AEpZBI3E5YM17wqivxaa4V14SB5FvzobZEPJWmJwQQL/8/bs5AMZMJFnmYSBYDK+qMFbQrZ2zuyqG2wkRYTSGLuWrFzzOUtxj1y2RjCA0we83ZJIZO4h6Hnwy/2w5tXg84gqlH4hcLkG6V1IZH4MqFJoOgwqhaiKWf9UT9sdrVFQ9N6q43/bj4NwK/GVqJsyRFuxaELWjylMwrXQ0ghk7iPoGj46QZvz0IikXQFg0kUpq7IYox/KRuqw9mQXsiCUa7Vj97clkVBpYWEUD/OtX0tdo5c7BMJ+wMu2EMikUgkZxGRAsCVKSKJ2dHY1UGNpZF/bBB5onedPxxD5mZxYORFHptie0ghk0gkkoFOhGjLNC9KRBl+k17Y1CkE+PNXxyiutpAcEcDV40JFkXHwmRQRKWQSiUQy0NECPsLrsrlkQgKqCve/v59qSyNrDxfw8haxNvbo5WMxHlgFdqsI8grruCyfJ5BrZBKJRDLQidG6dhQe4aFlaXx3ophDuZVMf/xrarUaqtdNT2b+iGh4c70YO/VWn+la7huzkEgkEon3cCQ0Fx8jPlDPyzdPIyHUzyliP5yaxB8uGwO2Rsj4Toz1QJ+xziItMolEIhnohA4CcyhYKqDkOBOSxvDNfedy8EwlMcHmpl6DGVugoQr8I1zqK3obaZFJJBLJQEdRIFazygoOAWA26JkyONy1Ye7B98V2xCKfcSuCFDKJRCKRAMSKdi4OIWtBWSbsfk3cT1vqmTl1EilkEolEImlaJ2tLyPatAnujaKI5UgqZRCKRSHyN2LFi25qQqSrsFV3hmX23T7kVQQqZRCKRSKApBL8qF2pLXY8VHYXyLNAZfc6tCFLIJBKJRALgFyKSnAEKDzftV1VYfY+4P2wBmAI9P7cOkEImkUgkEkFMKwEf6x+FrK3CGlvwsHfm1QFSyCQSiUQicEYuHhSW2IH3YPNfxL5Fjzcd9zFkQrREIpFIBHFawEfOLvj8V7DjP+LvcdfAjNu9N68OcLtFtnLlSqZNm0ZwcDAxMTFcfvnlpKenu/tlJBKJROJuBk0T28JDTSI2+nK45FlvzahTuF3INm7cyPLly/n+++9Zu3YtVquVCy+8kJqaGne/lEQikUjcSUgiDLug6e+gWLjinz4Z4NEcRVVVtTdfoKioiJiYGDZu3Mg555zT4fjKykpCQ0OpqKggJCSkN6cmkUgkkrOxNcKZnYACSdNF+Sov0Vk96PU1sooK0ZwtIiKi1eMWiwWLxeL8u7KysrenJJFIJJK20Bsgeaa3Z9ElejVq0W63s2LFCubMmcPYsWNbHbNy5UpCQ0Odt6SkpN6ckkQikUj6Gb3qWvzZz37GmjVr2Lx5M4MGDWp1TGsWWVJSknQtSiQSyQDH667Fu+66i9WrV7Np06Y2RQzAbDZjNpt7axoSiUQi6ee4XchUVeXnP/85H374IRs2bCA1NbXLjwe5ViaRSCQDHYcOdOQ4dLuQLV++nDfffJOPP/6Y4OBg8vPzAQgNDcXf37/Dx1dVVQHItTKJRCKRAEIXQkND2zzu9jUypY1QzZdffpmbb765w8fb7XZyc3MJDg5u87k6g2OtLTs7W661achz0hJ5Tloiz4kr8ny0xFPnRFVVqqqqSEhIQNdO65hecS32BJ1O1+6aWlcJCQmRH76zkOekJfKctESeE1fk+WiJJ85Je5aYA1k0WCKRSCR9GilkEolEIunT9FshM5vN/N///Z8M7W+GPCctkeekJfKcuCLPR0t87Zz0eq1FiUQikUh6k35rkUkkEolkYCCFTCKRSCR9GilkEolEIunTSCGTSCQSSZ9GCplEIpFI+jT9Usief/55UlJS8PPzY8aMGWzfvt3bU+o1Vq5cybRp0wgODiYmJobLL7+c9PR0lzH19fUsX76cyMhIgoKC+MEPfkBBQYHLmKysLJYuXUpAQAAxMTHcd999NDY2evKt9ApPPvkkiqKwYsUK576BeD7OnDnD9ddfT2RkJP7+/owbN46dO3c6j6uqysMPP0x8fDz+/v4sXLiQ48ePuzxHaWkpy5YtIyQkhLCwMG699Vaqq6s9/Vbcgs1m43e/+x2pqan4+/szdOhQHn30UZfKRP39nGzatIlLLrmEhIQEFEXho48+cjnurve/f/9+5s2bh5+fH0lJSTz99NPufzNqP2PVqlWqyWRSX3rpJfXQoUPqbbfdpoaFhakFBQXenlqvsGjRIvXll19WDx48qO7du1e96KKL1OTkZLW6uto55o477lCTkpLUdevWqTt37lRnzpypzp4923m8sbFRHTt2rLpw4UJ1z5496ueff65GRUWpDz74oDfektvYvn27mpKSoo4fP179xS9+4dw/0M5HaWmpOnjwYPXmm29Wt23bpp46dUr98ssv1RMnTjjHPPnkk2poaKj60Ucfqfv27VMvvfRSNTU1Va2rq3OOWbx4sTphwgT1+++/V7/99lt12LBh6nXXXeeNt9RjHn/8cTUyMlJdvXq1evr0afXdd99Vg4KC1L/+9a/OMf39nHz++efqb37zG/WDDz5QAfXDDz90Oe6O919RUaHGxsaqy5YtUw8ePKi+9dZbqr+/v/rPf/7Tre+l3wnZ9OnT1eXLlzv/ttlsakJCgrpy5UovzspzFBYWqoC6ceNGVVVVtby8XDUajeq7777rHHPkyBEVULdu3aqqqvhA63Q6NT8/3znmhRdeUENCQlSLxeLZN+Amqqqq1OHDh6tr165V58+f7xSygXg+7r//fnXu3LltHrfb7WpcXJz6xz/+0bmvvLxcNZvN6ltvvaWqqqoePnxYBdQdO3Y4x6xZs0ZVFEU9c+ZM702+l1i6dKn64x//2GXflVdeqS5btkxV1YF3Ts4WMne9/3/84x9qeHi4y/fm/vvvV0eOHOnW+fcr12JDQwO7du1i4cKFzn06nY6FCxeydetWL87Mc1RUVAAQEREBwK5du7BarS7nJC0tjeTkZOc52bp1K+PGjSM2NtY5ZtGiRVRWVnLo0CEPzt59LF++nKVLl7q8bxiY5+OTTz5h6tSpXH311cTExDBp0iT+/e9/O4+fPn2a/Px8l3MSGhrKjBkzXM5JWFgYU6dOdY5ZuHAhOp2Obdu2ee7NuInZs2ezbt06jh07BsC+ffvYvHkzS5YsAQbmOWmOu97/1q1bOeecczCZTM4xixYtIj09nbKyMrfNt9c6RHuD4uJibDabyw8QQGxsLEePHvXSrDyH3W5nxYoVzJkzh7FjxwKQn5+PyWQiLCzMZWxsbKyzV1x+fn6r58xxrK+xatUqdu/ezY4dO1ocG4jn49SpU7zwwgvce++9PPTQQ+zYsYO7774bk8nETTfd5HxPrb3n5uckJibG5bjBYCAiIqJPnpMHHniAyspK0tLS0Ov12Gw2Hn/8cZYtWwYwIM9Jc9z1/vPz81s0V27+XQoPD3fLfPuVkA10li9fzsGDB9m8ebO3p+I1srOz+cUvfsHatWvx8/Pz9nR8ArvdztSpU3niiScAmDRpEgcPHuTFF1/kpptu8vLsvMM777zDG2+8wZtvvsmYMWPYu3cvK1asICEhYcCek75Mv3ItRkVFodfrW0SgFRQUEBcX56VZeYa77rqL1atX880337j0c4uLi6OhoYHy8nKX8c3PSVxcXKvnzHGsL7Fr1y4KCwuZPHkyBoMBg8HAxo0b+dvf/obBYCA2NnZAnQ+A+Ph4Ro8e7bJv1KhRZGVlAU3vqb3vTVxcHIWFhS7HGxsbKS0t7ZPn5L777uOBBx7g2muvZdy4cdxwww3cc889rFy5EhiY56Q57nr/nvou9SshM5lMTJkyhXXr1jn32e121q1bx6xZs7w4s95DVVXuuusuPvzwQ9avX9/CjJ8yZQpGo9HlnKSnp5OVleU8J7NmzeLAgQMuH8q1a9cSEhLS4gfQ11mwYAEHDhxg7969ztvUqVNZtmyZ8/5AOh8Ac+bMaZGScezYMQYPHgxAamoqcXFxLueksrKSbdu2uZyT8vJydu3a5Ryzfv167HY7M2bM8MC7cC+1tbUtOg7r9XrsdjswMM9Jc9z1/mfNmsWmTZuwWq3OMWvXrmXkyJFucysC/TP83mw2q6+88op6+PBh9ac//akaFhbmEoHWn/jZz36mhoaGqhs2bFDz8vKct9raWueYO+64Q01OTlbXr1+v7ty5U501a5Y6a9Ys53FHuPmFF16o7t27V/3iiy/U6OjoPhtufjbNoxZVdeCdj+3bt6sGg0F9/PHH1ePHj6tvvPGGGhAQoL7++uvOMU8++aQaFhamfvzxx+r+/fvVyy67rNVQ60mTJqnbtm1TN2/erA4fPrzPhJqfzU033aQmJiY6w+8/+OADNSoqSv31r3/tHNPfz0lVVZW6Z88edc+ePSqgPvPMM+qePXvUzMxMVVXd8/7Ly8vV2NhY9YYbblAPHjyorlq1Sg0ICJDh953hueeeU5OTk1WTyaROnz5d/f777709pV4DaPX28ssvO8fU1dWpd955pxoeHq4GBASoV1xxhZqXl+fyPBkZGeqSJUtUf39/NSoqSv3lL3+pWq1WD7+b3uFsIRuI5+PTTz9Vx44dq5rNZjUtLU3917/+5XLcbrerv/vd79TY2FjVbDarCxYsUNPT013GlJSUqNddd50aFBSkhoSEqLfccotaVVXlybfhNiorK9Vf/OIXanJysurn56cOGTJE/c1vfuMSJt7fz8k333zT6m/HTTfdpKqq+97/vn371Llz56pms1lNTExUn3zySbe/F9mPTCKRSCR9mn61RiaRSCSSgYcUMolEIpH0aaSQSSQSiaRPI4VMIpFIJH0aKWQSiUQi6dNIIZNIJBJJn0YKmUQikUj6NFLIJBKJRNKnkUImkUgkkj6NFDKJRCKR9GmkkEkkEomkT/P/QOHWOAf8VhsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NRMS=38.17%\n",
      "RMSE=0.8031 V\n"
     ]
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.figure(figsize=(5,2))\n",
    "test_sim = model.simulate(test)\n",
    "plt.plot(test_sim.y) #simulate is apply_experiment equivilent\n",
    "plt.plot(test.y)\n",
    "plt.show()\n",
    "from nonlinear_benchmarks.error_metrics import NRMSE, RMSE\n",
    "print(f'NRMS={NRMSE(test.y[model.na:], test_sim.y[model.na:]):.2%}')\n",
    "print(f'RMSE={RMSE(test.y[model.na:], test_sim.y[model.na:]):.4f} V')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "73ed23e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert RMSE(test.y[model.na:], test_sim.y[model.na:]) < 1.0\n",
    "assert hash_array(test_sim.u)+hash_array(test_sim.y) == '3fc2c421f38edda7'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc551e9c-0501-486d-b8ba-e2d414d62b1e",
   "metadata": {},
   "source": [
    "### 1.4 Continous time SUBNET\n",
    "\n",
    "`deepSI` also contains options for continuous-time esitmation. For example: https://arxiv.org/abs/2204.09405 is implemented in `models.SUBNET_CT` given by the following structure.\n",
    "\n",
    "$$\n",
    "\\dot{x}(t) = f_\\theta^{\\text{CT}}(x(t), u(t))\n",
    "$$\n",
    "$$\n",
    "\\hat{y}(t) = h_\\theta(x(t))\n",
    "$$\n",
    "$$\n",
    "x(0) = \\psi_\\theta(u_\\text{past}, y_\\text{past})\n",
    "$$\n",
    "where $\\psi$ is the encoder that initializes the state.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "43bd6552-2884-4c99-a307-ca31111f3d8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 867\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "359cbef432854fb7890bcca8a349c5c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 1.00484!! nan samps/sec\n",
      "it      50 NRMS loss 0.43281 NRMS val 0.43648!! 9110.40 samps/sec\n",
      "it     100 NRMS loss 0.25465 NRMS val 0.17438!! 9343.13 samps/sec\n",
      "it     150 NRMS loss 0.21626 NRMS val 0.26487   9026.71 samps/sec\n",
      "it     200 NRMS loss 0.18967 NRMS val 0.24029   9202.77 samps/sec\n",
      "it     250 NRMS loss 0.16368 NRMS val 0.16510!! 9211.84 samps/sec\n",
      "it     300 NRMS loss 0.14797 NRMS val 0.11882!! 9299.40 samps/sec\n",
      "it     350 NRMS loss 0.13688 NRMS val 0.16736   9341.99 samps/sec\n",
      "it     400 NRMS loss 0.12764 NRMS val 0.25270   9384.95 samps/sec\n",
      "it     450 NRMS loss 0.11961 NRMS val 0.31058   9396.72 samps/sec\n",
      "it     500 NRMS loss 0.11332 NRMS val 0.38390   9439.81 samps/sec\n",
      "it     550 NRMS loss 0.10589 NRMS val 0.42049   9467.18 samps/sec\n",
      "it     600 NRMS loss 0.10072 NRMS val 0.43204   9425.18 samps/sec\n",
      "it     650 NRMS loss 0.09725 NRMS val 0.43822   9447.53 samps/sec\n",
      "it     700 NRMS loss 0.09236 NRMS val 0.41764   9460.37 samps/sec\n",
      "it     750 NRMS loss 0.08892 NRMS val 0.39759   9471.58 samps/sec\n",
      "it     800 NRMS loss 0.08804 NRMS val 0.37789   9494.57 samps/sec\n",
      "it     850 NRMS loss 0.08549 NRMS val 0.38551   9510.82 samps/sec\n",
      "it     900 NRMS loss 0.08297 NRMS val 0.38901   9523.33 samps/sec\n",
      "it     950 NRMS loss 0.08028 NRMS val 0.37251   9530.08 samps/sec\n",
      "it   1,000 NRMS loss 0.07858 NRMS val 0.33284   9475.45 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "import torch\n",
    "from nonlinear_benchmarks import Cascaded_Tanks\n",
    "from deepSI.networks import euler_integrator, rk4_integrator, rk45_integrator\n",
    "train, test = Cascaded_Tanks()\n",
    "\n",
    "nx, nb, na = 2, 5, 5\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train)\n",
    "torch.manual_seed(0)\n",
    "model_CT = dsi.SUBNET_CT(nu, ny, norm, nx=2, na=4, nb=4, integrator=euler_integrator)\n",
    "train_dict = dsi.fit(model_CT, train[:900], test[900:], n_its=1000, T=30, val_freq=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b8c93df4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbIAAADFCAYAAAAxI3fRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABccUlEQVR4nO2dd3wb5fnAvyfJlvfeju04dhJnx5lkMBMSQhhhjzBKKRQIu8y20MEI0NIfoxQKLasQQlgBAgFCIIvs7SxnOna8472HdL8/3pPkIW9Zku33+/noc+e7906PztI99zzvMxRVVVUkEolEIumj6FwtgEQikUgkPUEqMolEIpH0aaQik0gkEkmfRioyiUQikfRppCKTSCQSSZ9GKjKJRCKR9GmkIpNIJBJJn8bgagFaYjabycnJwd/fH0VRXC2ORCKRSFyEqqpUVFQQExODTte23eV2iiwnJ4e4uDhXiyGRSCQSNyErK4tBgwa1ud/tFJm/vz8gBA8ICHCxNBKJRCJxFeXl5cTFxVn1Qlu4nSKzuBMDAgKkIpNIJBJJh9NMMthDIpFIJH2aLiuydevWcfHFFxMTE4OiKCxfvrzZflVVefLJJ4mOjsbb25vZs2dz5MgRR8krkUgkEkkzuqzIqqqqGDduHK+99prd/S+88AKvvPIKb7zxBlu2bMHX15e5c+dSW1vbY2ElEolEImlJl+fI5s2bx7x58+zuU1WVl156iT/+8Y9ceumlALz//vtERkayfPlyrr322lbH1NXVUVdXZ/27vLy8qyINDDa+CunfgU4Hek/ba+w1kHKhq6WTSJxL0TH49mGor2p7jMEI5/8FYlKdJ5fEJTg02OPEiRPk5eUxe/Zs67bAwECmTp3Kpk2b7CqyxYsX85e//MWRYvQ/qk7DD3+0v+/ID3D/PvANda5MEokr+flZOLa643Ff3gO/XQs6fe/LJHEZDlVkeXl5AERGRjbbHhkZad3Xkscff5wHH3zQ+rcl3FLShIom1+6K/4KpXrxWPAAN1UKZjb/OdfJJJM6m4IBYTrgJks+noKIOg14hxMdTbFdN8NV9kJ8GGRtgyNmuk1XS67g8/N5oNGI0Gl0thntTfVosw0fAmCtt20szYf2LsOV1qcgkA4uGGrEcfwMf50fz6Bdp6BT418KJXDA6Suw7uAL2fQonN0pF1s9xaPh9VJT4AuXn5zfbnp+fb90n6QZVmiLzDWu+fcLNYpm3DxpkMI1kAKEpsga9Fy/+cBgAswqPfLqHkqp6MSZhmlju+QhU1RVSSpyEQxVZYmIiUVFRrF5t812Xl5ezZcsWpk2b5si3GlhYFJlPi3mwoHixTTVBzk7ny9UfUFWoLYND38Km16BWBhv1CTRFtiWrmoKKOoJ8PEgK96W8tpF3NmaIMYmaFVZ6Eg5/5xo5JU6hy4qssrKS3bt3s3v3bkAEeOzevZvMzEwUReH+++/n6aef5quvviItLY2bbrqJmJgYFixY4GDRBxAW16JvePPtigLJ54v19G+dK1N/4at74Ll4WHodfP97+ORmV0sk6QwN1QBsPSU8EZeOi+GB84cBsGTLSRpMZggbCsO1iN4DX7pETIlz6LIi2759O6mpqaSmipDWBx98kNTUVJ588kkAHnnkEe655x5uv/12Jk+eTGVlJd999x1eXl6OlXwgUVUoli1diwCJZ4lltrTIuszxtbDrf823HfsJTh91jTySzmFqBHMDAFtPCYU2OTGEuaOiCPMzcrqynp8PFYix0xaJZfpKcZykX9JlRXbOOeegqmqr17vvvguImlh//etfycvLo7a2lh9//JFhw4Y5Wu6BRVtzZABxU8UyczOU5zhPpv5A2jLb+n17Yehcsf7DH1wjj6RzNNZYV/fki/mwSQkheOh1XDEhFoBl20+JAfHThPu9thRO/uJsSSVOQtZa7AtY58jsKLKwZIgaK+bJsnc4V66+TF0F7Fkq1m/6EoITYM5T4u+jP0Jdpetkk7RPg02R1agehPkZiQoUHp+rJolWHz+nF1BYUSfyx4ZrBRwOrXC6qBLnIBVZX6C6HYsMIGKEWJ4+7Bx5+gO5e8HcCAGxMOQcsS18OAQliO3b3nKpeJJ20ObHGvXegEJyhK91V3KEP+MGBWIyq3y3X8u/TLlYLA99I6MX+ylSkfUFqtoI9rAQOlQs5dxO57E8nQ+a3Hz72Y+K5VapyNwWLdWkXhHJz0nhfs12zx8bDcA3ezVX+5BzwNMPyrNldG8/RSoyd8fUIPz7YN+1CMK9CFAkuwx0mmM/i2XTBHOAUQsARdz0KvJbHiVxBzSLrFYVhRSSI5orsnmjhSLbeqJYuBc9vCBZK5t3ULoX+yNSkbk71UViqejAO9j+mDAtmOb0Yek66Qx1lXA6Xay3tMg8fUXYNkDWFufKJekc2hxZpeoBtFZkcSE+jB0UiFnF5l4cYXEvSkXWH5GKzN1pmgyta+PfFTIEUERir2W8pG3y0kA1g38M+NupODN0jlge/Nq5ckk6h6bIyhuFImvpWgS4cIywyr7dmys2DD0fdB7iYa9QziX3N6Qic3csOWRtuRUBPLwhSCu0LAM+OiZnl1jGjLe/f8QlYnnkezCbnSKSpAtorsUaPPH20BMd2DpH9UKLezGjmLKaBvAKtNVbfGOG9Fz0M6Qic3csrsW2IhYtWNyLcp6sY6yKrI0+VbETwOAlLNziY86TS9I5GkWwR63qyaBgbxRFaTUkPtSHpHBfTGaV9Ue0h8FRl4ulqR4K050lrcQJSEXm7rRX1aMp1shFqcg6pCNFpveAmAlife/HzpFJ0nkswR4YiQ32bnPYeSkRAPx8SPsNjb/etvPTW6S13Y+QiszdqS0Ty7YCPSyEJIplaWbvytPXqS23Wa3R49seZ7npZchqEG6HNkdWgyexQW0rsnOHC0W29nABZrMqapPe8JnYWXAANr7c66JKnINUZO5OXYVYGv3bHxcQI5ayTFX75O4Ry8A48GsjLw8gdqJY5qXJJ3d3wzJHprZvkU0aHIKf0cDpynrSsrUHwiHn2gb8+GdYuhDKTsk5sz6OVGTuTp3WVkQqMseQsV4s2wr0sBA2FPRGqK+A0ozelkrSFbSE6Fo82rXIPA06ZiYLl/xPliLCOj3cu8s26NAK+L9R8JcgWPuCVGh9FKnI3B2rRRbQ/rgAUSyVyjxZ5bs9LH2phs9vf5zew1b66+Sm3pVJ0jWsUYtGBrVjkQGcmyKs7rWHC20bQ4bAb1aLcmRN+fkZeP8SqMhzqLiS3kcqMnens65F33DQGUR+VFVB78vVF1FVWxkvi+uwPSzFZmWvN7fCXG8J9vAkNsin3bFnDROKbO+pUkqr6207Bk2C+/fCnRshdpJt+4l18OGV0FiPpO8gFZm701lFptODv8idke7FNqjMh4YqUSUleHDH4+O1rub5+3tVLEnXqKkWnQnqFSMR/sZ2x0YHejMs0g+zChuO2ikWEDkKblsNDx0VXSRAzIt+dbejxZb0IlKRuTudVWTQRJFl9548fZkizRoLigeDZ8fjI0eJZckJ2dbFjairqQLAw+iLTtc6h6wlZw3V3IvphW0P8guHO9bDmb8Tf+/9WDRZlfQJpCJzd7qiyGTAR/sUacnNocmdG+8bBn5aCauCA70jk6TLNNYJRebp7dvBSIHFvbjuSCFqR8Ec0+4GD+28ssBwn0EqMnfHGrXYQbAH2AI+pEVmn+LjYhkypPPHWKwy6V50G0x1Yo7My6d1jUV7TEkMwctDR355HYfzO7CsfULgqnfF+t6PoaqoB5JKnIVUZO6MqnbTIsvtPZn6MhZFFpzY+WOsimyf4+WRdAtVC/bw9unEbwLw8tAzNTEUgHWH23EvWhh6PkSNgfpK2PdZt+WUOA+pyNyZhmoRhQhg9OdkURUPfrybez/aRXpeRevx0rXYPiUnxLIrFpmljNXRH2WOkbugVfbw9eucIgObe3FtZxSZosDIS8X6SVnZpS8gFZk7Y7HGFB2n6/Rc/9YWPt+VzVd7crj0tQ3sziptPl66FttGVaE4Q6yHdMEiGzZXtP8oyZDlv9wEnUkkRPv7d16RnT1MJEZvzSimpt7U8QHx08Uyc5N8gOkDSEXmzjRxK768+ijZpTUkhPowZXAItQ1mFn24k8q6JsnPAVrUYkWu/PG1pLpIVOlAaZ0I2x6evhCeItbz9vaKaJKuodcUWVBAJ+aNNZLC/YgN8qa+0czmE52Y94qdCHpPkbJhcUlL3BapyNwZLdDD5OHP0m3CGnj+irG8fctkBgV7k11aw2s/H7WNt0TYmept7V8kgmLNrRgQAx6t+1e1S7SWX5QrFZk74KlqiiwwsNPHKIrCWZpV1m4YvgUPL1vS/MmNXZZR4lykInNnNIusXPWiwaQyJjaQM4aE4mc08KeLRRDCu79kUFylVSEweIoKHyDnyVpimR/rSqCHhaaJshKXUtdowksV3/eQoKAuHXteSiQAK/bmUN/YiULQCU3cixK3xuGKzGQy8cQTT5CYmIi3tzdJSUk89dRTHedvSFqjKbL8OpG8uyA11rpr9ogIRsUEUNNg4tMdWbZj/Ju4FyU2LBZZyOCuHxs5UiwLDzlMHEn3KCyvwag0ABDYhTkygHOGhxPhb+R0ZT0/HOhEPUXLPJmrAz4O/wDLboIPr4aVj4GpwbXyuCEOV2TPP/88r7/+Ov/85z85ePAgzz//PC+88AKvvvqqo9+q/2NVZB4AzBkZad2lKAo3nCHmej7e1kSRychF+/TEIgsbrp0jA+qrHCaSpOsUl5Zb1xXP9usstsRDr+PayXEAvL3hRMcP13FTRDmzkgzX/Z62/ReWXAUHvoQj38OW1+GNM2Vh4xY4XJFt3LiRSy+9lPnz5zN48GCuvPJK5syZw9atWx39Vv0fTZFVqN4MCvYmLqT5D/eisdF46nUcK6ziSL4WGCItMvtYk6G7ocj8IkT/MlRRVFbiMsrKy2x/GNqvfG+PhWckYDTo2JlZyvojdmovNsUrQOSTgWvmyUqz4NuHWm8vPAjf/9758rgxDldk06dPZ/Xq1Rw+fBiAPXv2sGHDBubNm2d3fF1dHeXl5c1eEg0t2KNC9eaMIaGtdvt7eTAjWWz/fr/2hCYtstaoKpwW30dCkrp+vKJA8iyxLif+XYrl/lCveIKu67evyAAvFk4Vnoz/+/Fwx1ZZvIvmyUyNsORqkUcaOQbu3wePZcL8f4j9+5dD5hbnyuTGOFyRPfbYY1x77bWkpKTg4eFBamoq999/PwsXLrQ7fvHixQQGBlpfcXFxjhap76JZZJV4My4uyO6Q80YId+PGY1qUorTIWlORCzUloOhtofRdZdBksZQT/y6lslL8JhqU9qvet8cd5wzBy0PHrsxS1nSUIG0J+HD2A8z2t231Pec8BUFx4BUIk34Noy4H1QRrnnWuTG6MwxXZsmXL+PDDD1myZAk7d+7kvffe4+9//zvvvfee3fGPP/44ZWVl1ldWVpbdcQMRtVZTZKo3o2Ps58xMGxICwI6TJdQ1mmy5ZLJMlQ1LoEdwQtdD7y0knSeWp7ZBZSfCtyW9QlWV+E2Y9N1XZBH+XtyozS+/tKoDq8zSyqfgAFQXd/s9u4TZBD8/LdbnLoakc237FAXO+6NYP7EetHJdAx2HK7KHH37YapWNGTOGG2+8kQceeIDFixfbHW80GgkICGj2kghqq0oBqFJ8GBFt/7okhfsR5udJXaOZPVll4K+5Fiuka9FKpeZ2tVir3SEgBkKHivXcPT2XSdItaqpFsI25G/NjTfnt2Ul4e+jZc6qMbRklbQ/0C4ewYWI9y0muvO1vQ22ZCDQZe3Xr/SFDwC9SWGUySR/oBUVWXV2NroXvWq/XYzZ3Im9D0oyailIAjL6BeHno7Y5RFIWp2vzZ5uNFtjmymhJrTboBjyXCyy+y2eZTJdX85r1tjPvLD1z75ib2ZZfZObgJMePFMmO942WUdIparakmhm5a1hphfkYuGisebD7bcar9wfFniOWpbT16z06TvlIsJ/9GtBJqiaLA4Jlifce7zpHJzXG4Irv44ot55pln+Oabb8jIyOCLL77gH//4B5dddpmj36rf01Atbqy+AcHtjpuUIPbvySoVfnRLPyUZ8CGwKDL/KOumvLJarn5jEz8eLKCspoHNx4u58o2N7Mxs5+l8+IViefi7XhRW0h61WlNNPHpmkQFcOXEQAN+k5bZffzFC64BQmN7j9+yQ0iw4tlqsj7cfVwCIvmkAaZ9ArQyQc7gie/XVV7nyyiu56667GDFiBA899BC//e1veeqppxz9Vv0eVfuCBgaGtDtuTKwo1bMvp0w8rVkjF2XxYEDUywOrRWY2qzy4bDc5ZbUMCfflg1unMjM5jNoGM7e9t52Cilr750mYIZaF6bY6mBKn0lAr5oT0DlBkkweHEBfiTWVdoy3q1x7hWh6hMxLi9y61rUeMbHtc7AQIHgzmRsiSqU0OV2T+/v689NJLnDx5kpqaGo4dO8bTTz+Np2cnWstLmqFvEDfLkFA77oUmjIwJQFEgv7xO3IRlCH5zWlhk/91wgo3HivD20POfmyYxc2gY/75xIilR/hRV1fPMNwftn8c/UuswoMp5MhfRqDXV1Bu7lgxtD51O4bJUYZV9ubudh76IEWJZfBwa63r8vu1iiY4853FRcq49LA9WJzf0rkx9AFlr0Y3xbhQWWXhE+0EKPp4GksJFt9z92eUQKH6c0iLTaKLIsktreHGVcBE9cdFIhmjXzddo4G9XjkOnwJe7c9hxso0INUt/sr0f97bUkhbUNZqgUcz7enj5OuScl4wTv631R05TWl1vf5BfpHDZq+bedS+WZtkS7kdd3vF4qyKTuY1SkbkppoZ6/BBPnzHRHUfbWdyLadllNousTCoyVBXKtMn8gFie+voAtQ1mpgwO4bopzXMWxwwK5KqJYturPx1teSZt0JViaZmQlziNkqoGjAhlY3CARQaQHOFPSpQ/jWa1bfeiokD0eLGevcMh72uXDy4XrsKEmRA+rOPxlhy37J1Q1UGVkn6OVGRuSl6+7UcVFRHVzkjBSC08/1BeuXQtNqUyHxqqQNGxszyA7/bnodcp/HXBKBRFaTX8rnOT0CmwJr2QtFN2ohiTzxfLqkI4faSXhZc0paiqDi9EwVylu/mAdrBEL67Y207u5aBJYpm93WHv24zTR23VZybd0rljggeLzgzmBjiwvHfk6iNIReamFBQIRVaBLzqDocPxyZHCRXa0oFJ2im6KpcZiYByvrj0JwBUTYkmJsp+XlxDqy6XjxfV7Y92x1gOMfran8+1vO1paSTsUV9XjpVhaFvU82MPCRWPFg9/GY0UUVbYxBxarKbJTvWSRrbjftt4ZtyIIS9GSqF8wsDszSEXmppSXiEi7Gn3nWlUka3M9GaerafSzVPeQisyiyCp84/k5vRCdAnedk9zuIbedOQSA7/flkVdmJ4LxjDvF8tjPshO3EymuqscbTdE40CIbHObL6NgATGaV79pyL1qabBYecny4e225LTdx9p+7VkPSUnLNWTlubopUZG5KZanwedd5dK7SSWyQN14eOupNZrLNWrh+dRE0tBFKPlAoElbV3mqRNH7JuBgGh7UfKDAyJoApg0NoNKss2XKy9YChc4RFUHhQVlZwIsVV9XjheIsMbFbZij1tuBf9I20dEHJ2OfS92fSaWPpGwPT7unZs8mzQGSB3NxS0EW07AJCKzE2pKRdRcyZjUKfG63QKQ8KEVXa4zGD7oQ/0UlWaRba+SDwQ3DR9cKcOu2m6qMW3ZGumiJZrik8IJGg1+E710pyJpBXCtag1lXSgRQYwf4zwYmw5UdR2HqHFKnNkwEd9FaQtE+tTf9v1iv5+4eLBCuDgCsfJ1ceQisxNqa8QikzxDur0MckR2jxZYRUEWubJpCIDONIYTlK4L6ltdBFoydxRUUQFeHG6sp6VaXbcTZab2rGfHCSopCOKmlpkHo6JWrQQF+LD+LggzCp8t68N96I14MOBiuyXV8R31MMHxl3XvXMM0YoKZw7cMHypyNwUc40olWTwab88VVOsiqygUobgg5i/0irfZ6hRXD0pzm6koj089DoWTo0H4N2NGa0HDNf66x1aAfn7HSGtpANKquqt4fc9rbVoD2v0YlvuRWvAx3bHzI021MKu/4n18/9qe/jsKhbvQNZW0cdsACIVmZuiqy0FwNO//fJUTbFZZJUQYEmK7qAgan+m6jTUV2BWFXKUCC6b0LUbxbVT4vHU69idVSrqWDYldqLI9wE4+qNj5JW0S3OLzLFzZAAXau7FbSeL7Qf5RI8TPe0q8xwTSPX6NHEeT//ORyraI2IkGAOhvhLy9/Vcrj6IVGRuSF2jCaNW1cM3sP3yVE2xKLJjBZWolqe7sgGsyIpFoEcOocwYPogI/649xYf72yqk27fKLhDLjF96IuXAoTwXfnhCBDfkpXX58Obh9463yGKCvJmUEIyqikLCrfD0sdU/zNndszfLS7Olhsy8D3xbd4DvNDo9xE0R6wO08atUZG5IQXkdAYgq394Bnf+CJ4T6oFOgsq6RCqOWRD2AFVlDrnD5HTPHcM3k7nUe/9WMwQCs2JtDQXmLp3RLK42M9VBV1F0xe5fSLDj0Daz9m2urpJ8+Cq9NhY2vwPe/hzdmwgdXgqmh06co7mWLDJomR7cxtxw9Viy7oYibsX+5WHoFwcwHe3YusLkXpSKTuAu5ZbUEKkKRdSXYw2jQExciJsFPmTVLbgArsqxDIqLwlGci5w4P79Y5xg4KYlJCMA0mlQ82twjFjx4vKis0VMO2t3oorQPJ2QUvj4M/B8JLo2Hp9aLj8HNxrots++YBqGtRKeXoqk730zKZVUqr6/HuxTkyEO5FRYFdmaX2oxcteVtFbZQw6ww1JbD5dbE+7wVhUfWUeK1c1clNAzK3USoyNyS3rIZAzSLDK6hLxw7RcqSO1mvHDWBFVp0t5gvChozHoO/+V/2WGYkAfLglk9qGJqH4igJTbhPrx9d2+/wOpb4a3r8USjLs7/94Iez6wKkiUZppc7/euwv+WAghSeLvI6s6dYqymgbMKjbXYi9ZZBEBXoyOEXVL1x+2U78wWKRlUGonv7CzHPpGlE2LGGm/A3R3iEkFvRGqCnqmZPsoUpG5IXlltfgromAwXp1LiLZgqea+v1I7rq4cakodKF3f4EheGbF1Yg5i4uQZPTrX3FGRxAR6UVRVz9d7Wric4rTuwdk7oCK/R+/TY2pK4L2LoFazfIbOhUtehfvT4Ir/2sZ9uQi2v+M8udb9HVSTqNYeMkS0J7lGU6bHfupUwdviKlHRw9uSR9ZLFhnA2cOE9b72cGHrnUGaIis6Ct3ter/rQ7EccbF4GHIEHl62ebLjaxxzzj6EVGRuSG5Zrc2F4unXpWOHhAuL7HBxo4hkAqgscKR4fYIf160lWKmkTvEibEhqj85l0OusidRv/5KB2tR1EzYUwkeAqQ5+cnHz2LcvsOU4XfUuLFwGE26CoHhRtf/OJvMnK+6Ho6t7X6bcPbDzPbF+5u9s2yNHQswEUfB2z1L7xzahqFL8Hnp7jgzgbM0Nvf5IIWZzCzddxEgwBoiHhu5U+CjP1fK9FJhwc8+FbYplznYAJulLReaG5JXVNqkp17UfrKW6x/HTVSLrH4S7YQBR22Ci/IC4SVdFTuq4QWEnuHZyHF4eOg7mlrPlRJNeZYoCs/8k1g9+3aXgBYdSWWjrYDzltzDqstZjIkfCg03KGC29Hsym1uMcyS+viGXKRbYCtxYm3CiWaZ90eJriKqHAjE5QZKlxQfh46impbuBwQYtO4AZPiNes8JydXT/5Sc3FGj22+3ljbWEpZj0Am75KReaG5JdVNSnF07UKBkmaRZZVXI3ZR1NkA8wiW7E3lwkmEVUWNGq2Q84Z5OPJ5RNEbt5/N5xovnPoHPAJg9pS243K2TS9qc59pu1xATFw2ZtivbEWfm5nbE8xm+Dw92J95gOt3WhDzhHLgoMdJvIWV9ejx4QBTfH2omvRoNeRGh8EwLaMktYDIkeJZXcS4S3fj8Fndmr4scJKVh3I52hLhWqP6HFieTpdzJUOIKQic0NKyppEd3VRkYX7G/EzGjCrUOWphe4PMEX24eYMpuiE5aFLPMth5/21Forf6sai0zep9PGNw96vS1g6C4+/AfQe7Y8ddw3Mf1Gsr39RzGH1Bjm7oL5CJPzG2HHvBg0WNUFNdVByovX+JhRXNgm9h161yAAmJYhCBDsy7HQKjxwtlgUHun5iS9h+7IR2h+WX1/Kb97Yx68W13Pb+dmb/Yx23v7+dspp2LP6AaPCLEp2se6tvmpsiFZmb0WAyU1XZ5CbZxSdPRVGs82TFaHNkA8i1eCCnnFNZJwlUqlFRIGq0w86dHOHPnJGRAPx77fHmO1MuEstD3zg//NnUAHs/FuvD5nbumEm3wrl/EOs/PWXLa3IkFqU+bI79EHOdDiK0cPYOKrc3q+oBvWqRAUwaLErDtW+RHeja/1pVoTBdrIePaHPY3lOlXPzqBn48WIBOgRHRARh0Cj8cyOf297dT39hOkInFyh1gNUClInMzCivqMGrzY6rBu+vVsLGF4OeatMjFAWSRLdl6knhFRA8qgYPAYHTo+e84R4SNf7Erm5zSGtuOIeeI8OfybFvFBmeRs1t0rPYOtlmGHaEocPYjMOIS8ffnt0NJD0LK7ZG7G4CT/hNY/O1BXvwhncyiFi4vS6WMDhRZcVU93k2rejgq2q8NUuOD0SmQXVpDbllN850hQ0DRCWuzsguRquU5IopY0UOo/Z54uzJLWPjWFgoq6hgW6ccPD5zFyvvO5PO7puNnNLDlRDGLV7ZzrRK0fDJHFjbuA0hF5mbkltVYIxYVz+5V+LaE4GfWaRGPVXbCiPshVXWNLN+Vw1CdVgcvZIjD32NCfDBnDBG9yv6zvok7zMPL5j5LX+nw920XSzWH+OkduxVbcvlbEDVGuPcc2fFaVVFzRa+2e9eY+Pe647z601EueHkdOzObWDkRmmXSgZuuqKquVwsGt8TPaGBEtHgQ3N7SKjMYtd5kWPvddYpCTQGFJtkNQMosqubmt7dSUdfI1MQQPr9rBskRorHu2EFBvHzteADe+SWDLcfbqCRjcVnm7O5+ekAfRCoyNyOntBYftIoC3WxVkaQpsqPV2jzCALHIvt6TQ2VdI2d6Z4gNllYrDsbSYfqjrZmUVDVxd43UrJu1L0BdJybnHYXl6TtucteP9fCCGfeL9S1vOO5JviIPpfo0JlUhXY3jknExjIsLorrexD1LdtkSy62KrAPXYmXvtXBpi4kJwr3YTPFasFiSXYkQtKQ7xLSeH2swmVm0ZCfltY2kxgfxzi2T8TMamo2ZNSKSayYJBfrIZ3ubJ+dbCB8h5h3rym1RrAMAqcjcjNyyGpsLpZs/WMsc2YEyza3WiYTT/sCHWzIBmGbULKVB3bixd4Izh4YxKiaAmgZT82LCk38D3iGiFNNJJ9a8y9YiFruruFPmC2uysRZWPOCQOb6MdaI9ySE1nocvSuWV61L54NYpRAV4kV1aw+c7NavZMldUdBQa69o83+lmiqz3LTIQ1jfAzszS1jstvclObev8CS2BHknnttr18bYs0rLLCPLx4LXrJ+DjaWg1BuAPF40gwt/IyaLq1tGzAHoDWAKcLPOmAwCpyNyM3LJavLqZQ2YhMcwXRYGTdZoirCrs9/XX9p4qJS27jGB9LcFV2hyV5WbjYBRF4U5truy9TRlU1Wmh4wYjjNCCPk44qWRV1WkoywQUWx5RV/HwhoWfiTm+3D2Qt7dHIpXVNJCxU7S2ORp1oTXa09/Lg9+caSn3pc3HBcSIxH3V1GZpJbNZpbiqrknl+96NWLRgUWQHcspaWz9RY8TSErzRGSq0hp2WXoEalXWNvPTjYQAemD2MmKC2P1+AlwePXygCZF77+aj9djMTbhLLvcv6/e/eQq8osuzsbG644QZCQ0Px9vZmzJgxbN8+sMJBu0tuaS0+VkXWPYvMy0NPTKA3RaoWtdhYI1qq92OWaNbYrYklKKiilJBfRK+937zR0SSE+lBa3cBHWzNtOxLPFktn1V48fUQsg+Ks5cxOV9bx8bZMa4DFd/vy7LuhmuIbCsmzxPqenj3J/+nLfSQ2CmthzqzzmzUzvXzCIPQ6hf055ZwsqhJBG6Fa3cVi+yH4pZY6i062yOJCvAn19aTBpLI/p0XBY2vx4COdb2ZpCQzxi2q2+c11xzldWc/gUB+umxLf4WkWjI9lQrxw0/79BzuKNOk8EVBSkdN2zc1+hsMVWUlJCTNmzMDDw4OVK1dy4MABXnzxRYKDO9/peCDTNNiDbgZ7gHAvVmOkUaf96PtxwEd5bQNfaTUQLw3TaiH2kjVmQa9TuPNscQN+fc0xm1VmcevkpzmntYvWc42QJIqr6vnTl/uYtng1j36WZg2wuOODHcx8/if+s/44dY3tKLQxV4nltre6XWz6m7257NiziwRdAapiwDu+ubszxNeTaUNEfuOqA9qNPXiwWLaRS1ZUKR7sQo1a8IKTLDJFUUi1uBdPljbfGRgnHjRN9R3mwAFQXSzmrQD8bYost6yGN9eJ/+EjF6Tgaej4lqwoCn+8SMzRfbErm6ziFpGgnj627/+2/3QsWz/A4Yrs+eefJy4ujnfeeYcpU6aQmJjInDlzSEpKcvRb9UtyymrxVnrmWgS0iCuFCkOQ2NCP58m+3JVNdb2J5Ag/BtVoQQOxvavIAK6YOIjBoT4UVdXzzi/azcwvwhYIcPznXpfB4o7L9xzEvJfX8d6mkzSYVMbEBnLLjMFcNyWemEAvTlfW8/Q3B5n30nrW2SuGCzD6coifJm7On/66y+W2quoa+euK/YxThGtXiU0FO22IzhomWgxtPKYp+hDhbmzLejit1Vm0KjInWWQAExKCANiV1SLgQ6eDsGFivYNAFUD0rAMxJ9ikEPhzKw9R22BmUkIw80ZHtXGwHbnig5mZHIbJrPLvdXYiJyf/RiyP/TQg3IsOV2RfffUVkyZN4qqrriIiIoLU1FTeeqvtXk11dXWUl5c3ew1U6hvNnK6sa1JnsfsW2UgtdPi0WfvR9FOLTFVV/qf1Cbt+SjyKxT0VPrzX39tDr+OB88XN7N/rjlNarVnSllyuTvba6hGaIntzv4788jqSI/xYcttUvr5nJn+6eBSLLx/D2kfO5fkrxhDmZ+T46Spuensrd324o3kenIVLXxNzVllbbMV+O8mb646TX17HRB8tSraN/8H0JKHItp4optFkhmBNkbXhWrTUWQzx7P3yVC1JjWvDIgNbxGVn5snytfQCzVIymVX+b9Vhvtydg6LAExeNbOaC7QyLzhXRs8u2n6KwokWgTOJZoDOItIZjTigO7WIcrsiOHz/O66+/ztChQ/n++++58847uffee3nvPfs/isWLFxMYGGh9xcV1r5NvfyC/vBZVBX9d9+osNmVUjFBg2Q1aLll1/7TI1qQXcji/Ej+jgSsmxNqe6i3uql7m4rExpET5U1HbyOtrtSdji4sue2fv5vKYzTRkirnndFMM54+M5MtFM6yKwoKHXsc1k+P56aGz+fWMRPQ6hW/T8pj14lpeX3OseaWI0CQYf51Y/+Z3nbbKqusbrRGcCwK0G3vUOLtjR0QH4G80UFnXSHp+RceuRa2FS7BFkfVyeaqmjIsLRK9TyCuvba34LYq6oBM1F4u0ucywoeSV1bLwP5t5ebXY9tCc4YyLC+qybGcMCWF8XBD1jWZb8IwF/yiY+Cux3onuAn0dhysys9nMhAkTePbZZ0lNTeX222/ntttu44033rA7/vHHH6esrMz6ysrKcrRIfQbLDyXMS5tv6YEiGxLuh5eHjkJT/06KfkNTHtdPjSew9pQIbNF72hJWexmdTuHhueKG9u4vGSKKLGyYmMdpqLLNYfUCDYWH8ajKpVo1YkiczmvXT8DXaD9sG0TE25MXj2TFPTOZPDiYmgYTz393iMtf/6X5PIvlBgi2or8d8NmOU5TVNDA0xEBQiRb1mHKh3bF6ncKYQSIQae+pMptrsTTTbjV+i2sxyOB8RebjaSAlSiQl72oZhm9J7zixvmP3nRaUc0yN5sJX1rP5eDE+nnpevGqc1bLqKoqi8OuZ4tp9sPlk6/nPcdoDyf4voPBwt96jr+BwRRYdHc3IkSObbRsxYgSZmZl2xxuNRgICApq9Biq5WihtqOXJswfBHnqdwvCoAEoQP0Jq7CR19nF2ZZaw5UQxHnqFW2YMhswtYkdMqkNat3SW81IimJQQTF2jmZdXHxZ1BS01Hnuxpcb6zeLzZirRvHDNlE4FCoCwiJb9dhr/uHocwT4e7Msu55J/bmCrpT1NxAgYv1Csf3ZrhxGvZrNqzWm6Z6yKoppFZ/OAttuUjB0UBIi0CfyjxcOHudFukIkl2MPfw5Lm4DxFBlgr4bdKjB40BXQewttRav/+Bgglp1UAufeHCoqr6hkZHcCKe2ZyxcRBPZJt3ugoogLEHOjXe3Kb74ydKJqZmhvhwJc9eh93x+GKbMaMGaSnN/cZHz58mISEBEe/Vb8jR6vpFmT5wfbwyXNMbAClqkiO7o9doi2Fey8dH0t0oLct/8lO5YTeRFEUHpsnwrGXbT/FscJKW05XLzU5rKk3sWuPaOzoFTmUiICuF5e+fMIgVtx7JmMHBVJS3cDC/2zm852aIrE0wWyshU2vtXuuHw/mk1FUTYCXgbmh2vxY5Kh26yGO0yyyPVllQvFbOi/bCfjILxeKLEBv+V04b44MbPlkO062LFXl2STgo50SWxW50FCFCR2H68OYnhTKp3dOs5aS6wkeeh03TRfX7u0NJ5o3fVUUm5t71/+gsd7OGfoHDldkDzzwAJs3b+bZZ5/l6NGjLFmyhDfffJNFixY5+q36HbmlwiIL0DumFM/EhGDK0H4s/cwiO15YyfcHRILp7WdpNRUtlRMsyapOZNLgEGaPiMBkVvn79+kQN1XsOLW1V97vqz3ZhNULpROXNKrb54kN8ubj26dx4ZgoGkwqDy7bw4s/pGMOHgILXheDfn7GVj3EDm+tFw8U102Nx1hgeZhovyv3WG1OKD2/QuS4tTNPllcuHvD8Da6xyCYPFi1d9mWXUVnXImcsUvM+tdebTHMrnjRH4O/rwz/bqdzRHa6fEo+Xh44DLZu+Aoy9GnwjoPQk7HrfYe/pbjhckU2ePJkvvviCjz76iNGjR/PUU0/x0ksvsXDhQke/Vb/jVImYp/DX93yODERPpVJVKDJzdf9SZG+tP4GqwqyUCIZF+gv3jcUic4EiA3h4bgo6BVbuy+OEt6ZccvdAg53owB7y4ZZMBmtV/vVhPUtt8fbU88/rJnCXVq3k1Z+OcuPbW8iIPN826LvH7R6742QJ2zJKhHt3eqKtwWcHiiwm0IswP09MZpUDueXthuBbqlf4WYOgnGuRxYX4EB/iQ6NZZeuJFrmBllSL9qqhaOH5x9VofnvWEEJ8Hev2DvLx5Aqt6eu/17aYk/X0FU1NAXZ/5ND3dSd6pbLHRRddRFpaGrW1tRw8eJDbbrutN96m33FSm3D30znGIhsU7I3OW8w51lbYaRDYRymoqOUzzQX2Wy0pmbIsqC0TcxaWqgtOZniUP+dr/creSmsUFRzMjaISuQPZn1PG3lNlDNZZEooTe3xOnU7hkQtSeOHKsRgNOn45WsScf27nzWFa6kzWZlvzTo1Gk5mnVgiX2oLxsURRZJsT7KBxpKIojI4V7sV92WWiwza0coHXNZqswR4+FkXmZIsMYEaySOL+5WgLRWbp9Jz+XZtdmcvTRZWXvQzjmsm9E4R068xEDDqFn9ML+dGSaG5h5KVimbOz33lmLMhai26Cyaxyqlg8uXv3sNaiBUVRSIgUP8C6OsdbBa7i3V8yqG80MyE+iMlaA0SrWzE8xamBHi25ZYZQKp/vyqY+phuFZTvBt2m5GGhkkKKlVDiwXc3Vk+L44YGzOGtYOPUmM8/u9WWVWUTnHd65hpNFVRRW1LEvu4xFS3ayO6sUX089D80dLhLAzY1ijrITMo3RFFnaqTKbldXYvHZggTY/5mnQ4WG2dIVwrkUGtty3X462SGMZNAl8w0UbnEL7idGNOUK56+KnEOTTO9/NIeF+3KrVsXz8i7TmqQKBsWIuTzW3ehjpL0hF5ibklddSbzJj0Cl4qtoPtgdRixaSY8QPsLHeTnHRPkh5bQP/2yRyZn57dpItiVTrfUX0WBdJJpiaGMKI6ABqG8yk12i1LqsdV6pKVVW+TcsjRilCj0kkB/tHO+z8AAmhvrx3y2SW/GYqkwcHc8os5oh+2HWUs/+2hsnP/MhFr27g+/356HUKL12bSmSAl63uYyer8FsVWXaZzcpq4Ya1RPJGB3qhWJSck9q4NGV6knggPJRXwenKJsnHimJzL+bsanVcQ20lQXWibFrqxOm9KuN9s4YyPNKfwoo6bvjvFlHL0sIQreL+MSdUm3EBUpG5CZbOuYOCvVEsP2YH/GBT4sLFSmMtJnPfL1XzweaTVNQ1MjTCj/NHRNp2uHh+zIKiKFw/VRR+3X9am+t04BzZobwKTpyuItmgRQcGJ3ari3hHKIrC9OQwlv12GhdMFJF5cT4mjFqIf5CPB7NSIvjszulWd6p1fswSydcBllyyIwWVNOg0S6WFRWaZN44O9IIGbZ8TK3tYCPUzWqvlrElvkZM5RCsUfWRVq+P2b/oeHSr5hDB9XO+6vH08Dfz3V5OIDvTieGEVF726gc93nhKRjJbWMc4om+YCpCJzEzKLxdNTfKivzdfugMTP5GjxJOmp1ou5iD5MbYOJt7V8pTvPSUKn06wxUyOc/EWs91Izza4wf0w0Bp1CZrn24NBgf+6kO6zcJyI1Z0VUig290AW7KYqiEB0hHoYuTfEj/el5nFh8IbufnMN/fzWZ8ZaKFIXpkLFBrA+d3alzRwXYAj5OaR+npdI/cVr8LhLD/ESyOzg1IbopFoX9w/685jssFpmdogOn0sT8WG7IVDz0vX+7HRTsw/JFM5gQH0RFbSMPLtvDXR/upDhCi6ItyeiX82RSkbkJJzWLLCHEx3bj8+x5nonBU/zojTSwoaV/v4+xbHsWpyvriQ3y5uJxTXo6ZW0RgR7eIW6hyEJ8PTlneDi1aI1NHWiRrT4oJvLPNewTGyz1/noTo1akQOt6bbcmYNonYg5m6NxOK9emAR8ZZVoRgBYW2fFCociSwn1dapGBTZGtP3K6eVscTy1Xs0XieElVParmbo1Obj+K05FEBnix7LfTeGjOMAw6hZX78rjkzd2oes3qrats/wR9EKnI3ARLxGJ8sDfUa180yw+kJxjEzdRLaWBDW1XP+wANJrM1AfqOs4c0f7q1WGNJ54rkWjdgQWosNYgbh+ogiyyvrJb9OeUoikpUmTYfY2nk2ZsYteowljYk9rBYYyMv6dKpx2qK7GixphhaKP1jheK3MCTc1+UW2aiYAGKDvKlpMLHhSJOHwjYU2Ve7MklVRGmoyCTnzt0a9DruPm8oyxfNYFCwN6dKaqhWLQ9WjvMQuAtSkbkJxwrED3ZoqEE82YJDFRnA3szT1NR30GDRTfl85ymyS2sI8/PkqkktQpgtbTSi7RepdQXnpURYe8FVVVY45Jw/HRLzYmfF6tDVlgKKc1INLFU3Mtbbj3orONQk7L5r7XMsFtmh01pofROLzGRWyShq4lp0sUWmKIrVKvs2rUk5KIvnpN5m6aiqyvFNyxmknKbOEGBruOpkRscG8tr1IhWizOQhNvbDJrtSkbkBJrPKcW0uIDmoidvGEdFZTX70iqmWdUf6nlVW22Di/1YJF80dZyfh5dHE6qqrhKM/ivWIkXaOdg0+ngYSo8XcUpmDWhNZFNklsdqNKDDOOdZJ7ESbgnrvYvhzYPPXv6aKp/zwFAgb2qVTTxocgqLA0RJLYIxNkR3IKae2wYy/0UB8iI9NybnIIgO4ZLxwaX+7L5eKWk352rHINhw9TXLZJvHH2KsdEoHcXcbFBTErJYIaaZFJepOs4mrqG80YDTqivS0Vvn0c4ybT2/JWjDSwYm9uO4Pdk3d+ySCvvJbYIG9uOKNFzc7cPcLl5RsBQ85xiXxtMXqwaJRYU91zi6y2wWTNYZoWqE3Wh3WvanqX0engxi/af1BImAm/+qbL39kQX0/GDgqixjqfaFMGW7QqGpMTQ9DrFNsN2IWKLDUuiOQIP2obzLYivZYHTlO9tZ7hG2uPMV4nesUZk2a6QtRmXDVpENXaNTbXSYusT6CazahmM6WFOZTln+z4ABdzRHMrJoX7oW+0BHo4wK0IIs9Fs8qMNLD6YH6fci9mFVfz6k/CGnvw/GHNrTGAwkNiGTMe9B7OFa4DxieJ6u9ejeXNc3q6webjRdQ0mIgO9CK6UWt1FNo166dHeAXAr7+HBW/AhJtBb3NZM/0euPlr8A1r+/h2OG94BHmqyFWjusgaVLJOm4eakhgiIlPNllqLrnEtgnAvXqtV5/jPhuMipaVpUNbh71h/pJCSYzsYo8sQ25zQrbwjzhkeQZ0irlt2Yf+p8mOhXyqy6qpyGv8SRtBrIwh8fSym9xe4deXno5oiS47ws7knHBCxaEX74ScFQXW9iS92ZTvu3L2Iqqr8cfk+qutNTEkM4bJUO21BcneLpTOi97pIQMI4zOgYpJxm/c60Hp3L4lY8NyUCResK3VU3Xo/xChBNNy95BZ4ogD+Xidecp3uUy3btlDiqdX4Uqlp0ZNFR9ueUsV5zg88dFWUL9ACXWmQA106JJ8jHg+OFVXy1J1tUkgnQ2rEsu5EzP0zmW+Pvxd/+MRDYs1YtjsDLQ4/RW9xTjmbndzC679EvFVllWREeis3q0B//Gba95UKJ2qeZItOeRh2qyPyFi+v6EcLN+K81R/uEVfaf9SdYe7gQT72OxZePseWNNcXSgyzuDOcK1xm8AinxF003C9N+6vZpVFVl9UGhyGalRMBprUliqJNci71MZIAXN00bzHFVzD+9/PE3XP3GJlQVLhwTRWKYb7O5M1daZAB+RgO3nSlSDJ755pDol3bJK/YHn/1Iu+1snIlvoMgprcpup+VMH6VfKrKwqAQKbtvN7cH/4ZXGBWLjzv+5VKb2OJgrggGGRfo3scgc5FoEa4PD82LqiQ704lRJDU98uQ+zG1f6WHe4kOe+E27DP140giR7vZuqiuC01vvO0jbFzTAmnwVAZMl2yqobunWOIwWVZJfWYDTomD440FYhvpMVNPoCj81LwRyiKeaio1TVm0iND+LpBVqlFotFZvByC8Vw68xEhkX6cbqyjkVLdlIx6CzS5n/J1+bpnFYDyAudCjevgEm3uFpUK15jRGrE6LK1rbtJ93H6pSLTGwxExCaSNHwM7zZeIDYWHoRq9/MN1zWaOJwvrLAxgwJ7R5Fprg1jVS7PXzEWnQKf7jjFDf/dwk+H8m3RV4in/4KKWjYfLyI9r6J5oz4nsf5IIbe9vx2TWeWy1FhubBngYSFLs8bChoNvqPME7AJ+w0XY9WTlEL8c615CusUam54UindVlpgr8vCFgJgOjuw7eBp0TJsqahFePaiMJb+Zyqd3TLe1PHFx6H1LvDz0vHRNKr6eejYfL2bqs6u55PMq7qm/mz8kf0Hk3d9D4pmuFrMZ0RPnY0ZhsJJH+pEjrhbHoTiuu5sbMijYm2ICyDXEiQnyrK0w/AJXi9WMw3mVNJpVgn08iAn0cmwytAWLj77sFGedE84/rh7PI5/tZeOxIjYeK0KnQHSgN0YPHacr6iivtTUPnD0iklevS8Xbs/cTjRtNZv6z4QR/+z4dk1nlvJQInrtijP1KEiBaiwDEu6c1BlgtxaG6bJYcOs6FY7pe4PdnbX7svJQIOK25hUKT3MIycShaV+3o6nSik1sEjrg4GdoeI2MC+PC2M7j3o11kagUNLhobzd+vGtf2d9aFKN7BZHomk1B/hKJ9q2GEa9od9Qb9WpHFBYuw2N1KCtFkQeYmt1NkaVr9w9GxgeLL3xvBHpprkXIR5LEgNZbU+CD+u+EEa9ILySyuJrtJ2wedAjFB3uSW1fLjwXz+8EUa/7hmvOPkaYGqqvx4sIDnVh7kmFaSaMH4GJ6/cixGQzsKNNOiyKb1mmw9xjeMar94fCoz0R3+DlWd3qWbXGl1PdtPCk/CuSkRcOBLscPZgR7OIHosoIjvaWUB+EXY9rmZRWZhfFwQP/3ubA7mVhDo7UF8qOvyxTpDScQUEk4dwSPrF2CRq8VxGP1bkYWIL9W6umTm6VbZXFFuxL4cmyIDesm1qCmyMlu0YkKoL3+9dDQgGlVmFddQ32gm1M+T+BAfvDz0bDpWxA3/3cLnu7I5f2Qk87phTXREZlE1f1iexnot1DrYx4NHL0jhmslx7d/wq4vh1Hbtw8xwuFyOxDBhIaxbzNTaXzh+usr+fF8brD1ciFmFlCh/BgX72FqlODP03lkY/YWCPn1YPKQ0LXdVrbllvYNcIlp7GPQ6ayV/d8c49Bw49SHxFTtdLYpD6ZdzZBZigrxQFNjUoE0iZ++Exrr2D3IyuzJLAVtvpt4J9tBci+XZYGfOK8Lfi4kJwUxLCmVYpL81V2taUih3ah2Yn/hyH8VVjk1h2HSsiPmvrGf9kdN4GnTceU4Sax85l2unxHdstRz9EVQTRIyC4Dbm0NwEz+FzADhDd5D1h7qWkG6ZHzsvRbNOXBV67yySzxfLA8ubby/R8kGD3Pt/7e4kpM7GpCrEq7nkZ2e4WhyH0a8VmdGgJ9Lfiww1ikavENHFNX+fq8WyUlpdz6E8EbE4ebCWEGppsWAp1OoILEEB9ZWiSnwXuGdWshadVc9fvt7vMJG+Tcvl5re3UlHXyIT4IL6//ywevSCFAK9OJjWnfyuWbuYqtkv0OOoMfgQo1Zw6uLnThzWazKxJ18LuR0SIh5B+FnrfitFXiOWhb6GiSb5TvvbdC4p3vkz9CJ+AEHL0wkOTcWC7i6VxHP1akQHEhXgDijWfx9pJ2A3YfLwYVRX5Y+H+WqUES2h18GDHvZGnD/hoUX2lmV061GjQ88KV49Ap8OXuHJZu7drx9nh/UwaLluyk3mTmglFRLLntDJEr1FlU1Va8dlgfUGQ6PbWxIiLPN3sjjSZzpw7bcbKE8tpGgn08GB8XDJX5ovKFooPw4b0pseuInQAxqSK4Y8nVUHQMVj8Fuz8Q+4ee71r5+gEV/okAlGfucbEkjqPfK7JBWsBHho+YD+LQNy6Upjk/ar2lZjaN0CoWrUoISXTsm4VpNz5LpfguMD4uiLvPFRbA41+k8cWuU90SwWxWWbzyIE9+uR9VhRvPSOC1hRNal53qiPIccUNHcXlH6M7in3IeABPNadYAn46wVvMYHiFqDeZp3oTQZLeK3nMoigKTfi3Wc3fDqxNg/d/F30mzYLB7hbT3ReqjJwMQXWCnk0Efpd8rsrhg8YPf4KW1UTixztaB2YXUN5pZdUAosgtGR2kbq6BS6z4b7GBFFjlKLLvpWn3g/GHceEYCqgoPfbKXHw90rcxNdX0jd3yww9pT7HfnD+Ovl44SN+iusuV1sQwZ0mdu6LohWj6ZLp3NR/I6GC0iOb/dJ+bTZo0QrUOs/7vI0b0io9sw/gZRv7EpY66Gaz7ofykHLsB39HwAkuoOoLpx6b6u0O8V2SAtcnFnVbhoe2Gqg5MbXSwVLN+dTVlNA5EBRtv8mMWt6BUEPiGOfUOrIuvePJeiKPzlklFcPiEWk1ll0ZKdpOd1rqp7XlktV72xiR8O5ONp0PHyteO5Z9bQ7ufaZGiNNMdc2b3jXUHECGo8Q/BW6ik8tKHD4TszS8gqrsHXU28L9MjRIs2i+rki0+lE/cabV8CNy+HJYrjiLZe2QulPJAwfT7nqgzf15B/d5WpxHEK/V2SWuZfjp6tFB2GAY92ve+cIaupN1oruv56RaLNK8pskuzoaiwsud4/dyMXOoNMpvHDFWM4aFk5do5n7lu5q3vLdDvuyy7j0tQ3szykn1NeTj26byqXj7RT/7Sxpn9pu6Kk3dv88zkZRaIwXbrHg/E0dXrflu3IAUTDX21MPNaVwfI3Y6ebpBg4j8Uy36vrdX/D0MHDMU5Q3K0p3/UO9I+j3iixZy9nJLq2hJv4csdGFisxkVnn40z1kFdcQE+hl66+lqrbCxr1RADdqjOhNVn0aSk50+zQGvY4XrxpHqK8nh/IqeG7lIbvjVFVl6dZMrnh9I/nldQyN8GP5ohlMTOiBpVl1Gr75nViPnQRBce2PdzP8tHmyKexn58mSNseVVTfw+U4xD3nZBE3pr31eRJyGDXOLtiCSvk1J0FgAzKd2uFgSx9Driuy5555DURTuv//+3n4ruwT7ehLmJyICj/lNFBFfhQdFwICTsbQlWbE3Fw+9wt+uGoevUctJ3/KGSNjWecCEmxz/5gajtQQQWVt7dKpwfyN/v2ocAO9uzGBlWvPcqFMl1dzxwQ4e+zyNukYz56VE8Nld060J6t2ivgqW3Qy1pUIhX/dRDz6Ba1CGiALCE5QjbE8/3ua4/23OoKrexPBIfxEIVFMKW7WHnAueA32/rmMgcQJKnHgYCil1nyjuntCrimzbtm38+9//ZuzYsb35Nh0yLFJYZYfKDBAzQWx0slWmqirPrTzER1sz0Snw0jWpzLBEK57aAd9r/Ytm3AcRvVQDLW6KWDqgwsm5KRH8eoYISLl36S7+8vV+3lp3nN/+bzvn/G0N3+/PR69TeOSC4fznpkmdzw+zh6rCspvg5Aah6G9d1bx8UV8hZAhl/sl4KCaq9v9gd0huWQ1vaAExd56TJOYRD38H5gYIT4HkWc6UWNJPCRsuyrpFN2SidjG31B3pNUVWWVnJwoULeeuttwgODm5zXF1dHeXl5c1ejmZohFBkRwoqbHkoe5Y6/H0A0cAz7VNY8zx8frt4n+pili9fxtvrRDLr4svHMH+sVu4pewd88itQzTDqMjjvj70jFzRRZNsccrrfX5jCgvExNJhU3vklg2e+Pcj3+/NpNKtMTwrlm3tnctc5yfb7iHWFoz+Kl94ICz8R3aD7KF5DxTxtVPlejuQ3D5ZpMJl5+JO9VNY1khofxMXjtET2A1+J5YhLkEgcwdAhSZxSw9ChUnLE/Ur3dZVe81EsWrSI+fPnM3v2bJ5++uk2xy1evJi//OUvvSUGAMmRokrGkfxKuOwGWPsCZKwXOWUp8x3zJqoqFNjnv2m+fe/HAFwGXGapd/qN9mqKwQtm/al3w4sHaYqsYL9o4NnD6iEGvY7/u2Y888fGsOpAHtX1JpLC/Zg3JoqUqICey9tYLyxVy9zh2KtsATt9FGPiGbDzLSbojvDuxgyeuUwE4dQ1mnjok71sOHoaLw8dL1wxVgQBlWULJQ4w8lIXSi7pT3h56DnqmcKghg2UHN5IyJg5rhapR/SKIlu6dCk7d+5k27aOn/wff/xxHnzwQevf5eXlxMU5dhI/JUrcsPfnlEHgZJhwI+x4F5ZeD3fvgDCt3E9tOVQVQs4uqMiDH/7Q/ERnLIKyLNAZRJ5L0VGhwHZ/CIXpthyw7nDl245Pgm5JQLQo8VOaCRkbYPi8jo+pqxB17gJi7KYEKIrC+SMjOX9kpGNlzdkF/50r0iUAEmaKkOy+jtbWZZSSwVfbjjAxIRijQc8/fz7Kwdxy9DqFfy2cwFDt4YvvHhXXYNAUWwqFROIASoLGQuEGdDk9KyCsqirfpuVRUl3PlRMHdb3AgQNwuCLLysrivvvuY9WqVXh5ddxywWg0YjQaHS1GM0bHBKLXKeSX15FbVkP0uX8UigzgnxM7f6LNr9nW93/e9rhbvoPQZFbuz+PBL49jMNdz9dQh/HGKDmX93+HQCtvY4fPhkled1xhy+IUisOTAV+0rMrMZfnoKNvxD/K0zwMRbYMa9vVPvTlVFncmKXPjgSqhoEYxz/cdgdGBrG1cROAj8YzBU5DCaozy4zPbdD/T24J/Xp3Lm0HCxoabEVonmon/IZGCJYxk0CQohtHSv+P118/v1+tpjvPjdAW7Q/8i6TVXMueAycZ/ROS8o3uGKbMeOHRQUFDBhwgTrNpPJxLp16/jnP/9JXV0der1zNba3p57hkf4cyC1nV2Yp0WOi4ZaV8E4nLJKEGSLk+dhqqCmDOjsTo8ZAmHIbJJ4l5qE8vHl7wwn+uiITMHD5hAT+cOk4FJ0C137o8M/XJYbOEYrs2GrRCcBg5yHi5EYR5l5wwLbN3ChcfNvegsm/gQv/7rgba2Uh/PBH2Nti3tLDF85+RATA9JebuKLAkLNhz0c8FLuf+yon4eOp59yUCG4/cwihfk3+H5mbxdxpaHKfKcUl6TuEJk+mYaeeAFOJ8DR14wE1v7yWV1Yf4X2P55ih3w9lwMefip13bem9wLUWOFyRzZo1i7S0tGbbbrnlFlJSUnj00UedrsQsTEgI4kBuOZuPF4kuvQnTYdFWePciqCoA72AYd7240cy4r/2ouMpCcYPxj4SqIuFy0260qqry0qrDvLxaJDz/avpgnrxoZM8DHhxFwgzwjxaWz8GvW1fHOLkRllxrU9iJZ8Mlr4jSXt8+Ioq5bvsPRIwQCq2nlGTAv88WYfVNufr9/jsnNPpK2PMRE+u2s+GRd9tW0ukrxTLxLKeJJhk4pMRFcEiNY4ySQX3mNjy7ocg+3HyS0Y0HmGG0UzHoX1PhgQO2foi9iMMVmb+/P6NHNy+h4+vrS2hoaKvtzuSsoeF8sDmTNemFqKoqwprDh8PDR7p+Mr9w23oLl+BLPx6xKrGH5gxj0bnJ7tX23MMLxl8P618UuUljrhRuhSOrhBVWplW3DxsGCz+19foKHgwjLobvfg97loixNaVw5u+6by3teA++vtf2d/JsOOf3MKgL7t6+SMJ0kQtXfkpUd7fM0Talsd7mgu6vCl3iUsL9jazVD2eMmkHpkc1EjL2iS8c3msx8vD2L2/S2WIjJtf9ijsduntG/KTYUHOybisxdmZEchqdeR2ZxNen5FY6JqmvBd/vyrErsiYtGcuvMXg7e6C6Tb4ONr0LWZviznc62E24WEZQt5+28g+Hil0Xl+SPfizm0yFGdCxppydEfmyuxm78eOJaHp48I+shYD8d/tq/Idv1PXGe/KBHoIpE4GEVRqAgcDqXf05h3oOMDWvBzeiGny6u5zutnANQr/kvQqnA+LAjk8rGhTAyuFXPCTsAps3Fr1qzhpZdecsZbtYmv0cC5KcKSWro1y+HnP1pQyUOfiP4+v5mZ6L5KDET04sgFrbcnzRJRnJe80nbwicETrl1iUzofXWtrPdMZVBWW3wUfNHn6u37ZwFFiFoacI5bHfm69r7Ee1mmtS878nazkIek1DFFiDsuntOueqY+2ZpKsZONLLXj6oYy6nAWpsYDCP8rOgdl/dtocWb+vtdiU66cKN9nH27IoKK912Hkr60SLksq6RqYmhvDYPOf883rEhX8TrTEszH0WbvzcvnXQEr0BLnsTFG2+c8NLnXvPmlL47FaRrmDh0ZMwbG5npe4/JM8Wy+M/Q0OL7+LhlSJq0ycMJt7sfNkkA4agxIk0qjqCGvK71HQ3u7SGNekFjFYyxIbocaDTcYmWxL/xWJFD77EdMaAU2VlDw0iND6KmwcRjn6dhNnevCnxTVFXlkU/3cLSgksgAI/+8fgIGfR+4rN5BojXGn8vEa9qirh0fEC0sKRBusAL7xYOtNNTAu/Nh32e2bXdtFnIMRKLHQUAsNFQ3T+UoyxbluEAoO3tRpRKJgxgaH8NBVQR5qDmdb+mybFsWZhVmBWu5s1GiDGFciA8TE4JRVfh6b247Z3AsfeCO6zgUReHpBaMxGnT8dKiAJ7/ah6mHyuzNdcf5Ni0PD73CvxZOJNx/AN14ks4TUY2qGTb8X9vjdr4Pz0TZGkPOeUYoz4gRzpHTHVEUW8To8jvB1ABmEyy5xjbm7EdcI5tkwJAU7sdJRLm8spxjnTrGZFZZtl1Mz0zx0qZpom31dC8dL6yyL3dnO1DS9hlQigxgVEwgz18xFkWBDzZncuN/t3CqpHsdo7/cnc1z3wlL5MmLRjIxoe2akv0SnQ7O18qLpX0CObtbj9n+NnzVpNvvLSth+t1OEc/tmX4foEV8PhUG710C+VrqysLPeqcvnUTSBE+DjipvEVVYnnu0U8esO1xIblktod46QivSxcbocdb988dE46FX2HuqjIO5jq+da48Bp8gAFqTG8up1qXh56Nh4rIhZL67l2W8PklfWeZ/ul7uz+d2yPagq3DQtwdZXbKARkwpD54Jqgg+vEvNgFg58BSsesP196yoRei4R+IYKq9bCSa1z9ISbYehs18gkGXAowYMBMBd3rk/hR1vFXNp9yYUo9ZUimjlsuHV/qJ+ROSOjCPbx4GRRlcPltceADYe6aGwMo2ICefSzvWw9Ucyb647z1vrjpMYFMTImgNggH6IDvYgM8CI60IvYYG889Dpq6k28vPoIb6wVZviC8TH8+eJR7pUr5mwWvA7/OkMklj9vR6GHDYPrlkoLwx4LP4Wtb4rWOmVZMO46mPRrV0slGUD4RiVBHhgrT3U4tqCiltWHCgC4xEPra5hyUavI2j9dMpIALw+n1V0csIoMIDHMl49vP4M16YX8a81RtmWUsDOzlJ2Zpa3GGg06EsN8ySqupqpetKn/zcxEfn/hCPep2uEqfEPh+qWiRmJNcfN9Iy6By98ED2/XyObu6HRwxh3iJZG4gIj4FNgNoQ25osZqOzUSP91xCpNZZVKcP0EnvxMbRy1ofU7/juvsOpIBrchABICcmxLBuSkRZBVXszOzhPS8CvLKasktqyW/vJacshpqG8wcyhP9o+JDfPj9hSO4YHSUi6V3I2InwoMHRW81RSeqUXgFwvALXC2ZRCJphyFJw2hUdXgqDVQXZeETbn+apNFkZskW4VZclFQImwvBK0gEfLmYAa/ImhIX4kNciE+r7WazyomiKjKLqwn19WR0TKC0wuzh4SUsM4lE0mcIDfTjlBLOIPLJOn6A4W0ospX78jhVUkOIrydnNvwkNqZcBPoedH93EAMy2KOr6HQKSeF+nDs8grGDgqQSk0gk/YpSLxG5WJR12O5+VVV5c52o4HPTtAQMpzaLHW7icZGKTCKRSAY4DQHCCqsrsB+Cvya9kLTsMowGHTeNCxDFgKFZ2L0rkYpMIpFIBjge4SKi2LOsdQi+yazy3EqRL3vTtARCjnwq0m1Ch0JgnFPlbAupyCQSiWSAE5wgGrdG1R6nwWRutu/THVmk51cQ4GVg0bnJoj4owKRb3KbhrVRkEolEMsCJGjYJgARy2XEsz7q9oLyWZ74RbsR7zhtKkCdwcpPY6QbRihakIpNIJJIBjj4wlmqdHwbFzJ5dItFZVVV+/8U+ymsbGR0bwC0zBkPmRmioAt9wiBjpWqGbIBWZRCKRDHQUhbpQUcQ78+A2Kusa+deaY/x4MB8PvcILV4wTXT32fiLGD53bbuK0s5F5ZBKJRCIhcPB4KNxGfOMJ5r28jqziGgD+dPEoRsYEiAa6e5aIwSMucp2gdnAflSqRSCQSl6GLFK7CEbosqxJbdG4SC6eKfmXsWSpaNsVMgGHukT9mQVpkEolEIoHI0QBM88vjoUnDOGNIKJMGh4h9qgq7NWts2iK3iVa0IBWZRCKRSKyNbj2qC7h7agj4htj2FRwQ3Rn0njB8nosEbBvpWpRIJBIJGP0hSKuzWLDftl1V4ev7xXry+eDp63TROkIqMolEIpEIIkeJZf4B27Yf/wyntgprbPafXCJWR0hFJpFIJBKBVZHtE5bY3mXwy8ti29xnIXx428e6EDlHJpFIJBKBFvBB9g745kHY/rb4e+y1MOU218nVAQ63yBYvXszkyZPx9/cnIiKCBQsWkJ6e7ui3kUgkEomjiZ0olgUHbEps1OVw0f+5TqZO4HBFtnbtWhYtWsTmzZtZtWoVDQ0NzJkzh6qqKke/lUQikUgcSeAgEdBhwS8SFrwOnq0bDrsTiqqqam++QWFhIREREaxdu5azzjqrw/Hl5eUEBgZSVlZGQEBAb4omkUgkkpaYGiF7O6BA3BSX5ox1Vh/0+hxZWVkZACEhIXb319XVUVdXZ/27vLy8t0WSSCQSSVvoDRB/hqul6BK9GrVoNpu5//77mTFjBqNHj7Y7ZvHixQQGBlpfcXHu0ahNIpFIJH2DXnUt3nnnnaxcuZINGzYwaNAgu2PsWWRxcXHStSiRSCQDHJe7Fu+++25WrFjBunXr2lRiAEajEaPR2FtiSCQSiaSf43BFpqoq99xzD1988QVr1qwhMTGxy8eDnCuTSCSSgY5FD3TkOHS4Ilu0aBFLlizhyy+/xN/fn7w80TY7MDAQb2/vDo+vqKgAkHNlEolEIgGEXggMDGxzv8PnyJQ2QjXfeecdfvWrX3V4vNlsJicnB39//zbP1Rksc21ZWVlyrk1DXpPWyGvSGnlNmiOvR2ucdU1UVaWiooKYmBh07XSk7hXXYk/Q6XTtzql1lYCAAPnla4G8Jq2R16Q18po0R16P1jjjmrRniVmQRYMlEolE0qeRikwikUgkfZp+q8iMRiN/+tOfZGh/E+Q1aY28Jq2R16Q58nq0xt2uSa/XWpRIJBKJpDfptxaZRCKRSAYGUpFJJBKJpE8jFZlEIpFI+jRSkUkkEomkTyMVmUQikUj6NP1Skb322msMHjwYLy8vpk6dytatW10tUq+xePFiJk+ejL+/PxERESxYsID09PRmY2pra1m0aBGhoaH4+flxxRVXkJ+f32xMZmYm8+fPx8fHh4iICB5++GEaGxud+VF6heeeew5FUbj//vut2wbi9cjOzuaGG24gNDQUb29vxowZw/bt2637VVXlySefJDo6Gm9vb2bPns2RI0eanaO4uJiFCxcSEBBAUFAQt956K5WVlc7+KA7BZDLxxBNPkJiYiLe3N0lJSTz11FPNKhP192uybt06Lr74YmJiYlAUheXLlzfb76jPv3fvXs4880y8vLyIi4vjhRdecPyHUfsZS5cuVT09PdW3335b3b9/v3rbbbepQUFBan5+vqtF6xXmzp2rvvPOO+q+ffvU3bt3qxdeeKEaHx+vVlZWWsfccccdalxcnLp69Wp1+/bt6hlnnKFOnz7dur+xsVEdPXq0Onv2bHXXrl3qt99+q4aFhamPP/64Kz6Sw9i6das6ePBgdezYsep9991n3T7QrkdxcbGakJCg/upXv1K3bNmiHj9+XP3+++/Vo0ePWsc899xzamBgoLp8+XJ1z5496iWXXKImJiaqNTU11jEXXHCBOm7cOHXz5s3q+vXr1eTkZPW6665zxUfqMc8884waGhqqrlixQj1x4oT6ySefqH5+furLL79sHdPfr8m3336r/uEPf1A///xzFVC/+OKLZvsd8fnLysrUyMhIdeHCheq+ffvUjz76SPX29lb//e9/O/Sz9DtFNmXKFHXRokXWv00mkxoTE6MuXrzYhVI5j4KCAhVQ165dq6qqqpaWlqoeHh7qJ598Yh1z8OBBFVA3bdqkqqr4Qut0OjUvL8865vXXX1cDAgLUuro6534AB1FRUaEOHTpUXbVqlXr22WdbFdlAvB6PPvqoOnPmzDb3m81mNSoqSv3b3/5m3VZaWqoajUb1o48+UlVVVQ8cOKAC6rZt26xjVq5cqSqKomZnZ/ee8L3E/Pnz1V//+tfNtl1++eXqwoULVVUdeNekpSJz1Of/17/+pQYHBzf73Tz66KPq8OHDHSp/v3It1tfXs2PHDmbPnm3dptPpmD17Nps2bXKhZM6jrKwMgJCQEAB27NhBQ0NDs2uSkpJCfHy89Zps2rSJMWPGEBkZaR0zd+5cysvL2b9/vxOldxyLFi1i/vz5zT43DMzr8dVXXzFp0iSuuuoqIiIiSE1N5a233rLuP3HiBHl5ec2uSWBgIFOnTm12TYKCgpg0aZJ1zOzZs9HpdGzZssV5H8ZBTJ8+ndWrV3P48GEA9uzZw4YNG5g3bx4wMK9JUxz1+Tdt2sRZZ52Fp6endczcuXNJT0+npKTEYfL2WodoV3D69GlMJlOzGxBAZGQkhw4dcpFUzsNsNnP//fczY8YMRo8eDUBeXh6enp4EBQU1GxsZGWntFZeXl2f3mln29TWWLl3Kzp072bZtW6t9A/F6HD9+nNdff50HH3yQ3//+92zbto17770XT09Pbr75ZutnsveZm16TiIiIZvsNBgMhISF98po89thjlJeXk5KSgl6vx2Qy8cwzz7Bw4UKAAXlNmuKoz5+Xl9equXLT31JwcLBD5O1Ximygs2jRIvbt28eGDRtcLYrLyMrK4r777mPVqlV4eXm5Why3wGw2M2nSJJ599lkAUlNT2bdvH2+88QY333yzi6VzDcuWLePDDz9kyZIljBo1it27d3P//fcTExMzYK9JX6ZfuRbDwsLQ6/WtItDy8/OJiopykVTO4e6772bFihX8/PPPzfq5RUVFUV9fT2lpabPxTa9JVFSU3Wtm2deX2LFjBwUFBUyYMAGDwYDBYGDt2rW88sorGAwGIiMjB9T1AIiOjmbkyJHNto0YMYLMzEzA9pna+91ERUVRUFDQbH9jYyPFxcV98po8/PDDPPbYY1x77bWMGTOGG2+8kQceeIDFixcDA/OaNMVRn99Zv6V+pcg8PT2ZOHEiq1evtm4zm82sXr2aadOmuVCy3kNVVe6++26++OILfvrpp1Zm/MSJE/Hw8Gh2TdLT08nMzLRek2nTppGWltbsS7lq1SoCAgJa3QDdnVmzZpGWlsbu3butr0mTJrFw4ULr+kC6HgAzZsxolZJx+PBhEhISAEhMTCQqKqrZNSkvL2fLli3NrklpaSk7duywjvnpp58wm81MnTrVCZ/CsVRXV7fqOKzX6zGbzcDAvCZNcdTnnzZtGuvWraOhocE6ZtWqVQwfPtxhbkWgf4bfG41G9d1331UPHDig3n777WpQUFCzCLT+xJ133qkGBgaqa9asUXNzc62v6upq65g77rhDjY+PV3/66Sd1+/bt6rRp09Rp06ZZ91vCzefMmaPu3r1b/e6779Tw8PA+G27ekqZRi6o68K7H1q1bVYPBoD7zzDPqkSNH1A8//FD18fFRP/jgA+uY5557Tg0KClK//PJLde/eveqll15qN9Q6NTVV3bJli7phwwZ16NChfSbUvCU333yzGhsbaw2///zzz9WwsDD1kUcesY7p79ekoqJC3bVrl7pr1y4VUP/xj3+ou3btUk+ePKmqqmM+f2lpqRoZGaneeOON6r59+9SlS5eqPj4+Mvy+M7z66qtqfHy86unpqU6ZMkXdvHmzq0XqNQC7r3feecc6pqamRr3rrrvU4OBg1cfHR73sssvU3NzcZufJyMhQ582bp3p7e6thYWHq7373O7WhocHJn6Z3aKnIBuL1+Prrr9XRo0erRqNRTUlJUd98881m+81ms/rEE0+okZGRqtFoVGfNmqWmp6c3G1NUVKRed911qp+fnxoQEKDecsstakVFhTM/hsMoLy9X77vvPjU+Pl718vJShwwZov7hD39oFibe36/Jzz//bPfecfPNN6uq6rjPv2fPHnXmzJmq0WhUY2Nj1eeee87hn0X2I5NIJBJJn6ZfzZFJJBKJZOAhFZlEIpFI+jRSkUkkEomkTyMVmUQikUj6NFKRSSQSiaRPIxWZRCKRSPo0UpFJJBKJpE8jFZlEIpFI+jRSkUkkEomkTyMVmUQikUj6NFKRSSQSiaRP8/98Mo7FrawUPgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NRMS=43.98%\n",
      "RMSE=0.9248 V\n"
     ]
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "test_sim_CT = model_CT.simulate(test)\n",
    "plt.figure(figsize=(5,2))\n",
    "plt.plot(test_sim_CT.y) #simulate is apply_experiment equivilent\n",
    "plt.plot(test.y)\n",
    "plt.show()\n",
    "from nonlinear_benchmarks.error_metrics import NRMSE, RMSE\n",
    "print(f'NRMS={NRMSE(test.y[model_CT.na:], test_sim_CT.y[model_CT.na:]):.2%}')\n",
    "print(f'RMSE={RMSE(test.y[model_CT.na:], test_sim_CT.y[model_CT.na:]):.4f} V')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d41b5ea6",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert RMSE(test.y[model.na:], test_sim_CT.y[model.na:]) < 1.0\n",
    "assert hash_array(test_sim_CT.u)+hash_array(test_sim_CT.y) == '3fc2c4216721bc73'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c9663d7-1f72-4d64-ad57-a30854a2377a",
   "metadata": {},
   "source": [
    "### Changing the intergator\n",
    "\n",
    "You can change out the integrator as desired as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0fa87163-7e40-4cfb-9e7b-3a589b00a0f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE=0.9292 V\n"
     ]
    }
   ],
   "source": [
    "from deepSI.networks import euler_integrator, rk4_integrator, rk45_integrator\n",
    "from nonlinear_benchmarks.error_metrics import RMSE\n",
    "\n",
    "#this is a 3 step rk4_integrator\n",
    "integrator = rk4_integrator\n",
    "n_steps = 3\n",
    "model_CT.integrator = lambda f,x,u,dt: integrator(f,x,u,dt, n_steps) \n",
    "print(f'RMSE={RMSE(test.y, model_CT.simulate(test).y):.4f} V')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42d7ff99-7e07-40d1-8f86-dda7aa8af9d3",
   "metadata": {},
   "source": [
    "## 2.1 Cutomizable SUBNET\n",
    "\n",
    "If you want to parameterize differently (number of layers and different functions) than you can pass your desired `f`, `h` and `encoder` ($\\psi_\\theta$) into the SUBNET structure class.  \n",
    "\n",
    "Keep in mind that these function need to have the following function characteristics where $b$ is the batch dimension.\n",
    "\n",
    "$$f_\\theta : \\mathbb{R}^{b \\times n_x} \\times \\mathbb{R}^{b \\times n_u} \\rightarrow \\mathbb{R}^{b \\times n_x}$$\n",
    "\n",
    "$$h_\\theta : \\mathbb{R}^{b \\times n_x} \\times \\mathbb{R}^{b \\times n_u} \\rightarrow \\mathbb{R}^{b \\times n_y} \\text{\\ if feedthrough=True else }h_\\theta : \\mathbb{R}^{b \\times n_x} \\rightarrow \\mathbb{R}^{b \\times n_y}$$\n",
    "\n",
    "$$\\psi_\\theta : \\mathbb{R}^{b \\times n_b \\times n_u} \\times \\mathbb{R}^{b \\times n_a \\times n_y} \\rightarrow \\mathbb{R}^{b \\times n_x}$$\n",
    "\n",
    "Or in the case that the input and output are scalars (`nu = 'scalar'` and `ny = 'scalar'`)\n",
    "\n",
    "$$f_\\theta : \\mathbb{R}^{b \\times n_x} \\times \\mathbb{R}^{b} \\rightarrow \\mathbb{R}^{b \\times n_x}$$\n",
    "\n",
    "$$h_\\theta : \\mathbb{R}^{b \\times n_x} \\times \\mathbb{R}^{b} \\rightarrow \\mathbb{R}^{b} \\text{\\ if feedthrough=True else }h_\\theta : \\mathbb{R}^{b \\times n_x} \\rightarrow \\mathbb{R}^{b}$$\n",
    "\n",
    "$$\\psi_\\theta : \\mathbb{R}^{b \\times n_b} \\times \\mathbb{R}^{b \\times n_a} \\rightarrow \\mathbb{R}^{b \\times n_x}$$\n",
    "\n",
    "If a simple multi layer perceptron or feed forward neural network (with linear residual layer) is desired you can use `MLP_res_net`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1919a9b6-0052-428f-a420-2c45c8036dc8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IO_normalization_f(\n",
      "  (fun): MLP_res_net(\n",
      "    (net_res): Linear(in_features=3, out_features=2, bias=True)\n",
      "    (net_nonlin): Sequential(\n",
      "      (0): Linear(in_features=3, out_features=64, bias=True)\n",
      "      (1): Sigmoid()\n",
      "      (2): Linear(in_features=64, out_features=64, bias=True)\n",
      "      (3): Sigmoid()\n",
      "      (4): Linear(in_features=64, out_features=2, bias=True)\n",
      "    )\n",
      "  )\n",
      ")\n",
      "IO_normalization_h(\n",
      "  (fun): MLP_res_net(\n",
      "    (net_res): Linear(in_features=2, out_features=1, bias=True)\n",
      "    (net_nonlin): Sequential(\n",
      "      (0): Linear(in_features=2, out_features=10, bias=True)\n",
      "      (1): Tanh()\n",
      "      (2): Linear(in_features=10, out_features=10, bias=True)\n",
      "      (3): Tanh()\n",
      "      (4): Linear(in_features=10, out_features=10, bias=True)\n",
      "      (5): Tanh()\n",
      "      (6): Linear(in_features=10, out_features=10, bias=True)\n",
      "      (7): Tanh()\n",
      "      (8): Linear(in_features=10, out_features=10, bias=True)\n",
      "      (9): Tanh()\n",
      "      (10): Linear(in_features=10, out_features=1, bias=True)\n",
      "    )\n",
      "  )\n",
      ")\n",
      "IO_normalization_encoder(\n",
      "  (fun): MLP_res_net(\n",
      "    (net_res): Linear(in_features=10, out_features=2, bias=True)\n",
      "    (net_nonlin): Sequential(\n",
      "      (0): Linear(in_features=10, out_features=15, bias=True)\n",
      "      (1): Tanh()\n",
      "      (2): Linear(in_features=15, out_features=2, bias=True)\n",
      "    )\n",
      "  )\n",
      ")\n",
      "SUBNET(\n",
      "  (f): IO_normalization_f(\n",
      "    (fun): MLP_res_net(\n",
      "      (net_res): Linear(in_features=3, out_features=2, bias=True)\n",
      "      (net_nonlin): Sequential(\n",
      "        (0): Linear(in_features=3, out_features=64, bias=True)\n",
      "        (1): Sigmoid()\n",
      "        (2): Linear(in_features=64, out_features=64, bias=True)\n",
      "        (3): Sigmoid()\n",
      "        (4): Linear(in_features=64, out_features=2, bias=True)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (h): IO_normalization_h(\n",
      "    (fun): MLP_res_net(\n",
      "      (net_res): Linear(in_features=2, out_features=1, bias=True)\n",
      "      (net_nonlin): Sequential(\n",
      "        (0): Linear(in_features=2, out_features=10, bias=True)\n",
      "        (1): Tanh()\n",
      "        (2): Linear(in_features=10, out_features=10, bias=True)\n",
      "        (3): Tanh()\n",
      "        (4): Linear(in_features=10, out_features=10, bias=True)\n",
      "        (5): Tanh()\n",
      "        (6): Linear(in_features=10, out_features=10, bias=True)\n",
      "        (7): Tanh()\n",
      "        (8): Linear(in_features=10, out_features=10, bias=True)\n",
      "        (9): Tanh()\n",
      "        (10): Linear(in_features=10, out_features=1, bias=True)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (encoder): IO_normalization_encoder(\n",
      "    (fun): MLP_res_net(\n",
      "      (net_res): Linear(in_features=10, out_features=2, bias=True)\n",
      "      (net_nonlin): Sequential(\n",
      "        (0): Linear(in_features=10, out_features=15, bias=True)\n",
      "        (1): Tanh()\n",
      "        (2): Linear(in_features=15, out_features=2, bias=True)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from deepSI.networks import MLP_res_net\n",
    "from torch import nn\n",
    "from deepSI.models import SUBNET\n",
    "from deepSI import get_nu_ny_and_auto_norm\n",
    "from nonlinear_benchmarks import Cascaded_Tanks\n",
    "train, test = Cascaded_Tanks()\n",
    "nx, nb, na = 2, 5, 5\n",
    "nu, ny, norm = get_nu_ny_and_auto_norm(train)\n",
    "feedthrough = False\n",
    "\n",
    "f =       norm.f(       MLP_res_net(input_size = [nx , nu],                       output_size = nx, activation=nn.Sigmoid))\n",
    "h =       norm.h(       MLP_res_net(input_size = [nx, nu] if feedthrough else nx, output_size = ny, n_hidden_layers=5, n_hidden_nodes=10))\n",
    "encoder = norm.encoder( MLP_res_net(input_size = [(nb,nu) , (na,ny)],             output_size = nx, n_hidden_layers=1, n_hidden_nodes=15))\n",
    "\n",
    "print(f) #has f(x,u) -> x with batch\n",
    "print(h) #has h(x,u) -> y with batch if feedthrough = True else h(x) -> y\n",
    "print(encoder) #has encoder(upast, ypast) -> x\n",
    "\n",
    "print(SUBNET(nu, ny, norm, nx=nx, nb=nb, na=na, f=f, h=h, encoder=encoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40ed983a-71bd-479a-90bd-c8353745ad75",
   "metadata": {},
   "source": [
    "## 2.2 Available models\n",
    "\n",
    "You can construct your own networks but also `deepSI` provides multiple useful ones.\n",
    "\n",
    " * SUBNET encoder structue (`deepSI.models.SUBNET`)\n",
    "    * see: https://arxiv.org/abs/2012.07697 or https://www.sciencedirect.com/science/article/pii/S0005109823003710\n",
    "  * Continuous time SUBNET encoder structure (`deepSI.models.SUBNET_CT`)\n",
    "    * see: https://arxiv.org/abs/2204.09405 \n",
    "  * Base class for fully custom SUBNET structures with shared parameters between `f`, `h` or `encoder`. (`deepSI.models.Custom_SUBNET`)\n",
    "  * CNN SUBNET (`CNN_SUBNET`)\n",
    "    * see chapter 4: https://research.tue.nl/files/318935789/20240321_Beintema_hf.pdf\n",
    "  * LPV SUBNET (`SUBNET_LPV` and `SUBNET_LPV_ext_scheduled`)\n",
    "    * see: https://arxiv.org/abs/2204.04060\n",
    "  * port HNN SUBNET (`pHNN_SUBNET`)\n",
    "    * see: https://arxiv.org/abs/2305.01338"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80378da0-9035-40d9-95ce-84d9ffe0db68",
   "metadata": {},
   "source": [
    "## 3.1 Creating you own SUBNET structure\n",
    "\n",
    "A limitation in the `SUBNET` class is that the `f`, `h` and the `encoder` cannot share any parameters. If this is not the case than you can still create you own cutom SUBNET structure.\n",
    "\n",
    "This is done by inhereting `Custom_SUBNET` and defining a forward method with the characteristic:\n",
    "* `forward(upast, ypast, ufuture) = yfuture_sim`\n",
    "  * `upast.shape = (batch_dim, nb, nu)`,\n",
    "  * `ypast.shape = (batch_dim, na, ny)`,\n",
    "  * `ufuture.shape = (batch_dim, T, nu)`,\n",
    "  * `yfuture_sim.shape = (batch_dim, T, ny)`\n",
    "\n",
    "(and removing `nu` and `ny` from these shapes if they are `'scalar'`)\n",
    "\n",
    "Below you can find an example for a LPV structure, see: https://arxiv.org/abs/2204.04060\n",
    "\n",
    "This structure does not accept `nu='scalar'` or `ny = 'scalar'` and thus \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fc270508-30a7-4955-ab1c-b7709293956b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SUBNET_LPV(\n",
       "  (A): Bilinear()\n",
       "  (B): Bilinear()\n",
       "  (C): Bilinear()\n",
       "  (D): Bilinear()\n",
       "  (encoder): IO_normalization_encoder(\n",
       "    (fun): MLP_res_net(\n",
       "      (net_res): Linear(in_features=10, out_features=2, bias=True)\n",
       "      (net_nonlin): Sequential(\n",
       "        (0): Linear(in_features=10, out_features=64, bias=True)\n",
       "        (1): Tanh()\n",
       "        (2): Linear(in_features=64, out_features=64, bias=True)\n",
       "        (3): Tanh()\n",
       "        (4): Linear(in_features=64, out_features=2, bias=True)\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (scheduling_net): IO_normalization_f(\n",
       "    (fun): MLP_res_net(\n",
       "      (net_res): Linear(in_features=3, out_features=2, bias=True)\n",
       "      (net_nonlin): Sequential(\n",
       "        (0): Linear(in_features=3, out_features=64, bias=True)\n",
       "        (1): Tanh()\n",
       "        (2): Linear(in_features=64, out_features=64, bias=True)\n",
       "        (3): Tanh()\n",
       "        (4): Linear(in_features=64, out_features=2, bias=True)\n",
       "      )\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from deepSI.models import Custom_SUBNET, validate_custom_SUBNET_structure\n",
    "from deepSI import Norm, get_nu_ny_and_auto_norm\n",
    "from deepSI.networks import Bilinear, MLP_res_net\n",
    "import torch\n",
    "import numpy as np\n",
    "class SUBNET_LPV(Custom_SUBNET):\n",
    "    def __init__(self, nu, ny, norm:Norm, nx, n_schedual, na, nb, scheduling_net=None, A=None, B=None, C=None, D=None, encoder=None, feedthrough=True):\n",
    "        if np.any((2*abs(norm.ymean)>norm.ystd).numpy()) or np.any((2*abs(norm.umean)>norm.ustd).numpy()):\n",
    "            from warnings import warn\n",
    "            warn('SUBNET_LPV assumes that the data is approximatly zero mean. Not doing so can lead to unintended behaviour.')\n",
    "        assert isinstance(nu, int) and isinstance(ny, int) and isinstance(n_schedual, int) and feedthrough, 'SUBNET_LPV requires the input, output schedualing parameter to be vectors and feedthrough to be present'\n",
    "        super().__init__()\n",
    "        self.nu, self.ny, self.norm, self.nx, self.n_schedual, self.na, self.nb, self.feedthrough = nu, ny, norm, nx, n_schedual, na, nb, feedthrough\n",
    "        self.A = A if A is not None else Bilinear(n_in=nx, n_out=nx, n_schedual=n_schedual)\n",
    "        self.B = B if B is not None else Bilinear(n_in=nu, n_out=nx, n_schedual=n_schedual, std_input=norm.ustd)\n",
    "        self.C = C if C is not None else Bilinear(n_in=nx, n_out=ny, n_schedual=n_schedual, std_output=norm.ystd)\n",
    "        self.D = D if D is not None else Bilinear(n_in=nu, n_out=ny, n_schedual=n_schedual, std_output=norm.ystd, std_input=norm.ustd)\n",
    "        self.encoder = encoder if encoder is not None else norm.encoder(MLP_res_net(input_size = [(nb,nu) , (na,ny)], output_size = nx))\n",
    "        self.scheduling_net = scheduling_net if scheduling_net is not None else norm.f(MLP_res_net(input_size = [nx , nu], output_size = n_schedual))\n",
    "        validate_custom_SUBNET_structure(self) #does checks if forward is working as intended\n",
    "    \n",
    "    def forward(self, upast: torch.Tensor, ypast: torch.Tensor, ufuture: torch.Tensor, yfuture: torch.Tensor=None):\n",
    "        bmv = lambda A, x: torch.bmm(A, x[:, :, None])[:,:,0] #batched matrix vector multiply\n",
    "        yfuture_sim = []\n",
    "        x = self.encoder(upast, ypast)\n",
    "        for u in ufuture.swapaxes(0,1): #iterate over time\n",
    "            p = self.scheduling_net(x, u)\n",
    "            A, B, C, D = self.A(p), self.B(p), self.C(p), self.D(p)\n",
    "            y = bmv(C, x) + bmv(D, u)\n",
    "            x = bmv(A, x) + bmv(B, u)\n",
    "            yfuture_sim.append(y)\n",
    "        return torch.stack(yfuture_sim, dim=1)\n",
    "\n",
    "from nonlinear_benchmarks import Silverbox\n",
    "train, test = Silverbox(atleast_2d=True)\n",
    "nx, nb, na = 2, 5, 5\n",
    "nu, ny, norm = get_nu_ny_and_auto_norm(train)\n",
    "\n",
    "SUBNET_LPV(nu, ny, norm, nx, n_schedual=2, na=na, nb=nb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0551e5db",
   "metadata": {},
   "source": [
    "If needed you can use network provided in `deepSI.networks` to formulate your method. For example it contains:\n",
    "\n",
    " * `MLP_res_net` A Multi-Layer Perceptron with Residual Connection (MLP_res_net)\n",
    " * `euler_integrator`, `rk4_integrator`, `rk45_integrator` integrators.\n",
    " * `Bilinear` where `A(p) = A_0 + A_1 p_1 + A_2 p_2 + ... + A_n_schedual p_n_schedual`\n",
    " * `CNN_vec_to_image` and `CNN_encoder` to go from a vector to an image and from an image to a vector.\n",
    " * `ELU_lower_bound` lower bounds a function as `torch.nn.functional.elu(y - b) + b`\n",
    " * `Matrix_converter` converts a vector into a matrix.\n",
    " * `Skew_sym_converter`, `Sym_pos_semidef_converter`. Converts a matrix to a skew symetric matrix and semidefiniate positive matrix respectively. \n",
    " * `Ham_converter` converts a scalar output to haminonian by scaling the output.\n",
    " * `Bias_net` where `f(x)=b` where `b` is a bias (trainable)\n",
    " * `Contant_net` where `f(x)=c` where `c` is a constant (untrainable)\n",
    " * `Sum_net` conbine multiple networks `f_1(x) + f_2(x) + f_3(x) + ... + f_n(x)`\n",
    " * `Quadratic_net` where `x^T Q X`\n",
    "\n",
    "This is only a small sampling of function that you might require for you work so feel free to create new ones. They just need to be a `torch.nn.Module` and they will work in `deepSI`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ac292b-da4b-4c8d-98f8-7c99a4c8f1a3",
   "metadata": {},
   "source": [
    "## 4. Normalization\n",
    "\n",
    "Often to get the result you need to carfully consider the normalization of your data. I recommend two different strategies for this:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd1b93a8-a679-49cc-8840-bc3a63121c3b",
   "metadata": {},
   "source": [
    "### 4.1 Normalization strategy 1: Pre-processing data\n",
    "\n",
    "You can pre-process you data which is quite a simple approach:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d03f01ec-50a1-4054-a3bc-c1551df1388e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepSI as dsi\n",
    "import nonlinear_benchmarks as nlb\n",
    "\n",
    "#explicit: way:\n",
    "train_val_0, test_0 = nlb.WienerHammerBenchMark()\n",
    "umean, ustd, ymean, ystd = train_val_0.u.mean(), train_val_0.u.std(), train_val_0.y.mean(), train_val_0.y.std()\n",
    "train_val = dsi.Input_output_data(u=(train_val_0.u - umean)/ustd, y=(train_val_0.y - ymean)/ystd)\n",
    "test = dsi.Input_output_data(u=(test_0.u - umean)/ustd, y=(test_0.y - ymean)/test_0.y.std())\n",
    "model = dsi.SUBNET(nu='scalar', ny='scalar', norm=dsi.Norm(0,1,0,1)) #the data is normalized so the model can have a identity norm.\n",
    "\n",
    "#using deepSI functions.\n",
    "train_val_0, test_0 = nlb.WienerHammerBenchMark()\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train_val_0)\n",
    "train_val, test = norm.transform(train_val_0), norm.transform(test_0)\n",
    "model = dsi.SUBNET(nu='scalar', ny='scalar', norm=dsi.Norm(0,1,0,1)) #the data is normalized so the model can have a identity norm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad8a8dae-69f9-4868-8dfb-d067763bde19",
   "metadata": {},
   "source": [
    "### 4.2 Normalization strategy 2: Online\n",
    "\n",
    "You can change the functions `f`, `h` and `encoder` such that they normalize and unnormalize the data themselfs. For examples you can write:\n",
    "\n",
    "`f_actual(x,u) = f_norm(x, (u-umean)/ustd)`\n",
    "\n",
    "`h_actual(x, u) = ymean + ystd*h_norm(x, (u-umean)/ustd)`\n",
    "\n",
    "`encoder_actual(upast, ypast) -> encoder_norm((upast - umean)/ustd, (ypast - ymean)/ystd)`\n",
    "\n",
    "where `f_norm`, `h_norm`, `encoder_norm` are the functions in the normalized space. You can automaticly wrap any normalized function by using `norm.f`, `norm.h` and `norm.encoder` as follows:\n",
    "\n",
    "If `f`, `h`, or the `encoder` is not given in `SUBNET` this strategy is also applied with the given `norm`\n",
    "\n",
    "Also note that for Continuous time you can use `norm.f_CT` to also add a time/derivative normalization factor, see: https://arxiv.org/abs/2204.09405\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "cba54491-0f4c-45ba-88f7-8c96b04e00e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepSI as dsi\n",
    "import nonlinear_benchmarks as nlb\n",
    "from torch import nn\n",
    "\n",
    "train, test = nlb.WienerHammerBenchMark()\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train)\n",
    "feedthrough = False\n",
    "nx, na, nb = 6, 20, 20\n",
    "\n",
    "f =       norm.f(       dsi.networks.MLP_res_net(input_size = [nx , nu],                       output_size = nx, activation=nn.Sigmoid))\n",
    "h =       norm.h(       dsi.networks.MLP_res_net(input_size = [nx, nu] if feedthrough else nx, output_size = ny, n_hidden_layers=5, n_hidden_nodes=10))\n",
    "encoder = norm.encoder( dsi.networks.MLP_res_net(input_size = [(nb,nu) , (na,ny)],             output_size = nx, n_hidden_layers=1, n_hidden_nodes=15))\n",
    "\n",
    "model = dsi.SUBNET(nu, ny, norm, nx=nx, nb=nb, na=na, f=f, h=h, encoder=encoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57985b21-cc7d-471a-8e11-b49ecc5e7a56",
   "metadata": {},
   "source": [
    "## 5. Exporting functions via `torch.onnx.export`\n",
    "\n",
    "If you want to use your estimated function you can torch the ONNX framework to transport the functions to other languages like `c`, `MATLAB` and `CasaDI`. Below you can find an example for the MATLAB conversion of `model.f`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a301ea5f-a861-4e6e-9b04-531bf4782e4c",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Module onnx is not installed!\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "class Fun(nn.Module):\n",
    "    def __init__(self, f, model):\n",
    "        super().__init__()\n",
    "        self.f = f\n",
    "        self.model = model\n",
    "    def forward(self, *args, **kwargs):\n",
    "        with torch.no_grad():\n",
    "            return self.f(*args, **kwargs)\n",
    "\n",
    "import deepSI as dsi\n",
    "nx = 3\n",
    "model = dsi.SUBNET('scalar', 'scalar', dsi.Norm(0,1,0,1), nx=nx)\n",
    "f = Fun(model.f, model)\n",
    "x = torch.zeros((1, nx))\n",
    "u = torch.zeros((1,)) if model.nu == 'scalar' else torch.zeros((1,model.nu))\n",
    "\n",
    "try: \n",
    "    torch.onnx.export(\n",
    "        f,                  # the model (in this case Fun class wrapping your model)\n",
    "        (x, u),             # input tensors (in this case x and u)\n",
    "        \"f.onnx\",       # file name or file path where the model will be saved\n",
    "        opset_version=12    # specify the ONNX opset version\n",
    "    )\n",
    "except torch.onnx.OnnxExporterError as e:\n",
    "    print(f'{e}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d038b3f",
   "metadata": {},
   "source": [
    "Than in matlab you can import the function as:\n",
    "```matlab\n",
    "params = importONNXFunction('f.onnx', 'fmatlab.m')\n",
    "x = zeros(1,6)\n",
    "u = zeros(1)\n",
    "xnext = fmatlab(x, u, params)\n",
    "```\n",
    "\n",
    "keep in mind that this still includes the batch dimension (here 1). Also `importONNXFunction` function is missing a number of operations that are included ONNX, so it might sometimes complain.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2d8c67a-76d5-4b8a-a01a-1e281a248e60",
   "metadata": {},
   "source": [
    "## 6. Reproducability\n",
    "\n",
    "If you set a manual seed with `torch.manual_seed(0)` before creating the networks and the model than the results is always the same. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "57f19cd0-2605-4bc5-b0ce-27b23f1eb1cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51e4730669c043898e9d98302e2af99b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.99626!! nan samps/sec\n",
      "it      10 NRMS loss 0.96911 NRMS val 0.91461!! 6994.61 samps/sec\n",
      "it      20 NRMS loss 0.88075 NRMS val 0.87685!! 6956.33 samps/sec\n",
      "it      30 NRMS loss 0.84420 NRMS val 0.84332!! 6925.06 samps/sec\n",
      "it      40 NRMS loss 0.75531 NRMS val 0.58872!! 6914.19 samps/sec\n",
      "it      50 NRMS loss 0.51822 NRMS val 0.54371!! 6911.41 samps/sec\n",
      "it      60 NRMS loss 0.45830 NRMS val 0.49202!! 6832.69 samps/sec\n",
      "it      70 NRMS loss 0.40896 NRMS val 0.41274!! 6842.26 samps/sec\n",
      "it      80 NRMS loss 0.36910 NRMS val 0.36029!! 6851.76 samps/sec\n",
      "it      90 NRMS loss 0.32021 NRMS val 0.31022!! 6846.92 samps/sec\n",
      "it     100 NRMS loss 0.27207 NRMS val 0.26480!! 6855.89 samps/sec\n",
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4bf63455b42541c08f6338f244c52a68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.99626!! nan samps/sec\n",
      "it      10 NRMS loss 0.96911 NRMS val 0.91461!! 6416.10 samps/sec\n",
      "it      20 NRMS loss 0.88075 NRMS val 0.87685!! 6679.66 samps/sec\n",
      "it      30 NRMS loss 0.84420 NRMS val 0.84332!! 6781.43 samps/sec\n",
      "it      40 NRMS loss 0.75531 NRMS val 0.58872!! 6838.03 samps/sec\n",
      "it      50 NRMS loss 0.51822 NRMS val 0.54371!! 6854.07 samps/sec\n",
      "it      60 NRMS loss 0.45830 NRMS val 0.49202!! 6880.16 samps/sec\n",
      "it      70 NRMS loss 0.40896 NRMS val 0.41274!! 6883.03 samps/sec\n",
      "it      80 NRMS loss 0.36910 NRMS val 0.36029!! 6887.50 samps/sec\n",
      "it      90 NRMS loss 0.32021 NRMS val 0.31022!! 6880.70 samps/sec\n",
      "it     100 NRMS loss 0.27207 NRMS val 0.26480!! 6883.56 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "from nonlinear_benchmarks import Silverbox\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "train_val, tests = Silverbox()\n",
    "test = tests[0]\n",
    "s = int(0.95*len(train_val))\n",
    "train, val = train_val[:s], train_val[s:]\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train_val)\n",
    "\n",
    "for _ in range(2):\n",
    "    torch.manual_seed(0)\n",
    "    subnet = dsi.SUBNET(nu, ny, norm, nx=4, nb=20, na=20)\n",
    "    fit_res_subnet = dsi.fit(subnet, train, val, n_its=100, val_freq=10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34e44ddb-2419-40d3-adda-7969af433b6a",
   "metadata": {},
   "source": [
    "## 7. Regularization and custom cost functions\n",
    "\n",
    "In `fit` you can give custom cost function like by giving an argument for `loss_fun`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0721ca49-a48b-4092-be7b-62bf95314776",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 980\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f5132f997f334cfd8bae68eba4355c08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/101 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 1.04559!! nan samps/sec\n",
      "it     100 NRMS loss 0.81684 NRMS val 0.98126!! 2338.65 samps/sec\n"
     ]
    }
   ],
   "source": [
    "from deepSI import SUBNET, get_nu_ny_and_auto_norm\n",
    "from deepSI.fitting import compute_NMSE\n",
    "import torch\n",
    "from nonlinear_benchmarks import Cascaded_Tanks\n",
    "import numpy as np\n",
    "train, test = Cascaded_Tanks()\n",
    "\n",
    "nx, nb, na = 2, 5, 5\n",
    "nu, ny, norm = get_nu_ny_and_auto_norm(train)\n",
    "model = SUBNET(nu, ny, norm, nx=nx, nb=nb, na=na)\n",
    "\n",
    "def ridge_L2_regularization(model, upast, ypast, ufuture, yfuture, kappa=1):\n",
    "    yfuture_model = model(upast, ypast, ufuture)\n",
    "    amp = torch.stack([(p**2).sum() for p in model.parameters()])/torch.as_tensor([p.numel() for p in model.parameters()]) #parameter amplitude\n",
    "    return ((yfuture-yfuture_model)**2/model.norm.ystd**2).mean() + torch.mean(amp)*kappa\n",
    "\n",
    "from deepSI import fit\n",
    "kappa = 0.1\n",
    "my_loss = lambda *args: ridge_L2_regularization(*args, kappa=kappa)\n",
    "train_dict = fit(model, train, test, n_its=101, T=40, batch_size=64, val_freq=100, loss_fun=my_loss, val_fun=compute_NMSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b68aa1d3-166a-41bd-8d11-400495979e0a",
   "metadata": {},
   "source": [
    "## 8. Training/fitting details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "57f5d31b-9c30-4add-94eb-55f3e829583c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c2a6697851d40e68a2c8e92b42255e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/201 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.94222!! nan samps/sec\n",
      "it      10 NRMS loss 0.83779 NRMS val 0.71031!! 6918.32 samps/sec\n",
      "it      20 NRMS loss 0.53195 NRMS val 0.37894!! 6890.73 samps/sec\n",
      "it      30 NRMS loss 0.35080 NRMS val 0.31227!! 6912.53 samps/sec\n",
      "it      40 NRMS loss 0.27582 NRMS val 0.28759!! 6902.38 samps/sec\n",
      "it      50 NRMS loss 0.25068 NRMS val 0.25593!! 6900.05 samps/sec\n",
      "it      60 NRMS loss 0.22850 NRMS val 0.23592!! 6912.52 samps/sec\n",
      "it      70 NRMS loss 0.20390 NRMS val 0.21209!! 6818.73 samps/sec\n",
      "it      80 NRMS loss 0.18199 NRMS val 0.18607!! 6794.85 samps/sec\n",
      "it      90 NRMS loss 0.16168 NRMS val 0.15986!! 6812.44 samps/sec\n",
      "it     100 NRMS loss 0.13624 NRMS val 0.13161!! 6826.58 samps/sec\n",
      "it     110 NRMS loss 0.11548 NRMS val 0.11078!! 6825.74 samps/sec\n",
      "it     120 NRMS loss 0.09990 NRMS val 0.09464!! 6824.30 samps/sec\n",
      "it     130 NRMS loss 0.08603 NRMS val 0.08397!! 6837.11 samps/sec\n",
      "it     140 NRMS loss 0.08553 NRMS val 0.07959!! 6841.59 samps/sec\n",
      "it     150 NRMS loss 0.07856 NRMS val 0.07453!! 6812.06 samps/sec\n",
      "it     160 NRMS loss 0.07228 NRMS val 0.07198!! 6807.32 samps/sec\n",
      "it     170 NRMS loss 0.06824 NRMS val 0.07117!! 6813.78 samps/sec\n",
      "it     180 NRMS loss 0.07021 NRMS val 0.08080   6814.52 samps/sec\n",
      "it     190 NRMS loss 0.06583 NRMS val 0.07901   6824.72 samps/sec\n",
      "it     200 NRMS loss 0.06688 NRMS val 0.08364   6828.46 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "import nonlinear_benchmarks as nlb\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "train_val, test = nlb.Silverbox()\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train_val)\n",
    "train_val, test = norm.transform(train_val), norm.transform(test)\n",
    "s = int(len(train_val)*0.95)\n",
    "train, val = train_val[:s], train_val[s:]\n",
    "model = dsi.models.SUBNET(nu, ny, dsi.Norm(0, 1, 0, 1), nx=4)\n",
    "train_dict = dsi.fit(model, train, val, n_its=201, val_freq=10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70be2b71-6e96-45cf-98f1-a74164c5a160",
   "metadata": {},
   "source": [
    "The training dict contains a lot of information as seen below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "96590657-6295-4cd7-bf13-fb925ecdbe05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['best_model', 'best_optimizer_state', 'last_model', 'last_optimizer_state', 'NRMS_train', 'NRMS_val', 'samples/sec', 'val_freq', 'batch_size', 'it_counter'])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dict.keys()\n",
    "# - 'best_model': The best model (with the lowest validation loss).\n",
    "# - 'best_optimizer_state': Optimizer state when the best model was found.\n",
    "# - 'last_model': The model at the end of training.\n",
    "# - 'last_optimizer_state': The optimizer state at the end of training.\n",
    "# - 'NRMS_train': Training loss history (normalized root mean square error).\n",
    "# - 'NRMS_val': Validation loss history (normalized root mean square error).\n",
    "# - 'samples/sec': Number of data samples processed per second. (batch_size * iterations per second)\n",
    "# - 'val_freq': Validation frequency.\n",
    "# - 'batch_size': Batch size used during training.\n",
    "# - 'it_counter': List of iteration counts corresponding to each validation point."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "294811fd-8496-460b-b9ab-1fa62c332e1f",
   "metadata": {},
   "source": [
    "### Trainig curve plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "64fdc636-5ad4-4398-ba82-40334d81b0ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAACyCAYAAABhsukCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCmElEQVR4nO3dd3gU1frA8e/uZje9QUjvoUNI6MUrIC0BroJYQAFBr6AIKHKxYEOQC4KKXgXlWhDwB4KIIopSjIQuUgKhBgiBhFRISC+72Z3fH5usCdkku2HT4HyeZ57szpyZOXOymzczc+Y9MkmSJARBEATBQuSNXQFBEAThziICiyAIgmBRIrAIgiAIFiUCiyAIgmBRIrAIgiAIFiUCiyAIgmBRIrAIgiAIFmXV2BW4E+l0OlJSUnB0dEQmkzV2dQRBEG6bJEnk5eXh7e2NXF7zOYkILPUgJSUFPz+/xq6GIAiCxSUlJeHr61tjGRFY6oGjoyOg/wU4OTmZvJ5Go2Hnzp0MGzYMpVJZX9VrlkTbGCfapXqibYyra7vk5ubi5+dn+PtWExFY6kH55S8nJyezA4udnR1OTk7ii3AL0TbGiXapnmgb4263XUy5vC9u3guCIAgWJQJLE6HVSfwcm8r/zskp1eoauzqCIAh1JgJLE1Gs0fLOtvOczZbz08nUxq6OIAhCnYl7LE2EvbUVU+4NZOmOiyyPvsxDPfxRKkTcv5totVo0Go1Z62g0GqysrCguLkar1dZTzZon0TbGVdcuSqUShUJhkX2IwNKEjO/lx6d/XODazSI2H7vGuF7+jV0loQFIkkRaWhrZ2dl1WtfT05OkpCTxzNQtRNsYV1O7uLi44OnpedvtJQJLE2KnsmKIt44tVxV88sclxnTzRWUlzlrudOVBxd3dHTs7O7O+1Dqdjvz8fBwcHGp9aO1uI9rGOGPtIkkShYWFZGRkAODl5XVb+xCBpYm5x0PiQKaK5OwivjuaxIQ+AY1dJaEeabVaQ1Bp2bKl2evrdDrUajU2Njbij+ctRNsYV1272NraApCRkYG7u/ttXRYTrd3EqBTwTP8gAFbsvkRJqbg2fCcrv6diZ2fXyDURhL8/h+be67uVCCxN0Lgevng4WZOaU8zGI0mNXR2hAYh7AEJTYKnPoQgsTZC1UsGM+1oD+rOWYo04axEEofkQgaWJerSnH97ONqTnlrD+cGJjV0cQBMFkIrA0UdZWCmYMagPAp9HxFKnFWYsgNAUDBw5k1qxZhveBgYF89NFHNa4jk8nYsmXLbe/bUtupbyKwNGEPd/fF19WWG/klrDt8tbGrIwiVTJ48GZlMxrvvvltp/pYtWypdq4+OjkYmkxmmVq1aMWLECE6dOmV0e88++2yVfU2fPh2ZTMbkyZMN865fv860adPw9/fH2toaT09PIiIiOHDggGUPtBZHjhxh6tSpFt3m22+/TXh4eJX5qampDB8+3KL7qg8isDRhKis5Mwfp77V8Fh1PobrUeEFJgpQY/U9BaEA2NjYsWbKEmzdv1lo2Li6O1NRUduzYQUlJCSNHjkStVlcq4+fnx4YNGygqKjLMKy4uZv369fj7V35g+KGHHiImJoY1a9Zw4cIFtm7dysCBA8nMzLTMwZmoVatWDdarz9PTE2tr6wbZ1+0QgaWJG9PNF/8WdmQWqFl7yMhZi04HX9wHnw+E5GMNXj/B8iRJolBdavJUpNaaVb6mSTLzn5MhQ4bg6enJ4sWLay3r7u6Op6cn3bp1Y9asWSQlJXH+/PlKZbp164afnx8//PCDYd4PP/yAv78/Xbt2NczLzs5m3759LFmyhPvuu4+AgAB69erF3LlzeeCBB4zuf+fOndjY2FTJcPDCCy8waNAgADIzM3nsscfw8fHBzs6O0NBQvv322xqP69ZLYRcvXqR///7Y2NjQsWNHdu3aVWWdV155hbZt22JnZ0dwcDBvvvmmoYvv6tWrmT9/PidPnjSc5a1evRqoeins1KlTDBo0CFtbW1q2bMnUqVPJz883LJ88eTKjR4/m/fffx8vLi5YtWzJjxozb7k5cG/GAZBOnVMh5fnAb5mw6yf/2xDOhTwAO1hV+bXI5uLXVn7HEfAO+PRqvsoJFFGm0dHxrR6Ps++yCCOxUpv9ZUCgULFq0iMcff5znn3++1pEFAXJyctiwYQMAKpWqyvKnnnqKr7/+mvHjxwOwatUqnnzySaKjow1lHBwccHBwYMuWLfTp08ek/+IHDx6Mi4sLmzdv5l//+hegf0B148aN/Oc//wH0Z0fdu3fnlVdewcnJiW3btjFx4kRCQkLo1atXrfvQ6XSMGTMGDw8PDh8+TE5OTqX7MeUcHR1ZvXo13t7enDp1iilTpuDo6MjLL7/M2LFjOX36NNu3b+f3338HwNnZuco2CgoKiIiIoG/fvhw5coSMjAyefvppZsyYYQhEALt378bLy4vdu3dz6dIlxo4dS7t27Zg5c2atx1NX4oylGRgd7k2Qmz03CzWsOXilaoGuE/Q/T20GdUGD1k0QHnzwQcLDw5k3b16N5Xx9fXFwcMDFxYX169fzwAMP0L59+yrlJkyYwP79+7l69SpXr17lwIEDTJgwoVIZKysrVq9ezZo1a3BxceGee+7htddeIzY2ttr9KxQKxo0bx/r16w3zoqKiyM7O5qGHHgLAx8eHOXPmEB4eTnBwMDNnziQyMpLvvvvOpLb4/fffOX/+PGvXriUsLIz+/fuzaNGiKuXeeOMN+vXrR2BgIPfffz9z5swx7MPW1hYHBwesrKzw9PTE09PT8FR8RevXr6e4uJi1a9fSuXNnBg0axPLly/nmm29IT083lHN1dWX58uW0b9+ef/7zn4wYMYI9e/aYdDx1Jc5YmgErhZwXBrdh1sYTfL73MhP7BuBkU2Hkt4B/gGsg3LwCZ7dC+GONVVXBAmyVCs4uiDCprE6nIy83D0cnR4ukLbFV1i2Nx5IlSxg0aBBz5syptsy+ffuws7Pjzz//ZNGiRaxcudJouVatWjFy5EhWr16NJEmMHDkSNze3KuUeeughRo4cyb59+/jzzz/57bffWLp0KV9++WWlm/wVjR8/nj59+pCSkoK3tzfr1q1j5MiRuLi4APozmEWLFvHdd9+RnJyMWq2mpKTE5Hso586dw8/PD29vb8O8vn37Vim3ceNGPv74Y+Lj48nPz6e0tNSs0WbL9xUWFoa9vb1h3j333INOpyMuLg4PDw8AOnXqVCk9i5eXFydOnDBrX+YSZyzNxP1h3oS0sienSMPqA1cqL5TLIbzsP7qYbxq8boJlyWQy7FRWJk+2KoVZ5Wua6vrkdf/+/YmIiGDu3LnVlgkKCqJdu3ZMmjSJp59+mrFjx1Zb9qmnnjKckTz11FPVlrOxsWHo0KG8+eabHDx4kMmTJ9d45tSzZ09CQkIMHQR+/PFHwyU3gPfee4///ve/vPLKK+zevZsTJ04QERFRpZPB7Th06BDjx49nxIgR/PLLL8TExPD6669bdB8V3Tr8sEwmQ6er38EERWBpJhRyGS8MaQvAF/suk1N0y8238McBGVw9AJnxDV9B4a737rvv8vPPP3Po0KFay06fPp3Tp0/z448/Gl0eGRmJWq1Go9EQEWHa2RtAx44dKSio+XLw+PHjWbduHT///DNyuZyRI0calh04cIBRo0YxYcIEwsLCCA4O5sKFCybvv0OHDiQlJZGa+vdgfX/++WelMgcPHiQgIIDXX3+dHj160KZNG65erdwxR6VS1TqGTIcOHTh58mSl4z1w4AByuZx27dqZXOf6IAJLMzIy1Iu2Hg7kFZfy1f6EygudfaD1YP3rE+savnLCXS80NJTx48fz8ccf11rWzs6OKVOmMG/ePKM90RQKBefOnePs2bNGs+xmZmYyaNAg/u///o/Y2FgSEhLYtGkTS5cuZdSoUTXue/z48Rw/fpz//Oc/PPzww5Vu/Ldp04Zdu3Zx8OBBzp07xzPPPFPpfkVthgwZQtu2bZk0aRInT55k3759vP7665XKtGnThsTERDZs2EB8fDwff/xxlQAbGBhIQkICJ06c4MaNG5SUlBg9DhsbGyZNmsTp06fZvXs3M2fOZOLEiYbLYI1FBJZmRCGXMavsrGXV/gSyC285de46Uf/zxHrQVvPMiyDUowULFph8mWXGjBmcO3eOTZs2GV3u5ORU7X0HBwcHevfuzYcffkj//v3p3Lkzb775JlOmTGH58uU17rd169b06tWL2NjYSpfBQH9TvVu3bkRERDBw4EA8PT0ZPXq0SccDIJfL+fHHHykqKqJXr148/fTThh5n5R544AFefPFFZsyYQXh4OAcPHuTNN9+sVOahhx4iMjKS++67j1atWhnt8mxnZ8eOHTvIysqiZ8+ePPzwwwwePLjW428Q0h1g+fLlUkBAgGRtbS316tVLOnz4cLVlBwwYIAFVphEjRhjKTJo0qcryiIgIk+uTk5MjAVJOTo5Zx6FWq6UtW7ZIarW62jJarU6K+HCPFPDKL9LS7ecqL9QUS9K7gZI0z0mS4naYte+mzpS2aY6Kioqks2fPSkVFRXVaX6vVSjdv3pS0Wq2Fa9b8ibYxrqZ2qenzaM7ftWZ/xrJx40Zmz57NvHnzOH78OGFhYURERBhGQrvVDz/8QGpqqmE6ffo0CoWCRx55pFK5yMjISuVqe0iqocjlMl4cqj9rWX3gClkFFc5arKyhS9kN0Zi1jVA7QRCEOnY3joqKIioqioyMjCqnvatWrbJIxUy1bNkypkyZwpNPPgnAypUr2bZtG6tWreLVV1+tUr5FixaV3m/YsAE7O7sqgaU891BTNKyjB528nTiTksvney/z6vAKzwJ0mwiHP4O436DgBthX7aYpCIJQn8wOLPPnz2fBggX06NEDLy+vRh2gSK1Wc+zYsUpdHOVyOUOGDDGpZwrAV199xbhx4yr1BQd94jx3d3dcXV0ZNGgQCxcurHbo2JKSkko313JzcwH9KGzmpE4oL2vKOjPvC+bZdSdYczCByX18aelQdgOyRVsUXuHIU0+gjVmHrvdzJu+/KTOnbZoTjUaDJEnodLo6dQGVym58l29D+JtoG+NqahedTockSWg0miqdJsz57pkdWFauXMnq1auZOHGiuata3I0bN9BqtVV6QHh4eFTJQWTMX3/9xenTp/nqq68qzY+MjGTMmDEEBQURHx/Pa6+9xvDhwzl06JDRHiqLFy9m/vz5Vebv3LmzTsnpjOUWupUkgZ+9gqQCHXPX7mZ04N8fkECrcMI4QcH+/7H7RgDcQaMTmtI2zUn509X5+fm39RxDXl6eBWt1ZxFtY5yxdlGr1RQVFbF3715KSyt3ACosLDR522YHFrVaTb9+/cxdrUn66quvCA0NrZIDaNy4cYbXoaGhdOnShZCQEKKjoxk8eHCV7cydO5fZs2cb3ufm5uLn58ewYcPMeppWo9Gwa9cuhg4dWuWhJmMc2lzn6W9iOHjdioUT78XdseyspfgepP9uwKk4mZHhnkg+3U2uQ1Nlbts0F8XFxSQlJeHg4ICNjY3Z60uSRF5eHo6OjmJ441uItjGupnYpLi7G1tbWkESzovIrMaYwO7A8/fTTrF+/vkr3uMbg5uaGQqGo0s88PT291vsjBQUFbNiwgQULFtS6n+DgYNzc3Lh06ZLRwGJtbW00CZ5SqazTH0FT1xvc0Yuu/gnEJGbz5YGrzLu/U9kG3KDjKIjdiNWpbyGwj9l1aKrq2qZNlVarRSaTIZfL65SSpfxSRvk2hL+JtjGupnaRy+XIZDKj3zNzvndmt3ZxcTHLli1jwIABzJw5k9mzZ1eaTLVv3z4mTJhA3759SU5OBuCbb75h//79Jm9DpVLRvXt3oqKiDPN0Oh1RUVFG8/NUtGnTJkpKSqoktzPm2rVrZGZm4uXlZXLdGoJMJmN2WQ+xdYcTSc8t/nthpcSUpp/CCoIg3C6zA0tsbCzh4eHI5XJOnz5NTEyMYTI1sdnmzZuJiIjA1taWmJgYw43vnJwco5lAazJ79my++OIL1qxZw7lz55g2bRoFBQWGXmJPPPGE0fxFX331FaNHj65yQz4/P5+XXnqJP//8kytXrhAVFcWoUaNo3bq1WaklGso/WrvRPcAVdamO749d+3tBeWJKdR6c/anR6icIwt3H7Ethu3fvvu2dLly4kJUrV/LEE08YxmUAfWbOhQsXmrWtsWPHcv36dd566y3S0tIIDw9n+/bthhv6iYmJVU734uLi2L9/Pzt37qyyPYVCQWxsLGvWrCE7Oxtvb2+GDRvGO++80yRHbpPJZIzt4cexqzfZEpPMcwND9NdNyxNT7l4IMf8nMh4LQj0YOHAg4eHhtY55f7e5rQuP165d49q1a7UXvEVcXBz9+/evMt/Z2bnK6G6mmDFjBlevXqWkpITDhw/Tu3dvw7Lo6OhKg94AtGvXDkmSGDp0aJVt2drasmPHDjIyMlCr1Vy5coXPP/+80XPv1CQy1BOVlZyLGfmcS63Q08OQmHK/SEwpWFxzHPO+fERFS/nhhx945513LLa9O4XZgUWn07FgwQKcnZ0JCAggICAAFxcX3nnnHZP7int6enLp0qUq8/fv309wcLC5VbrrOdkoGdTOHYCfTiT/vUAkphTq2Z065r2pz2y0aNECR0fH297fncbswPL666+zfPly3n33XcO9lUWLFvHJJ5+Y3FNsypQpvPDCCxw+fBiZTEZKSgrr1q1jzpw5TJs2zeyDEGB0V/3AQltPpqDTVcgWW34TXySmFOpBcxrz/u2332bNmjX89NNPhrOn6Ohorly5gkwmY+PGjQwYMAAbGxvWrVtHZmYmjz32GD4+PtjZ2REaGloltdPAgQMrDT0cGBjIokWLeOqpp3B0dMTf35/PP//clKa8o5gdWNasWcOXX37JtGnT6NKlC126dOG5557jiy++qHLJqTqvvvoqjz/+OIMHDyY/P5/+/fvz9NNP88wzz9TrOMx3soHt3HG0sSI1p5jDCVl/L2g3AmxbQF4qxP/ReBUUTCdJ+iGmTZ00heaVr2kyksK+JuVj3n/yyScmXxY3dcz7cuVj3ldUccx7YynljZkzZw6PPvpopTyAFZ/Je/XVV3nhhRc4d+4cERERFBcX0717d7Zt28bp06eZOnUqEydO5K+//qpxPx988AE9evQgJiaG5557jmnTphEXF2dSHe8UZt+8z8rKMjpOdfv27cnKyjKyRmVarZYDBw4wffp0XnrpJS5dukR+fj4dO3bEwcHB3OoIZWyUCkZ09mLj0SR+OpFM35Cy3m7liSkPf6ZPTNl2WONWVKidphAWeddeDv1/hi6W3PdrKaCyr71cBRXHvL81i0VFvr6+AIaBqWoa837u3LmGwa8OHDjAhg0biI6ONpQpH/N+ypQprFy5km7dujFgwADGjRtHly5djO7fwcEBW1tbSkpKjD7nNmvWLMaMGVNpXsWhlmfOnMmOHTv47rvvqjxUXdGIESN47jl9KqVXXnmFDz/8kN27dzf64FsNyewzlrCwMKP5/pcvX05YWFit6ysUCoYNG8bNmzdRqVR07NiRXr16iaBiAaPKLof9eiqVktIKo891K0u/U56YUhAsbMmSJYYu/9XZt28fx44dY/Xq1bRt29akMe+//vrrGse8T0lJYevWrURGRhIdHU23bt1MvnJyqx49elR6r9VqeeeddwgNDaVFixY4ODiwY8cOEhMTa9xOxcAmk8nw9PSsNtv6ncrsM5alS5cycuRIfv/9d8NDiIcOHSIpKYlff/3VpG107tyZy5cvExQUZO7uhRr0CWqJp5MNabnFRMddJ6JT2X9lHp3AuyukxEDsRug7vXErKtRMaac/czCBTqcjNy8PJ0dHyzxdrjQ/tx1UHvO+Ys+tioKCgnBxcaFdu3ZkZGQwduxY9u7da7TsU089xYwZMwBYsWJFtfstH/O+fNz7p59+mnnz5lVbh5rcmoj2vffe47///S8fffQRoaGh2NvbM2vWrFpzujXGGPNNjdmfxAEDBnDhwgUefPBBsrOzyc7OZsyYMcTFxXHvvfeatI2FCxcyZ84cfvnlF1JTU8nNza00CXUjl8t4IFx/1lKpdxj8Pbrk8W/Mvo4uNDCZTH85ytRJaWde+Zqm28ip1RzGvDdlLPlyBw4cYNSoUUyYMIGwsDCCg4O5cOGCyXW5m9VpPBZvb+8qw22aY8SIEYD+GmvF/u6SJCGTyUz+xQtVPRDmzed7L/P7uQxyizU42ZT999T5IdjxGlw/B8nHwbf5J6YUmpa6jnk/evToKskQy8e8L399q8zMTB555BGeeuopunTpgqOjI0ePHq11zPvAwEB27NhBXFwcLVu2xNnZudqybdq04fvvv+fgwYO4urqybNky0tPT6dixY63Hd7czKbDExsbSuXNn5HI5sbGxNZat7sZZRZZ4el8wrpO3E63dHbiUkc/202k82sNPv8DWxZCYkpi1IrAI9WLBggVs3LjRpLIzZsxg2bJlbNq0iUcffbTK8poyg1cc8z4+Ph6NRoOfnx9Tpkzhtddeq3a9KVOmEB0dTY8ePcjPz2f37t0EBgYaLfvGG29w+fJlIiIisLOzY+rUqYwePZqcnByTju9uJpOk2q+LyOVy0tLScHd3N2S/NLaaONvQy83NxdnZmZycHLPT5v/666+MGDHitjL4Lv/jIu/vvMA9rVuy7ukKmY0T9sKa+0HlCHMugKpu19Mbg6XapqkpLi4mISGBoKCgOqXN1+l05Obm4uTkJDL43kK0jXE1tUtNn0dz/q6ZdMaSkJBAq1atDK9vV3U37MoZS/cimG5UuA/v77zAwfhM0nOL8XAq+4AE/ANcAiD7qj4xpcgfJghCPTApsAQEBBheX716lX79+mFlVXnV0tJSDh48WKlsdQYOHFhlXsVrrOKs5/b4tbCje4Arx67e5OeTKTx9b1maHLlcfxNfJKYUBKEemX1+eN999xl9EDInJ4f77rvPpG3cvHmz0pSRkcH27dvp2bOn0YzDgvlGl/UO23Jr77DwxxCJKQVBqE9mB5bynlu3yszMrNIPvDrOzs6VJjc3N4YOHcqSJUt4+eWXza2SYMTILt5YyWWcTs7lUkb+3wucfUViSkEQ6pXJ3Y3LUx2Up66uODaJVqslNja2Ut6duvDw8LjrcurUlxb2Kvq3bcUf5zPYeiKZ2cMqpJPoOgEu/a5PTDnwNVDUqde5YEEm9KERhHpnqc+hyX9Ryvt7S5KEo6Mjtra2hmUqlYo+ffowZcoUk7Z1a5dlSZJITU3l3XffJTw83NQqCbUYFe7NH+cz2HIihReHtv37TPPWxJQif1ijKe/hVlhYWOk7JQiNobBQP4z57fa8NDmwlGcbDQwMZM6cOSZf9jImPDzcaJflPn36sGrVqjpvV6hsaEcP7FQKErMKiUnKppu/q35BpcSU34jA0ogUCgUuLi6GXFJ2dnZGLzVXR6fToVarKS4uFl1qbyHaxjhj7SJJEoWFhWRkZODi4mL0oVRzmH0NZN68ebe1Q6jaZVkul9OqVas69eMXqmensmJYRw+2nEjhp5jkvwML6BNTHv7s78SU9lWT/AkNozzTbl0SFUqSRFFREba2tmYFpLuBaBvjamoXFxcXo5mfzVWni+vff/893333HYmJiVUSsh0/frzW9Y11Sc7OzhaBpR6M6urDlhMp/BKbyhv/7IhSUfafW8XElAf+C8PE8KqNRSaT4eXlhbu7u8kjF5bTaDTs3buX/v3731EPjlqCaBvjqmsXpVJ522cq5cwOLB9//DGvv/46kydP5qeffuLJJ58kPj6eI0eOMH26aVlzlyxZQmBgIGPHjgXg0Ucf5fvvv8fLy4tff/3VpPT7Fa1YsYL33nuPtLQ0wsLC+OSTT6odL2H16tVVBg2ytramuLjY8F6SJObNm8cXX3xBdnY299xzD5999hlt2rQxq15Nwb2t3WhpryKzQM3+Sze4r2wIYwAGvALfjoNDy6HTg+DTrfEqKqBQKMz+YisUCkpLS7GxsRF/PG8h2sa4hmgXsy88fvrpp3z++ed88sknqFQqXn75ZXbt2sXzzz9vcg6dlStX4uenz2G1a9cudu3axfbt2xk+fDgvvfSSWfXZuHEjs2fPZt68eRw/fpywsDAiIiJqvKzg5ORkGEEuNTXVMKBQuaVLl/Lxxx+zcuVKDh8+jL29vWFEuebGSiHnn128ANh64pZU7O2G65NTSjrYOhO05v23LAiCYIzZZyyJiYmGbsW2trbk5eUBMHHiRPr06WN0ELBbpaWlGQLLL7/8wqOPPsqwYcMIDAykd+/eZtVn2bJlTJkyxXAWsnLlSrZt28aqVat49dVXja5TPviOMZIk8dFHH/HGG28YsqSuXbsWDw8PtmzZwrhx46qsU1JSUml41PLU/xqNxqxLG+Vlzb0cUpuRoR6sOXSVHWfSyCkowk5V4dc+ZCFW8buRpZ9Gu/cDdP/4t0X3bSn11TbNnWiX6om2Ma6u7WJOebMDi6enJ1lZWQQEBODv78+ff/5JWFgYCQkJJveBdnV1JSkpCT8/P7Zv387ChQsB/R91c9K5qNVqjh07xty5cw3z5HI5Q4YMqXFMiPz8fAICAtDpdHTr1o1FixbRqVMnQN+xIC0tjSFDhhjKOzs707t3bw4dOmQ0sCxevJj58+dXmb9z507s7MxP9Lhr1y6z16mJJEFLawWZJVo+2LCL7m6Vf0++7o/S/epK2Pse+9KdyLP1sej+LcnSbXOnEO1SPdE2xpnbLuVdkU1hdmAZNGgQW7dupWvXrjz55JO8+OKLfP/99xw9erTKeNHVGTNmDI8//jht2rQhMzOT4cOHAxATE0Pr1q1NrsuNGzfQarV4eHhUmu/h4cH58+eNrtOuXTtWrVpFly5dyMnJ4f3336dfv36cOXMGX19f0tLSDNu4dZvly241d+5cZs+ebXifm5uLn58fw4YNMzu78a5duxg6dKjFr31esL7Ep3sukyjz4M0Rt9xLkYaj+y4exaVdDMz7Ae2Dv4DcMjfxLKU+26Y5E+1SPdE2xtW1XcwZhNHswPL5558bhtmcPn06LVu25ODBgzzwwAM888wzJm3jww8/JDAwkKSkJJYuXWoY7z41NZXnnnvO3CqZpW/fvoYhlQH69etHhw4d+N///sc779StZ5S1tXWlTATllEplnT7QdV2vJmO6+/Hpnsvsv5RJnlqihb2qcoH7P4IVfZAnH0Eesxr6PGvR/VtKfbTNnUC0S/VE2xhnbruYU9bswCKXyys9bDRu3Dijl4dqolQqmTNnTpX5L774olnbcXNzQ6FQkJ6eXml+enq6yX2xlUolXbt25dKlS8DfzxSkp6fj5eVVaZvNOStAa3cHOvs4cTo5l22xKUzsG1i5gLMvDJ0P22ZD1Hz9jX3X2jNVC4Ig3MrkESRNZcoIkuXOnj1r9FmYBx54wKT1VSoV3bt3JyoqitGjRwP6p0qjoqKYMWOGSdvQarWcOnXKMFxyUFAQnp6eREVFGQJJbm4uhw8fZtq0aaYdWBM1OtyH08m5bDlhJLAAdH8STm+Gqwfg5+dh4pbbGgNdEIS7k0mBpWIKltqeYDXl5vvly5d58MEHOXXqVKXULuXbNucG/uzZs5k0aRI9evSgV69efPTRRxQUFBh6iT3xxBP4+PiwePFiQD90ap8+fWjdujXZ2dm89957XL16laefftpQh1mzZrFw4ULatGlDUFAQb775Jt7e3obg1VzdH+bNf349x7GrN0nKKsSvxS0dC+RyeOAT+KwfXI7WZz/uOqFR6ioIQvNl0nMsCQkJXL58mYSEBDZv3kxQUBCffvopMTExxMTE8OmnnxISEsLmzZtN2ukLL7xAUFAQGRkZ2NnZcebMGfbu3UuPHj2Ijo426wDGjh3L+++/z1tvvUV4eDgnTpxg+/bthpvviYmJpKamGsrfvHmTKVOm0KFDB0aMGEFubi4HDx6kY8eOhjIvv/wyM2fOZOrUqfTs2ZP8/Hy2b9/e7DMDeDjZ0C+kJQBbT6YYL9QyBO4rGzN8x2uQZ7zDgiAIQrUkM/Xs2VPatm1blfnbtm2TunXrZtI2WrZsKZ08eVKSJElycnKSzp8/L0mSJEVFRUnh4eHmVqnJycnJkQApJyfHrPXUarW0ZcsWSa1W11PNJGnjkUQp4JVfpCEfREs6nc54oVKNJK3sL0nznCRpw/h6q4s5GqJtmiPRLtUTbWNcXdvFnL9rZj95f+rUKYKCgqrMDwoK4uzZsyZtQ6vV4ujoCOhvwKek6P97DggIEOOx1LPIzp6orORczMjnbGo13QcVVjBqOcit4NzPcPanhq2kIAjNmtmBpUOHDixevLjSDXe1Ws3ixYvp0KGDSdvo3LkzJ0+eBKB3794sXbqUAwcOsGDBAoKDg82tkmAGJxslg9vr84X9dGuKl4o8Q+EfZb30ts2BwqrDUQuCIBhjdmBZuXIlO3bswNfXlyFDhjBkyBB8fX3ZsWMHK1euNGkbb7zxhuFZmAULFpCQkMC9997Lr7/+yscff2xulQQzjQrXP1m/9UQKWl0N2RL6vwRu7aAgA3a83kC1EwShuTP7OZZevXpx+fJl1q1bZ3i6fezYsTz++OMmD/4VERFheN26dWvOnz9PVlYWrq6uYtyEBnBf+1Y42ViRllvM4YRM+oVUMxaLlbX+kthXw+Dkegh9CFoPMV5WEAShTJ3GY7G3t2fq1Km3vfNLly4RHx9P//79adGihRj3u4FYWykYEerFhiNJ/BSTUn1gAfDrBb2f1Q8K9vMseO4QWDs2WF0FQWh+TAosW7duZfjw4SiVSrZu3VpjWVMebszMzOTRRx9l9+7dyGQyLl68SHBwMP/6179wdXXlgw8+MK32Qp2NCvdhw5Ekfj2dyoLRnbC2qiE32KA3IG4bZCdC1DswYmnDVVQQhGbHpMAyevRo0tLScHd3r/EhQZlMZtLDjS+++CJKpZLExMRKN/zHjh3L7NmzRWBpAL2DWuDpZENabjFzN5/i1RHtcXes5jkdawe4/7/wzYPw1+fQeQz492nYCguC0GyYdPNep9Ph7u5ueF3dZOoT8zt37mTJkiX4+vpWmt+mTZsqg24J9UMul/HcfSEA/BCTzH3vRbNi9yWKNdX8DkMGlT2FL8FPM0DT/AY9EwShYZjdK8wSCgoKjI5TkpWVZTRLsFA/nugbyPfP9iXM15kCtZb3dsQx+IM9/Hwyxfj9rmELwcEDMi/CXnE5TBAE40y6FGZOF+Dnn3++1jL33nsva9euNaSpl8lk6HQ6li5dyn333WfyvoTb1yOwBT8+dw8/nUxmyW9xJGcXMfPbGNYcvMKb/+xImJ/L34VtXWHkB7BxAuz/CALugdaDG6vqgiA0USYFlg8//NCkjclkMpMCy9KlSxk8eDBHjx5FrVbz8ssvc+bMGbKysjhw4IBJ+xIsRy6X8WBXXyI7efH53sus3BPP0as3GbXiAA929eHlyHZ4OdvqC3e4HzqO0j+N/39joOtEGPaOPugIgiBgYmBJSEiw6E47d+5MXFwcK1aswNHRkfz8fMaMGcP06dMrjYEiNCxblYIXhrRhbE8/3tsRx+bj1/gxJpnfTqcytX8Izw4Ixk5lBaM+BTs3OPoVxHwDF3bA8CXQ6UGRZl8QhLo9x2IJNjY2DB06lLCwMMNT+EeOHAFMH49FqB+ezjZ88GgYk/oF8M4vZzly5SYfR11k45FEXo5oz4NdfZD/cxmEPqIft+XGBfj+SYjdqL9U5uxb+04EQbhj1SmwXLt2ja1btxodpGvZsmW1rr99+3YmTpxIVlZWlZvEpnZZFupfF18XvnumL7+dTmPxb+dIyiri35tOsubQFd4Y2ZGegX2QPbsf9i2DfR/Ahe1wZT8Mngc9/wXyGp6NEQThjmV2YImKiuKBBx4gODiY8+fP07lzZ65cuYIkSXTr1s2kbcycOZNHH32Ut956yzBuitA0yWQyRoR6Mai9O6sPXmH5H5eIvZbDo/87RCtHa8J8XQjzfZi+wwcQfmIeVsl/wW8vwanv9IOGuZuWmFQQhDuH2d2N586dy5w5czh16hQ2NjZs3ryZpKQkBgwYwCOPPGLSNtLT05k9e7YIKs2IjVLBswNCiH5pII/39kepkHE9r4Tfz6Xzwa4LPLz5Jm3in+cj62coltvBtSPoVt5L6e/viGdeBOEuY3ZgOXfuHE888QQAVlZWFBUV4eDgwIIFC1iyZIlJ23j44YfNHilSaBrcHKxZ9GAosfMi2DytL2/+syOjwr0JbGmHhJyPcgYwsHAJO7Xdkes0WO1/n6TF3fn8m//juyNJxKXloaspo7IgCM2e2ZfC7O3tDfdVvLy8iI+Pp1OnTgDcuHHDpG0sX76cRx55hH379hEaGopSqay03JQuy0LjslUp6B7Qgu4BLQzzbhaoiU3OITYpm++S2vN74g7maL/ET3eNqfHTWR83iEdKH8PK3pV+IS25t40b/2jTCh8X20Y8EkEQLM3swNKnTx/2799vGDP+3//+N6dOneKHH36gTx/T8kd9++237Ny5ExsbG6Kjoyulyjf1WRih6XG1VzGgbSsGtG0FgCT1JDV9Conb38D/yiYet/qDIYoYfirpy4nTrfnkVAivSG4EuznwjzZu/KO1G31DWuJoo6xlT4IgNGVmXwpbtmwZvXv3BmD+/PkMHjyYjRs3EhgYyFdffWXSNl5//XXmz59PTk4OV65cISEhwTBdvnzZ3CqxYsUKAgMDsbGxoXfv3vz111/Vlv3iiy+49957cXV1xdXVlSFDhlQpP3nyZGQyWaUpMjLS7Hrd7WQyGd6envhP/hImb4OWrXGX3WSK1a+sUH3MfusXOGo9jbk583H66yPWrlvNvQu28NBnB/lw1wWOXslCo9U19mEIgmAms89YFi1axIQJEwD9ZTFTR42sSK1WM3bsWOTy209VtnHjRmbPns3KlSvp3bs3H330EREREcTFxRkSZ1YUHR3NY489Rr9+/bCxsWHJkiUMGzaMM2fO4OPjYygXGRnJ119/bXgvcpjdpsB/wLMH4NxWSPoLko9C2incdLkMVRxnqOK4oWh8qhcnUlqzJbo1S63a4RoUTu8Qd/LyIL+kFFelOKMRhKbM7MBy/fp1IiMjadWqFePGjWPChAmEhYWZtY1JkyaxceNGXnvtNXN3X8WyZcuYMmUKTz75JKAfOnnbtm2sWrWKV199tUr5devWVXr/5ZdfsnnzZqKiogydEkAfSDw9PU2qQ0lJCSUlJYb3ubm5AGg0GjQajcnHUl7WnHWaFwV0eFA/AZQWI0s7hSzlGLKU48iSjyPLvkKIPJUQUnlIsQ+AkgQlpy8HcloXyOLF+7hh3xa5RweCPFrQxt2Bth4OhLSyx0Z59z03c+d/ZupOtI1xdW0Xc8rLpDoM23jz5k02bdrE+vXr2bdvH+3bt2f8+PE8/vjjBAYG1rr+888/z9q1awkLC6NLly5Vbt6b8pAl6M987Ozs+P777yuNEzNp0iSys7P56aefat1GXl4e7u7ubNq0iX/+85+A/lLYli1bUKlUuLq6MmjQIBYuXEjLli2NbuPtt99m/vz5VeavX7/eaBZnoXoqTS6uhZdxKbyMa0E8zgWXsdEVVCmnkRRckrw5KwVwVhfAWSmAdFUA9rb2eNmBl52Ep62Euy1YNUoOb0G4sxQWFvL444+Tk5ODk5NTjWXrFFgqunbtGt9++y2rVq3i4sWLlJaW1rpOTRmMZTIZf/zxh0n7TklJwcfHh4MHD9K3b1/D/Jdffpk9e/Zw+PDhWrfx3HPPsWPHDs6cOYONjX6gqw0bNmBnZ0dQUBDx8fG89tprODg4cOjQIRSKqv8VGztj8fPz48aNG7X+AirSaDTs2rWLoUOHVgm2dy1JgpuX0SUeIenINvxUucgzzqBUZxstfk1y45wugDNlASeOIGzdAujs60JnHydCvZ1o5+GA9R1ydiM+M9UTbWNcXdslNzcXNzc3kwLLbeUK02g0HD16lMOHD3PlyhWTH3jcvXv37ezWYt599102bNhAdHS0IagAjBs3zvA6NDSULl26EBISQnR0NIMHV00Tb21tbfQejFKprNMHuq7r3bE82qNpEcKZFCcCRoxAaWUFuSmQFgtpp5DSYtGlnESRk4iv7Aa+ihsM5Zhh9awcB2JvhnAyNpgVuhDOEIKbpx9dfJ0J9XGhi68zbT0cUTXjUxvxmameaBvjzG0Xc8rWKbDs3r2b9evXs3nzZnQ6HWPGjOGXX35h0KBBddlcnbm5uaFQKEhPT680Pz09vdb7I++//z7vvvsuv//+O126dKmxbHBwMG5ubly6dMloYBEamEwGzj76qd1wZIACoCgb0s9A2ilIi0VKi4WM87TQ5TNQcZKBnDRs4lqmG7HXgzl5LISfpWDOy0Lw9/Kgs4+zIeC09XDAStF8g40gNBazA4uPjw9ZWVlERkby+eefc//99zdajymVSkX37t2Jiooy3GPR6XRERUUxY8aMatdbunQp//nPf9ixYwc9evSodT/Xrl0jMzNTpPRv6mxdIPAe/QTIAEpL9MEm+RgkH0dKOQ7X4wxnNiMU+q7mOklGfIY3J9NDOHk0mHW6EK5YBdHB142u/q5083ehW4Arbg6id6Ag1MbswPL222/zyCOP4OLiUg/VMd/s2bOZNGkSPXr0oFevXnz00UcUFBQYeok98cQT+Pj4sHjxYgCWLFnCW2+9xfr16wkMDCQtLQ0ABwcHHBwcyM/PZ/78+Tz00EN4enoSHx/Pyy+/TOvWrYmIiGi04xTqyMoafLrpJ8qCTXEupJ4sCzbHkFKOI8+5RhtZMm1I5mHFXqCsg0CyN+euBfDX/gDWSP7kOLcnJCCAbv6udPV3oYOXE0pxViMIlZgdWKZMmVIf9aizsWPHcv36dd566y3S0tIIDw9n+/bthvs9iYmJlZ6X+eyzz1Cr1Tz88MOVtjNv3jzefvttFAoFsbGxrFmzhuzsbLy9vRk2bBjvvPOOeJblTmHjBEH36ifKgk1+BiQf1weblONIycdQFt2kgyyJDiQxRrFfv24RpJ5rwbkz/uyVAvhKFgQenfEM6kjXgJZ0C3DB3dGm2l0Lwt2g0Qb6sqQZM2ZUe+nr1mSXV65cqXFbtra27Nixw0I1E5oNB3doF6mfAJkkQW5y2f0a/aRLPYU8OwEvWRZeiiwGcUK/7g0ouG7N+cP+7NT5k2rTGiuvTriHhNM5JICO3uKsRri73BGBRRAsTibTj4Tp7AvthgNl+Y+KcyHjbFlvtFOUXDuJ8sY57HUldJddpLv8IpRGQRKQBCm7W3BI8iPboTVyj460CO5K607dcG/h2phHJwj1SgQWQTCHjRP49wH/PsgAGwBtKWTFQ9op1MknyE88hTLzHI4l6XjLsvCWZUHhSUjYDAmg/V1GktyTLPvWSO4dcA0Mx7ttN5St2oBCfCWF5k98igXhdimsoFU7aNUOVejDGAYSKM5ByjjH9csnyLlyEvn1c7gVXsKZPPykVPzyUyF/H1wG/oASrLnu1AF8euDW/h5sAnvru1QLQjMjAosg1BcbZ2T+fXD374MhHaokkZeZzJWzR8lKOAkZZ2lRcIlgKQl7WQm+uScg9wSc+xKAXKUbRe5dcQzpi11wb/DuCir7RjogQTCNCCyC0JBkMhzdfAnt7wv9RwOg00lcvp5L3JkYci4ewi7jOG00cbSXJeKkuYFT8i5I3gV7QYecPKc2WAX0wj6oN3h2BUkMLSA0LSKwCEIjk8tltPZwprXHQBg0EIDk7CJ+vXiN1PN/wrWj+BWeIVwej5csC+fcODgVB6e+QQlEyqwpudYefLqg9A4F947g0QnsWtS0W0GoNyKwCEIT5ONii0/PNtCzDTCRzPwSjly5ycYL5ym6fJgW2bGEyy8RKkvAjhKsb5yEGyepkLWGYptW6Fp1xMY3FLlHZ/DoCG7tQCmesxHqlwgsgtAMtHSwJrKzJ5GdPYGB5BVrOJ6YzfJLqcSfOIibNh3Pksu0lyXRXpaIn/w6NsXXIWmPfiqjkykodgxE7tkJa6+OyFoEg2ugfnJw13ezFoTbJAKLIDRDjjZKBrRtRb8gF37VJDBixL/IV0ucT8sjKi2Xy8nplKScxibrPEG6q7SXJ9FOloQr+djlxkNuPFzYWmmbpQpb1I5+KFoEomoVjMw16O+g4+IPKjG2kGAaEVgE4Q7haq+ib0hL+oa0BIKAPuh0Ekk3CzmflsfalFzSUi6jSzuLS94FgknFT5aBvzwDLzKx0hZhlX0Bsi/ou0DfosTWHcnZH5VbEHJX/7IHSP3/fpDU2qGhD1lookRgEYQ7mFwuI6ClPQEt7Yno5Am0BSIp1mi5fL2AxKwCtmUWcu1GDoXXryDdvIJdfhK+sgz8DVM6TrIirIsyoCgD0o4a3VexlRNFdt6UOvogd/ZD1dIf21aBWJUHIQcPkIvUNncDEVgE4S5ko1TQ0duJjt4VRwLUZ4DWaHUk3yzialYhMZkFbLlRwPUb6ZRmJmCVk4iXLg0vWSbeskx8ZTfwlt3AWVaITWkuNrm5kHseko3vV4cc/ZC1MiSZTP8TeVkmUFnZPR79T1n5a7kCndIBrJ3A1hm5jTNyWyfkNs76TAjWTmB4XWGewg4rbVGT646t0eq4WaAms0BNVtmk1Uko5DKs5DL9T4UMhVyOQlbxfYXlcjkKuQytTqJQXUqRRkuxRkuRWkeRRqt/r9YaXhepy5ZrtBSWlJKUImdohI76Gv9MBBZBECpRKuQEutkT6GYPtCqb2wkASZLILFBzI7+EG3lq4vJL2J9XQm5OFqU3k1DkXsO6MAWH4jRcNel4yTLxkd3AkyysZDrkVPgjX3FQ9NoGSC/JhnwzjwMYCehin6VYYYdaYY9GYY/Gyh6NlQOlVnZolQ5olY5oVQ7olA5IKkcklT2SlS06KxuwskGnsIYK7yUraySFLShtkCsUyMuCokwGuUUaQ7DILFBXCSBZ+UUUFqtRoEWJFgVarNChQUExKkpQUhZl65mcYo0Wu3rqICgCiyAIJpPJZLg5WOsHPKs0SGswUHnQPK1O4mahPgj9mVNIQXYGJepSijWllJRqKdGUUqLRodZoUJfq/6NWl5ai1ugoLtWiKS2lRKOltFSDXJOPUpOPjTYfBwpxpBBHWSGOFOEkK8CJorL3+smp7LVCJiFHwkZbgI22wOLtUSJZUYKSElQUSyrcZRIKdFihxapC4CgPJHKZVJZgroZtYk2JzJoSmYoSrClGVRZ0VBRhTTFKiiRriiQlyBQo5aCUS4afKrmElRyUMrCSS1jJyiY5ZcFdoiAvB5WuH1A/HTJEYBEEoV4o5BWDkBO3RKI6kSSJYo2OQnUphWWXegrVWgrVpRSotVxX6y/7FKpLyS9Sc/F8LG39vLDSFiLX5GOlyUOhKcSqNB9laT7K0kKU2nxU2kKsy4KPta4QlVSCSlLrzyEkDSqpBGtJjQo1SkoN9bGWlWJNKVBksRMNa0qwlkpqP4sDfRlt2WQmjcKUHdSNCCyCIDQbMpkMW5UCW5WClrWU1Wg0/FpwnhEjeqO05M0EnRY0RVBarJ80xVBapP8pk4NcAXIrUCj1P+UKkJe/ttInLS1/LVfqOzRoS//ehqawbPtF+p/lU2mFZeWTpNPvUybXb6f8tUxR4XVZnWQykMkp1cGp06fpbFV/AxeKwCIIgmAOuULftdqS3asVVqBwBGtHy22zGpJGQ2Lqr3RWqOptH6LvnyAIgmBRIrAIgiAIFiUuhdUDSdLfFMvNzTVrPY1GQ2FhIbm5uZa9JnwHEG1jnGiX6om2Ma6u7VL+96z871tNRGCpB3l5eQD4+fk1ck0EQRAsKy8vD2dn5xrLyCRTwo9gFp1OR0pKCo6OjmVPD5smNzcXPz8/kpKScHJyqn2Fu4hoG+NEu1RPtI1xdW0XSZLIy8vD29sbeS2pecQZSz2Qy+X4+vrWeX0nJyfxRaiGaBvjRLtUT7SNcXVpl9rOVMqJm/eCIAiCRYnAIgiCIFiUCCxNiLW1NfPmzcPauv6eiG2uRNsYJ9qleqJtjGuIdhE37wVBEASLEmcsgiAIgkWJwCIIgiBYlAgsgiAIgkWJwCIIgiBYlAgsTciKFSsIDAzExsaG3r1789dffzV2lRrU22+/jaxsrPPyqX379oblxcXFTJ8+nZYtW+Lg4MBDDz1Eenp6I9a4fuzdu5f7778fb29vZDIZW7ZsqbRckiTeeustvLy8sLW1ZciQIVy8eLFSmaysLMaPH4+TkxMuLi7861//Ij/fzLF9m6Da2mby5MlVPkORkZGVytyJbbN48WJ69uyJo6Mj7u7ujB49mri4uEplTPn+JCYmMnLkSOzs7HB3d+ell16itLQUc4nA0kRs3LiR2bNnM2/ePI4fP05YWBgRERFkZGQ0dtUaVKdOnUhNTTVM+/fvNyx78cUX+fnnn9m0aRN79uwhJSWFMWPGNGJt60dBQQFhYWGsWLHC6PKlS5fy8ccfs3LlSg4fPoy9vT0REREUFxcbyowfP54zZ86wa9cufvnlF/bu3cvUqVMb6hDqTW1tAxAZGVnpM/Ttt99WWn4nts2ePXuYPn06f/75J7t27UKj0TBs2DAKCv4ejrm2749Wq2XkyJGo1WoOHjzImjVrWL16NW+99Zb5FZKEJqFXr17S9OnTDe+1Wq3k7e0tLV68uBFr1bDmzZsnhYWFGV2WnZ0tKZVKadOmTYZ5586dkwDp0KFDDVTDhgdIP/74o+G9TqeTPD09pffee88wLzs7W7K2tpa+/fZbSZIk6ezZsxIgHTlyxFDmt99+k2QymZScnNxgda9vt7aNJEnSpEmTpFGjRlW7zt3SNhkZGRIg7dmzR5Ik074/v/76qySXy6W0tDRDmc8++0xycnKSSkpKzNq/OGNpAtRqNceOHWPIkCGGeXK5nCFDhnDo0KFGrFnDu3jxIt7e3gQHBzN+/HgSExMBOHbsGBqNplIbtW/fHn9//7uqjRISEkhLS6vUDs7OzvTu3dvQDocOHcLFxYUePXoYygwZMgS5XM7hw4cbvM4NLTo6Gnd3d9q1a8e0adPIzMw0LLtb2iYnJweAFi1aAKZ9fw4dOkRoaCgeHh6GMhEREeTm5nLmzBmz9i8CSxNw48YNtFptpV8ogIeHB2lpaY1Uq4bXu3dvVq9ezfbt2/nss89ISEjg3nvvJS8vj7S0NFQqFS4uLpXWudvaqPxYa/qspKWl4e7uXmm5lZUVLVq0uOPbKjIykrVr1xIVFcWSJUvYs2cPw4cPR6vVAndH2+h0OmbNmsU999xD586dAUz6/qSlpRn9XJUvM4fIbiw0GcOHDze87tKlC7179yYgIIDvvvsOW1vbRqyZ0FyMGzfO8Do0NJQuXboQEhJCdHQ0gwcPbsSaNZzp06dz+vTpSvcnG5o4Y2kC3NzcUCgUVXpopKen4+np2Ui1anwuLi60bduWS5cu4enpiVqtJjs7u1KZu62Nyo+1ps+Kp6dnlU4fpaWlZGVl3VVtBRAcHIybmxuXLl0C7vy2mTFjBr/88gu7d++uNHSHKd8fT09Po5+r8mXmEIGlCVCpVHTv3p2oqCjDPJ1OR1RUFH379m3EmjWu/Px84uPj8fLyonv37iiVykptFBcXR2Ji4l3VRkFBQXh6elZqh9zcXA4fPmxoh759+5Kdnc2xY8cMZf744w90Oh29e/du8Do3pmvXrpGZmYmXlxdw57aNJEnMmDGDH3/8kT/++IOgoKBKy035/vTt25dTp05VCry7du3CycmJjh07ml0hoQnYsGGDZG1tLa1evVo6e/asNHXqVMnFxaVSD4073b///W8pOjpaSkhIkA4cOCANGTJEcnNzkzIyMiRJkqRnn31W8vf3l/744w/p6NGjUt++faW+ffs2cq0tLy8vT4qJiZFiYmIkQFq2bJkUExMjXb16VZIkSXr33XclFxcX6aeffpJiY2OlUaNGSUFBQVJRUZFhG5GRkVLXrl2lw4cPS/v375fatGkjPfbYY411SBZTU9vk5eVJc+bMkQ4dOiQlJCRIv//+u9StWzepTZs2UnFxsWEbd2LbTJs2TXJ2dpaio6Ol1NRUw1RYWGgoU9v3p7S0VOrcubM0bNgw6cSJE9L27dulVq1aSXPnzjW7PiKwNCGffPKJ5O/vL6lUKqlXr17Sn3/+2dhValBjx46VvLy8JJVKJfn4+Ehjx46VLl26ZFheVFQkPffcc5Krq6tkZ2cnPfjgg1Jqamoj1rh+7N69WwKqTJMmTZIkSd/l+M0335Q8PDwka2trafDgwVJcXFylbWRmZkqPPfaY5ODgIDk5OUlPPvmklJeX1whHY1k1tU1hYaE0bNgwqVWrVpJSqZQCAgKkKVOmVPnn7E5sG2NtAkhff/21oYwp358rV65Iw4cPl2xtbSU3Nzfp3//+t6TRaMyuj0ibLwiCIFiUuMciCIIgWJQILIIgCIJFicAiCIIgWJQILIIgCIJFicAiCIIgWJQILIIgCIJFicAiCIIgWJQILIIgCIJFicAiCM3QwIEDmTVrVmNXQxCMEk/eC0IzlJWVhVKpxNHRkcDAQGbNmiUCjdBkiPFYBKEZKh8ZUBCaInHGIgjN0MCBAwkPD+fEiRPs2bOn0jLxlRYam7jHIgjN2A8//ICvry8LFiwgNTWV1NTUxq6SIIhLYYLQnLVo0QKFQoGjo+MdMQKicGcQZyyCIAiCRYnAIgiCIFiUCCyC0MypVCq0Wm1jV0MQDERgEYRmLjAwkL1795KcnMyNGzcauzqCIAKLIDR3CxYs4MqVK4SEhNCqVavGro4giOdYBEEQBMsSZyyCIAiCRYnAIgiCIFiUCCyCIAiCRYnAIgiCIFiUCCyCIAiCRYnAIgiCIFiUCCyCIAiCRYnAIgiCIFiUCCyCIAiCRYnAIgiCIFiUCCyCIAiCRf0/yh9Hauta8KEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 400x150 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "plt.figure(figsize=(4,1.5))\n",
    "plt.plot(train_dict['it_counter'], train_dict['NRMS_val'])\n",
    "plt.plot(train_dict['it_counter'], train_dict['NRMS_train'])\n",
    "plt.legend(['NRMS validation', 'NRMS train'])\n",
    "plt.grid(); plt.xlabel('it'); plt.ylabel('validation\\nmeasure')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "700858a9-93f6-424c-b0de-e5b539ef8b4f",
   "metadata": {},
   "source": [
    "## 9. Restarting training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca4d9e4-f380-4fad-b9b0-f1f44f815df1",
   "metadata": {},
   "source": [
    "If you want to restart training with the same optimizer you can use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b6a38035-5322-4512-bfc4-e07282f49ce5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f224f9a2b11040f6ab5c4e822d92981a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/201 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.06717!! nan samps/sec\n",
      "it      10 NRMS loss 0.06928 NRMS val 0.10416   6719.25 samps/sec\n",
      "it      20 NRMS loss 0.08278 NRMS val 0.09314   6808.57 samps/sec\n",
      "it      30 NRMS loss 0.07425 NRMS val 0.06692!! 6796.49 samps/sec\n",
      "it      40 NRMS loss 0.06091 NRMS val 0.05533!! 6365.85 samps/sec\n",
      "it      50 NRMS loss 0.06163 NRMS val 0.08280   6495.54 samps/sec\n",
      "it      60 NRMS loss 0.06084 NRMS val 0.05746   6556.87 samps/sec\n",
      "it      70 NRMS loss 0.05917 NRMS val 0.05523!! 6604.02 samps/sec\n",
      "it      80 NRMS loss 0.05511 NRMS val 0.06172   6650.44 samps/sec\n",
      "it      90 NRMS loss 0.05593 NRMS val 0.05128!! 6673.43 samps/sec\n",
      "it     100 NRMS loss 0.06536 NRMS val 0.11851   6704.21 samps/sec\n",
      "it     110 NRMS loss 0.06302 NRMS val 0.05014!! 6724.78 samps/sec\n",
      "it     120 NRMS loss 0.05203 NRMS val 0.05096   6746.47 samps/sec\n",
      "it     130 NRMS loss 0.04926 NRMS val 0.05559   6760.80 samps/sec\n",
      "it     140 NRMS loss 0.05323 NRMS val 0.06890   6769.30 samps/sec\n",
      "it     150 NRMS loss 0.05186 NRMS val 0.06291   6782.67 samps/sec\n",
      "it     160 NRMS loss 0.05317 NRMS val 0.05860   6780.86 samps/sec\n",
      "it     170 NRMS loss 0.04853 NRMS val 0.05156   6622.51 samps/sec\n",
      "it     180 NRMS loss 0.05845 NRMS val 0.04722!! 6626.77 samps/sec\n",
      "it     190 NRMS loss 0.08121 NRMS val 0.17444   6640.62 samps/sec\n",
      "it     200 NRMS loss 0.07852 NRMS val 0.09504   6653.12 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "model = train_dict['last_model']\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters()) #or the optimizer that you selected\n",
    "optimizer.load_state_dict(train_dict['last_optimizer_state'])\n",
    "\n",
    "train_dict = dsi.fit(model, train, val, n_its=201, val_freq=10, optimizer=optimizer)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79f39981",
   "metadata": {},
   "source": [
    "## 10. Saving and loading models\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f4c1029",
   "metadata": {},
   "source": [
    "Saving and loading model can be done in multiple ways. The first and most straightforward way is to use `cloudpickle`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4e342d1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepSI as dsi\n",
    "model = dsi.SUBNET(nu='scalar', ny='scalar', norm=dsi.Norm(0,1,0,1), nx=2, na=4, nb=4)\n",
    "\n",
    "import cloudpickle\n",
    "cloudpickle.dump(model, open('model.pth','wb'))\n",
    "model = cloudpickle.load(open('model.pth','rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00926837",
   "metadata": {},
   "source": [
    "Another good option is `safetensors` with state dicts. This makes it much safer to share since safetensors files cannot execute arbitrary code unlike cloupickle, reducing risks associated with loading untrusted models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ad6143d4",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "safetensors not installed, skipping cell\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from safetensors.torch import save_file\n",
    "except ModuleNotFoundError:\n",
    "    print(\"safetensors not installed, skipping cell\")\n",
    "else:\n",
    "    import deepSI as dsi\n",
    "    model = dsi.SUBNET(nu='scalar', ny='scalar', norm=dsi.Norm(0,1,0,1), nx=2, na=4, nb=4)\n",
    "    save_file(model.state_dict(), \"model_state-1.safetensors\")\n",
    "    del model\n",
    "    \n",
    "    from safetensors.torch import load_file\n",
    "    model = dsi.SUBNET(nu='scalar', ny='scalar', norm=dsi.Norm(0,1,0,1), nx=2, na=4, nb=4) #recreate the model structure\n",
    "    state = load_file(\"model_state-1.safetensors\")\n",
    "    model.load_state_dict(state) #load the parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88f1325-88e0-4432-aece-532fc798e0b7",
   "metadata": {},
   "source": [
    "### 10.1 Recovering from a crash\n",
    "\n",
    "After a crash (e.g. a forced windows update) you can recover the training progress with the following steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6d9f327d-415d-4d82-9b84-6b1cda366a36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 271\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "03b64971aa0242b1aad00211965bfef1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.97981!! nan samps/sec\n",
      "it      25 NRMS loss 1.03293 NRMS val 1.32194   6870.23 samps/sec\n",
      "it      50 NRMS loss 0.87532 NRMS val 0.80628!! 6852.13 samps/sec\n",
      "it      75 NRMS loss 0.70106 NRMS val 0.60638!! 6850.91 samps/sec\n",
      "it     100 NRMS loss 0.44274 NRMS val 2.05038   6847.93 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "from nonlinear_benchmarks import Cascaded_Tanks\n",
    "import torch\n",
    "train, test = Cascaded_Tanks()\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train)\n",
    "train, val = train[:-700], train[-700:]\n",
    "torch.random.manual_seed(0)\n",
    "model = dsi.SUBNET(nu, ny, norm, nx=2, na=4, nb=4)\n",
    "train_dict = dsi.fit(model, train, val, n_its=100, val_freq=25)\n",
    "del train_dict, model # Oops I deleted the results, time to recover them from the checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e70bdf13-8a5f-4113-8d1b-f6cf0763d5c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training results saving directory: C:\\Users\\gerben\\AppData\\Local\\deepSI-checkpoints/\n"
     ]
    }
   ],
   "source": [
    "import os, cloudpickle\n",
    "folder = dsi.fitting.get_checkpoint_dir()\n",
    "print('Training results saving directory:', folder) #you can manually go here\n",
    "\n",
    "# get the most resent file with :\n",
    "most_recent_file = max((os.path.join(folder, f) for f in os.listdir(folder)), key=os.path.getmtime)\n",
    "train_dict = cloudpickle.load(open(most_recent_file,'rb'))\n",
    "# train_dict.keys() contains all the info \n",
    "model = train_dict['best_model']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3adc6e0e-9124-4249-ab54-8c7f82dfa5ce",
   "metadata": {},
   "source": [
    "## 11. n-step errors\n",
    "\n",
    "One of the best ways to analyise and develop models is in my opinion to analyse the n-step-error.\n",
    "\n",
    "n-step-error is very simple; it is the (normalized) expected error after taking \n",
    " steps forward in time after state initialization. (see: https://arxiv.org/abs/2012.07697 for details or docs for details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "753363ef-50c1-4206-90f8-7fafe08e8987",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43ec2cc7365f4bae8a36ff5686a69bdd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1001 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.92862!! nan samps/sec\n",
      "it     100 NRMS loss 0.45504 NRMS val 0.19720!! 6115.92 samps/sec\n",
      "it     200 NRMS loss 0.13398 NRMS val 0.10192!! 6478.72 samps/sec\n",
      "it     300 NRMS loss 0.08147 NRMS val 0.08834!! 6615.01 samps/sec\n",
      "it     400 NRMS loss 0.05968 NRMS val 0.06005!! 6611.59 samps/sec\n",
      "it     500 NRMS loss 0.05644 NRMS val 0.07996   6664.74 samps/sec\n",
      "it     600 NRMS loss 0.07515 NRMS val 0.10694   6694.42 samps/sec\n",
      "it     700 NRMS loss 0.04553 NRMS val 0.04793!! 6486.69 samps/sec\n",
      "it     800 NRMS loss 0.03738 NRMS val 0.05132   6495.77 samps/sec\n",
      "it     900 NRMS loss 0.04360 NRMS val 0.04306!! 6490.63 samps/sec\n",
      "it   1,000 NRMS loss 0.03887 NRMS val 0.04736   6468.51 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "import nonlinear_benchmarks as nlb\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "train_val, test = nlb.Silverbox()\n",
    "nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train_val)\n",
    "train_val, test = norm.transform(train_val), norm.transform(test) #normalize data\n",
    "s = int(len(train_val)*0.95)\n",
    "train, val = train_val[:s], train_val[s:]\n",
    "model = dsi.models.SUBNET(nu, ny, dsi.Norm(0, 1, 0, 1), nx=4)\n",
    "train_dict = dsi.fit(model, train, val, n_its=1001, val_freq=100)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "35c8407a-8318-443a-95df-6bdf0f49e22a",
   "metadata": {},
   "outputs": [],
   "source": [
    "(upast, ypast, ufuture, yfuture), ids = model.create_arrays(test, T=100, stride=10) #stride to not evaluate on all available datapoints and thus safe compute and memory.\n",
    "import torch\n",
    "with torch.no_grad():\n",
    "    yfuture_pred = model(upast[ids], ypast[ids], ufuture[ids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "db15c44f-179b-40dc-9319-2c4363524f88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAADZCAYAAAB7Etp/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEjUlEQVR4nO3dd1xV9f/A8ddlXJYMFZkyVFRcgAIiampK4Sg1LUeWIxuWpkVLs9SmVl/72bCvWd8clTkqrUxNw9y4UHGCC8TBEJC97z2/P9BbNy56EeSqvJ+PB49HnPM5577PR7tvP/OoFEVREEIIIcRNMzN1AEIIIcSdTpKpEEIIUUOSTIUQQogakmQqhBBC1JAkUyGEEKKGJJkKIYQQNSTJVAghhKghSaZCCCFEDVmYOoDbkVar5dKlS9jb26NSqUwdjhBCCBNRFIW8vDw8PDwwM6u6/SnJ1IBLly7h5eVl6jCEEELcJs6fP0/Tpk2rPG/yZDp//nw++ugjUlNTCQwM5LPPPqNz585Vll+1ahVvvvkmSUlJtGzZkg8++ID+/fvrzufn5zN16lTWrFlDZmYmzZo1Y/LkyUyYMMHomOzt7YGKynNwcLj5hxNCCHFHy83NxcvLS5cXqmLSZLpixQqioqJYsGABYWFhzJs3j8jISBISEnBxcalUfteuXYwcOZLZs2fzwAMPsGzZMgYPHsyBAwdo3749AFFRUWzevJnvvvsOX19fNm7cyHPPPYeHhwcDBw40Kq5rXbsODg6STIUQQtxwyE9lyo3uw8LCCA0N5fPPPwcqxiq9vLx4/vnnmTp1aqXyw4cPp6CggLVr1+qOdenShaCgIBYsWABA+/btGT58OG+++aauTHBwMP369ePdd981Kq7c3FwcHR3JycmRZCqEEPWYsfnAZLN5S0tLiY2NJSIi4u9gzMyIiIggJibG4DUxMTF65QEiIyP1ynft2pVff/2VixcvoigKf/31FydPnuT++++vMpaSkhJyc3P1foQQQghjmSyZZmRkoNFocHV11Tvu6upKamqqwWtSU1NvWP6zzz6jbdu2NG3aFLVaTd++fZk/fz49evSoMpbZs2fj6Oio+5HJR0IIcWNarcLt8hbPgpJydp3J4HJeiUk+3+QTkGrbZ599xu7du/n111/x8fFh27ZtTJw4EQ8Pj0qt2mumTZtGVFSU7vdrA85CCKEoym2zRG7FvmT+OJaGv5s9gV5OBHk54epgbZJYzmUWMPDznZibqejs24jOzRrRo5Uzfi7Xn6hzq7z202HWHk4BwKexLcE+DQn2aciQjk2xUZvf8s83WTJ1dnbG3NyctLQ0veNpaWm4ubkZvMbNze265YuKinj99ddZvXo1AwYMACAgIIBDhw7xn//8p8pkamVlhZWVVU0fSQhRCxJS8zifVUiAlyMu9qZJFNfsPpvJuEX7sLe2oLWbPa1d7enV2oXuLZ3rPJbswlJm/HKMknItm+PTdcef6dGcaf3b1Hk83+xIJKeoDIANx1LZcCwVczMVXz4WTERb1xtcXbsuZhex7khFIlWp4FxmIecyC1l7OIWHg6tezlKbTJZM1Wo1wcHBREdHM3jwYKBiAlJ0dDSTJk0yeE14eDjR0dG88MILumObNm0iPDwcgLKyMsrKyiotrDU3N0er1d6S5xBC1J4rBaU89MVOCks1ADRtaEOwT0Nevr81Xo1s6zyeT6NPUVSmoahMQ3peCdtPZfD1jkQWjQ3lXv/KKw5upZ8OXKSkXItvY1vCmjUm7kI28al5fLntLIOCPGnrUXeTJfNLyvnpwEUA3hrYjoLScv48nsaB5GzeW3eCnq2bYGled6OIy/acQ6tA1xaN+e9jwRxMvkLsuSvkl5RjZXHrW6Vg4m7eqKgoxowZQ0hICJ07d2bevHkUFBQwbtw4AEaPHo2npyezZ88GYMqUKfTs2ZO5c+cyYMAAli9fzv79+1m4cCFQsZSlZ8+evPLKK9jY2ODj48PWrVtZunQpH3/8scmeUwhhnFWx5yks1WBtaUZJuZYLV4q4cKWIxIwCfpnYrU67W0+m5bHrTCZmKvhqdAiX80rYcCyVLQmXeW/dCe5p6YzFTSSM/JJyYs9dwcPRmpauxnWJKorC93vOAfBUj+aMCvMBYNKyA6w9nMLHmxL4ekxoteLIKSzjRGouJ1JyuZRdxOhwX6P/wbL64EXyS8pp3sSO0eE+qFQqRof70uujv0jMKGD5vvM83sXnhvfRahXS8oo5e7mA45dyOZ6Sy7FLOThYW/K/saE42lje8B4l5RqW7z0PwOhwHxxtLOnV2oVerev2HzsmTabDhw/n8uXLzJgxg9TUVIKCgtiwYYNuklFycrJeK7Nr164sW7aMN954g9dff52WLVuyZs0a3RpTgOXLlzNt2jRGjRpFVlYWPj4+vPfee9XatEGI+qagpJzDF3I4dD6bIxezadGkAVH3tarT5KXVKny/JxmAmQ+244EAdw6dz2bCt7EcvpDDhqOp9OvgbvT9iss0ZOSXkJlfSmZBCVot9PZ3wczMuGdaGpMEwH1tXenTpuI7qV8Hd3p99Ben0/NZvu88j10nYWi0Cmm5xaTkFJOSU8Tp9Hx2nc7kQPIVyrUKNpbmrJ9yD77OdjeMJeZsJmcvF2CnNmdQkKfueNR9rVh/NJU/T6QTe+4KwT4NDV6fnFnINzsTuZhdRGpORUwZ+foTdWLOZrLmuW43/AeCoih8e7VuHu/io/s70sDKgsl9WjLjl2N88ucphnT0xM6qcopRFIX/bEzgz+PpnMsqoLjMcK/hnPUnmD0k4LqxAKw/kkpmQSnujtZEtKnb7uV/Muk609uVrDMV9clPsRd47afDlGv1vwp+nBBOiG+jOotj28nLjP5mL/ZWFuyZ3gdbdcUX8ccbE/h082laNLHjjxd6VPllv+l4GktjkkjLLSYtt0Q3nvdPk+714+XI1jeMJbe4jC7vR1NYqmHZU2F0bfH3GOninYnM+u04je3UbHmlF/bWlVtP6bnFPPj5DtJyDc8stbY0o7hMS4hPQ1Y8E475DRL8xO8P8PuRFB7r4s27gzvonXvtx8Os2H+eLs0b8cNTXQz+A2jkwt3EnM2sdNzTyYY27g7sS8oip6iMqf38mdCzxXVj2X02kxELd2Njac6e6X1w+Mfzl2m03PfxVpIyC3khoiUvRLSqdH3MmUxGfrVb97uFmQqvRra0drWnnYcDDjaWzPz1GAA/PNWF8BaNrxvPkC92ciA5m5fua8XzfVpet+zNMDYf3HWzeYUQ1fO/HYmUaxXcHKzp5ONERl4pe5Oy+N+ORKOTaXZhKb/FXeLg+WyKyzQUlWoo1WgZFOjJsFDjZsZ/t7uiG3NIJ09dIgV4skdzlu4+x5nLBfx88CLDQirfr7hMw2s/HSaroFTvuNrcDOcGahxsLIlPzeOLLafp08aFjt6GW3DXrNp/gcJSDa1d7Qlvrv9lPqqLD0tjznE2o4D/bjnDq339K13/5bazpOWWYGGmwtXBGndHa5o2tCG0WSPu8WuCmRlE/t829p+7wqKdiTx5T/MqY0nPK+aPYxXL/x7tXLklPDmiJasPXmT32Sx2ns6sNDnqYPIVYs5mYmGm4s0H2uLpZIObozVejWx13air9p/nlR8P83+bThLZzo1m12ktfxtT8ec0uKOnXiIFsDQ345VIfyYuO8DCbWcZFeZDE3v9yZ2fbT4FwEMdPZnSpyWeDW0qja+eTMvj+z3JvL76COun3IO1peFxz6MXcziQnI2luYoRnb2rjLkuSDIVoh47l1nA8ZRczM1UrJ9yDw3t1CSk5hE5bxt/HEvlfFbhdcfRdp3O4Ps9yWw6nkappnJ33Z6zWXTycbrhcomUnCL+PFExU3/Uv7pOHawtea5XC95fF88nf55iUJBHpUklv8ZdIqugFE8nGz4YGoCrgxUu9tY42FjoWmqTfzjIr3GXeGllHL9PvqfK5RJa7d/dmKO7+lRq6VmamzG1nz9PfxvL/3YkMqqLD55ONrrzWQWlLLvaXf3VmBDurWLsbvqAtry++ggf/ZHAvf4utGjSwGC5lfvOU65V6OTtZHCSkaeTDaO6eLNoZxIf/RFPNz/9seUvtpwBKpLfmK6+Bj/j4eCm/Bp3ie2nMpj602F+eKqLwe7wtNy/E/vocMNd3P07uBHY1JG4Czl8En1SryUdey6LXWcysTRX8XJka716+6fX+vnz54k0EjMK+DT6lMF/sMDfib1fe/dKSbuuyftMhajH1h+t+GIMb96YhnZqAFq72XNPS2e0CizZlVTltZvj03j06z38fiSFUo2WNu4ORN3XircHtePDoQF0bdGYcq3CjF+O3XBh/w97z6NVoHOzRrQyMClndLgvbg7WXMwu4vvdyXrnFEVh0c6kq+V86N7SmZau9jjaWuollbcHtcPF3oqzGQV8sCG+yli2nrpMUmYh9tYWDP7H+OQ/3dfWlS7NG1FSruWtX/Wfb/HORIrKNLTzcKBXqyZVfs7Izl7c09KZknItL6+KQ6OtXEcarcIPVyfXXG989rleftiqzYm7kMN3e/6un5NpeWw6noZKxXW7b1UqFe8/1KGi6zYxi+X7zhsst2xPMuVahVDfhrRxN9zlqVKpmNqvYqnOd7uTWXv4ku7cZ5tPAzC0U9MqEylU/APq7UEVc2G+3HaWY5dyKpVJzSnml7iKGcVVJfa6JMlUiLtIdadArL+6Nq9fB/213U90bwbAin3nyS8pN3jtd1eTWh9/F9Y+3531U+5hcp+WjA73ZVioF3OGBGBlYcauM5m6xfSGlGm0LN9bca+qEoa1pTmTr46Hzf/rNNmFf3fn7k3M4kRKLtaWZgy/Tpeyk62aDx+umNCyeFcSu05nVCqjKArf7EgEYFiIl8EJNFCRMN58oC2W5io2Hk/jy21nAcgrLmPx1X+ATLzX77oTuFQqFR8MDcDeyoKDydm89/uJSn9+3+85x8XsIpxsLel/nclXTeytmHK1fmb9eoztpy4DsGBrRas0sq0bfi6GW77XeDWy1Y0nv7/uBDFn9MdYNx1P479XW7mPh/te917hLRrzRLeKv0MvrYyrmNh2IYctCZcxN1PxbK/rj8sCRLZzo197NzRahfGL9+sl1JScIkYsjKG4TEsHT8cqJ17VJUmmQpjYmcv5LNqZSEEVScsYseeyGPLFTsLejyY+1bi9pS9cKSTuQg5mKri/rX4y7dmyCc2b2JFXUs5KA62UjPwStp6s+MJ+fUAb2ns6Virj3diW53r5AfDu78erTMobj6WRnleCcwM1fdsZ3rAF4JGQpjRvYkdmQSnP/3CQ8qvdytdapUM6NcXJVn3dZ+7V2oVHwyrG1iYvP8ief0zKKddoee2nw2w/lYGZ6satnXYejsx8sB0AH26IZ9vJy3y3O5nc4nJaNLG77rNc4+Fkw1uDKu7xzc5Epq85ivZqC/Xr7WeZ8UvFRJzx3ZpVOW54zdM9mjOkoycarcJz3x9gS0I6vx6qaBU+d++NkxfA2K6+hDdvTH5JOY//b4/uz37dkRSe/S6WUo2W/h3cGGDErOrpA9rQ29+FknItTy7Zz3vrjgMwKNADn8Y3nsEM8Pag9vi5NCA1t5hHFsTwV3w6F7OLGP7lbpIyC2na0IYvRnW6LXaokmQqhIkoisKyPckM+HQ7b/12XPfFWR0XrhQyadkBhv43hgPJ2aTnlTDh21iDM1n/bcPVLt5Q30aVxpvMzFS6lsXiXUmVuiDXxl1Co1UIbOpY5VgfwDM9m+PT2Ja03BI++fNkpfPnswqZ8ctRAEaEeqO2qPorydLcjPmPdsLG0pztpzL4YEM8F64UsvF4xXOMrWI88N+m929DW3cHMvJLefTrPXy17SzFZRomfHeAlfsvYKaCOUMDjPrCHxXmzfAQL7QKPP/DQb7eXtFCfbaXn9FLcIZ0asqcIR1QqSq6UaNWHmLuxgTe/f0EULHD0aTefje8j0qlYvbQDoT6NiSvuJxxi/dRrlXo7udMQFMno2IxN1OxaFwoDwS4U65VePWnw0z4NrbiHy9ahUFBHnw6ouMNZx9fu9enIzvi72ZPRn4Ju89moVIZn9ihosX907Nd6dqiMYWlGsYv2cegz3eSnFWIdyNbVjwTbpLNPAyRZCqECeQUlTFp2UFeX31Et87upwMX2J+UZdT1GfklvLv2OL3nbmXt4RRUKhge4oWnkw1JmYW8tPKQroVTlWvjpVV1Hw7t1BQnW0uSswrZdFx/G8/VByvGqgZ3NDymeI21pTmzBl5reSXp3Se/pJwnl+wns6CUdh4ORn3JtnF34D+PBALw1fZEJn5/AK0C3fwaGxxrNcTOyoIfnw3noautuPfWnaD7B5v580Qaagsz/vtYsMEZw4aoVCreHtyOIC8ncorKyLw6CWpQkIdR118zorM3n47oiIWZijWHLunGFl+JbM3Ufv5Gt7ysLMz58vEQvBvZcq3H+DkjulT/ydrSnE9HdGTy1QS+4VgqGq3Cw8FN+XhYULU2qmhgZcH/xobi3KDiH2v927tXe+9eRxtLFo/rzCPBTdEqFX/3fRvbsuKZLtcdd61rkkyFqGN5xWUM/HwHvx9JwcJMxbR+/gwLqdg/9M1fjum6Lw3JKSzjoz/i6fHhX3y9I5HSci3hzRuz9vnufPBwAAseC0ZtYcafJ9L5YsvpKu+TmlNM7LkrAPRtb7g70kZtzqNXlxu8v+6Erhv6zOV84i7kYG6m4sHAGyeNe1u70L9DxdjXU0v389LKOLILS5n8w0ES0vJoYm/F12NC9JbDXM+AAHcmXk28cRcqxtHGdW1m1LXX2Kot+HhYIG8PaoeluYqM/FLsrSxY+kRnIo3onv0nKwtzFjwWrEsYE3o2v6mt9B4M9GDh6GBd6/ydQe1uOO5qSCM7Nd+MDcXF3op7Wze54TpNQ8zMVETd35r/Gx6IcwMrnujWjA+HBhjVIv03TycbvnuyM4918ebNB9pW+3oAtYUZHz4cwMwH2/JgoAfLnw7H3fH2SaQgmzYYJJs2iFvp+z3nmL76KK4OVnz5eAhBXk5k5pdw73+2kFtczlsD2xlcwlCu0dL/0+2cTMsHILCpIy/d35p7WjrrfeGu3HeeV386jEoFi8d1pqeBGaXXNh4I8WnIj892rTLW3OIy+s3bzsXsIt2GAf/5I4HP/zpNb38Xvhlr3BZ2xWUa/m/TSRZuP4uigI2lOUVlGqwszFj5TDiBXk5G3ecajVbhySX7+CvhMt6NbPnr5V439UUPFeswV8Ve4PEuPlXOUDVGYkYB+xKzeDi4qdFdvIYkZRSQX1JucBy6OrRapUZxXHM7vTXHFG77l4MLUV/9fHWD8PHdmxF0NYk0bmDFK1dnUv5nY0Klrd4A1h1N5WRaPk62lix8PJg1E7vRo1WTSl90w0K9GNnZC0WB55cd4HR6vsF7QdWt0mscrC11M2C/253MloR0XRfvQzfo4v0na0tzpvVvw48TwvFtbEtRWcVG9nOHBVY7kcLf43GT7vXjkxFBN51IATp6N+T9hzrUKJECNHO2Y1ioV40TmK+zXY0TKVAriRSo14m0OiSZClGHzmUWEHvuCmYq9PZYBXg0zId2Hg7kFZfzwXr9dZCKovDV1eUX47o24/52btf9kpv5YDuCfRqSW1zO+CX79HYG2nA0hX1Xx2aN2eu2m58zY67ObJ34/QEuZhfRwMqC+27iNVvBPo1YP6UHr/X159ORHXkgoHpji/9kb23Jy5Gtb7ibkRB1QZKpEHXoWqu0m59zpZc6m5updAvVV8VeYMepv9dB7knM4sjFHKwszHisy423TbO2NOfLx4Np2tCGc5mFTPg2luIyDR9uiGfCdwdQFBjQwd3oCRxT+7WhmbMdBVdfjdavvdsNl2pUxUZtzrO9WjDQiPFWIe4UkkyFqCOKoui6SId2MvzC4mCfhrpXV73202HyiiuWuFxrlT4c3JTGDYzbNs25gRWLxoZib2XB3qQs7vnwL93Wck92b8YnI4KMjt1Gbc5/HgnkWs/hQ52M7+IVoj6QZCrETVp7+BLd5mzmrd+OUXx1DPB6Ys9dITmrEFu1Ofe3q7qLdGo/f7wb2XIxu4j3fj/B6fR8ouPTUakqxlmro6WrPfNHdcLcTMXlvBKsLc34ZEQQbzzQttrv4gz2acjnj3bijQFtKm3+LkR9JxvdC1FNiqLw5bazzLk6rrloZxK7z2bx2ciO192y7eerrdJ+7d2vuwzEzsqCjx4OYPjC3Szfd55jlyp2NIpo40rz62yQUJUerZrwyYggfj10iRciWhncLN1Y19vSToj6TFqmQlRDuUbLm78c1SXSgYEeNLZTcyIllwc/28HK/ecN7o9bXKZhbVzF1m5DjOgiDWvemHHdfAE4crFiLeXTPap+TdeNPBDgwcLRITVKpEKIqknLVIgbUBSFEyl5bD15mQ1HU4i7kINKBW8MaMv47s1Izy3mhRWH2HUmk1d/PMzGY6m8O7gDbo5/TzDaHJ9ObnE57o7WdDGyi/TVSH+2JFwmMaOAIC8nQm6DzbyFEIZJMhXiOvYnZfH8DwdJySnWHbOyqBh37Nu+osvTxcGab8eHsWDrGeb9eZI/T6SzJ3Er0/u3wd7akt+PXGJzfDpQsf2esWsibdTmzH+0E3M3JjC5T0tZ7yfEbUx2QDJAdkC6+yiKQlJmITtOZ7Dj1GX2JmYR5OXEJyM74mBtafCa0nItkfO2kZhRgI2lOeEtGtOzVRMi2rpWuaTkZFoer6yK021z90/+bvYsfaIzLv9aEiOEuH0Zmw8kmRogyfTu8+GGeN2ykH9q7WrPonGheBhIjt/sSOTttcdxbqBm88u9qky6/1au0fLNzkS+2HIGB+uK91A+EOBOOw8HaV0KcYeRZFoDkkzvLqk5xXT/YDPlWoWwZo24p6Uzfi4NmPHLMdLzSnB1sOKbsaG08/h7C7fswlJ6frSFnKIy3n+og+4dmNVV3/c1FeJOZ2w+kDFTcdf7fs85yrUKob4NWfFMuO54h6ZOjFu0l5Np+QxbEMM7g9vzUEdPVCoVn20+TU5RGa1cG+je6HIzJJEKUT/I0hhxVysu07BsTzIA47rpb3jg6WTDqgkVLx4uKNUQtTKOp5bGsi8pi6UxSQBMH1D9zQ2EEPWPfEuIu9pvcZfILCjFw9Ga+w1szO5oY8mSJzrz8v2tsDRX8eeJNB5ZEEOZRqFHqyYGX18mhBD/JslU3LUURWHxriQAHg/3rbKFaWluxqTeLfnt+e6096wYEzFTwfT+beoqVCHEHU7GTMVda1/SFY5dysXa0owRoV43LO/v5sDq57qx+sBFnO3VtHazr4MohRB3A0mm4q61eFciUPES64Z2aqOusTQ3Y5gRiVcIIf5JunnFXel0eh5/HEsDYExXX9MGI4S460kyFXedY5dyGLFwNxqtwj0tnfF3k7XCQohb67ZIpvPnz8fX1xdra2vCwsLYu3fvdcuvWrUKf39/rK2t6dChA+vWrdM7r1KpDP589NFHt/IxxC1QrtGyNzGL5XuTmbM+nue+j2XWr8dIzys2WH5vYhYjvtxNRn4pbd0d+L/hQXUbsBCiXjL5mOmKFSuIiopiwYIFhIWFMW/ePCIjI0lISMDFxaVS+V27djFy5Ehmz57NAw88wLJlyxg8eDAHDhygffv2AKSkpOhds379esaPH8/QoUPr5JlE7diflMWbvxzjREpupXM/xl7ghYiWjOnqi6W5GRezi9gcn867a49TUq6ls28jvh4bYvQWgEIIURPV2k6wvLyc999/nyeeeIKmTW9+V5h/CgsLIzQ0lM8//xwArVaLl5cXzz//PFOnTq1Ufvjw4RQUFLB27VrdsS5duhAUFMSCBQsMfsbgwYPJy8sjOjraqJhkO0HTysgvYc76eH6MvQCAvbUFnbwb4tvYlqYNbVl7+JJuI/lmznaUa7WczyrSXX9v6yZ8MSoYG7W5SeIXQtw9bsl2ghYWFnz00UeMHj26xgEClJaWEhsby7Rp03THzMzMiIiIICYmxuA1MTExREVF6R2LjIxkzZo1BsunpaXx+++/s2TJkirjKCkpoaSkRPd7bm7llpCoG3nFZTz42Q7dK8+Gh3jxat/WNG5gpSszvnszVu4/zwcb4knMKADA3ExFB09H+vi7MKFXCyxl1yIhRB2qdjdv79692bp1K76+vjX+8IyMDDQaDa6u+jvTuLq6Eh8fb/Ca1NRUg+VTU1MNll+yZAn29vYMGTKkyjhmz57NW2+9Vc3oxa0QfSKdlJxi3Bys+eKxTnTyrvxCbDMzFSM6e9O3vRt/HEvFzdGGYJ+GNLAy+aiFEKKeqva3T79+/Zg6dSpHjhwhODgYOzs7vfMDBw6steBqwzfffMOoUaOwtq76HZLTpk3Ta+3m5ubi5SVrDU1h/dGK8e5HQpoaTKT/5GSrZnjozb3NRQghalO1k+lzzz0HwMcff1zpnEqlQqPRGH0vZ2dnzM3NSUtL0zuelpaGm5ubwWvc3NyMLr99+3YSEhJYsWLFdeOwsrLCysrqumVEzWTkl5CaU0x7T8cqyxSWlrP15GUA+rY3/OcvhBC3o2oPLGm12ip/qpNIAdRqNcHBwXoTg7RaLdHR0YSHhxu8Jjw8vNJEok2bNhks/7///Y/g4GACAwOrFZeoPcVlGub/dZoeH/7FA5/t4JdDF6ssuyXhMsVlWrwb2dLWXSZ+CSHuHCYfZIqKimLMmDGEhITQuXNn5s2bR0FBAePGjQNg9OjReHp6Mnv2bACmTJlCz549mTt3LgMGDGD58uXs37+fhQsX6t03NzeXVatWMXfu3Dp/JlFhw9FU3v39OBeu/D3Tduavx+jawpkm9pV7AtYfrRj37tfeTd4DKoS4o9zUlMetW7fy4IMP4ufnh5+fHwMHDmT79u03FcDw4cP5z3/+w4wZMwgKCuLQoUNs2LBBN8koOTlZb91o165dWbZsGQsXLiQwMJAff/yRNWvW6NaYXrN8+XIURWHkyJE3FZeomV2nM5jwXSwXrhTh7mjN/w0PpJ2HA9mFZcz45Wil8sVlGjafqOi+ly5eIcSdplrrTAG+++47xo0bx5AhQ+jWrRsAO3fuZPXq1SxevJhHH330lgRal2Sdac1NX32E7/ckc39bV+aNCMJWbcHxS7kM/HwH5VqF+Y92YkCAu678puNpPLV0P+6O1ux8rTdmZtIyFUKYnrH5oNot0/fee48PP/yQFStWMHnyZCZPnsyKFSuYM2cO77zzTo2CFneP7acyABjR2QtbdcVoQlsPB5671w+AGb8cJTP/77W912bxRrZzk0QqhLjjVDuZnj17lgcffLDS8YEDB5KYmFgrQYk7W1JGAclZhViaqwhr1ljv3KR7/Wjtak9mQSlPfxtL3PlsSsu1/Hm8oou3n3TxCiHuQNVOpl5eXga35fvzzz9lbaYAYPupiuUtwT4NsfvXRgpqCzM+eiQAKwszYs9dYdD8nQz97y5yi8txbmBFiG8jU4QshBA1Uu3ZvC+99BKTJ0/m0KFDdO3aFagYM128eDGffPJJrQco7jzbrnbx3tOyicHzAU2d2PhiDz6NPs3qgxc4crFin93Idq6YSxevEOIOVO1k+uyzz+Lm5sbcuXNZuXIlAG3atGHFihUMGjSo1gMUd5YyjZaYM5kA9GxlOJkC+DS2Y+6wQCb19uOzzac4djGXcd186yhKIYSoXdVKpv98a8yOHTtuVUziDnbofDb5JeU0slMbtfFCM2c7Ph4WdOsDE0KIW6haY6YWFhZ8+OGHlJeX36p4xB1u29XtALv7OcusXCFEvVHtCUh9+vRh69attyIWcRf4e7zU2cSRCCFE3bnr3xoj6k52YSmHL2QDVU8+EkKIu5FJ3xoj7i47T2eiKNDa1R43x6pfeSeEEHebaidTrVZ7K+IQt7EyjRZL8xuPCFwbL5UuXiFEfVOtMdOysjIsLCw4erTyRuXi7rRsTzItp6+n77xtzN2YwOEL2Rjazjm7sJQ/r25Uf891lsQIIcTdqFotU0tLS7y9vaUrt55QFIWvd5wFID41j/jUPD7bfJp2Hg4sfaIzjRv8/Rq1N9YcJbOglOZN7Ahv3riqWwohxF2p2rN5p0+fzuuvv05WVtatiEfcRo5dyuXs5QKsLMz48OEA+ndww1ZtzrFLuYxdtI/8koolUr/GXWLt4RTMzVT837Ag1BY39WY/IYS4Y1V7zPTzzz/n9OnTeHh44OPjU2k274EDB2otOGFav8ZdAqBPGxeGhXgxLMSLs5fzeXhBDEcu5vD00v3MGRLAm2squv2f7+1HoJeTCSMWQgjTqHYyHTx48C0IQ9xutFqF364m04GBnrrjzZs0YMm4zoxYGMOuM5n0/WQbhaUaApo6MvHq69WEEKK+qXYynTlz5q2IQ9xm9iVlkZJTjL2VBb1a608o6tDUka/GhDD2m30UlmqwsjDj42GBRs34FUKIu9FNfftlZ2fz9ddfM23aNN3Y6YEDB7h48WKtBidM51oXb9/2blhbmlc637WFM/NHdaKlSwPmDO2An4t9XYcohBC3jWq3TA8fPkxERASOjo4kJSXx1FNP0ahRI37++WeSk5NZunTprYhT1KEyjZZ1R1IAGBjkUWW5+9q6cl9b17oKSwghblvVbplGRUUxduxYTp06hbX137vc9O/fn23bttVqcMI0dpzK4EphGc4NrGSZixBCGKHayXTfvn0888wzlY57enqSmppaK0EJ0/rlUEV3/QMB7ljIOKgQQtxQtb8praysyM3NrXT85MmTNGkiO9/c6QpKytl4vGIno+t18QohhPhbtZPpwIEDefvttykrKwMqNrdPTk7mtddeY+jQobUeoKhbC7edpbBUQ3NnOzrKmlEhhDBKtZPp3Llzyc/Px8XFhaKiInr27Imfnx/29va89957tyJGUUfSc4tZuK1i+8CXI1ujUsnLvYUQwhjVns3r6OjIpk2b2LlzJ3FxceTn59OpUyciIiJuRXyiDv3fnycpKtPQ0duJfu3dTB2OEELcMaqdTK/p1q0b3bp1q81YhAmdSstjxb7zAEzv30ZapUIIUQ0yVVMAMGd9PFoFItu5EuLbyNThCCHEHUWSqSDmTCbR8emYm6l4ta+/qcMRQog7jiTTeq60XMtbvx0D4NHO3rRo0sDEEQkhxJ3H5Ml0/vz5+Pr6Ym1tTVhYGHv37r1u+VWrVuHv74+1tTUdOnRg3bp1lcqcOHGCgQMH4ujoiJ2dHaGhoSQnJ9+qR7ijLdh6hvjUPBrZqXnxvlamDkcIIe5IRifT8vJySkpK9I6lpaXx1ltv8eqrr7Jjx45qf/iKFSuIiopi5syZHDhwgMDAQCIjI0lPTzdYfteuXYwcOZLx48dz8OBBBg8ezODBgzl69KiuzJkzZ+jevTv+/v5s2bKFw4cP8+abb+ptfVjfaLQKS3YlMe3nw6TnFuuOn0rL47PNpwCY+WBbGtmpTRWiEELc0VSKoijGFBw3bhxqtZovv/wSgLy8PNq1a0dxcTHu7u4cP36cX375hf79+xv94WFhYYSGhvL5558DoNVq8fLy4vnnn2fq1KmVyg8fPpyCggLWrl2rO9alSxeCgoJYsGABACNGjMDS0pJvv/3W6Dj+LTc3F0dHR3JycnBwcLjp+9wOTqfn8cqPhzmYnA2AcwMr5j/akRDfRjy8YBcHk7Pp4+/C12NCZAavEEL8i7H5wOiW6c6dO/V2OFq6dCkajYZTp04RFxdHVFQUH330kdEBlpaWEhsbq7c+1czMjIiICGJiYgxeExMTU2k9a2RkpK68Vqvl999/p1WrVkRGRuLi4kJYWBhr1qy5biwlJSXk5ubq/dzpFEXhv1vO0P/THRxMzqaBlQXNm9iRkV/Co1/v4ckl+3TH332ovSRSIYSoAaOT6cWLF2nZsqXu9+joaIYOHYqjoyMAY8aM4dixY0Z/cEZGBhqNBldX/Vd4ubq6Vrlhfmpq6nXLp6enk5+fz5w5c+jbty8bN27koYceYsiQIWzdurXKWGbPno2jo6Pux8vLy+jnuF39djiFDzbEU1qupVfrJmx8sQdrn+/OQx090WgV/kq4DMC0/v64O9qYOFohhLizGZ1Mra2tKSoq0v2+e/duwsLC9M7n5+fXbnTVpNVqARg0aBAvvvgiQUFBTJ06lQceeEDXDWzItGnTyMnJ0f2cP3++rkK+ZdYdrngf6diuviwaG4qHkw22ags+HhbIO4PbY2VhRm9/F0aGeps4UiGEuPMZvQNSUFAQ3377LbNnz2b79u2kpaXRu3dv3fkzZ87g4WH8W0acnZ0xNzcnLS1N73haWhpuboa3snNzc7tueWdnZywsLGjbtq1emTZt2lx3gpSVlRVWVlZGx367KynXsP1URctzSCdPvS5clUrF4118eCS4KWpzM8zMpHtXCCFqyuiW6YwZM/jkk09o0aIFkZGRjB07Fnd3d9351atXV2t7QbVaTXBwMNHR0bpjWq2W6OhowsPDDV4THh6uVx5g06ZNuvJqtZrQ0FASEhL0ypw8eRIfHx+jY7vT7T6bRUGpBhd7K9p7OBosY21pLolUCCFqidEt0549exIbG8vGjRtxc3PjkUce0TsfFBRE586dq/XhUVFRjBkzhpCQEDp37sy8efMoKChg3LhxAIwePRpPT09mz54NwJQpU+jZsydz585lwIABLF++nP3797Nw4ULdPV955RWGDx9Ojx49uPfee9mwYQO//fYbW7ZsqVZsd7LoExWt9z5tXCRhCiFEXVBM7LPPPlO8vb0VtVqtdO7cWdm9e7fuXM+ePZUxY8bolV+5cqXSqlUrRa1WK+3atVN+//33Svf83//+p/j5+SnW1tZKYGCgsmbNmmrFlJOTowBKTk7OTT2TKWm1WqXr7GjF57W1yqZjqaYORwgh7mjG5gOj15lu27bNqOTco0ePGqT228OdvM70REou/T7ZjpWFGYdm3I+N2tzUIQkhxB3L2HxgdDdvr169dBNZqsq/KpUKjUZTzVBFbbrWxdvdz1kSqRBC1BGjk2nDhg2xt7dn7NixPP744zg7O9/KuMRN+vNExVaMfdq43qCkEEKI2mL0bN6UlBQ++OADYmJi6NChA+PHj2fXrl04ODjobXggTOdyXglxF7KBislHQggh6obRyVStVjN8+HD++OMP4uPjCQgIYNKkSXh5eTF9+nTKy8tvZZzCCH/Fp6MoENDUEVeH+ruxvxBC1LWbegWbt7c3M2bM4M8//6RVq1bMmTPnrtjP9k7357UlMf7SxSuEEHWp2sm0pKSEZcuWERERQfv27XF2dub333+nUaNGtyI+YaT03GK2nKzY9Ui6eIUQom4ZPQFp7969LFq0iOXLl+Pr68u4ceNYuXKlJNHbxIKtZykt19LJ24l2HnfWch4hhLjTGZ1Mu3Tpgre3N5MnTyY4OBjA4H63AwcOrL3ohFHS84r5fs85AKZEtJLXqQkhRB0zOpkCJCcn884771R5XtaZmsZX285SUq4lyMuJHi1lyZIQQtQ1o5PptdebidtLRn4J3+6+1iptKa1SIYQwgZuazVuVf77vVNSNr7adpbhMS2BTR3q1amLqcIQQol6qlWRaUlLC3LlzadasWW3cThgpM7+EpTHSKhVCCFMzOpmWlJQwbdo0QkJC6Nq1K2vWrAFg0aJFNGvWjHnz5vHiiy/eqjjFv2QVlPLMt7EUlWno4OnIva1lOYwQQpiK0WOmM2bM4MsvvyQiIoJdu3bxyCOPMG7cOHbv3s3HH3/MI488grm5bKxeF5IyChi3eB+JGQXYW1vw9qB20ioVQggTMjqZrlq1iqVLlzJw4ECOHj1KQEAA5eXlxMXFyRd5HYo9l8WTS/ZzpbAMTycbFo8LpaWrvanDEkKIes3oZHrhwgXd+tL27dtjZWXFiy++KIm0DmUVlDLmm33kl5QT0NSRr8eE4GIve/AKIYSpGZ1MNRoNarX67wstLGjQoMEtCUoY9t3uc+SXlOPvZs/yp7tgq67WMmEhhBC3iNHfxoqiMHbsWKysrAAoLi5mwoQJ2NnZ6ZX7+eefazdCAUBxmYYlu5IAeLZXC0mkQghxGzH6G3nMmDF6vz/22GO1Hoyo2s8HLpJZUIqnkw39O7ibOhwhhBD/YHQyXbRo0a2MQ1yHVqvw9fazADzRvRmW5rW614YQQogakm/lO0B0fDpnry6DGR7qZepwhBBC/Isk0zvAwm1nABgV5kMDKxkrFUKI240k09vcgeQr7Eu6gqW5inHdfE0djhBCCAMkmd7mvtpWMVY6KMgTVwdZUyqEELcjSaa3saSMAjYcSwXgqXuamzgaIYQQVZFkehv7avtZFAV6+7vQ2k22DBRCiNuVJNPbVEZ+CatiLwDwTA9plQohxO1MkultaumuJErLtQR6OdG5WSNThyOEEOI6JJnehgpKylly9aXfE3o0l5cJCCHEbe62SKbz58/H19cXa2trwsLC2Lt373XLr1q1Cn9/f6ytrenQoQPr1q3TOz927FhUKpXeT9++fW/lI9SqlfvPk1NUhm9jW+5v52bqcIQQQtyAyZPpihUriIqKYubMmRw4cIDAwEAiIyNJT083WH7Xrl2MHDmS8ePHc/DgQQYPHszgwYM5evSoXrm+ffuSkpKi+/nhhx/q4nFqrFyj5evtiQA81aM55mbSKhVCiNudSlEUxZQBhIWFERoayueffw6AVqvFy8uL559/nqlTp1YqP3z4cAoKCli7dq3uWJcuXQgKCmLBggVARcs0OzubNWvW3FRMubm5ODo6kpOTg4ODw03d42b9cSyVZ76NpbGdmp1Te2NtaV6nny+EEOJvxuYDk7ZMS0tLiY2NJSIiQnfMzMyMiIgIYmJiDF4TExOjVx4gMjKyUvktW7bg4uJC69atefbZZ8nMzKwyjpKSEnJzc/V+TGX53mQAHg5pKolUCCHuECZNphkZGWg0GlxdXfWOu7q6kpqaavCa1NTUG5bv27cvS5cuJTo6mg8++ICtW7fSr18/NBqNwXvOnj0bR0dH3Y+Xl2k2k7+YXcSWk5cBGBHqbZIYhBBCVN9duWv6iBEjdP/doUMHAgICaNGiBVu2bKFPnz6Vyk+bNo2oqCjd77m5uSZJqCv3nUdRILx5Y5o52934AiGEELcFk7ZMnZ2dMTc3Jy0tTe94Wloabm6GZ7G6ublVqzxA8+bNcXZ25vTp0wbPW1lZ4eDgoPdT1zRahZX7zwMworO8Zk0IIe4kJk2marWa4OBgoqOjdce0Wi3R0dGEh4cbvCY8PFyvPMCmTZuqLA9w4cIFMjMzcXd3r53Ab4GtJ9NJySnGydaSSFkOI4QQdxSTL42Jioriq6++YsmSJZw4cYJnn32WgoICxo0bB8Do0aOZNm2arvyUKVPYsGEDc+fOJT4+nlmzZrF//34mTZoEQH5+Pq+88gq7d+8mKSmJ6OhoBg0ahJ+fH5GRkSZ5RmP8sLeiVTq0k0w8EkKIO43Jx0yHDx/O5cuXmTFjBqmpqQQFBbFhwwbdJKPk5GTMzP7O+V27dmXZsmW88cYbvP7667Rs2ZI1a9bQvn17AMzNzTl8+DBLliwhOzsbDw8P7r//ft555x2srKxM8ow3kp5bzOb4inW1I6WLVwgh7jgmX2d6O6rrdabz/zrNR38kEOLTkB+f7XrLP08IIYRx7oh1pgJyCsv4envFC8BHdpblMEIIcSeSZGpi86JPcqWwjFauDRgU5GHqcIQQQtwESaYmdDo9j6VX3w4z44F2WJjLH4cQQtyJ5NvbRBRF4e21J9BoFSLauNK9pbOpQxJCCHGTJJmayF8J6Ww7eRlLcxVvDGhj6nCEEELUgCRTEygt1/LO2hMAPNGtGb6ydaAQQtzRJJmawJqDF0nMKMC5gZpJvf1MHY4QQogakmRaxxRF4ZudFS//fvKe5thbW5o4IiGEEDUlybSO7UnMIj41D2tLM0aEym5HQghxN5BkWscWXW2VDunUFCdbtYmjEUIIURskmdah81mFbDpe8fq4sV19TRuMEEKIWiPJtA4tjUlCq0B3P2daudqbOhwhhBC1RJLpLZKcWcj4xfv4Ne4SRaUaCkrKWb6v4jVr47r5mjY4IYQQtcrkr2C7W/0ad5Ho+HSi49OxVZvT2s2evOJyfBvbcm9rF1OHJ4QQohZJMr1FBgR4UFKuZc2hi5zPKuJgcjYAY7r6YmamMm1wQgghapW8z9SA2nyfqaIoHEjO5tdDFyku0zJrYDts1Oa1FKkQQohbydh8IC3TW0ylUhHs05Bgn4amDkUIIcQtIhOQhBBCiBqSZCqEEELUkCRTIYQQooYkmQohhBA1JBOQDLg2wTk3N9fEkQghhDCla3ngRgtfJJkakJeXB4CXl7zVRQghREVecHR0rPK8rDM1QKvVcunSJezt7VGpbn6DhdzcXLy8vDh//nyN16vebaRuqiZ1UzWpm6pJ3VStJnWjKAp5eXl4eHhgZlb1yKi0TA0wMzOjadOmtXY/BwcH+ctdBambqkndVE3qpmpSN1W72bq5Xov0GpmAJIQQQtSQJFMhhBCihiSZ3kJWVlbMnDkTKysrU4dy25G6qZrUTdWkbqomdVO1uqgbmYAkhBBC1JC0TIUQQogakmQqhBBC1JAkUyGEEKKGJJkKIYQQNSTJ9BaZP38+vr6+WFtbExYWxt69e00dUp2bPXs2oaGh2Nvb4+LiwuDBg0lISNArU1xczMSJE2ncuDENGjRg6NChpKWlmShi05kzZw4qlYoXXnhBd6w+183Fixd57LHHaNy4MTY2NnTo0IH9+/frziuKwowZM3B3d8fGxoaIiAhOnTplwojrhkaj4c0336RZs2bY2NjQokUL3nnnHb19Y+tT3Wzbto0HH3wQDw8PVCoVa9as0TtvTF1kZWUxatQoHBwccHJyYvz48eTn51c/GEXUuuXLlytqtVr55ptvlGPHjilPPfWU4uTkpKSlpZk6tDoVGRmpLFq0SDl69Khy6NAhpX///oq3t7eSn5+vKzNhwgTFy8tLiY6OVvbv36906dJF6dq1qwmjrnt79+5VfH19lYCAAGXKlCm64/W1brKyshQfHx9l7Nixyp49e5SzZ88qf/zxh3L69GldmTlz5iiOjo7KmjVrlLi4OGXgwIFKs2bNlKKiIhNGfuu99957SuPGjZW1a9cqiYmJyqpVq5QGDRoon3zyia5MfaqbdevWKdOnT1d+/vlnBVBWr16td96Yuujbt68SGBio7N69W9m+fbvi5+enjBw5stqxSDK9BTp37qxMnDhR97tGo1E8PDyU2bNnmzAq00tPT1cAZevWrYqiKEp2drZiaWmprFq1SlfmxIkTCqDExMSYKsw6lZeXp7Rs2VLZtGmT0rNnT10yrc9189prryndu3ev8rxWq1Xc3NyUjz76SHcsOztbsbKyUn744Ye6CNFkBgwYoDzxxBN6x4YMGaKMGjVKUZT6XTf/TqbG1MXx48cVQNm3b5+uzPr16xWVSqVcvHixWp8v3by1rLS0lNjYWCIiInTHzMzMiIiIICYmxoSRmV5OTg4AjRo1AiA2NpaysjK9uvL398fb27ve1NXEiRMZMGCAXh1A/a6bX3/9lZCQEB555BFcXFzo2LEjX331le58YmIiqampenXj6OhIWFjYXV83Xbt2JTo6mpMnTwIQFxfHjh076NevH1C/6+bfjKmLmJgYnJycCAkJ0ZWJiIjAzMyMPXv2VOvzZKP7WpaRkYFGo8HV1VXvuKurK/Hx8SaKyvS0Wi0vvPAC3bp1o3379gCkpqaiVqtxcnLSK+vq6kpqaqoJoqxby5cv58CBA+zbt6/SufpcN2fPnuW///0vUVFRvP766+zbt4/JkyejVqsZM2aM7vkN/T92t9fN1KlTyc3Nxd/fH3NzczQaDe+99x6jRo0CqNd182/G1EVqaiouLi565y0sLGjUqFG160uSqagTEydO5OjRo+zYscPUodwWzp8/z5QpU9i0aRPW1tamDue2otVqCQkJ4f333wegY8eOHD16lAULFjBmzBgTR2daK1eu5Pvvv2fZsmW0a9eOQ4cO8cILL+Dh4VHv68bUpJu3ljk7O2Nubl5p1mVaWhpubm4misq0Jk2axNq1a/nrr7/0Xm3n5uZGaWkp2dnZeuXrQ13FxsaSnp5Op06dsLCwwMLCgq1bt/Lpp59iYWGBq6trva0bd3d32rZtq3esTZs2JCcnA+ievz7+P/bKK68wdepURowYQYcOHXj88cd58cUXmT17NlC/6+bfjKkLNzc30tPT9c6Xl5eTlZVV7fqSZFrL1Go1wcHBREdH645ptVqio6MJDw83YWR1T1EUJk2axOrVq9m8eTPNmjXTOx8cHIylpaVeXSUkJJCcnHzX11WfPn04cuQIhw4d0v2EhIQwatQo3X/X17rp1q1bpSVUJ0+exMfHB4BmzZrh5uamVze5ubns2bPnrq+bwsLCSi+oNjc3R6vVAvW7bv7NmLoIDw8nOzub2NhYXZnNmzej1WoJCwur3gfWaPqUMGj58uWKlZWVsnjxYuX48ePK008/rTg5OSmpqammDq1OPfvss4qjo6OyZcsWJSUlRfdTWFioKzNhwgTF29tb2bx5s7J//34lPDxcCQ8PN2HUpvPP2byKUn/rZu/evYqFhYXy3nvvKadOnVK+//57xdbWVvnuu+90ZebMmaM4OTkpv/zyi3L48GFl0KBBd+3yj38aM2aM4unpqVsa8/PPPyvOzs7Kq6++qitTn+omLy9POXjwoHLw4EEFUD7++GPl4MGDyrlz5xRFMa4u+vbtq3Ts2FHZs2ePsmPHDqVly5ayNOZ28tlnnyne3t6KWq1WOnfurOzevdvUIdU5wODPokWLdGWKioqU5557TmnYsKFia2urPPTQQ0pKSorpgjahfyfT+lw3v/32m9K+fXvFyspK8ff3VxYuXKh3XqvVKm+++abi6uqqWFlZKX369FESEhJMFG3dyc3NVaZMmaJ4e3sr1tbWSvPmzZXp06crJSUlujL1qW7++usvg98xY8aMURTFuLrIzMxURo4cqTRo0EBxcHBQxo0bp+Tl5VU7FnkFmxBCCFFDMmYqhBBC1JAkUyGEEKKGJJkKIYQQNSTJVAghhKghSaZCCCFEDUkyFUIIIWpIkqkQQghRQ5JMhbjLbNmyBZVKVWlf39rQq1cvXnjhhVq/b3x8PF26dMHa2pqgoKBav78Qt5okUyHqwKxZs+osSXTt2pWUlBQcHR1v+h63MiEbMnPmTOzs7EhISNDbS/V6Fi9eXOkVdUKYiiRTIe4yarUaNzc3VCqVqUMx2pkzZ+jevTs+Pj40bty4zj+/rKyszj9T3F0kmQphQK9evZg8eTKvvvoqjRo1ws3NjVmzZl33mi1bttC5c2fs7OxwcnKiW7dunDt3jsWLF/PWW28RFxeHSqVCpVKxePFiALKzs3nyySdp0qQJDg4O9O7dm7i4ON09r7Vov/zyS7y8vLC1tWXYsGHk5ORcN45/tiqvteD++OMP2rRpQ4MGDejbty8pKSkGr09KSuLee+8FoGHDhqhUKsaOHas7r9Vqr1svN3qmf1OpVMTGxvL222+jUqmYNWuWwZbxoUOHUKlUJCUlsWXLFsaNG0dOTo6uTq/FoVKpWLNmjd5nODk56eo8KSkJlUrFihUr6NmzJ9bW1nz//fcAfP3117Rp0wZra2v8/f354osvqoxbCD0132pYiLtPz549FQcHB2XWrFnKyZMnlSVLligqlUrZuHGjwfJlZWWKo6Oj8vLLLyunT59Wjh8/rixevFg5d+6cUlhYqLz00ktKu3btKr05JyIiQnnwwQeVffv2KSdPnlReeuklpXHjxkpmZqaiKIoyc+ZMxc7OTundu7dy8OBBZevWrYqfn5/y6KOPVhn7tc2/r1y5oiiKoixatEixtLRUIiIilH379imxsbFKmzZtqrxHeXm58tNPPymAkpCQoKSkpCjZ2dlG18uNnunfUlJSlHbt2ikvvfSSkpKSouTl5VV6BkVRdG8GSUxMVEpKSpR58+YpDg4Oujq9tjk5oKxevVrvMxwdHXUvWEhMTFQAxdfXV/npp5+Us2fPKpcuXVK+++47xd3dXXfsp59+Uho1aqQsXry4yroW4hpJpkIY0LNnT6V79+56x0JDQ5XXXnvNYPnMzEwFULZs2WLw/MyZM5XAwEC9Y9u3b1ccHByU4uJiveMtWrRQvvzyS9115ubmyoULF3Tn169fr5iZmVX5BhlDyRRQTp8+rSszf/58xdXV1eD1hu5xzY3qxZhnMiQwMFCZOXPmdT//n8n02nM5OjpWupexyXTevHmVYly2bJnesXfeeadevPZO1JyFKVrDQtwJAgIC9H53d3cnPT3dYNlGjRoxduxYIiMjue+++4iIiGDYsGG4u7tXef+4uDjy8/MrjREWFRVx5swZ3e/e3t54enrqfg8PD0er1ZKQkICbm5tRz2Jra0uLFi2MepYbuV69GPtMt4OQkBDdfxcUFHDmzBnGjx/PU089pTteXl5eo4lcov6QZCpEFSwtLfV+V6lUaLXaKssvWrSIyZMns2HDBlasWMEbb7zBpk2b6NKli8Hy+fn5uLu7s2XLlkrnanuWqqFnUW7y7YvXq5faeiYzs4rpHP+M0dhJQoaezdC1dnZ2uv/Oz88H4KuvviIsLEyvnLm5uXFBi3pNkqkQtahjx4507NiRadOmER4ezrJly+jSpQtqtRqNRqNXtlOnTqSmpmJhYYGvr2+V90xOTubSpUt4eHgAsHv3bszMzGjduvUtew61Wg1QKeYbMfaZbqRJkyYApKSk0LBhQ6BiAtK/YzQUX5MmTfQmV506dYrCwsLrfp6rqyseHh6cPXuWUaNG3XTcov6S2bxC1ILExESmTZtGTEwM586dY+PGjZw6dYo2bdoA4OvrS2JiIocOHSIjI4OSkhIiIiIIDw9n8ODBbNy4kaSkJHbt2sX06dPZv3+/7t7W1taMGTOGuLg4tm/fzuTJkxk2bJjRXbw3w8fHB5VKxdq1a7l8+bKu5XYjxj7Tjfj5+eHl5cWsWbM4deoUv//+O3PnztUr4+vrS35+PtHR0WRkZOgSZu/evfn88885ePAg+/fvZ8KECZVa04a89dZbzJ49m08//ZSTJ09y5MgRFi1axMcff2x03KL+kmQqRC2wtbUlPj6eoUOH0qpVK55++mkmTpzIM888A8DQoUPp27cv9957L02aNOGHH35ApVKxbt06evTowbhx42jVqhUjRozg3LlzuLq66u7t5+fHkCFD6N+/P/fffz8BAQG3fMmGp6cnb731FlOnTsXV1ZVJkyYZdZ2xz3QjlpaW/PDDD8THxxMQEMAHH3zAu+++q1ema9euTJgwgeHDh9OkSRM+/PBDAObOnYuXlxf33HMPjz76KC+//DK2trY3/Mwnn3ySr7/+mkWLFtGhQwd69uzJ4sWLadasmdFxi/pLpdzswIkQ4pabNWsWa9asqdTFKYS4vUjLVAghhKghSaZCCCFEDUk3rxBCCFFD0jIVQgghakiSqRBCCFFDkkyFEEKIGpJkKoQQQtSQJFMhhBCihiSZCiGEEDUkyVQIIYSoIUmmQgghRA1JMhVCCCFq6P8B+F4fUyNjS6AAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "RMS_time = ((yfuture_pred - yfuture[ids])**2).mean(0)**0.5\n",
    "plt.figure(figsize=(5,2))\n",
    "plt.plot(RMS_time)\n",
    "plt.xlabel('n step in the future')\n",
    "plt.ylabel('RMS error')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a4cb271-1f7a-4beb-8699-476a31765d98",
   "metadata": {},
   "source": [
    "Here you can see that the encoder predicts the initial state quite accurately and the performance degredes when increasing the number of steps that need to be predicted in the future. Generally you can use this graph also to pick $T$ since it should be somewhere where the n-step-error starts to plateau. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "103b40fc-757d-4015-b729-f6792e03a6e7",
   "metadata": {},
   "source": [
    "## 12. Cuda\n",
    "\n",
    "Depending on the model size, structure, and the degree of parallelism it is sometimes faster to run on a GPU instead of a CPU. Below an example is given with a small model where it is not faster to run using cuda."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "dbc421d0-5e3d-48fc-aa24-96234bb7d6c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "device = 'cpu'\n",
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d0f6d1e1c94498295695b9ebd6cdf86",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1001 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.99626!! nan samps/sec\n",
      "it     100 NRMS loss 0.60581 NRMS val 0.27648!! 6492.02 samps/sec\n",
      "it     200 NRMS loss 0.18159 NRMS val 0.15365!! 6403.86 samps/sec\n",
      "it     300 NRMS loss 0.11793 NRMS val 0.12647!! 6325.17 samps/sec\n",
      "it     400 NRMS loss 0.09611 NRMS val 0.09750!! 6392.34 samps/sec\n",
      "it     500 NRMS loss 0.08544 NRMS val 0.09348!! 6376.74 samps/sec\n",
      "it     600 NRMS loss 0.07418 NRMS val 0.09384   6433.99 samps/sec\n",
      "it     700 NRMS loss 0.07000 NRMS val 0.08232!! 6433.79 samps/sec\n",
      "it     800 NRMS loss 0.06342 NRMS val 0.05766!! 6462.30 samps/sec\n",
      "it     900 NRMS loss 0.05231 NRMS val 0.05636!! 6484.89 samps/sec\n",
      "it   1,000 NRMS loss 0.05433 NRMS val 0.05975   6492.68 samps/sec\n",
      "device = 'cuda'\n",
      "Number of samples to train on = 61739\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae6dfc3feb3349d4a32d20d6b9523615",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1001 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "it       0 NRMS loss nan NRMS val 0.99626!! nan samps/sec\n",
      "it     100 NRMS loss 0.60581 NRMS val 0.27648!! 5068.56 samps/sec\n",
      "it     200 NRMS loss 0.18159 NRMS val 0.15365!! 4975.40 samps/sec\n",
      "it     300 NRMS loss 0.11793 NRMS val 0.12647!! 4999.81 samps/sec\n",
      "it     400 NRMS loss 0.09611 NRMS val 0.09750!! 5023.89 samps/sec\n",
      "it     500 NRMS loss 0.08544 NRMS val 0.09348!! 5049.34 samps/sec\n",
      "it     600 NRMS loss 0.07418 NRMS val 0.09384   5041.56 samps/sec\n",
      "it     700 NRMS loss 0.07000 NRMS val 0.08243!! 5041.09 samps/sec\n",
      "it     800 NRMS loss 0.06336 NRMS val 0.05820!! 5021.37 samps/sec\n",
      "it     900 NRMS loss 0.05244 NRMS val 0.05599!! 5015.86 samps/sec\n",
      "it   1,000 NRMS loss 0.05469 NRMS val 0.05394!! 5012.38 samps/sec\n"
     ]
    }
   ],
   "source": [
    "import deepSI as dsi\n",
    "import nonlinear_benchmarks as nlb\n",
    "import torch\n",
    "if torch.cuda.is_available():\n",
    "    train_val, test = nlb.Silverbox()\n",
    "    nu, ny, norm = dsi.get_nu_ny_and_auto_norm(train_val)\n",
    "    train_val, test = norm.transform(train_val), norm.transform(test) #normalize data\n",
    "    s = int(len(train_val)*0.95)\n",
    "    train, val = train_val[:s], train_val[s:]\n",
    "\n",
    "    for device in ['cpu', 'cuda']:\n",
    "        print(f'{device = }')\n",
    "        torch.manual_seed(0)\n",
    "        model = dsi.models.SUBNET(nu, ny, dsi.Norm(0, 1, 0, 1), nx=4)\n",
    "        model.to(device) #you should move it before calling the fit\n",
    "        optimizer = torch.optim.Adamax(model.parameters()) #create the optimizer after the .to call\n",
    "        train_dict = dsi.fit(model, train, val, n_its=1001, val_freq=100, device=device, optimizer=optimizer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02833423-d705-419c-89ee-de7dc3ae8cad",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}