06 - Docker and Singularity.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/agave\n",
      "Requirement already satisfied: setvar in /opt/conda/lib/python3.6/site-packages\n",
      "CMD=jobs-output-get 8979877899256983065-242ac11b-0001-007 fork-command-1.out\n",
      "INPUTS={\"datafile\":\"agave://nectar-storage-stevenrbrandt/funwave-tvd-docker/rundock.tgz\"}\n",
      "JOB_FILE=job-remote-13262.txt\n",
      "JOB_ID=8979877899256983065-242ac11b-0001-007\n",
      "OUTPUT=Successfully submitted job 8979877899256983065-242ac11b-0001-007\n",
      "REMOTE_COMMAND=tar xzf rundock.tgz && bash rundock.sh\n",
      "REQUESTBIN_URL=https://requestbin.agaveapi.co/pkxilxpk\n",
      "STAT=FINISHED\n",
      "VM_IPADDRESS=18.216.53.253\n",
      "\u001b[1;0mToken for agave.prod:stevenrbrandt successfully refreshed and cached for 14400 seconds\n",
      "a283883161e6f7fb10274d6a1921d153\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p ~/agave/funwave-tvd-docker\n",
    "\n",
    "%cd ~/agave\n",
    "\n",
    "!pip3 install setvar\n",
    "\n",
    "import re\n",
    "import os\n",
    "import sys\n",
    "from setvar import *\n",
    "from time import sleep\n",
    "\n",
    "# This cell enables inline plotting in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "loadvar()\n",
    "!auth-tokens-refresh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Creating the Docker Image</h2>\n",
    "To start, we need a Dockerfile, which has a number of simple commands.\n",
    "It starts with \"FROM\" which can specify any docker image available from DockerHub. That not only includes basic operating systems such as \"ubunto\", \"fedora\", \"centos\", etc. but specialized containers made by anyone with a dockerhub account. I've provided \"science-base\" which has OpenMPI 2.1.1 and some standard compilers, i.e. gfortran, gcc, and g++.\n",
    "\n",
    "MAINTAINER is a bit of metadata that (hopefully) will allow you to contact the container's creator, if need be.\n",
    "\n",
    "WORKDIR is the dockerfile equivalent of the \"cd\" command. Note that running \"cd\" will not change your directory.\n",
    "\n",
    "RUN simply runs the command that follows. Because the container is saved after each step, we want to avoid creating files that we don't want to keep (we want containers to be as small as possible).\n",
    "\n",
    "USER specifies the user id for running subsequent RUN commands.\n",
    "\n",
    "COPY can be used to copy files into the container from the build directory.\n",
    "\n",
    "ENTRYPOINT is a script that runs when the container starts up. What our script does is create a new user on the docker image with a user id and name that is convenient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `funwave-tvd-docker/Dockerfile'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"funwave-tvd-docker/Dockerfile\",\"\"\"\n",
    "FROM stevenrbrandt/science-base\n",
    "\n",
    "LABEL baseImage=\"stevenrbrandt/science-base:latest\"\n",
    "LABEL version=\"3\"\n",
    "LABEL software=\"FUNWAVE-TVD\"\n",
    "LABEL softwareVersion=\"v3.2-beta\"\n",
    "LABEL description=\"FUNWAVE–TVD is the TVD version of the fully nonlinear Boussinesq wave model (FUNWAVE) initially developed by Kirby et al. (1998)\"\n",
    "LABEL website=\"https://fengyanshi.github.io/build/html/index.html\"\n",
    "LABEL documentation=\"https://fengyanshi.github.io/build/html/index.html\"\n",
    "LABEL license=\"BSD 2-Clause\"\n",
    "LABEL tags=\"crc,fortran,tvd\"\n",
    "\n",
    "MAINTAINER Steven R. Brandt <sbrandt@cct.lsu.edu>\n",
    "\n",
    "USER root\n",
    "RUN mkdir -p /home/install\n",
    "RUN chown jovyan /home/install\n",
    "USER jovyan\n",
    "\n",
    "RUN cd /home/install && \\\n",
    "    git clone https://github.com/fengyanshi/FUNWAVE-TVD && \\\n",
    "    cd FUNWAVE-TVD/src && \\\n",
    "    perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile && \\\n",
    "    make\n",
    "\n",
    "WORKDIR /home/install/FUNWAVE-TVD/src\n",
    "RUN mkdir -p /home/jovyan/rundir\n",
    "WORKDIR /home/jovyan/rundir\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've create our Dockerfile and entrypoint.sh, bundle them up in a tarball and send them somewhere that agave can access them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;0mSuccessfully created folder funwave-tvd-docker\u001b[0m\n",
      "Uploading dockerjob.tgz...\n",
      "######################################################################## 100.0%\n"
     ]
    }
   ],
   "source": [
    "!tar -czf dockerjob.tgz -C funwave-tvd-docker Dockerfile\n",
    "!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N funwave-tvd-docker\n",
    "!files-upload -F dockerjob.tgz -S ${AGAVE_STORAGE_SYSTEM_ID} funwave-tvd-docker/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'runagavecmd' from '/home/jovyan/agave/runagavecmd.py'>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import runagavecmd as r\n",
    "import imp\n",
    "imp.reload(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the docker build command. We will \"tag\" this build with the name \"funwave-tvd\" when it is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REMOTE_COMMAND=tar xzf dockerjob.tgz && sudo docker build --rm -t funwave-tvd-2 .\n",
      "REQUESTBIN_URL=https://requestbin.agaveapi.co/qyc1k8qy\n",
      "\n",
      " ** QUERY STRING FOR REQUESTBIN **\n",
      "https://requestbin.agaveapi.co/qyc1k8qy?inspect\n",
      "\n",
      "INPUTS={\"datafile\":\"agave://nectar-storage-stevenrbrandt/funwave-tvd-docker/dockerjob.tgz\"}\n",
      "JOB_FILE=job-remote-19597.txt\n",
      "Writing file `job-remote-19597.txt'\n",
      "OUTPUT=Successfully submitted job 4040708458529943065-242ac11b-0001-007\n",
      "JOB_ID=4040708458529943065-242ac11b-0001-007\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PROCESSING_INPUTS\n",
      "STAT=STAGING_INPUTS\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=FINISHED\n",
      "CMD=jobs-output-get 4040708458529943065-242ac11b-0001-007 fork-command-1.out\n",
      "All done! Output follows.\n",
      "Reading file `fork-command-1.out'\n",
      "======================================================================\n",
      "Sending build context to Docker daemon  12.29kB\n",
      "\n",
      "Step 1/10 : FROM stevenrbrandt/science-base\n",
      " ---> e93db47f971c\n",
      "Step 2/10 : USER root\n",
      " ---> Using cache\n",
      " ---> b94fc308f479\n",
      "Step 3/10 : RUN mkdir -p /home/install\n",
      " ---> Using cache\n",
      " ---> beccae742726\n",
      "Step 4/10 : RUN chown jovyan /home/install\n",
      " ---> Using cache\n",
      " ---> 81c2bd496517\n",
      "Step 5/10 : USER jovyan\n",
      " ---> Using cache\n",
      " ---> c16d5c774336\n",
      "Step 6/10 : MAINTAINER Steven R. Brandt <sbrandt@cct.lsu.edu>\n",
      " ---> Using cache\n",
      " ---> f992707e0150\n",
      "Step 7/10 : RUN cd /home/install &&     git clone https://github.com/fengyanshi/FUNWAVE-TVD &&     cd FUNWAVE-TVD/src &&     perl -p -i -e 's/FLAG_8 = -DCOUPLING/#$&/' Makefile &&     make\n",
      " ---> Using cache\n",
      " ---> 04bd5f3529fb\n",
      "Step 8/10 : WORKDIR /home/install/FUNWAVE-TVD/src\n",
      " ---> Using cache\n",
      " ---> 396b8c1fb201\n",
      "Step 9/10 : RUN mkdir -p /home/jovyan/rundir\n",
      " ---> Using cache\n",
      " ---> 3aecce97d0e7\n",
      "Step 10/10 : WORKDIR /home/jovyan/rundir\n",
      " ---> Using cache\n",
      " ---> 22beb9c32c31\n",
      "Successfully built 22beb9c32c31\n",
      "Successfully tagged funwave-tvd-2:latest\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r.runagavecmd(\n",
    "    \"tar xzf dockerjob.tgz && sudo docker build --rm -t funwave-tvd-2 .\",\n",
    "    \"agave://${AGAVE_STORAGE_SYSTEM_ID}/funwave-tvd-docker/dockerjob.tgz\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n"
     ]
    }
   ],
   "source": [
    "!jobs-output-get ${JOB_ID} fork-command-1.err\n",
    "!cat fork-command-1.err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Running the Docker Image</h2>\n",
    "It is possible to run docker interactively, but that isn't convenient inside scripts. So instead, we start it in detached mode, with the -d flag.\n",
    "\n",
    "Because your docker image has its own internal file system, it can't see files on the host machine. You can, however, transfer them using the \"docker cp\" command.\n",
    "\n",
    "Running docker is slightly tricky. When a Docker image starts up, you can execute any command you want--but when you type \"exit\" all the changes you've made to the file system vanish. Therefore it's necessary to copy them out before the docker container stops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing file `rundock.sh'\n"
     ]
    }
   ],
   "source": [
    "writefile(\"rundock.sh\",\"\"\"\n",
    "rm -fr cid.txt out.tgz\n",
    "\n",
    "# Start a docker image running in detached mode, write the container id to cid.txt\n",
    "sudo docker run -d -it --rm --cidfile cid.txt funwave-tvd-2 bash\n",
    "\n",
    "# Store the container id in CID for convenience\n",
    "CID=\\$(cat cid.txt)\n",
    "\n",
    "# Copy the input.txt file into the running image\n",
    "sudo docker cp input.txt \\$CID:/home/jovyan/rundir/\n",
    "\n",
    "# Run funwave on the image\n",
    "sudo docker exec --user jovyan \\$CID mpirun -np 2 /home/install/FUNWAVE-TVD/src/funwave_vessel\n",
    "\n",
    "# Extract the output files from the running image\n",
    "# Having them in a tgz makes it more convenient to fetch them with jobs-output-get\n",
    "sudo docker exec --user jovyan \\$CID tar czf - output > out.tgz\n",
    "\n",
    "# Stop the image\n",
    "sudo docker stop \\$CID\n",
    "\n",
    "# List the output files\n",
    "tar tzf out.tgz\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the input.txt file and the rundock.sh script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading rundock.tgz...\n",
      "######################################################################## 100.0%\n"
     ]
    }
   ],
   "source": [
    "!tar czf rundock.tgz rundock.sh input.txt\n",
    "!files-upload -F rundock.tgz -S ${AGAVE_STORAGE_SYSTEM_ID} funwave-tvd-docker/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the rundock.sh script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REMOTE_COMMAND=tar xzf rundock.tgz && bash rundock.sh\n",
      "REQUESTBIN_URL=https://requestbin.agaveapi.co/1bl4xtf1\n",
      "\n",
      " ** QUERY STRING FOR REQUESTBIN **\n",
      "https://requestbin.agaveapi.co/1bl4xtf1?inspect\n",
      "\n",
      "INPUTS={\"datafile\":\"agave://nectar-storage-stevenrbrandt/funwave-tvd-docker/rundock.tgz\"}\n",
      "JOB_FILE=job-remote-19597.txt\n",
      "Writing file `job-remote-19597.txt'\n",
      "OUTPUT=Successfully submitted job 799296640238743065-242ac11b-0001-007\n",
      "JOB_ID=799296640238743065-242ac11b-0001-007\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=STAGING_INPUTS\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=FINISHED\n",
      "CMD=jobs-output-get 799296640238743065-242ac11b-0001-007 fork-command-1.out\n",
      "All done! Output follows.\n",
      "Reading file `fork-command-1.out'\n",
      "======================================================================\n",
      "10a6e2bb73f5dc8718306f090faaebe814345abee91ff763773ab127f4049798\n",
      " BATHY_CORRECTION DOES NOT EXIST. USE DEFAULT VALUE\n",
      " INI_UVZ DOES NOT EXIST. USE DEFAULT VALUE\n",
      " WaterLevel DOES NOT EXIST. USE DEFAULT VALUE\n",
      " ETA_LIMITER DOES NOT EXIST. USE DEFAULT VALUE\n",
      " EqualEnergy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " PERIODIC DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DIFFUSION_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DIRECT_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " FRICTION_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OBSTACLE_FILE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " BREAKWATER_FILE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DISPERSION DOES NOT EXIST. USE DEFAULT VALUE\n",
      "         DISPERSION Default:  DISPERSION\n",
      " Gamma1 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "        Gamma1 Default:  1.0: DISPERSION\n",
      " Gamma2 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "              Gamma2 Default:  1.0: Full nonlinear\n",
      " Beta_ref DOES NOT EXIST. USE DEFAULT VALUE\n",
      "               Beta_ref Default:  -0.531\n",
      " Gamma3 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                      Gamma3 Default:  1.0: NOT fully linear\n",
      " VISCOSITY_BREAKING DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                   VISCOSITY_BREAKING Default:  SWE Breaking\n",
      " SWE_ETA_DEP DOES NOT EXIST. USE DEFAULT VALUE\n",
      "               SWE_ETA_DEP Default:  0.8\n",
      " Friction_Matrix DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             Friction_Matrix Default:  constant Cd\n",
      " Cd DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                  Cd_fixed Default:  0.0\n",
      " Time_Scheme DOES NOT EXIST. USE DEFAULT VALUE\n",
      "       Time_Scheme Default:  Runge_Kutta\n",
      " CONSTRUCTION DOES NOT EXIST. USE DEFAULT VALUE\n",
      " HIGH_ORDER DOES NOT EXIST. USE DEFAULT VALUE\n",
      "  HIGH_ORDER                     NOT DEFINED, USE FOURTH-ORDER\n",
      " CFL DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                       CFL Default:  0.5\n",
      " DT_fixed DOES NOT EXIST. USE DEFAULT VALUE\n",
      " FroudeCap DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                 FroudeCap Default:  3.0\n",
      " MinDepth DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                MinDepth Default:  0.1 m\n",
      " MinDepthFrc DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             MinDepthFrc Default:  0.1 m\n",
      " SHOW_BREAKING DOES NOT EXIST. USE DEFAULT VALUE\n",
      "            SHOW_BREAKING Default:  TRUE\n",
      " Cbrk1 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    Cbrk1 Default:  0.65\n",
      " Cbrk2 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    Cbrk2 Default:  0.35\n",
      " WAVEMAKER_Cbrk DOES NOT EXIST. USE DEFAULT VALUE\n",
      "            WAVEMAKER_Cbrk Default:  1.0\n",
      " WAVEMAKER_VIS DOES NOT EXIST. USE DEFAULT VALUE\n",
      "           WAVEMAKER_VIS Default:  FALSE\n",
      " T_INTV_mean DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             T_INTV_mean Default:  LARGE\n",
      " STEADY_TIME DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             STEADY_TIME Default:  LARGE\n",
      " C_smg DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                     C_smg Default:  0.0\n",
      " nu_bkg DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    nu_bkg Default:  0.0\n",
      " FIELD_IO_TYPE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " NumberStations DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OUTPUT_RES DOES NOT EXIST. USE DEFAULT VALUE\n",
      "         OUTPUT_RES NOT FOUND, OUTPUT_RES=1: full resolution\n",
      " DEPTH_OUT DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Hmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Hmin DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Umax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MFmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " VORmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MASK DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MASK9 DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Umean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Vmean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " ETAmean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " WaveHeight DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SXL DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SXR DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SYL DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SYR DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SourceX DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SourceY DOES NOT EXIST. USE DEFAULT VALUE\n",
      " P DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Q DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Fx DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Fy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Gx DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Gy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " AGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OUT_NU DOES NOT EXIST. USE DEFAULT VALUE\n",
      " TMP DOES NOT EXIST. USE DEFAULT VALUE\n",
      "      EtaBlowVal Default:  100xmax_depth\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.4428E-01  0.4428E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1937E+04  0.1247E+08  0.2996E+01 -0.4596E-06  0.4050E-01  0.4050E-01\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.4050E-01  0.1066E+02  0.3800E-02\n",
      "   PRINTING FILE NO.     1  TIME/TOTAL:        0.044 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.1017E+01  0.4414E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1927E+04  0.1246E+08  0.1900E+01 -0.1727E-05  0.7329E+00  0.7325E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.7338E+00  0.1050E+02  0.6987E-01\n",
      "   PRINTING FILE NO.     2  TIME/TOTAL:        1.017 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.2037E+01  0.4473E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1906E+04  0.1246E+08  0.7393E+00 -0.2109E-05  0.9186E+00  0.9148E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.9215E+00  0.1024E+02  0.8998E-01\n",
      "   PRINTING FILE NO.     3  TIME/TOTAL:        2.037 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.3032E+01  0.4566E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1890E+04  0.1246E+08  0.7188E+00 -0.9413E+00  0.7352E+00  0.6932E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.7378E+00  0.1019E+02  0.7239E-01\n",
      "   PRINTING FILE NO.     4  TIME/TOTAL:        3.032 /       3.000\n",
      " Simulation takes   17.199041255997145      seconds\n",
      " Normal Termination!\n",
      "10a6e2bb73f5dc8718306f090faaebe814345abee91ff763773ab127f4049798\n",
      "output/\n",
      "output/eta_00001\n",
      "output/v_00003\n",
      "output/u_00001\n",
      "output/eta_00003\n",
      "output/v_00001\n",
      "output/u_00002\n",
      "output/eta_00004\n",
      "output/eta_00002\n",
      "output/u_00003\n",
      "output/v_00004\n",
      "output/v_00002\n",
      "output/u_00004\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r.runagavecmd(\n",
    "    \"tar xzf rundock.tgz && bash rundock.sh\",\n",
    "    \"agave://${AGAVE_STORAGE_SYSTEM_ID}/funwave-tvd-docker/rundock.tgz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the output of the job back to our local machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;0m.agave.archive\n",
      ".agave.log\n",
      "cid.txt\n",
      "fork-command-1.err\n",
      "fork-command-1.ipcexe\n",
      "fork-command-1.out\n",
      "fork-command-1.pid\n",
      "fork-test.txt\n",
      "fork-wrapper.txt\n",
      "input.txt\n",
      "out.tgz\n",
      "rundock.sh\n",
      "rundock.tgz\u001b[0m\n",
      "######################################################################## 100.0%                       66.2%\n"
     ]
    }
   ],
   "source": [
    "!jobs-output-list ${JOB_ID}\n",
    "!jobs-output-get ${JOB_ID} out.tgz\n",
    "!tar xzf out.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "head: cannot open 'output/eta_00010' for reading: No such file or directory\r\n"
     ]
    }
   ],
   "source": [
    "!head output/eta_00010"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Running with Singularity</h2>\n",
    "If we have a public docker image, we can run it directly with Singularity. Singularity is desiged to be more HPC friendly than Docker. First, because it doesn't all the running user to access any user id but their own inside the container, and second, because singularity images can be run through MPI, making it easier to scale up to a distributed cluser.\n",
    "\n",
    "In this first step, we build the singularity installation. Because the result of this job is intended to be an installation for subsequent jobs, we install it to a hard-coded directory rather than using the normal Agave job directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;0mSuccessfully created folder sing\u001b[0m\n",
      "Uploading input.txt...\n",
      "######################################################################## 100.0%\n",
      "REMOTE_COMMAND=mkdir -p ~/singu && cd ~/singu && rm -f funwave-tvd.img && singularity create funwave-tvd.img --size 2000 && singularity import funwave-tvd.img docker://stevenrbrandt/funwave-tvd-2:latest\n",
      "REQUESTBIN_URL=https://requestbin.agaveapi.co/yj6r3eyj\n",
      "\n",
      " ** QUERY STRING FOR REQUESTBIN **\n",
      "https://requestbin.agaveapi.co/yj6r3eyj?inspect\n",
      "\n",
      "INPUTS={}\n",
      "JOB_FILE=job-remote-19597.txt\n",
      "Writing file `job-remote-19597.txt'\n",
      "OUTPUT=Successfully submitted job 4296939754503082471-242ac11b-0001-007\n",
      "JOB_ID=4296939754503082471-242ac11b-0001-007\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "CMD=jobs-output-get 4296939754503082471-242ac11b-0001-007 fork-command-1.out\n",
      "All done! Output follows.\n",
      "Reading file `fork-command-1.out'\n",
      "======================================================================\n",
      "Initializing Singularity image subsystem\n",
      "\u001b[0mOpening image file: funwave-tvd.img\n",
      "\u001b[0mCreating 768MiB image\n",
      "\u001b[0mBinding image to loop\n",
      "\u001b[0mCreating file system within image\n",
      "\u001b[0mImage is done: funwave-tvd.img\n",
      "\u001b[0mDocker image path: index.docker.io/stevenrbrandt/funwave-tvd-2:latest\n",
      "Cache folder set to /home/jovyan/.singularity/docker\n",
      "\n",
      "[1/13] ||----------------------------------|   0.0% \n",
      "[1/13] |==|--------------------------------|   7.7% \n",
      "[2/13] |=====|-----------------------------|  15.4% \n",
      "[3/13] |========|--------------------------|  23.1% \n",
      "[4/13] |==========|------------------------|  30.8% \n",
      "[5/13] |=============|---------------------|  38.5% \n",
      "[6/13] |================|------------------|  46.2% \n",
      "[7/13] |==================|----------------|  53.8% \n",
      "[8/13] |=====================|-------------|  61.5% \n",
      "[9/13] |========================|----------|  69.2% \n",
      "[10/13] |==========================|--------|  76.9% \n",
      "[11/13] |=============================|-----|  84.6% \n",
      "[12/13] |================================|--|  92.3% \n",
      "[13/13] |===================================| 100.0% \n",
      "Importing: base Singularity environment\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:be96bf654103f9670434de9ccc428ee55fb1f14531619dd6cb8fbd85b5765393.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:3f38f85a45654716a81c8ad5c89a0cfc820932422078150624f1f83661b6fec7.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:15bd671928b4d58adb14870cf74c013f9781d4979c944d851827afde27d5bdb8.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:203b799f83cfd0d284472fdd78efc556302c353e270b9974d03cec71a2faf14b.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:c20f403f88792630033b7760769f60fe2da3271d0fdecce4eeeed17c5adfaf21.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:c912233ab5d5939d997ab7a9110bda1f64e2d3cdf3808a21d95736b1da58766f.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:022aa0a604e76c9b9223aa277639f27104b61b6e1c1428ed4174d4c133cd685c.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:630ceed02486a128742ac2b5c3105c88ad28f410a0b2e08ff4ddae2878483924.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:25bb6f291ceb2dcdd3e24aa3a00ef625e6e1a022779283fc2c1b2ccecbc0d162.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:a7cad26d035791e5bf04d6479b03dee59d40a504cfd0b54840966f98b39abfc4.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:f7d512d8250289b154a8e982ff1272ed1376e407bf21f314cd55627c22e5598e.tar.gz\n",
      "Importing: /home/jovyan/.singularity/docker/sha256:60730f9603637b2b4d71f56eca6b7cf26f3c3a59478c45f1e53733901ea89b02.tar.gz\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!files-mkdir -S ${AGAVE_STORAGE_SYSTEM_ID} -N sing\n",
    "!files-upload -F input.txt -S ${AGAVE_STORAGE_SYSTEM_ID} sing/\n",
    "r.runagavecmd(\n",
    "            \"mkdir -p ~/singu && \"+\n",
    "            \"cd ~/singu && \"+\n",
    "            \"rm -f funwave-tvd.img && \"+\n",
    "            \"singularity create funwave-tvd.img --size 2000 && \"+\n",
    "            \"singularity import funwave-tvd.img docker://stevenrbrandt/funwave-tvd-2:latest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the Singularity image is built, we can run it with mpi. Notice that mpi executes the singularity command. The tricky part here is to make sure you've got the same version of mpi running inside and outside the container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading input.txt...\n",
      "######################################################################## 100.0%\n",
      "REMOTE_COMMAND=export LD_LIBRARY_PATH=/usr/local/lib && mpirun -np 2 singularity exec ~/singu/funwave-tvd.img /home/install/FUNWAVE-TVD/src/funwave_vessel && tar cvzf singout.tgz output\n",
      "REQUESTBIN_URL=https://requestbin.agaveapi.co/17u7mgg1\n",
      "\n",
      " ** QUERY STRING FOR REQUESTBIN **\n",
      "https://requestbin.agaveapi.co/17u7mgg1?inspect\n",
      "\n",
      "INPUTS={\"datafile\":\"agave://nectar-storage-stevenrbrandt/input.txt\"}\n",
      "JOB_FILE=job-remote-19597.txt\n",
      "Writing file `job-remote-19597.txt'\n",
      "OUTPUT=Successfully submitted job 5360771817640956391-242ac11b-0001-007\n",
      "JOB_ID=5360771817640956391-242ac11b-0001-007\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PROCESSING_INPUTS\n",
      "STAT=STAGING_INPUTS\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=CLEANING_UP\n",
      "STAT=FINISHED\n",
      "CMD=jobs-output-get 5360771817640956391-242ac11b-0001-007 fork-command-1.out\n",
      "All done! Output follows.\n",
      "Reading file `fork-command-1.out'\n",
      "======================================================================\n",
      " BATHY_CORRECTION DOES NOT EXIST. USE DEFAULT VALUE\n",
      " INI_UVZ DOES NOT EXIST. USE DEFAULT VALUE\n",
      " WaterLevel DOES NOT EXIST. USE DEFAULT VALUE\n",
      " ETA_LIMITER DOES NOT EXIST. USE DEFAULT VALUE\n",
      " EqualEnergy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " PERIODIC DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DIFFUSION_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DIRECT_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " FRICTION_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OBSTACLE_FILE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " BREAKWATER_FILE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DISPERSION DOES NOT EXIST. USE DEFAULT VALUE\n",
      "         DISPERSION Default:  DISPERSION\n",
      " Gamma1 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "        Gamma1 Default:  1.0: DISPERSION\n",
      " Gamma2 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "              Gamma2 Default:  1.0: Full nonlinear\n",
      " Beta_ref DOES NOT EXIST. USE DEFAULT VALUE\n",
      "               Beta_ref Default:  -0.531\n",
      " Gamma3 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                      Gamma3 Default:  1.0: NOT fully linear\n",
      " VISCOSITY_BREAKING DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                   VISCOSITY_BREAKING Default:  SWE Breaking\n",
      " SWE_ETA_DEP DOES NOT EXIST. USE DEFAULT VALUE\n",
      "               SWE_ETA_DEP Default:  0.8\n",
      " Friction_Matrix DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             Friction_Matrix Default:  constant Cd\n",
      " Cd DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                  Cd_fixed Default:  0.0\n",
      " Time_Scheme DOES NOT EXIST. USE DEFAULT VALUE\n",
      "       Time_Scheme Default:  Runge_Kutta\n",
      " CONSTRUCTION DOES NOT EXIST. USE DEFAULT VALUE\n",
      " HIGH_ORDER DOES NOT EXIST. USE DEFAULT VALUE\n",
      "  HIGH_ORDER                     NOT DEFINED, USE FOURTH-ORDER\n",
      " CFL DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                       CFL Default:  0.5\n",
      " DT_fixed DOES NOT EXIST. USE DEFAULT VALUE\n",
      " FroudeCap DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                 FroudeCap Default:  3.0\n",
      " MinDepth DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                MinDepth Default:  0.1 m\n",
      " MinDepthFrc DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             MinDepthFrc Default:  0.1 m\n",
      " SHOW_BREAKING DOES NOT EXIST. USE DEFAULT VALUE\n",
      "            SHOW_BREAKING Default:  TRUE\n",
      " Cbrk1 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    Cbrk1 Default:  0.65\n",
      " Cbrk2 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    Cbrk2 Default:  0.35\n",
      " WAVEMAKER_Cbrk DOES NOT EXIST. USE DEFAULT VALUE\n",
      "            WAVEMAKER_Cbrk Default:  1.0\n",
      " WAVEMAKER_VIS DOES NOT EXIST. USE DEFAULT VALUE\n",
      "           WAVEMAKER_VIS Default:  FALSE\n",
      " T_INTV_mean DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             T_INTV_mean Default:  LARGE\n",
      " STEADY_TIME DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             STEADY_TIME Default:  LARGE\n",
      " C_smg DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                     C_smg Default:  0.0\n",
      " nu_bkg DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    nu_bkg Default:  0.0\n",
      " FIELD_IO_TYPE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " NumberStations DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OUTPUT_RES DOES NOT EXIST. USE DEFAULT VALUE\n",
      "         OUTPUT_RES NOT FOUND, OUTPUT_RES=1: full resolution\n",
      " DEPTH_OUT DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Hmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Hmin DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Umax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MFmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " VORmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MASK DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MASK9 DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Umean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Vmean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " ETAmean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " WaveHeight DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SXL DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SXR DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SYL DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SYR DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SourceX DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SourceY DOES NOT EXIST. USE DEFAULT VALUE\n",
      " P DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Q DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Fx DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Fy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Gx DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Gy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " AGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OUT_NU DOES NOT EXIST. USE DEFAULT VALUE\n",
      " TMP DOES NOT EXIST. USE DEFAULT VALUE\n",
      "      EtaBlowVal Default:  100xmax_depth\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.4428E-01  0.4428E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1937E+04  0.1247E+08  0.2996E+01 -0.4596E-06  0.4050E-01  0.4050E-01\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.4050E-01  0.1066E+02  0.3800E-02\n",
      "   PRINTING FILE NO.     1  TIME/TOTAL:        0.044 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.1017E+01  0.4414E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1927E+04  0.1246E+08  0.1900E+01 -0.1727E-05  0.7329E+00  0.7325E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.7338E+00  0.1050E+02  0.6987E-01\n",
      "   PRINTING FILE NO.     2  TIME/TOTAL:        1.017 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.2037E+01  0.4473E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1906E+04  0.1246E+08  0.7393E+00 -0.2109E-05  0.9186E+00  0.9148E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.9215E+00  0.1024E+02  0.8998E-01\n",
      "   PRINTING FILE NO.     3  TIME/TOTAL:        2.037 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.3032E+01  0.4566E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1890E+04  0.1246E+08  0.7188E+00 -0.9413E+00  0.7352E+00  0.6932E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.7378E+00  0.1019E+02  0.7239E-01\n",
      "   PRINTING FILE NO.     4  TIME/TOTAL:        3.032 /       3.000\n",
      " Simulation takes   15.861142730995198      seconds\n",
      " Normal Termination!\n",
      "output/\n",
      "output/eta_00001\n",
      "output/v_00003\n",
      "output/u_00001\n",
      "output/eta_00003\n",
      "output/v_00001\n",
      "output/u_00002\n",
      "output/eta_00004\n",
      "output/eta_00002\n",
      "output/u_00003\n",
      "output/v_00004\n",
      "output/v_00002\n",
      "output/u_00004\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!files-upload -F input.txt -S ${AGAVE_STORAGE_SYSTEM_ID} ./\n",
    "r.runagavecmd(\n",
    "    \"export LD_LIBRARY_PATH=/usr/local/lib && \"+\n",
    "    \"mpirun -np 2 singularity exec ~/singu/funwave-tvd.img /home/install/FUNWAVE-TVD/src/funwave_vessel && \"+\n",
    "    \"tar cvzf singout.tgz output\",\n",
    "    \"agave://${AGAVE_STORAGE_SYSTEM_ID}/input.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "######################################################################## 100.0%\n"
     ]
    }
   ],
   "source": [
    "!jobs-output-get ${JOB_ID} singout.tgz\n",
    "!rm -fr output\n",
    "!tar xzf singout.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   -0.370381E-07   -0.264216E-07   -0.453174E-07   -0.615901E-07   -0.867047E-07   -0.916501E-07   -0.873622E-07   -0.295986E-07    0.365446E-08   -0.118685E-07   -0.359654E-07   -0.657907E-07   -0.491518E-07   -0.167981E-07   -0.261493E-07   -0.331495E-07   -0.531462E-07   -0.380234E-07   -0.715021E-08   -0.531698E-07   -0.288188E-07   -0.224122E-07   -0.117357E-07   -0.334322E-07   -0.607799E-07   -0.793287E-07   -0.801748E-07   -0.278102E-07    0.820207E-08   -0.176744E-07   -0.376689E-07   -0.505262E-07   -0.474473E-07   -0.120814E-07    0.521564E-08   -0.202734E-07   -0.502107E-07   -0.495059E-07   -0.170843E-07   -0.794791E-09    0.287557E-07    0.286107E-08   -0.407979E-07   -0.571593E-07   -0.166976E-07   -0.917486E-08   -0.159806E-08   -0.293239E-07   -0.487221E-07   -0.363046E-07   -0.312466E-07   -0.454842E-07   -0.737199E-07   -0.372320E-07   -0.263822E-07   -0.606216E-07   -0.521675E-07   -0.414295E-08   -0.285707E-08   -0.387530E-07   -0.759180E-07   -0.594871E-07   -0.381240E-07   -0.243736E-07   -0.351791E-07   -0.454820E-07   -0.330221E-07   -0.431807E-07   -0.492954E-07   -0.323381E-07   -0.366857E-07   -0.224607E-07    0.536307E-08   -0.184430E-07   -0.379402E-07   -0.367824E-07   -0.425240E-07   -0.580965E-07   -0.256130E-07   -0.361566E-07    0.143026E-07   -0.978389E-08   -0.586753E-07   -0.855737E-07   -0.700330E-07   -0.243344E-07   -0.316075E-07   -0.630901E-08    0.129229E-07   -0.638592E-08   -0.243215E-07   -0.272797E-07   -0.252633E-07   -0.508527E-08   -0.886411E-09   -0.405799E-07   -0.709902E-07   -0.646340E-07   -0.539962E-07   -0.289094E-07   -0.267936E-07   -0.608653E-07   -0.683233E-07   -0.270277E-07   -0.373515E-07   -0.293033E-07   -0.103415E-07   -0.379871E-07   -0.164734E-07   -0.107011E-07   -0.271246E-07   -0.197447E-07   -0.627131E-07   -0.101714E-06   -0.107446E-06   -0.865976E-07   -0.311182E-07   -0.163863E-07   -0.350941E-07   -0.514499E-07   -0.847307E-07   -0.991687E-07   -0.107440E-06   -0.122014E-06   -0.114511E-06   -0.866306E-07   -0.825142E-07   -0.107459E-06   -0.113634E-06   -0.118026E-06   -0.150423E-06   -0.185461E-06   -0.204589E-06   -0.216434E-06   -0.267158E-06   -0.300989E-06   -0.346729E-06   -0.403227E-06   -0.468551E-06   -0.555254E-06   -0.634795E-06   -0.711520E-06   -0.806948E-06   -0.944412E-06   -0.107287E-05   -0.122608E-05   -0.140268E-05   -0.162333E-05   -0.185759E-05   -0.212830E-05   -0.244532E-05   -0.280490E-05   -0.321622E-05   -0.368944E-05   -0.421665E-05   -0.482467E-05   -0.552103E-05   -0.631197E-05   -0.721552E-05   -0.824129E-05   -0.940552E-05   -0.107353E-04   -0.122488E-04   -0.139686E-04   -0.159261E-04   -0.181496E-04   -0.206773E-04   -0.235424E-04   -0.267869E-04   -0.304680E-04   -0.346422E-04   -0.393683E-04   -0.447096E-04   -0.507457E-04   -0.575648E-04   -0.652570E-04   -0.739240E-04   -0.836955E-04   -0.946992E-04   -0.107069E-03   -0.120968E-03   -0.136570E-03   -0.154071E-03   -0.173679E-03   -0.195627E-03   -0.220168E-03   -0.247578E-03   -0.278161E-03   -0.312244E-03   -0.350183E-03   -0.392363E-03   -0.439197E-03   -0.491132E-03   -0.548648E-03   -0.612254E-03   -0.682492E-03   -0.759938E-03   -0.845200E-03   -0.938916E-03   -0.104175E-02   -0.115441E-02   -0.127761E-02   -0.141208E-02   -0.155862E-02   -0.171796E-02   -0.189089E-02   -0.207820E-02   -0.228065E-02   -0.249899E-02   -0.273395E-02   -0.298621E-02   -0.325641E-02   -0.354511E-02   -0.385282E-02   -0.417993E-02   -0.452676E-02   -0.489348E-02   -0.528014E-02   -0.568666E-02   -0.611276E-02   -0.655801E-02   -0.702178E-02   -0.750324E-02   -0.800136E-02   -0.851490E-02   -0.904234E-02   -0.958194E-02   -0.101322E-01   -0.106911E-01   -0.112536E-01   -0.118194E-01   -0.123917E-01   -0.129585E-01   -0.135200E-01   -0.140730E-01   -0.146135E-01   -0.151396E-01   -0.156474E-01   -0.161338E-01   -0.165953E-01   -0.170292E-01   -0.174322E-01   -0.178011E-01   -0.181325E-01   -0.184244E-01   -0.186761E-01   -0.188885E-01   -0.190593E-01   -0.191750E-01   -0.192263E-01   -0.192388E-01   -0.192263E-01   -0.191750E-01   -0.190593E-01   -0.188885E-01   -0.186761E-01   -0.184244E-01   -0.181325E-01   -0.178011E-01   -0.174322E-01   -0.170292E-01   -0.165953E-01   -0.161338E-01   -0.156474E-01   -0.151396E-01   -0.146135E-01   -0.140730E-01   -0.135200E-01   -0.129585E-01   -0.123917E-01   -0.118194E-01   -0.112536E-01   -0.106911E-01   -0.101322E-01   -0.958194E-02   -0.904234E-02   -0.851490E-02   -0.800136E-02   -0.750324E-02   -0.702178E-02   -0.655801E-02   -0.611276E-02   -0.568666E-02   -0.528014E-02   -0.489348E-02   -0.452676E-02   -0.417993E-02   -0.385282E-02   -0.354511E-02   -0.325641E-02   -0.298621E-02   -0.273395E-02   -0.249899E-02   -0.228065E-02   -0.207820E-02   -0.189089E-02   -0.171796E-02   -0.155862E-02   -0.141208E-02   -0.127761E-02   -0.115441E-02   -0.104175E-02   -0.938916E-03   -0.845200E-03   -0.759938E-03   -0.682492E-03   -0.612254E-03   -0.548648E-03   -0.491132E-03   -0.439197E-03   -0.392364E-03   -0.350183E-03   -0.312244E-03   -0.278161E-03   -0.247578E-03   -0.220168E-03   -0.195627E-03   -0.173679E-03   -0.154072E-03   -0.136571E-03   -0.120968E-03   -0.107069E-03   -0.946992E-04   -0.836955E-04   -0.739240E-04   -0.652569E-04   -0.575642E-04   -0.507449E-04   -0.447087E-04   -0.393671E-04   -0.346414E-04   -0.304668E-04   -0.267858E-04   -0.235420E-04   -0.206784E-04   -0.181505E-04   -0.159257E-04   -0.139692E-04   -0.122502E-04   -0.107376E-04   -0.940862E-05   -0.824220E-05   -0.721090E-05   -0.630980E-05   -0.552089E-05   -0.482670E-05   -0.421979E-05   -0.368889E-05   -0.322201E-05   -0.281092E-05   -0.245560E-05   -0.213557E-05   -0.186686E-05   -0.162406E-05   -0.140119E-05   -0.123302E-05   -0.108787E-05   -0.949514E-06   -0.812611E-06   -0.699016E-06   -0.633429E-06   -0.567527E-06   -0.484464E-06   -0.407605E-06   -0.339964E-06   -0.312465E-06   -0.281380E-06   -0.220302E-06   -0.190919E-06   -0.157776E-06   -0.143017E-06   -0.158178E-06   -0.164528E-06   -0.131794E-06   -0.101306E-06   -0.867126E-07   -0.995368E-07   -0.114116E-06   -0.118863E-06   -0.141764E-06   -0.130892E-06   -0.800921E-07   -0.537856E-07   -0.361008E-07   -0.806649E-08   -0.429416E-07   -0.782987E-07   -0.731513E-07   -0.457386E-07   -0.272762E-07   -0.199942E-07   -0.289818E-07   -0.153540E-07   -0.192661E-07   -0.755925E-07   -0.521052E-07   -0.275858E-07   -0.355991E-07   -0.519737E-07   -0.440696E-07   -0.208959E-07   -0.441005E-07   -0.183000E-07   -0.556731E-07   -0.713152E-07   -0.383023E-07   -0.265801E-07   -0.324680E-07   -0.457190E-07   -0.307102E-07   -0.169014E-07   -0.129009E-07    0.532344E-08   -0.195307E-07   -0.499705E-07   -0.528522E-07   -0.725266E-07   -0.541334E-07   -0.368727E-07   -0.431557E-07   -0.259984E-07   -0.129517E-07   -0.429034E-07   -0.691278E-07   -0.736828E-07   -0.566911E-07   -0.394391E-07   -0.537709E-07   -0.330791E-07    0.300045E-09    0.661530E-11   -0.478729E-07   -0.783663E-07   -0.821362E-07   -0.828671E-07   -0.951237E-07   -0.401482E-07    0.151596E-07   -0.149809E-07   -0.524474E-07   -0.928579E-07   -0.732621E-07   -0.287197E-07   -0.144439E-08    0.240076E-07   -0.431736E-07   -0.695510E-07   -0.485310E-07   -0.489752E-07   -0.505464E-07   -0.599041E-08   -0.403792E-08   -0.293470E-07   -0.905036E-07   -0.942252E-07   -0.490246E-07   -0.202922E-07   -0.180157E-07   -0.241625E-07   -0.694954E-07   -0.895467E-07   -0.622533E-07   -0.368261E-07   -0.128154E-07   -0.295913E-07   -0.635859E-07   -0.990896E-07   -0.742199E-07   -0.408184E-07   -0.283015E-08    0.945373E-08    0.125971E-07   -0.301275E-07   -0.569471E-07   -0.781747E-07   -0.748916E-07   -0.440774E-07   -0.337795E-07   -0.964221E-08   -0.464799E-07   -0.831190E-07   -0.917154E-07   -0.713160E-07   -0.537729E-07   -0.575811E-07   -0.426448E-07   -0.537945E-07   -0.683450E-07   -0.672369E-07   -0.460913E-07   -0.270848E-07   -0.275067E-07   -0.250059E-07   -0.372548E-07   -0.338110E-07   -0.294900E-07   -0.580005E-07   -0.932209E-07   -0.792152E-07   -0.339107E-07\r\n",
      "   -0.774590E-09    0.244346E-07    0.114468E-07   -0.174623E-07   -0.716177E-07   -0.103320E-06   -0.100045E-06    0.118483E-07    0.757278E-07    0.425873E-07   -0.781542E-08   -0.393849E-07   -0.609445E-08    0.301945E-07    0.110521E-07    0.400187E-08   -0.927601E-08    0.108788E-07    0.334078E-07   -0.778556E-08   -0.404745E-08    0.188577E-07    0.467406E-07    0.273384E-07   -0.699353E-08   -0.550968E-07   -0.572451E-07    0.310849E-07    0.841461E-07    0.414149E-07   -0.126095E-08   -0.598285E-08    0.234102E-07    0.776079E-07    0.846676E-07    0.211216E-07   -0.663077E-07   -0.595288E-07    0.154215E-07    0.748034E-07    0.124162E-06    0.687626E-07   -0.711947E-08   -0.326631E-07    0.455169E-07    0.743092E-07    0.592146E-07   -0.555122E-08   -0.299976E-07   -0.326588E-08    0.472432E-07    0.489664E-08   -0.448447E-07   -0.182709E-07    0.160077E-07   -0.630359E-08    0.889716E-08    0.786829E-07    0.845471E-07   -0.776813E-08   -0.723394E-07   -0.667815E-07    0.370151E-08    0.310380E-07    0.163808E-07    0.241722E-07    0.500987E-07    0.100893E-07   -0.224043E-07   -0.388007E-08   -0.977577E-09    0.440141E-07    0.110755E-06    0.753374E-07    0.754767E-08   -0.302793E-07   -0.504430E-07   -0.548098E-07    0.806263E-08    0.469743E-07    0.115147E-06    0.788433E-07   -0.124046E-07   -0.848779E-07   -0.593120E-07    0.168095E-07    0.323456E-07    0.601356E-07    0.803928E-07    0.354741E-07    0.219140E-09    0.202971E-07    0.384538E-07    0.834231E-07    0.757007E-07    0.149302E-07   -0.608033E-07   -0.867807E-07   -0.519805E-07    0.109245E-07    0.537161E-07    0.969121E-08   -0.846035E-08    0.117486E-07   -0.390474E-08    0.899846E-08    0.362640E-07    0.297944E-07    0.496054E-07    0.421846E-07    0.214698E-07    0.156166E-07   -0.617144E-07   -0.119376E-06   -0.114822E-06   -0.541323E-07    0.224453E-07    0.217207E-07   -0.219806E-07   -0.636484E-07   -0.120255E-06   -0.131688E-06   -0.136277E-06   -0.145316E-06   -0.132071E-06   -0.848409E-07   -0.109528E-06   -0.155503E-06   -0.178931E-06   -0.204031E-06   -0.277090E-06   -0.350872E-06   -0.404389E-06   -0.457131E-06   -0.545135E-06   -0.627485E-06   -0.737475E-06   -0.861620E-06   -0.999610E-06   -0.118020E-05   -0.136497E-05   -0.154509E-05   -0.177336E-05   -0.205848E-05   -0.236173E-05   -0.270255E-05   -0.311796E-05   -0.359776E-05   -0.413599E-05   -0.473572E-05   -0.543174E-05   -0.623969E-05   -0.717534E-05   -0.822930E-05   -0.941968E-05   -0.107839E-04   -0.123461E-04   -0.141432E-04   -0.161837E-04   -0.184939E-04   -0.211233E-04   -0.241254E-04   -0.275404E-04   -0.314283E-04   -0.358546E-04   -0.408900E-04   -0.466055E-04   -0.530904E-04   -0.604504E-04   -0.688026E-04   -0.782732E-04   -0.889991E-04   -0.101140E-03   -0.114872E-03   -0.130384E-03   -0.147898E-03   -0.167652E-03   -0.189937E-03   -0.215043E-03   -0.243292E-03   -0.275057E-03   -0.310743E-03   -0.350800E-03   -0.395720E-03   -0.446040E-03   -0.502348E-03   -0.565292E-03   -0.635581E-03   -0.713984E-03   -0.801332E-03   -0.898526E-03   -0.100654E-02   -0.112643E-02   -0.125932E-02   -0.140642E-02   -0.156901E-02   -0.174846E-02   -0.194620E-02   -0.216376E-02   -0.240273E-02   -0.266478E-02   -0.295163E-02   -0.326505E-02   -0.360691E-02   -0.397903E-02   -0.438330E-02   -0.482162E-02   -0.529584E-02   -0.580780E-02   -0.635927E-02   -0.695195E-02   -0.758739E-02   -0.826703E-02   -0.899212E-02   -0.976369E-02   -0.105825E-01   -0.114492E-01   -0.123638E-01   -0.133262E-01   -0.143359E-01   -0.153919E-01   -0.164928E-01   -0.176366E-01   -0.188210E-01   -0.200431E-01   -0.212991E-01   -0.225851E-01   -0.238973E-01   -0.252311E-01   -0.265754E-01   -0.279270E-01   -0.292946E-01   -0.306514E-01   -0.319950E-01   -0.333195E-01   -0.346149E-01   -0.358759E-01   -0.370939E-01   -0.382607E-01   -0.393686E-01   -0.404102E-01   -0.413781E-01   -0.422642E-01   -0.430609E-01   -0.437627E-01   -0.443682E-01   -0.448782E-01   -0.452871E-01   -0.455650E-01   -0.456935E-01   -0.457252E-01   -0.456935E-01   -0.455650E-01   -0.452871E-01   -0.448782E-01   -0.443682E-01   -0.437627E-01   -0.430609E-01   -0.422642E-01   -0.413781E-01   -0.404102E-01   -0.393686E-01   -0.382607E-01   -0.370939E-01   -0.358759E-01   -0.346149E-01   -0.333195E-01   -0.319950E-01   -0.306514E-01   -0.292946E-01   -0.279270E-01   -0.265754E-01   -0.252311E-01   -0.238973E-01   -0.225851E-01   -0.212991E-01   -0.200431E-01   -0.188210E-01   -0.176366E-01   -0.164928E-01   -0.153919E-01   -0.143359E-01   -0.133262E-01   -0.123638E-01   -0.114492E-01   -0.105825E-01   -0.976369E-02   -0.899212E-02   -0.826704E-02   -0.758739E-02   -0.695195E-02   -0.635927E-02   -0.580780E-02   -0.529584E-02   -0.482162E-02   -0.438330E-02   -0.397903E-02   -0.360691E-02   -0.326505E-02   -0.295163E-02   -0.266478E-02   -0.240273E-02   -0.216376E-02   -0.194620E-02   -0.174846E-02   -0.156901E-02   -0.140642E-02   -0.125932E-02   -0.112643E-02   -0.100654E-02   -0.898526E-03   -0.801333E-03   -0.713985E-03   -0.635582E-03   -0.565292E-03   -0.502349E-03   -0.446041E-03   -0.395721E-03   -0.350801E-03   -0.310743E-03   -0.275057E-03   -0.243292E-03   -0.215043E-03   -0.189937E-03   -0.167652E-03   -0.147897E-03   -0.130383E-03   -0.114870E-03   -0.101139E-03   -0.889968E-04   -0.782709E-04   -0.688010E-04   -0.604482E-04   -0.530896E-04   -0.466076E-04   -0.408921E-04   -0.358537E-04   -0.314279E-04   -0.275443E-04   -0.241320E-04   -0.211298E-04   -0.184969E-04   -0.161763E-04   -0.141412E-04   -0.123599E-04   -0.107908E-04   -0.943098E-05   -0.823482E-05   -0.717534E-05   -0.625193E-05   -0.544503E-05   -0.473794E-05   -0.412172E-05   -0.356740E-05   -0.308589E-05   -0.271524E-05   -0.238797E-05   -0.207924E-05   -0.177447E-05   -0.153500E-05   -0.137127E-05   -0.121061E-05   -0.102729E-05   -0.863602E-06   -0.729414E-06   -0.636982E-06   -0.574655E-06   -0.467208E-06   -0.373953E-06   -0.310523E-06   -0.265863E-06   -0.283279E-06   -0.289816E-06   -0.235032E-06   -0.146098E-06   -0.818967E-07   -0.910248E-07   -0.127687E-06   -0.143742E-06   -0.181208E-06   -0.193866E-06   -0.126029E-06   -0.607151E-07   -0.676629E-08    0.459368E-07    0.330189E-08   -0.585058E-07   -0.568347E-07   -0.547540E-08    0.394257E-07    0.187043E-07    0.530845E-08    0.335159E-07    0.210619E-07   -0.592217E-07   -0.303054E-07    0.249156E-07    0.345130E-07    0.133214E-08   -0.800218E-08    0.197049E-07   -0.381084E-08    0.220936E-07   -0.406661E-07   -0.502859E-07    0.138074E-07    0.305977E-07    0.133714E-07    0.162652E-07    0.406131E-07    0.497889E-07    0.544327E-07    0.602489E-07    0.421851E-07   -0.655496E-08   -0.666280E-07   -0.685340E-07   -0.289730E-07   -0.209081E-07    0.125114E-07    0.684504E-07    0.972422E-07    0.329112E-07   -0.503201E-07   -0.914355E-07   -0.688374E-07   -0.187943E-07    0.608812E-08    0.384358E-07    0.793996E-07    0.831636E-07    0.951214E-08   -0.419420E-07   -0.880056E-07   -0.117966E-06   -0.887392E-07    0.608430E-08    0.865081E-07    0.627000E-07   -0.313353E-07   -0.965983E-07   -0.511543E-07    0.145939E-07    0.835815E-07    0.104196E-06    0.117584E-07   -0.363720E-07   -0.500153E-07   -0.469978E-07   -0.887187E-08    0.740349E-07    0.707971E-07   -0.159814E-07   -0.105235E-06   -0.825761E-07    0.385412E-08    0.604969E-07    0.559995E-07    0.136028E-07   -0.611353E-07   -0.107055E-06   -0.485498E-07   -0.870359E-09    0.594435E-07    0.201136E-07   -0.519615E-07   -0.929957E-07   -0.623953E-07   -0.290340E-08    0.572289E-07    0.837256E-07    0.848998E-07    0.256161E-07   -0.334347E-07   -0.743709E-07   -0.518353E-07   -0.213192E-07    0.577678E-08    0.628525E-07    0.161502E-07   -0.699367E-07   -0.927898E-07   -0.289846E-07   -0.122917E-08   -0.164549E-07   -0.138486E-07   -0.338664E-07   -0.459918E-07   -0.372331E-07    0.649753E-08    0.499435E-07    0.306088E-07   -0.158438E-08   -0.866671E-08    0.268501E-07    0.298320E-07   -0.282160E-07   -0.811647E-07   -0.722003E-07   -0.207232E-07\r\n",
      "   -0.339909E-07    0.358749E-07    0.587911E-07    0.212184E-07   -0.859902E-07   -0.165358E-06   -0.141265E-06   -0.267459E-08    0.894392E-07    0.368367E-07   -0.422009E-07   -0.595034E-07    0.379870E-08    0.349192E-07    0.109388E-07   -0.190885E-07   -0.283507E-07   -0.149404E-07   -0.318034E-08   -0.248512E-07   -0.625180E-07   -0.336676E-07    0.458928E-07    0.718956E-07    0.423826E-07   -0.486977E-07   -0.706578E-07   -0.107132E-07    0.385368E-07    0.213267E-07   -0.216384E-07    0.152317E-07    0.883231E-07    0.131455E-06    0.816120E-07   -0.409054E-07   -0.189887E-06   -0.162097E-06   -0.168788E-07    0.118880E-06    0.161387E-06    0.578567E-07   -0.432996E-07   -0.547122E-07    0.316060E-07    0.833476E-07    0.300079E-07   -0.630857E-07   -0.587934E-07    0.301428E-07    0.979274E-07    0.248290E-07   -0.440631E-07   -0.419158E-07   -0.113431E-08    0.257176E-07    0.509196E-07    0.109239E-06    0.947763E-07   -0.360380E-07   -0.115876E-06   -0.106591E-06   -0.217004E-07    0.299943E-07    0.225230E-07    0.611427E-07    0.989252E-07    0.247859E-07   -0.596327E-07   -0.659551E-07   -0.163846E-07    0.847739E-07    0.171026E-06    0.122569E-06   -0.347017E-08   -0.114792E-06   -0.161132E-06   -0.128151E-06   -0.101605E-07    0.112389E-06    0.175895E-06    0.125477E-06   -0.713570E-08   -0.117323E-06   -0.103018E-06   -0.329518E-07    0.252204E-07    0.831128E-07    0.869814E-07    0.520853E-08   -0.319358E-07    0.634073E-08    0.510285E-07    0.118482E-06    0.119191E-06    0.281424E-07   -0.993467E-07   -0.186576E-06   -0.128964E-06    0.283498E-08    0.848808E-07    0.677615E-07    0.104231E-08   -0.476158E-07   -0.699391E-07   -0.503002E-07    0.168812E-07    0.656364E-07    0.673110E-07    0.474118E-07    0.194882E-07   -0.245830E-07   -0.112027E-06   -0.164049E-06   -0.121564E-06   -0.267442E-07    0.293148E-07    0.161310E-07   -0.582301E-07   -0.150092E-06   -0.226424E-06   -0.230406E-06   -0.218817E-06   -0.171373E-06   -0.125105E-06   -0.107288E-06   -0.180792E-06   -0.253464E-06   -0.309390E-06   -0.338909E-06   -0.446816E-06   -0.556542E-06   -0.648153E-06   -0.764717E-06   -0.896806E-06   -0.998506E-06   -0.113856E-05   -0.131985E-05   -0.154288E-05   -0.180540E-05   -0.210512E-05   -0.238561E-05   -0.271844E-05   -0.313153E-05   -0.360466E-05   -0.412967E-05   -0.475771E-05   -0.547623E-05   -0.631056E-05   -0.723876E-05   -0.830820E-05   -0.954187E-05   -0.109713E-04   -0.125911E-04   -0.144388E-04   -0.165477E-04   -0.189463E-04   -0.217018E-04   -0.248501E-04   -0.284179E-04   -0.324849E-04   -0.371420E-04   -0.424580E-04   -0.484982E-04   -0.553707E-04   -0.632054E-04   -0.721259E-04   -0.822479E-04   -0.937606E-04   -0.106854E-03   -0.121709E-03   -0.138562E-03   -0.157669E-03   -0.179310E-03   -0.203780E-03   -0.231440E-03   -0.262710E-03   -0.298032E-03   -0.337878E-03   -0.382787E-03   -0.433359E-03   -0.490260E-03   -0.554227E-03   -0.626077E-03   -0.706690E-03   -0.797040E-03   -0.898204E-03   -0.101136E-02   -0.113778E-02   -0.127886E-02   -0.143610E-02   -0.161114E-02   -0.180575E-02   -0.202184E-02   -0.226143E-02   -0.252670E-02   -0.281997E-02   -0.314370E-02   -0.350048E-02   -0.389305E-02   -0.432426E-02   -0.479708E-02   -0.531459E-02   -0.587998E-02   -0.649646E-02   -0.716731E-02   -0.789585E-02   -0.868535E-02   -0.953904E-02   -0.104601E-01   -0.114514E-01   -0.125159E-01   -0.136561E-01   -0.148743E-01   -0.161724E-01   -0.175519E-01   -0.190138E-01   -0.205586E-01   -0.221861E-01   -0.238955E-01   -0.256853E-01   -0.275531E-01   -0.294957E-01   -0.315091E-01   -0.335882E-01   -0.357270E-01   -0.379186E-01   -0.401562E-01   -0.424320E-01   -0.447273E-01   -0.470351E-01   -0.493730E-01   -0.516947E-01   -0.539929E-01   -0.562598E-01   -0.584777E-01   -0.606371E-01   -0.627233E-01   -0.647222E-01   -0.666207E-01   -0.684060E-01   -0.700650E-01   -0.715840E-01   -0.729497E-01   -0.741533E-01   -0.751932E-01   -0.760698E-01   -0.767708E-01   -0.772425E-01   -0.774601E-01   -0.775125E-01   -0.774601E-01   -0.772425E-01   -0.767708E-01   -0.760698E-01   -0.751932E-01   -0.741533E-01   -0.729497E-01   -0.715840E-01   -0.700650E-01   -0.684060E-01   -0.666207E-01   -0.647222E-01   -0.627233E-01   -0.606371E-01   -0.584777E-01   -0.562598E-01   -0.539930E-01   -0.516947E-01   -0.493730E-01   -0.470351E-01   -0.447273E-01   -0.424320E-01   -0.401562E-01   -0.379186E-01   -0.357270E-01   -0.335882E-01   -0.315091E-01   -0.294957E-01   -0.275531E-01   -0.256853E-01   -0.238955E-01   -0.221861E-01   -0.205586E-01   -0.190138E-01   -0.175519E-01   -0.161724E-01   -0.148743E-01   -0.136561E-01   -0.125159E-01   -0.114514E-01   -0.104601E-01   -0.953904E-02   -0.868535E-02   -0.789585E-02   -0.716731E-02   -0.649646E-02   -0.587998E-02   -0.531459E-02   -0.479708E-02   -0.432426E-02   -0.389305E-02   -0.350048E-02   -0.314370E-02   -0.281997E-02   -0.252670E-02   -0.226142E-02   -0.202184E-02   -0.180575E-02   -0.161114E-02   -0.143610E-02   -0.127886E-02   -0.113778E-02   -0.101136E-02   -0.898204E-03   -0.797040E-03   -0.706691E-03   -0.626077E-03   -0.554228E-03   -0.490260E-03   -0.433360E-03   -0.382787E-03   -0.337878E-03   -0.298032E-03   -0.262711E-03   -0.231441E-03   -0.203779E-03   -0.179308E-03   -0.157667E-03   -0.138559E-03   -0.121705E-03   -0.106850E-03   -0.937585E-04   -0.822451E-04   -0.721279E-04   -0.632123E-04   -0.553710E-04   -0.484950E-04   -0.424619E-04   -0.371536E-04   -0.324981E-04   -0.284239E-04   -0.248466E-04   -0.216966E-04   -0.189426E-04   -0.165427E-04   -0.144586E-04   -0.126136E-04   -0.109936E-04   -0.956846E-05   -0.831831E-05   -0.723069E-05   -0.628219E-05   -0.542438E-05   -0.472178E-05   -0.413040E-05   -0.362219E-05   -0.316737E-05   -0.272307E-05   -0.237596E-05   -0.211413E-05   -0.185553E-05   -0.157254E-05   -0.131835E-05   -0.112896E-05   -0.100605E-05   -0.907646E-06   -0.760371E-06   -0.609427E-06   -0.494414E-06   -0.425570E-06   -0.446915E-06   -0.459260E-06   -0.378341E-06   -0.235731E-06   -0.115410E-06   -0.752937E-07   -0.133587E-06   -0.192550E-06   -0.253800E-06   -0.310099E-06   -0.236302E-06   -0.135601E-06   -0.328200E-07    0.477023E-07   -0.107290E-09   -0.839724E-07   -0.839256E-07   -0.119941E-07    0.502888E-07    0.491705E-08   -0.280186E-07    0.569559E-09    0.254859E-08   -0.781070E-07   -0.794271E-07    0.192614E-07    0.673388E-07    0.126927E-07   -0.352115E-07   -0.341979E-08    0.671950E-08   -0.466496E-08   -0.556639E-07   -0.640718E-07   -0.143710E-07    0.213572E-07    0.237582E-07    0.545037E-07    0.840586E-07    0.769608E-07    0.736305E-07    0.376354E-07    0.402320E-07   -0.208712E-07   -0.100545E-06   -0.668832E-07   -0.280675E-07   -0.559087E-07   -0.161168E-07    0.858118E-07    0.133745E-06    0.505692E-07   -0.709063E-07   -0.175418E-06   -0.148704E-06   -0.559908E-07    0.264537E-07    0.741986E-07    0.114073E-06    0.124657E-06    0.588912E-07   -0.356444E-07   -0.137266E-06   -0.191595E-06   -0.119701E-06    0.199071E-07    0.111062E-06    0.738313E-07   -0.687209E-07   -0.163613E-06   -0.108647E-06    0.156674E-07    0.132949E-06    0.121772E-06    0.227118E-07   -0.661391E-07   -0.116140E-06   -0.101392E-06    0.174942E-07    0.118605E-06    0.109350E-06   -0.531355E-07   -0.152886E-06   -0.102738E-06    0.384355E-08    0.911110E-07    0.100468E-06    0.233121E-07   -0.100398E-06   -0.163877E-06   -0.940302E-07   -0.329399E-08    0.679556E-07    0.174672E-07   -0.864709E-07   -0.144815E-06   -0.959227E-07   -0.559383E-08    0.699319E-07    0.107582E-06    0.741349E-07    0.382070E-08   -0.795627E-07   -0.127771E-06   -0.770264E-07   -0.388753E-07    0.253358E-07    0.607022E-07    0.165984E-07   -0.825140E-07   -0.850162E-07   -0.120665E-08    0.302934E-07    0.932569E-08   -0.299949E-07   -0.760222E-07   -0.748601E-07   -0.477093E-07    0.341479E-07    0.850819E-07    0.188453E-07   -0.480806E-07   -0.132449E-07    0.664012E-07    0.735072E-07   -0.812449E-09   -0.705410E-07   -0.720208E-07   -0.560352E-07\r\n",
      "   -0.448524E-07    0.892067E-07    0.132991E-06    0.846673E-07   -0.634472E-07   -0.185576E-06   -0.157938E-06   -0.319074E-07    0.668351E-07    0.415890E-07   -0.283464E-07   -0.587575E-08    0.610539E-07    0.740529E-07    0.390847E-07   -0.370164E-07   -0.562303E-07   -0.255232E-07    0.521875E-08   -0.438531E-07   -0.119632E-06   -0.690526E-07    0.649681E-07    0.152648E-06    0.131940E-06    0.112380E-07   -0.701276E-07   -0.624942E-07   -0.179961E-07   -0.258699E-07   -0.214468E-07    0.877910E-07    0.181932E-06    0.157989E-06    0.362510E-07   -0.123586E-06   -0.259260E-06   -0.191785E-06   -0.914927E-08    0.164781E-06    0.199999E-06    0.960668E-07   -0.306587E-07   -0.533508E-07    0.181194E-08    0.596020E-07   -0.239699E-08   -0.824471E-07   -0.157651E-07    0.121641E-06    0.168822E-06    0.759848E-07   -0.305833E-07   -0.364200E-07    0.153828E-07    0.621399E-07    0.809430E-07    0.117786E-06    0.842096E-07   -0.195553E-07   -0.845825E-07   -0.630396E-07   -0.212168E-07    0.124424E-07    0.233432E-07    0.102554E-06    0.161126E-06    0.811570E-07   -0.451108E-07   -0.832647E-07    0.149198E-07    0.149469E-06    0.193858E-06    0.108256E-06   -0.476180E-07   -0.186058E-06   -0.243128E-06   -0.152616E-06    0.459277E-07    0.202025E-06    0.233449E-06    0.147658E-06    0.718734E-08   -0.914952E-07   -0.877784E-07   -0.373965E-07    0.286929E-07    0.106381E-06    0.113435E-06    0.633922E-08   -0.520151E-07   -0.620190E-08    0.739782E-07    0.152289E-06    0.187620E-06    0.568101E-07   -0.117117E-06   -0.209477E-06   -0.147327E-06    0.157017E-07    0.116225E-06    0.116988E-06   -0.981042E-08   -0.126105E-06   -0.167834E-06   -0.117330E-06    0.211315E-07    0.127391E-06    0.137844E-06    0.106745E-06    0.466600E-07   -0.423064E-07   -0.116392E-06   -0.149625E-06   -0.942654E-07    0.141292E-07    0.580393E-07    0.397921E-07   -0.582074E-07   -0.205545E-06   -0.302641E-06   -0.311847E-06   -0.251643E-06   -0.138777E-06   -0.627926E-07   -0.954162E-07   -0.232070E-06   -0.339842E-06   -0.409905E-06   -0.459961E-06   -0.559579E-06   -0.694393E-06   -0.833186E-06   -0.100549E-05   -0.114943E-05   -0.127269E-05   -0.143345E-05   -0.164778E-05   -0.194135E-05   -0.228613E-05   -0.265151E-05   -0.301305E-05   -0.343180E-05   -0.395952E-05   -0.456607E-05   -0.525953E-05   -0.606213E-05   -0.699119E-05   -0.805794E-05   -0.924624E-05   -0.106275E-04   -0.122175E-04   -0.140533E-04   -0.161328E-04   -0.185331E-04   -0.212732E-04   -0.243850E-04   -0.279355E-04   -0.320108E-04   -0.366531E-04   -0.419517E-04   -0.480241E-04   -0.549640E-04   -0.628617E-04   -0.718458E-04   -0.820865E-04   -0.937567E-04   -0.107041E-03   -0.122163E-03   -0.139365E-03   -0.158904E-03   -0.181096E-03   -0.206284E-03   -0.234841E-03   -0.267173E-03   -0.303773E-03   -0.345204E-03   -0.392050E-03   -0.444962E-03   -0.504680E-03   -0.572010E-03   -0.647865E-03   -0.733253E-03   -0.829284E-03   -0.937173E-03   -0.105826E-02   -0.119402E-02   -0.134607E-02   -0.151618E-02   -0.170627E-02   -0.191844E-02   -0.215495E-02   -0.241826E-02   -0.271102E-02   -0.303608E-02   -0.339649E-02   -0.379550E-02   -0.423657E-02   -0.472336E-02   -0.525974E-02   -0.584975E-02   -0.649760E-02   -0.720769E-02   -0.798456E-02   -0.883281E-02   -0.975716E-02   -0.107624E-01   -0.118532E-01   -0.130342E-01   -0.143101E-01   -0.156851E-01   -0.171635E-01   -0.187490E-01   -0.204449E-01   -0.222542E-01   -0.241791E-01   -0.262213E-01   -0.283815E-01   -0.306597E-01   -0.330549E-01   -0.355650E-01   -0.381869E-01   -0.409161E-01   -0.437470E-01   -0.466726E-01   -0.496842E-01   -0.527724E-01   -0.559275E-01   -0.591378E-01   -0.623782E-01   -0.656374E-01   -0.689390E-01   -0.722223E-01   -0.754728E-01   -0.786802E-01   -0.818195E-01   -0.848761E-01   -0.878302E-01   -0.906612E-01   -0.933507E-01   -0.958803E-01   -0.982314E-01   -0.100384E+00   -0.102319E+00   -0.104026E+00   -0.105502E+00   -0.106747E+00   -0.107741E+00   -0.108404E+00   -0.108707E+00   -0.108778E+00   -0.108707E+00   -0.108404E+00   -0.107741E+00   -0.106747E+00   -0.105502E+00   -0.104026E+00   -0.102319E+00   -0.100384E+00   -0.982314E-01   -0.958803E-01   -0.933507E-01   -0.906612E-01   -0.878302E-01   -0.848761E-01   -0.818195E-01   -0.786802E-01   -0.754728E-01   -0.722223E-01   -0.689390E-01   -0.656374E-01   -0.623782E-01   -0.591378E-01   -0.559275E-01   -0.527724E-01   -0.496842E-01   -0.466726E-01   -0.437470E-01   -0.409161E-01   -0.381869E-01   -0.355650E-01   -0.330549E-01   -0.306597E-01   -0.283815E-01   -0.262213E-01   -0.241791E-01   -0.222542E-01   -0.204449E-01   -0.187490E-01   -0.171635E-01   -0.156851E-01   -0.143101E-01   -0.130342E-01   -0.118532E-01   -0.107624E-01   -0.975716E-02   -0.883281E-02   -0.798456E-02   -0.720769E-02   -0.649760E-02   -0.584975E-02   -0.525974E-02   -0.472336E-02   -0.423657E-02   -0.379550E-02   -0.339649E-02   -0.303608E-02   -0.271102E-02   -0.241825E-02   -0.215495E-02   -0.191844E-02   -0.170627E-02   -0.151618E-02   -0.134607E-02   -0.119402E-02   -0.105826E-02   -0.937173E-03   -0.829284E-03   -0.733253E-03   -0.647865E-03   -0.572010E-03   -0.504679E-03   -0.444962E-03   -0.392049E-03   -0.345205E-03   -0.303774E-03   -0.267172E-03   -0.234839E-03   -0.206282E-03   -0.181093E-03   -0.158902E-03   -0.139362E-03   -0.122160E-03   -0.107036E-03   -0.937553E-04   -0.820916E-04   -0.718468E-04   -0.628555E-04   -0.549674E-04   -0.480391E-04   -0.419677E-04   -0.366574E-04   -0.320109E-04   -0.279302E-04   -0.243606E-04   -0.212674E-04   -0.185616E-04   -0.161624E-04   -0.140797E-04   -0.122391E-04   -0.106349E-04   -0.924426E-05   -0.800621E-05   -0.692890E-05   -0.604758E-05   -0.525962E-05   -0.457708E-05   -0.398620E-05   -0.344440E-05   -0.300123E-05   -0.264952E-05   -0.232124E-05   -0.197889E-05   -0.165489E-05   -0.141218E-05   -0.126904E-05   -0.111871E-05   -0.958384E-06   -0.789570E-06   -0.612422E-06   -0.528033E-06   -0.534442E-06   -0.550771E-06   -0.464594E-06   -0.296793E-06   -0.141673E-06   -0.320343E-07   -0.855714E-07   -0.196869E-06   -0.287127E-06   -0.354247E-06   -0.287766E-06   -0.150092E-06   -0.242170E-07    0.592212E-07    0.130928E-07   -0.960467E-07   -0.110003E-06   -0.173484E-07    0.736489E-07    0.472810E-07    0.561912E-08    0.892464E-08    0.135360E-07   -0.678017E-07   -0.917859E-07    0.114424E-07    0.754305E-07    0.247134E-07   -0.454143E-07   -0.170291E-07    0.353826E-07    0.991317E-09   -0.766540E-08   -0.219464E-07   -0.359049E-07    0.457476E-08    0.590405E-07    0.106188E-06    0.134065E-06    0.914526E-07    0.677357E-07    0.400182E-07    0.612814E-07    0.608855E-08   -0.452417E-07    0.619793E-08    0.415655E-07   -0.149663E-07   -0.357967E-07    0.535808E-07    0.107446E-06    0.342734E-07   -0.869922E-07   -0.199519E-06   -0.154717E-06   -0.598978E-07    0.244414E-07    0.861956E-07    0.154800E-06    0.194990E-06    0.140868E-06    0.786732E-08   -0.116607E-06   -0.173612E-06   -0.944944E-07    0.597120E-07    0.131339E-06    0.696556E-07   -0.751755E-07   -0.162498E-06   -0.123350E-06    0.363154E-07    0.164201E-06    0.151886E-06    0.554309E-07   -0.652668E-07   -0.134828E-06   -0.858884E-07    0.794793E-07    0.180089E-06    0.132666E-06   -0.459076E-07   -0.152330E-06   -0.128557E-06   -0.205075E-07    0.111254E-06    0.166164E-06    0.987244E-07   -0.656203E-07   -0.162170E-06   -0.103157E-06    0.287618E-07    0.984976E-07    0.449217E-07   -0.750700E-07   -0.169929E-06   -0.100951E-06    0.184474E-07    0.111389E-06    0.142371E-06    0.713708E-07   -0.452088E-07   -0.136365E-06   -0.141081E-06   -0.504402E-07    0.297134E-07    0.955887E-07    0.622507E-07   -0.174395E-07   -0.699898E-07   -0.363931E-07    0.406289E-07    0.885139E-07    0.732306E-07   -0.699724E-08   -0.984331E-07   -0.119803E-06   -0.451660E-07    0.791174E-07    0.113327E-06    0.142335E-07   -0.578263E-07    0.148465E-07    0.132511E-06    0.149574E-06    0.678050E-07   -0.283161E-07   -0.399297E-07   -0.414983E-07\r\n",
      "   -0.774479E-07    0.113111E-06    0.175444E-06    0.107871E-06   -0.490468E-07   -0.200723E-06   -0.222428E-06   -0.128809E-06   -0.148345E-07    0.226646E-07    0.544636E-08    0.557210E-07    0.106283E-06    0.854488E-07    0.475201E-08   -0.115704E-06   -0.133553E-06   -0.568701E-07   -0.899981E-08   -0.773569E-07   -0.156570E-06   -0.751329E-07    0.809259E-07    0.190219E-06    0.169389E-06    0.217632E-07   -0.114851E-06   -0.126668E-06   -0.710731E-07   -0.823219E-07   -0.443161E-07    0.116754E-06    0.195784E-06    0.117139E-06   -0.553367E-07   -0.218403E-06   -0.267760E-06   -0.162036E-06    0.131093E-07    0.177928E-06    0.222124E-06    0.145258E-06   -0.117558E-07   -0.910127E-07   -0.725313E-07   -0.208554E-07   -0.507046E-07   -0.793227E-07    0.306858E-07    0.168898E-06    0.195151E-06    0.741228E-07   -0.757504E-07   -0.603416E-07    0.959597E-08    0.363317E-07    0.353560E-07    0.645534E-07    0.527286E-07    0.460161E-08   -0.298347E-07   -0.117402E-07   -0.214180E-07   -0.454981E-07   -0.379559E-07    0.690499E-07    0.163995E-06    0.131915E-06    0.303093E-08   -0.595442E-07    0.451341E-07    0.155181E-06    0.128488E-06    0.570221E-08   -0.154201E-06   -0.271803E-06   -0.290704E-06   -0.135572E-06    0.901929E-07    0.249908E-06    0.215227E-06    0.893406E-07   -0.317490E-07   -0.793487E-07   -0.582443E-07   -0.142325E-07    0.353866E-07    0.942777E-07    0.112553E-06   -0.280882E-08   -0.105622E-06   -0.458664E-07    0.832573E-07    0.166024E-06    0.188246E-06    0.354387E-07   -0.140401E-06   -0.197368E-06   -0.133794E-06    0.403889E-07    0.120381E-06    0.900944E-07   -0.814288E-07   -0.231967E-06   -0.268660E-06   -0.181060E-06    0.107278E-08    0.157679E-06    0.214491E-06    0.164860E-06    0.418761E-07   -0.899337E-07   -0.166830E-06   -0.193328E-06   -0.119998E-06    0.178037E-07    0.793454E-07    0.502795E-07   -0.827628E-07   -0.278163E-06   -0.398673E-06   -0.392903E-06   -0.273277E-06   -0.137785E-06   -0.782586E-07   -0.145282E-06   -0.322071E-06   -0.458517E-06   -0.523871E-06   -0.607679E-06   -0.707259E-06   -0.869262E-06   -0.105182E-05   -0.125177E-05   -0.139415E-05   -0.153554E-05   -0.175337E-05   -0.201375E-05   -0.234988E-05   -0.274307E-05   -0.314817E-05   -0.359565E-05   -0.412374E-05   -0.475212E-05   -0.548919E-05   -0.634091E-05   -0.732785E-05   -0.846063E-05   -0.975212E-05   -0.111898E-04   -0.128513E-04   -0.147791E-04   -0.169976E-04   -0.195337E-04   -0.224477E-04   -0.257641E-04   -0.295646E-04   -0.339053E-04   -0.388720E-04   -0.445463E-04   -0.510351E-04   -0.584739E-04   -0.669809E-04   -0.766837E-04   -0.877194E-04   -0.100295E-03   -0.114644E-03   -0.131009E-03   -0.149650E-03   -0.170854E-03   -0.194957E-03   -0.222343E-03   -0.253453E-03   -0.288752E-03   -0.328779E-03   -0.374146E-03   -0.425533E-03   -0.483682E-03   -0.549431E-03   -0.623708E-03   -0.707546E-03   -0.802100E-03   -0.908648E-03   -0.102860E-02   -0.116351E-02   -0.131508E-02   -0.148521E-02   -0.167596E-02   -0.188960E-02   -0.212860E-02   -0.239564E-02   -0.269365E-02   -0.302582E-02   -0.339555E-02   -0.380655E-02   -0.426276E-02   -0.476843E-02   -0.532805E-02   -0.594641E-02   -0.662857E-02   -0.737983E-02   -0.820575E-02   -0.911209E-02   -0.101049E-01   -0.111902E-01   -0.123743E-01   -0.136634E-01   -0.150641E-01   -0.165824E-01   -0.182245E-01   -0.199963E-01   -0.219034E-01   -0.239510E-01   -0.261436E-01   -0.284853E-01   -0.309793E-01   -0.336280E-01   -0.364325E-01   -0.393932E-01   -0.425089E-01   -0.457770E-01   -0.491935E-01   -0.527530E-01   -0.564481E-01   -0.602696E-01   -0.642065E-01   -0.682464E-01   -0.723766E-01   -0.765812E-01   -0.808290E-01   -0.851043E-01   -0.894326E-01   -0.937441E-01   -0.980137E-01   -0.102229E+00   -0.106356E+00   -0.110376E+00   -0.114262E+00   -0.117988E+00   -0.121528E+00   -0.124859E+00   -0.127956E+00   -0.130791E+00   -0.133341E+00   -0.135590E+00   -0.137536E+00   -0.139179E+00   -0.140488E+00   -0.141358E+00   -0.141755E+00   -0.141847E+00   -0.141755E+00   -0.141358E+00   -0.140488E+00   -0.139179E+00   -0.137536E+00   -0.135590E+00   -0.133341E+00   -0.130791E+00   -0.127956E+00   -0.124859E+00   -0.121528E+00   -0.117988E+00   -0.114262E+00   -0.110376E+00   -0.106356E+00   -0.102229E+00   -0.980137E-01   -0.937441E-01   -0.894326E-01   -0.851043E-01   -0.808290E-01   -0.765812E-01   -0.723766E-01   -0.682464E-01   -0.642065E-01   -0.602696E-01   -0.564481E-01   -0.527530E-01   -0.491935E-01   -0.457770E-01   -0.425089E-01   -0.393932E-01   -0.364325E-01   -0.336280E-01   -0.309793E-01   -0.284853E-01   -0.261436E-01   -0.239510E-01   -0.219034E-01   -0.199963E-01   -0.182245E-01   -0.165824E-01   -0.150641E-01   -0.136634E-01   -0.123743E-01   -0.111902E-01   -0.101049E-01   -0.911209E-02   -0.820575E-02   -0.737983E-02   -0.662857E-02   -0.594641E-02   -0.532805E-02   -0.476843E-02   -0.426276E-02   -0.380655E-02   -0.339555E-02   -0.302582E-02   -0.269365E-02   -0.239564E-02   -0.212859E-02   -0.188960E-02   -0.167596E-02   -0.148521E-02   -0.131508E-02   -0.116351E-02   -0.102860E-02   -0.908648E-03   -0.802100E-03   -0.707546E-03   -0.623708E-03   -0.549430E-03   -0.483681E-03   -0.425532E-03   -0.374147E-03   -0.328779E-03   -0.288752E-03   -0.253452E-03   -0.222341E-03   -0.194956E-03   -0.170852E-03   -0.149646E-03   -0.131005E-03   -0.114642E-03   -0.100296E-03   -0.877195E-04   -0.766782E-04   -0.669815E-04   -0.584844E-04   -0.510535E-04   -0.445544E-04   -0.388689E-04   -0.338884E-04   -0.295443E-04   -0.257854E-04   -0.224728E-04   -0.195424E-04   -0.170018E-04   -0.147932E-04   -0.128656E-04   -0.111819E-04   -0.968961E-05   -0.841729E-05   -0.733158E-05   -0.636573E-05   -0.551542E-05   -0.476286E-05   -0.410593E-05   -0.356592E-05   -0.312697E-05   -0.275649E-05   -0.238959E-05   -0.200095E-05   -0.171012E-05   -0.151469E-05   -0.134328E-05   -0.116661E-05   -0.967827E-06   -0.778750E-06   -0.669524E-06   -0.636517E-06   -0.634100E-06   -0.544822E-06   -0.384858E-06   -0.193519E-06   -0.758103E-07   -0.120465E-06   -0.243123E-06   -0.332872E-06   -0.379946E-06   -0.316177E-06   -0.148966E-06   -0.292373E-08    0.520349E-07   -0.942643E-08   -0.154495E-06   -0.184841E-06   -0.858429E-07    0.422418E-07    0.853478E-07    0.741030E-07    0.546297E-07    0.242017E-08   -0.835027E-07   -0.115678E-06   -0.393944E-07    0.218228E-07   -0.625068E-08   -0.782769E-07   -0.586171E-07    0.143945E-07    0.207528E-07    0.558834E-07    0.153736E-07   -0.455798E-07   -0.337878E-07    0.403493E-07    0.928658E-07    0.950874E-07    0.205000E-07   -0.435951E-08    0.474480E-07    0.106392E-06    0.440794E-07    0.542991E-08    0.630559E-07    0.104673E-06    0.381799E-07   -0.330781E-07   -0.141953E-07    0.122215E-07   -0.472907E-07   -0.141160E-06   -0.201950E-06   -0.138332E-06   -0.755528E-07   -0.238332E-07    0.455549E-07    0.151252E-06    0.224682E-06    0.183482E-06    0.601262E-07   -0.578933E-07   -0.121077E-06   -0.748288E-07    0.304085E-07    0.788161E-07    0.283750E-07   -0.100360E-06   -0.151015E-06   -0.100357E-06    0.497559E-07    0.148880E-06    0.135493E-06    0.734343E-07   -0.408878E-07   -0.115642E-06   -0.695480E-07    0.850779E-07    0.173869E-06    0.102680E-06   -0.479647E-07   -0.177276E-06   -0.203943E-06   -0.974495E-07    0.756991E-07    0.184386E-06    0.142572E-06   -0.332452E-07   -0.138499E-06   -0.102467E-06    0.280283E-07    0.102307E-06    0.390282E-07   -0.877416E-07   -0.177428E-06   -0.101698E-06    0.417331E-07    0.136069E-06    0.132113E-06    0.267592E-07   -0.122445E-06   -0.209098E-06   -0.154641E-06   -0.126266E-07    0.121185E-06    0.153185E-06    0.460420E-07   -0.745163E-07   -0.104394E-06   -0.623425E-07    0.346181E-07    0.115842E-06    0.118302E-06    0.167708E-07   -0.120924E-06   -0.190276E-06   -0.112471E-06    0.364779E-07    0.827687E-07    0.325057E-08   -0.581581E-07    0.276681E-07    0.151428E-06    0.166680E-06    0.646376E-07   -0.383388E-07   -0.563523E-07   -0.412036E-07\r\n",
      "   -0.872746E-07    0.107023E-06    0.219588E-06    0.168049E-06    0.137136E-07   -0.151332E-06   -0.240963E-06   -0.210482E-06   -0.103399E-06    0.361617E-08    0.688021E-07    0.136373E-06    0.178120E-06    0.112203E-06   -0.394854E-07   -0.181904E-06   -0.177338E-06   -0.648563E-07   -0.608298E-08   -0.507332E-07   -0.105387E-06   -0.115594E-07    0.142477E-06    0.224839E-06    0.180683E-06    0.318029E-07   -0.116845E-06   -0.141708E-06   -0.888254E-07   -0.845353E-07   -0.468265E-07    0.843605E-07    0.166414E-06    0.836593E-07   -0.126720E-06   -0.275521E-06   -0.223310E-06   -0.807969E-07    0.736861E-07    0.211513E-06    0.272239E-06    0.201780E-06    0.108669E-07   -0.116279E-06   -0.123160E-06   -0.833195E-07   -0.785148E-07   -0.444944E-07    0.782565E-07    0.187285E-06    0.179023E-06    0.382096E-07   -0.958402E-07   -0.619894E-07    0.118270E-07    0.528720E-08   -0.301991E-07   -0.150557E-07    0.126859E-07    0.403754E-07    0.431061E-07    0.691915E-07    0.261921E-07   -0.662541E-07   -0.105080E-06   -0.153184E-07    0.126601E-06    0.188046E-06    0.968799E-07    0.185794E-07    0.719028E-07    0.130206E-06    0.614574E-07   -0.835339E-07   -0.220338E-06   -0.290153E-06   -0.256845E-06   -0.687421E-07    0.130069E-06    0.242276E-06    0.165027E-06    0.291203E-07   -0.716608E-07   -0.693367E-07   -0.994651E-08    0.365761E-07    0.668739E-07    0.956587E-07    0.998196E-07   -0.656512E-08   -0.133030E-06   -0.679316E-07    0.950253E-07    0.181084E-06    0.156268E-06    0.183791E-08   -0.136917E-06   -0.163157E-06   -0.672982E-07    0.105933E-06    0.148858E-06    0.601448E-07   -0.145033E-06   -0.299983E-06   -0.296325E-06   -0.188318E-06    0.901417E-08    0.188380E-06    0.284133E-06    0.207995E-06    0.277750E-08   -0.157012E-06   -0.245700E-06   -0.250039E-06   -0.132873E-06    0.408419E-07    0.132433E-06    0.837840E-07   -0.105887E-06   -0.337893E-06   -0.453754E-06   -0.413956E-06   -0.277331E-06   -0.152092E-06   -0.136962E-06   -0.244957E-06   -0.417742E-06   -0.551820E-06   -0.618840E-06   -0.706364E-06   -0.832759E-06   -0.103317E-05   -0.125024E-05   -0.146655E-05   -0.164510E-05   -0.178630E-05   -0.202130E-05   -0.234372E-05   -0.273955E-05   -0.315758E-05   -0.362418E-05   -0.416130E-05   -0.479620E-05   -0.554225E-05   -0.639483E-05   -0.739725E-05   -0.855410E-05   -0.988025E-05   -0.113785E-04   -0.130777E-04   -0.150323E-04   -0.172893E-04   -0.198805E-04   -0.228584E-04   -0.262587E-04   -0.301448E-04   -0.346132E-04   -0.397312E-04   -0.455810E-04   -0.522641E-04   -0.599105E-04   -0.686757E-04   -0.787100E-04   -0.901693E-04   -0.103216E-03   -0.118090E-03   -0.135073E-03   -0.154452E-03   -0.176532E-03   -0.201657E-03   -0.230234E-03   -0.262718E-03   -0.299628E-03   -0.341547E-03   -0.389136E-03   -0.443107E-03   -0.504263E-03   -0.573516E-03   -0.651877E-03   -0.740465E-03   -0.840535E-03   -0.953483E-03   -0.108086E-02   -0.122436E-02   -0.138589E-02   -0.156751E-02   -0.177153E-02   -0.200046E-02   -0.225707E-02   -0.254437E-02   -0.286565E-02   -0.322450E-02   -0.362481E-02   -0.407077E-02   -0.456693E-02   -0.511815E-02   -0.572964E-02   -0.640699E-02   -0.715609E-02   -0.798321E-02   -0.889493E-02   -0.989815E-02   -0.110001E-01   -0.122081E-01   -0.135300E-01   -0.149734E-01   -0.165463E-01   -0.182568E-01   -0.201125E-01   -0.221213E-01   -0.242907E-01   -0.266277E-01   -0.291389E-01   -0.318303E-01   -0.347069E-01   -0.377731E-01   -0.410318E-01   -0.444850E-01   -0.481331E-01   -0.519749E-01   -0.560074E-01   -0.602259E-01   -0.646237E-01   -0.691920E-01   -0.739193E-01   -0.787922E-01   -0.837954E-01   -0.889128E-01   -0.941247E-01   -0.993940E-01   -0.104700E+00   -0.110068E+00   -0.115423E+00   -0.120727E+00   -0.125964E+00   -0.131095E+00   -0.136093E+00   -0.140926E+00   -0.145560E+00   -0.149964E+00   -0.154109E+00   -0.157963E+00   -0.161492E+00   -0.164666E+00   -0.167466E+00   -0.169890E+00   -0.171936E+00   -0.173565E+00   -0.174646E+00   -0.175138E+00   -0.175251E+00   -0.175138E+00   -0.174646E+00   -0.173565E+00   -0.171936E+00   -0.169890E+00   -0.167466E+00   -0.164666E+00   -0.161492E+00   -0.157963E+00   -0.154109E+00   -0.149964E+00   -0.145560E+00   -0.140926E+00   -0.136093E+00   -0.131095E+00   -0.125964E+00   -0.120727E+00   -0.115423E+00   -0.110068E+00   -0.104700E+00   -0.993940E-01   -0.941247E-01   -0.889128E-01   -0.837954E-01   -0.787922E-01   -0.739193E-01   -0.691920E-01   -0.646237E-01   -0.602259E-01   -0.560074E-01   -0.519749E-01   -0.481331E-01   -0.444850E-01   -0.410318E-01   -0.377731E-01   -0.347069E-01   -0.318303E-01   -0.291389E-01   -0.266277E-01   -0.242907E-01   -0.221213E-01   -0.201125E-01   -0.182568E-01   -0.165463E-01   -0.149734E-01   -0.135299E-01   -0.122081E-01   -0.110001E-01   -0.989815E-02   -0.889493E-02   -0.798321E-02   -0.715609E-02   -0.640699E-02   -0.572964E-02   -0.511815E-02   -0.456693E-02   -0.407077E-02   -0.362481E-02   -0.322450E-02   -0.286565E-02   -0.254437E-02   -0.225707E-02   -0.200046E-02   -0.177153E-02   -0.156751E-02   -0.138589E-02   -0.122436E-02   -0.108086E-02   -0.953483E-03   -0.840535E-03   -0.740465E-03   -0.651876E-03   -0.573516E-03   -0.504262E-03   -0.443106E-03   -0.389137E-03   -0.341548E-03   -0.299628E-03   -0.262717E-03   -0.230233E-03   -0.201656E-03   -0.176529E-03   -0.154451E-03   -0.135072E-03   -0.118090E-03   -0.103216E-03   -0.901652E-04   -0.787072E-04   -0.686815E-04   -0.599320E-04   -0.522853E-04   -0.455780E-04   -0.397073E-04   -0.346023E-04   -0.301692E-04   -0.262744E-04   -0.228518E-04   -0.198586E-04   -0.172846E-04   -0.150475E-04   -0.130762E-04   -0.113530E-04   -0.987483E-05   -0.856575E-05   -0.742579E-05   -0.641864E-05   -0.551049E-05   -0.472904E-05   -0.410668E-05   -0.359145E-05   -0.316271E-05   -0.274270E-05   -0.230681E-05   -0.197582E-05   -0.176295E-05   -0.158791E-05   -0.138808E-05   -0.114135E-05   -0.917982E-06   -0.783909E-06   -0.730655E-06   -0.704978E-06   -0.608497E-06   -0.461177E-06   -0.269829E-06   -0.160337E-06   -0.189271E-06   -0.288093E-06   -0.356422E-06   -0.374513E-06   -0.279513E-06   -0.102793E-06    0.451040E-07    0.815594E-07   -0.179806E-08   -0.174709E-06   -0.248970E-06   -0.173447E-06   -0.315702E-07    0.859914E-07    0.135178E-06    0.129080E-06    0.330770E-07   -0.687007E-07   -0.105427E-06   -0.771597E-07   -0.303401E-07   -0.341210E-07   -0.829856E-07   -0.772153E-07   -0.318925E-08    0.680980E-07    0.121107E-06    0.586957E-07   -0.267175E-07   -0.338431E-07    0.212497E-07    0.518273E-07    0.322761E-08   -0.690649E-07   -0.484904E-07    0.550619E-07    0.134014E-06    0.898842E-07    0.635322E-07    0.122462E-06    0.157869E-06    0.879940E-07   -0.745572E-08   -0.216634E-07   -0.232164E-07   -0.810794E-07   -0.159377E-06   -0.183881E-06   -0.120097E-06   -0.751870E-07   -0.466755E-07    0.660160E-08    0.109091E-06    0.208244E-06    0.204177E-06    0.121574E-06    0.223980E-07   -0.443034E-07   -0.441589E-07   -0.273578E-08    0.170709E-07   -0.311478E-07   -0.119801E-06   -0.120715E-06   -0.373793E-07    0.949256E-07    0.168439E-06    0.136717E-06    0.934272E-07   -0.102456E-08   -0.507737E-07   -0.185494E-07    0.731569E-07    0.131807E-06    0.772871E-07   -0.470601E-07   -0.193687E-06   -0.237732E-06   -0.138833E-06    0.528515E-07    0.181430E-06    0.153592E-06    0.349791E-09   -0.784551E-07   -0.535112E-07    0.322496E-07    0.779354E-07    0.755020E-08   -0.108428E-06   -0.126362E-06   -0.369734E-07    0.991018E-07    0.174555E-06    0.133353E-06   -0.594580E-08   -0.165521E-06   -0.240591E-06   -0.154842E-06    0.282916E-07    0.196006E-06    0.218873E-06    0.561982E-07   -0.103192E-06   -0.149661E-06   -0.831490E-07    0.567234E-07    0.175665E-06    0.188284E-06    0.713232E-07   -0.102187E-06   -0.213355E-06   -0.170324E-06   -0.352010E-07    0.448280E-07    0.195842E-07   -0.177251E-07    0.642388E-07    0.163437E-06    0.149981E-06    0.192675E-07   -0.725929E-07   -0.703182E-07   -0.579635E-08\r\n",
      "   -0.917789E-07    0.586070E-07    0.204025E-06    0.221330E-06    0.991894E-07   -0.746918E-07   -0.230912E-06   -0.270681E-06   -0.186525E-06   -0.282900E-07    0.108142E-06    0.205731E-06    0.234191E-06    0.102601E-06   -0.929330E-07   -0.220767E-06   -0.189820E-06   -0.561871E-07    0.195796E-07   -0.670947E-08   -0.367938E-07    0.443242E-07    0.181392E-06    0.241120E-06    0.162975E-06    0.210363E-07   -0.101732E-06   -0.999621E-07   -0.655636E-07   -0.820332E-07   -0.621207E-07    0.146818E-07    0.105348E-06    0.472081E-07   -0.161903E-06   -0.294854E-06   -0.211664E-06   -0.315983E-07    0.135930E-06    0.248553E-06    0.305312E-06    0.216079E-06    0.861866E-08   -0.145535E-06   -0.176101E-06   -0.135877E-06   -0.928164E-07   -0.212323E-08    0.113529E-06    0.161366E-06    0.102042E-06   -0.188446E-07   -0.905048E-07   -0.603594E-07    0.113217E-07   -0.180883E-07   -0.107327E-06   -0.123605E-06   -0.401139E-07    0.244704E-07    0.737384E-07    0.129677E-06    0.796225E-07   -0.655773E-07   -0.174894E-06   -0.118172E-06    0.493724E-07    0.198722E-06    0.173386E-06    0.999756E-07    0.660939E-07    0.559137E-07   -0.145071E-07   -0.135186E-06   -0.234622E-06   -0.258147E-06   -0.175182E-06    0.475568E-08    0.148544E-06    0.166324E-06    0.770382E-07   -0.262618E-07   -0.959378E-07   -0.733110E-07    0.166143E-07    0.635024E-07    0.993218E-07    0.105924E-06    0.542324E-07   -0.344752E-07   -0.126304E-06   -0.762966E-07    0.774634E-07    0.151551E-06    0.914617E-07   -0.303195E-07   -0.139599E-06   -0.136253E-06   -0.120000E-07    0.129858E-06    0.169840E-06    0.427625E-07   -0.172673E-06   -0.319021E-06   -0.287604E-06   -0.155429E-06    0.348157E-07    0.197215E-06    0.276258E-06    0.164767E-06   -0.834099E-07   -0.247938E-06   -0.337586E-06   -0.299200E-06   -0.125440E-06    0.677030E-07    0.173032E-06    0.837089E-07   -0.147737E-06   -0.381699E-06   -0.477369E-06   -0.425477E-06   -0.309707E-06   -0.222944E-06   -0.233619E-06   -0.375495E-06   -0.546339E-06   -0.650151E-06   -0.705012E-06   -0.807620E-06   -0.938180E-06   -0.115033E-05   -0.142426E-05   -0.168378E-05   -0.188875E-05   -0.205033E-05   -0.227098E-05   -0.264344E-05   -0.311416E-05   -0.360275E-05   -0.414400E-05   -0.477076E-05   -0.550476E-05   -0.635826E-05   -0.733647E-05   -0.849568E-05   -0.981310E-05   -0.113026E-04   -0.130014E-04   -0.149608E-04   -0.172067E-04   -0.197994E-04   -0.227664E-04   -0.261571E-04   -0.300493E-04   -0.345304E-04   -0.396604E-04   -0.455304E-04   -0.522610E-04   -0.599635E-04   -0.687693E-04   -0.788450E-04   -0.903825E-04   -0.103583E-03   -0.118646E-03   -0.135825E-03   -0.155428E-03   -0.177805E-03   -0.203308E-03   -0.232342E-03   -0.265389E-03   -0.302975E-03   -0.345694E-03   -0.394242E-03   -0.449391E-03   -0.511957E-03   -0.582894E-03   -0.663265E-03   -0.754257E-03   -0.857184E-03   -0.973524E-03   -0.110492E-02   -0.125318E-02   -0.142033E-02   -0.160859E-02   -0.182040E-02   -0.205849E-02   -0.232582E-02   -0.262567E-02   -0.296161E-02   -0.333755E-02   -0.375774E-02   -0.422679E-02   -0.474971E-02   -0.533190E-02   -0.597915E-02   -0.669771E-02   -0.749421E-02   -0.837574E-02   -0.934979E-02   -0.104243E-01   -0.116074E-01   -0.129079E-01   -0.143348E-01   -0.158971E-01   -0.176044E-01   -0.194663E-01   -0.214923E-01   -0.236920E-01   -0.260748E-01   -0.286498E-01   -0.314256E-01   -0.344104E-01   -0.376114E-01   -0.410350E-01   -0.446864E-01   -0.485695E-01   -0.526868E-01   -0.570389E-01   -0.616247E-01   -0.664409E-01   -0.714820E-01   -0.767400E-01   -0.822045E-01   -0.878620E-01   -0.936965E-01   -0.996897E-01   -0.105822E+00   -0.112069E+00   -0.118390E+00   -0.124756E+00   -0.131192E+00   -0.137620E+00   -0.143989E+00   -0.150278E+00   -0.156441E+00   -0.162444E+00   -0.168250E+00   -0.173818E+00   -0.179111E+00   -0.184092E+00   -0.188724E+00   -0.192966E+00   -0.196781E+00   -0.200147E+00   -0.203062E+00   -0.205523E+00   -0.207481E+00   -0.208778E+00   -0.209367E+00   -0.209503E+00   -0.209367E+00   -0.208778E+00   -0.207481E+00   -0.205523E+00   -0.203062E+00   -0.200147E+00   -0.196781E+00   -0.192966E+00   -0.188724E+00   -0.184092E+00   -0.179111E+00   -0.173818E+00   -0.168250E+00   -0.162444E+00   -0.156441E+00   -0.150278E+00   -0.143989E+00   -0.137620E+00   -0.131192E+00   -0.124756E+00   -0.118390E+00   -0.112069E+00   -0.105822E+00   -0.996897E-01   -0.936965E-01   -0.878620E-01   -0.822045E-01   -0.767400E-01   -0.714820E-01   -0.664409E-01   -0.616247E-01   -0.570389E-01   -0.526868E-01   -0.485695E-01   -0.446864E-01   -0.410350E-01   -0.376114E-01   -0.344104E-01   -0.314256E-01   -0.286498E-01   -0.260748E-01   -0.236920E-01   -0.214923E-01   -0.194663E-01   -0.176044E-01   -0.158971E-01   -0.143348E-01   -0.129079E-01   -0.116074E-01   -0.104243E-01   -0.934979E-02   -0.837574E-02   -0.749421E-02   -0.669771E-02   -0.597915E-02   -0.533190E-02   -0.474971E-02   -0.422679E-02   -0.375774E-02   -0.333755E-02   -0.296161E-02   -0.262567E-02   -0.232582E-02   -0.205849E-02   -0.182040E-02   -0.160859E-02   -0.142033E-02   -0.125318E-02   -0.110492E-02   -0.973524E-03   -0.857184E-03   -0.754256E-03   -0.663264E-03   -0.582892E-03   -0.511957E-03   -0.449391E-03   -0.394244E-03   -0.345695E-03   -0.302976E-03   -0.265390E-03   -0.232344E-03   -0.203308E-03   -0.177806E-03   -0.155428E-03   -0.135823E-03   -0.118647E-03   -0.103581E-03   -0.903823E-04   -0.788526E-04   -0.687873E-04   -0.599864E-04   -0.522651E-04   -0.455222E-04   -0.396502E-04   -0.345239E-04   -0.300534E-04   -0.261561E-04   -0.227342E-04   -0.197726E-04   -0.172149E-04   -0.149758E-04   -0.130042E-04   -0.113072E-04   -0.980463E-05   -0.848132E-05   -0.731388E-05   -0.628094E-05   -0.541473E-05   -0.473030E-05   -0.413255E-05   -0.359276E-05   -0.307171E-05   -0.260501E-05   -0.224957E-05   -0.202688E-05   -0.184657E-05   -0.162334E-05   -0.133346E-05   -0.106411E-05   -0.900091E-06   -0.817284E-06   -0.762164E-06   -0.685611E-06   -0.562577E-06   -0.403706E-06   -0.283239E-06   -0.295110E-06   -0.346898E-06   -0.386329E-06   -0.373221E-06   -0.248597E-06   -0.853010E-07    0.629817E-07    0.117608E-06    0.176098E-07   -0.168542E-06   -0.289282E-06   -0.261301E-06   -0.142918E-06    0.156931E-07    0.134520E-06    0.168350E-06    0.848470E-07   -0.211139E-07   -0.699092E-07   -0.110888E-06   -0.917575E-07   -0.871087E-07   -0.110033E-06   -0.901750E-07   -0.162676E-07    0.892350E-07    0.129471E-06    0.776205E-07   -0.143015E-07   -0.256625E-07    0.698689E-08   -0.403321E-08   -0.101422E-06   -0.157724E-06   -0.845991E-07    0.293835E-07    0.103726E-06    0.979164E-07    0.894968E-07    0.140230E-06    0.153775E-06    0.818419E-07    0.418236E-08    0.119560E-07    0.115955E-07   -0.615535E-07   -0.154022E-06   -0.175517E-06   -0.118509E-06   -0.805051E-07   -0.627864E-07   -0.591419E-07    0.130066E-07    0.121875E-06    0.161928E-06    0.139268E-06    0.630044E-07   -0.130902E-07   -0.143457E-08    0.947890E-08   -0.177908E-07   -0.102280E-06   -0.168631E-06   -0.139967E-06   -0.239644E-07    0.103516E-06    0.174161E-06    0.160406E-06    0.949428E-07    0.123172E-07   -0.239634E-07    0.452559E-08    0.488738E-07    0.755178E-07    0.382913E-07   -0.608043E-07   -0.190042E-06   -0.236702E-06   -0.140411E-06    0.371410E-07    0.146140E-06    0.107325E-06    0.854537E-08   -0.335856E-07    0.112575E-07    0.515350E-07    0.173695E-07   -0.798746E-07   -0.158585E-06   -0.767280E-07    0.555943E-07    0.163790E-06    0.190971E-06    0.112170E-06   -0.441525E-07   -0.202504E-06   -0.271591E-06   -0.181035E-06    0.257196E-07    0.197408E-06    0.231468E-06    0.772725E-07   -0.956268E-07   -0.187342E-06   -0.855563E-07    0.957028E-07    0.227726E-06    0.242986E-06    0.950532E-07   -0.973568E-07   -0.210308E-06   -0.195046E-06   -0.888124E-07    0.112079E-07    0.311275E-07    0.212418E-07    0.994368E-07    0.164758E-06    0.971470E-07   -0.583725E-07   -0.149199E-06   -0.119858E-06    0.122704E-07\r\n",
      "   -0.794728E-07    0.121276E-07    0.143228E-06    0.236370E-06    0.169769E-06    0.775815E-09   -0.181016E-06   -0.247631E-06   -0.203893E-06   -0.345647E-07    0.125833E-06    0.235906E-06    0.255357E-06    0.880939E-07   -0.134365E-06   -0.229286E-06   -0.161083E-06   -0.110301E-07    0.777684E-07    0.324937E-07    0.157183E-07    0.682476E-07    0.178910E-06    0.238443E-06    0.149088E-06    0.431677E-08   -0.850818E-07   -0.523113E-07   -0.161447E-07   -0.545700E-07   -0.808693E-07   -0.273783E-07    0.493744E-07    0.808259E-08   -0.156708E-06   -0.276746E-06   -0.210190E-06   -0.200962E-07    0.172075E-06    0.280297E-06    0.313450E-06    0.199220E-06   -0.129116E-08   -0.163678E-06   -0.221900E-06   -0.182804E-06   -0.837357E-07    0.528229E-07    0.142786E-06    0.129579E-06    0.270267E-07   -0.658167E-07   -0.654870E-07   -0.310959E-07    0.303584E-07   -0.158491E-07   -0.144308E-06   -0.182129E-06   -0.860663E-07   -0.193485E-07    0.749387E-07    0.155274E-06    0.103084E-06   -0.622508E-07   -0.198575E-06   -0.171960E-06   -0.212527E-07    0.175060E-06    0.225787E-06    0.164896E-06    0.494567E-07   -0.388580E-07   -0.883150E-07   -0.129535E-06   -0.172782E-06   -0.189419E-06   -0.747324E-07    0.764788E-07    0.143158E-06    0.790457E-07   -0.180537E-07   -0.628755E-07   -0.789748E-07   -0.343925E-07    0.475970E-07    0.828767E-07    0.109316E-06    0.101660E-06    0.259680E-07   -0.478909E-07   -0.992933E-07   -0.576624E-07    0.554197E-07    0.104393E-06    0.347484E-07   -0.498706E-07   -0.118337E-06   -0.926354E-07    0.117177E-07    0.111277E-06    0.147073E-06    0.428719E-07   -0.165434E-06   -0.284611E-06   -0.238610E-06   -0.971688E-07    0.735151E-07    0.201859E-06    0.217074E-06    0.653324E-07   -0.169930E-06   -0.319785E-06   -0.377189E-06   -0.311354E-06   -0.101277E-06    0.994758E-07    0.171800E-06    0.562103E-07   -0.170976E-06   -0.372769E-06   -0.466185E-06   -0.430105E-06   -0.364055E-06   -0.328440E-06   -0.358195E-06   -0.494474E-06   -0.638644E-06   -0.725751E-06   -0.784912E-06   -0.899169E-06   -0.104001E-05   -0.123677E-05   -0.154226E-05   -0.185215E-05   -0.208870E-05   -0.227752E-05   -0.252300E-05   -0.293636E-05   -0.346757E-05   -0.403792E-05   -0.466414E-05   -0.538064E-05   -0.620113E-05   -0.715356E-05   -0.826438E-05   -0.956386E-05   -0.110327E-04   -0.126953E-04   -0.145978E-04   -0.168001E-04   -0.193323E-04   -0.222471E-04   -0.255808E-04   -0.293970E-04   -0.337847E-04   -0.388288E-04   -0.446115E-04   -0.512341E-04   -0.588299E-04   -0.675343E-04   -0.774783E-04   -0.888510E-04   -0.101870E-03   -0.116781E-03   -0.133831E-03   -0.153287E-03   -0.175479E-03   -0.200808E-03   -0.229694E-03   -0.262601E-03   -0.300072E-03   -0.342709E-03   -0.391200E-03   -0.446331E-03   -0.508966E-03   -0.580067E-03   -0.660722E-03   -0.752145E-03   -0.855699E-03   -0.972899E-03   -0.110544E-02   -0.125521E-02   -0.142431E-02   -0.161504E-02   -0.182997E-02   -0.207194E-02   -0.234407E-02   -0.264982E-02   -0.299294E-02   -0.337760E-02   -0.380831E-02   -0.429001E-02   -0.482806E-02   -0.542828E-02   -0.609693E-02   -0.684079E-02   -0.766710E-02   -0.858364E-02   -0.959865E-02   -0.107209E-01   -0.119597E-01   -0.133246E-01   -0.148258E-01   -0.164739E-01   -0.182797E-01   -0.202542E-01   -0.224088E-01   -0.247548E-01   -0.273035E-01   -0.300660E-01   -0.330530E-01   -0.362749E-01   -0.397412E-01   -0.434606E-01   -0.474409E-01   -0.516882E-01   -0.562073E-01   -0.610013E-01   -0.660711E-01   -0.714155E-01   -0.770308E-01   -0.829108E-01   -0.890463E-01   -0.954249E-01   -0.102031E+00   -0.108847E+00   -0.115849E+00   -0.123016E+00   -0.130320E+00   -0.137712E+00   -0.145159E+00   -0.152683E+00   -0.160206E+00   -0.167659E+00   -0.175019E+00   -0.182234E+00   -0.189260E+00   -0.196056E+00   -0.202574E+00   -0.208771E+00   -0.214603E+00   -0.220026E+00   -0.224993E+00   -0.229460E+00   -0.233401E+00   -0.236816E+00   -0.239698E+00   -0.241991E+00   -0.243508E+00   -0.244194E+00   -0.244352E+00   -0.244194E+00   -0.243508E+00   -0.241991E+00   -0.239698E+00   -0.236816E+00   -0.233401E+00   -0.229460E+00   -0.224993E+00   -0.220026E+00   -0.214603E+00   -0.208771E+00   -0.202574E+00   -0.196056E+00   -0.189260E+00   -0.182234E+00   -0.175019E+00   -0.167659E+00   -0.160206E+00   -0.152683E+00   -0.145159E+00   -0.137712E+00   -0.130320E+00   -0.123016E+00   -0.115849E+00   -0.108847E+00   -0.102031E+00   -0.954249E-01   -0.890463E-01   -0.829108E-01   -0.770308E-01   -0.714155E-01   -0.660711E-01   -0.610013E-01   -0.562073E-01   -0.516882E-01   -0.474409E-01   -0.434606E-01   -0.397412E-01   -0.362749E-01   -0.330530E-01   -0.300660E-01   -0.273035E-01   -0.247548E-01   -0.224088E-01   -0.202542E-01   -0.182797E-01   -0.164739E-01   -0.148258E-01   -0.133246E-01   -0.119597E-01   -0.107209E-01   -0.959865E-02   -0.858364E-02   -0.766710E-02   -0.684079E-02   -0.609693E-02   -0.542828E-02   -0.482806E-02   -0.429001E-02   -0.380831E-02   -0.337760E-02   -0.299294E-02   -0.264982E-02   -0.234408E-02   -0.207194E-02   -0.182997E-02   -0.161504E-02   -0.142431E-02   -0.125521E-02   -0.110544E-02   -0.972898E-03   -0.855699E-03   -0.752145E-03   -0.660721E-03   -0.580067E-03   -0.508966E-03   -0.446332E-03   -0.391201E-03   -0.342710E-03   -0.300073E-03   -0.262603E-03   -0.229697E-03   -0.200812E-03   -0.175480E-03   -0.153283E-03   -0.133827E-03   -0.116784E-03   -0.101874E-03   -0.888582E-04   -0.774935E-04   -0.675492E-04   -0.588393E-04   -0.512377E-04   -0.446075E-04   -0.388086E-04   -0.337630E-04   -0.293801E-04   -0.255549E-04   -0.222284E-04   -0.193371E-04   -0.168135E-04   -0.146112E-04   -0.126846E-04   -0.109976E-04   -0.949259E-05   -0.818012E-05   -0.704623E-05   -0.612143E-05   -0.536712E-05   -0.469353E-05   -0.404195E-05   -0.342392E-05   -0.291965E-05   -0.254142E-05   -0.228374E-05   -0.207018E-05   -0.181121E-05   -0.149180E-05   -0.122728E-05   -0.102720E-05   -0.900697E-06   -0.820636E-06   -0.752939E-06   -0.655614E-06   -0.520706E-06   -0.407099E-06   -0.408139E-06   -0.420644E-06   -0.431357E-06   -0.376687E-06   -0.264185E-06   -0.108019E-06    0.588919E-07    0.142279E-06    0.483807E-07   -0.139635E-06   -0.293252E-06   -0.320233E-06   -0.235666E-06   -0.704306E-07    0.110420E-06    0.188925E-06    0.145399E-06    0.617720E-07   -0.298699E-07   -0.108861E-06   -0.128436E-06   -0.159651E-06   -0.182858E-06   -0.113409E-06   -0.100409E-07    0.110953E-06    0.147029E-06    0.881664E-07   -0.144722E-08   -0.180427E-07   -0.850643E-08   -0.484790E-07   -0.166698E-06   -0.214573E-06   -0.107245E-06    0.243588E-07    0.863794E-07    0.908746E-07    0.798565E-07    0.112180E-06    0.105388E-06    0.469153E-07    0.107376E-07    0.512362E-07    0.665251E-07   -0.134551E-07   -0.126800E-06   -0.152863E-06   -0.865840E-07   -0.474820E-07   -0.767834E-07   -0.137551E-06   -0.939729E-07   -0.704167E-08    0.732585E-07    0.101037E-06    0.509790E-07    0.154460E-07    0.617935E-07    0.794275E-07    0.171454E-07   -0.129148E-06   -0.206619E-06   -0.177388E-06   -0.580008E-07    0.661818E-07    0.155642E-06    0.185916E-06    0.108899E-06    0.213328E-07   -0.256170E-07   -0.252659E-07    0.126560E-07    0.378557E-07    0.830864E-08   -0.531448E-07   -0.147735E-06   -0.192384E-06   -0.101600E-06    0.201371E-07    0.907289E-07    0.521685E-07    0.206279E-07    0.259952E-07    0.922893E-07    0.868482E-07   -0.351884E-07   -0.169841E-06   -0.194979E-06   -0.403960E-07    0.122624E-06    0.206510E-06    0.189520E-06    0.679962E-07   -0.842026E-07   -0.212114E-06   -0.275701E-06   -0.199662E-06   -0.124068E-07    0.134491E-06    0.194692E-06    0.110098E-06   -0.329537E-07   -0.150952E-06   -0.683950E-07    0.113868E-06    0.244910E-06    0.251895E-06    0.934737E-07   -0.961735E-07   -0.197829E-06   -0.193394E-06   -0.815575E-07    0.278873E-07    0.372461E-07    0.447789E-07    0.118672E-06    0.153413E-06    0.552605E-07   -0.115245E-06   -0.208259E-06   -0.155086E-06    0.113346E-08\r\n",
      "   -0.700790E-07   -0.139915E-07    0.668191E-07    0.181559E-06    0.159613E-06    0.361981E-07   -0.102380E-06   -0.157392E-06   -0.178249E-06   -0.637635E-07    0.103993E-06    0.205929E-06    0.212469E-06    0.540293E-07   -0.161700E-06   -0.229934E-06   -0.129057E-06    0.325171E-07    0.119151E-06    0.496608E-07    0.130758E-07    0.439834E-07    0.145965E-06    0.198890E-06    0.103639E-06   -0.363625E-07   -0.874458E-07   -0.358450E-07    0.655754E-08   -0.386392E-07   -0.859262E-07   -0.451862E-07    0.930258E-08   -0.553231E-07   -0.166661E-06   -0.251059E-06   -0.212563E-06   -0.468956E-07    0.140678E-06    0.256257E-06    0.275161E-06    0.157520E-06   -0.278837E-07   -0.190676E-06   -0.253302E-06   -0.209013E-06   -0.721025E-07    0.858408E-07    0.145227E-06    0.760296E-07   -0.274383E-07   -0.895921E-07   -0.341699E-07    0.155126E-07    0.412086E-07   -0.182187E-07   -0.143610E-06   -0.188868E-06   -0.130263E-06   -0.479831E-07    0.377317E-07    0.984138E-07    0.508785E-07   -0.901835E-07   -0.187985E-06   -0.162478E-06   -0.455599E-07    0.126246E-06    0.220933E-06    0.161902E-06    0.100648E-07   -0.127627E-06   -0.160659E-06   -0.110200E-06   -0.823949E-07   -0.100455E-06   -0.109615E-07    0.859325E-07    0.914280E-07   -0.840426E-08   -0.845954E-07   -0.724422E-07   -0.404796E-07    0.200588E-07    0.651720E-07    0.762134E-07    0.597372E-07    0.404995E-07   -0.796598E-08   -0.645409E-07   -0.858280E-07   -0.316908E-07    0.440616E-07    0.289155E-07   -0.322621E-07   -0.608269E-07   -0.854200E-07   -0.798503E-07   -0.320442E-07    0.384704E-07    0.762112E-07    0.102235E-07   -0.157721E-06   -0.251627E-06   -0.194759E-06   -0.469585E-07    0.101051E-06    0.195673E-06    0.148812E-06   -0.329297E-07   -0.257778E-06   -0.386051E-06   -0.391009E-06   -0.289056E-06   -0.865570E-07    0.750362E-07    0.100320E-06   -0.150521E-07   -0.190620E-06   -0.353956E-06   -0.459423E-06   -0.447103E-06   -0.422300E-06   -0.437022E-06   -0.484281E-06   -0.588681E-06   -0.708215E-06   -0.800060E-06   -0.887467E-06   -0.101814E-05   -0.117300E-05   -0.137612E-05   -0.166425E-05   -0.196136E-05   -0.222113E-05   -0.247750E-05   -0.279444E-05   -0.326183E-05   -0.383911E-05   -0.446828E-05   -0.517232E-05   -0.597061E-05   -0.686757E-05   -0.792503E-05   -0.915749E-05   -0.105784E-04   -0.121952E-04   -0.140464E-04   -0.161610E-04   -0.185975E-04   -0.214006E-04   -0.246325E-04   -0.283376E-04   -0.325785E-04   -0.374405E-04   -0.430312E-04   -0.494527E-04   -0.568211E-04   -0.652654E-04   -0.749336E-04   -0.859970E-04   -0.986546E-04   -0.113144E-03   -0.129740E-03   -0.148722E-03   -0.170408E-03   -0.195151E-03   -0.223388E-03   -0.255606E-03   -0.292333E-03   -0.334169E-03   -0.381791E-03   -0.435993E-03   -0.497635E-03   -0.567682E-03   -0.647239E-03   -0.737526E-03   -0.839915E-03   -0.955942E-03   -0.108733E-02   -0.123598E-02   -0.140405E-02   -0.159389E-02   -0.180812E-02   -0.204967E-02   -0.232174E-02   -0.262789E-02   -0.297203E-02   -0.335847E-02   -0.379192E-02   -0.427755E-02   -0.482096E-02   -0.542830E-02   -0.610619E-02   -0.686183E-02   -0.770294E-02   -0.863784E-02   -0.967542E-02   -0.108252E-01   -0.120972E-01   -0.135020E-01   -0.150508E-01   -0.167554E-01   -0.186277E-01   -0.206804E-01   -0.229261E-01   -0.253781E-01   -0.280493E-01   -0.309530E-01   -0.341018E-01   -0.375085E-01   -0.411849E-01   -0.451422E-01   -0.493905E-01   -0.539389E-01   -0.587947E-01   -0.639636E-01   -0.694491E-01   -0.752527E-01   -0.813728E-01   -0.878056E-01   -0.945438E-01   -0.101577E+00   -0.108891E+00   -0.116468E+00   -0.124287E+00   -0.132322E+00   -0.140548E+00   -0.148931E+00   -0.157419E+00   -0.165972E+00   -0.174606E+00   -0.183247E+00   -0.191808E+00   -0.200263E+00   -0.208550E+00   -0.216621E+00   -0.224429E+00   -0.231917E+00   -0.239036E+00   -0.245736E+00   -0.251966E+00   -0.257671E+00   -0.262802E+00   -0.267331E+00   -0.271254E+00   -0.274566E+00   -0.277200E+00   -0.278940E+00   -0.279726E+00   -0.279906E+00   -0.279726E+00   -0.278940E+00   -0.277200E+00   -0.274566E+00   -0.271254E+00   -0.267331E+00   -0.262802E+00   -0.257671E+00   -0.251966E+00   -0.245736E+00   -0.239036E+00   -0.231917E+00   -0.224429E+00   -0.216621E+00   -0.208550E+00   -0.200263E+00   -0.191808E+00   -0.183247E+00   -0.174606E+00   -0.165972E+00   -0.157419E+00   -0.148931E+00   -0.140548E+00   -0.132322E+00   -0.124287E+00   -0.116468E+00   -0.108891E+00   -0.101577E+00   -0.945438E-01   -0.878056E-01   -0.813728E-01   -0.752527E-01   -0.694491E-01   -0.639636E-01   -0.587947E-01   -0.539389E-01   -0.493905E-01   -0.451422E-01   -0.411849E-01   -0.375085E-01   -0.341018E-01   -0.309530E-01   -0.280493E-01   -0.253781E-01   -0.229261E-01   -0.206804E-01   -0.186277E-01   -0.167554E-01   -0.150508E-01   -0.135020E-01   -0.120972E-01   -0.108252E-01   -0.967542E-02   -0.863784E-02   -0.770294E-02   -0.686183E-02   -0.610619E-02   -0.542830E-02   -0.482096E-02   -0.427755E-02   -0.379192E-02   -0.335847E-02   -0.297203E-02   -0.262789E-02   -0.232174E-02   -0.204967E-02   -0.180812E-02   -0.159389E-02   -0.140405E-02   -0.123598E-02   -0.108733E-02   -0.955942E-03   -0.839915E-03   -0.737526E-03   -0.647239E-03   -0.567682E-03   -0.497635E-03   -0.435993E-03   -0.381792E-03   -0.334171E-03   -0.292337E-03   -0.255610E-03   -0.223392E-03   -0.195151E-03   -0.170403E-03   -0.148716E-03   -0.129740E-03   -0.113148E-03   -0.986547E-04   -0.860019E-04   -0.749451E-04   -0.652787E-04   -0.568350E-04   -0.494574E-04   -0.430131E-04   -0.374022E-04   -0.325340E-04   -0.283150E-04   -0.246179E-04   -0.213931E-04   -0.185951E-04   -0.161713E-04   -0.140321E-04   -0.121533E-04   -0.104797E-04   -0.904334E-05   -0.781836E-05   -0.682838E-05   -0.598982E-05   -0.524063E-05   -0.450978E-05   -0.383643E-05   -0.328506E-05   -0.286205E-05   -0.253187E-05   -0.225367E-05   -0.197206E-05   -0.165623E-05   -0.138482E-05   -0.114997E-05   -0.100613E-05   -0.917929E-06   -0.838982E-06   -0.733348E-06   -0.605981E-06   -0.508698E-06   -0.495539E-06   -0.520636E-06   -0.516545E-06   -0.427010E-06   -0.323758E-06   -0.168948E-06   -0.201387E-07    0.950739E-07    0.639468E-07   -0.109454E-06   -0.302302E-06   -0.377335E-06   -0.307544E-06   -0.131344E-06    0.882937E-07    0.201471E-06    0.180424E-06    0.999003E-07   -0.310496E-08   -0.809513E-07   -0.155579E-06   -0.245021E-06   -0.277451E-06   -0.186742E-06   -0.262111E-07    0.121366E-06    0.158482E-06    0.111744E-06    0.177307E-07   -0.350893E-07   -0.501082E-07   -0.798734E-07   -0.182045E-06   -0.231279E-06   -0.128115E-06    0.185555E-07    0.779595E-07    0.655243E-07    0.285885E-07    0.417770E-07    0.351680E-07   -0.121301E-07   -0.310667E-07    0.350870E-07    0.762274E-07    0.821252E-08   -0.101799E-06   -0.121310E-06   -0.505222E-07   -0.811298E-08   -0.794643E-07   -0.188799E-06   -0.203767E-06   -0.151333E-06   -0.515614E-07    0.448378E-08    0.782089E-08    0.368623E-07    0.118643E-06    0.157492E-06    0.648431E-07   -0.101706E-06   -0.198837E-06   -0.204976E-06   -0.130042E-06   -0.253260E-07    0.101458E-06    0.181088E-06    0.113431E-06    0.127904E-07   -0.742844E-07   -0.964233E-07   -0.702264E-07   -0.230045E-07   -0.109932E-07   -0.257196E-07   -0.792881E-07   -0.121164E-06   -0.719695E-07   -0.282535E-07    0.190828E-07    0.183383E-07    0.424637E-07    0.877042E-07    0.138318E-06    0.926816E-07   -0.657306E-07   -0.201470E-06   -0.206133E-06   -0.485519E-07    0.908666E-07    0.154636E-06    0.138209E-06    0.221218E-07   -0.101907E-06   -0.206998E-06   -0.267027E-06   -0.209599E-06   -0.826628E-07    0.510236E-07    0.134728E-06    0.117194E-06    0.894221E-08   -0.105106E-06   -0.597877E-07    0.100727E-06    0.219950E-06    0.182759E-06    0.233682E-07   -0.144430E-06   -0.217592E-06   -0.169263E-06   -0.517343E-07    0.551317E-07    0.325186E-07    0.243087E-07    0.756872E-07    0.930857E-07    0.159314E-07   -0.128128E-06   -0.211501E-06   -0.169894E-06   -0.467291E-07\r\n",
      "   -0.694485E-07   -0.224378E-07    0.221401E-07    0.836066E-07    0.860432E-07    0.491671E-07   -0.669545E-08   -0.696102E-07   -0.129047E-06   -0.937561E-07    0.402438E-07    0.135540E-06    0.149598E-06    0.208053E-07   -0.161068E-06   -0.217021E-06   -0.105352E-06    0.484587E-07    0.120888E-06    0.537277E-07    0.118156E-08    0.269338E-08    0.931195E-07    0.139754E-06    0.513238E-07   -0.639400E-07   -0.106632E-06   -0.440832E-07    0.121775E-07   -0.229683E-07   -0.729073E-07   -0.524572E-07   -0.387171E-07   -0.106852E-06   -0.189536E-06   -0.221171E-06   -0.188663E-06   -0.621178E-07    0.894821E-07    0.193790E-06    0.212407E-06    0.115938E-06   -0.507050E-07   -0.202161E-06   -0.250866E-06   -0.177374E-06   -0.368849E-07    0.999744E-07    0.110361E-06    0.360437E-08   -0.841377E-07   -0.851415E-07    0.689834E-08    0.761140E-07    0.489977E-07   -0.331880E-07   -0.112899E-06   -0.152876E-06   -0.121838E-06   -0.460047E-07    0.559288E-08    0.813540E-08   -0.562482E-07   -0.131840E-06   -0.155442E-06   -0.987869E-07   -0.473705E-08    0.933745E-07    0.157866E-06    0.101932E-06   -0.387892E-07   -0.178805E-06   -0.190619E-06   -0.768422E-07   -0.705869E-09   -0.100708E-07    0.164751E-07    0.376901E-07    0.188341E-08   -0.651132E-07   -0.801101E-07   -0.477235E-07    0.157766E-07    0.685136E-07    0.717596E-07    0.342603E-07   -0.299430E-07   -0.311202E-07   -0.322710E-07   -0.767779E-07   -0.725369E-07    0.154624E-08    0.620740E-07   -0.231309E-07   -0.979338E-07   -0.789127E-07   -0.480766E-07   -0.909703E-07   -0.113284E-06   -0.448274E-07    0.607988E-08   -0.342468E-07   -0.158397E-06   -0.233235E-06   -0.166530E-06   -0.102502E-09    0.143710E-06    0.191821E-06    0.906804E-07   -0.960291E-07   -0.309744E-06   -0.415529E-06   -0.374299E-06   -0.230153E-06   -0.607787E-07    0.325874E-07    0.242335E-08   -0.104976E-06   -0.234904E-06   -0.372155E-06   -0.466619E-06   -0.451784E-06   -0.440951E-06   -0.492843E-06   -0.563039E-06   -0.664022E-06   -0.768607E-06   -0.879071E-06   -0.994832E-06   -0.113137E-05   -0.131484E-05   -0.154962E-05   -0.181444E-05   -0.206876E-05   -0.233224E-05   -0.264476E-05   -0.304648E-05   -0.358880E-05   -0.421438E-05   -0.491228E-05   -0.566852E-05   -0.652931E-05   -0.750234E-05   -0.864049E-05   -0.999422E-05   -0.115371E-04   -0.132961E-04   -0.153198E-04   -0.176468E-04   -0.203251E-04   -0.233983E-04   -0.269381E-04   -0.309954E-04   -0.356498E-04   -0.409827E-04   -0.471101E-04   -0.541450E-04   -0.622301E-04   -0.715006E-04   -0.821184E-04   -0.942863E-04   -0.108208E-03   -0.124133E-03   -0.142378E-03   -0.163257E-03   -0.187111E-03   -0.214352E-03   -0.245448E-03   -0.280933E-03   -0.321407E-03   -0.367525E-03   -0.420052E-03   -0.479862E-03   -0.547904E-03   -0.625257E-03   -0.713146E-03   -0.812933E-03   -0.926145E-03   -0.105450E-02   -0.119991E-02   -0.136452E-02   -0.155070E-02   -0.176111E-02   -0.199866E-02   -0.226663E-02   -0.256862E-02   -0.290861E-02   -0.329099E-02   -0.372060E-02   -0.420273E-02   -0.474317E-02   -0.534827E-02   -0.602492E-02   -0.678059E-02   -0.762339E-02   -0.856206E-02   -0.960599E-02   -0.107652E-01   -0.120505E-01   -0.134733E-01   -0.150455E-01   -0.167799E-01   -0.186896E-01   -0.207886E-01   -0.230909E-01   -0.256113E-01   -0.283645E-01   -0.313655E-01   -0.346293E-01   -0.381705E-01   -0.420036E-01   -0.461421E-01   -0.505989E-01   -0.553857E-01   -0.605128E-01   -0.659887E-01   -0.718200E-01   -0.780110E-01   -0.845631E-01   -0.914752E-01   -0.987426E-01   -0.106357E+00   -0.114307E+00   -0.122577E+00   -0.131146E+00   -0.139990E+00   -0.149080E+00   -0.158387E+00   -0.167874E+00   -0.177481E+00   -0.187163E+00   -0.196930E+00   -0.206713E+00   -0.216406E+00   -0.225977E+00   -0.235359E+00   -0.244496E+00   -0.253333E+00   -0.261810E+00   -0.269868E+00   -0.277451E+00   -0.284502E+00   -0.290960E+00   -0.296767E+00   -0.301891E+00   -0.306333E+00   -0.310082E+00   -0.313062E+00   -0.315028E+00   -0.315914E+00   -0.316116E+00   -0.315914E+00   -0.315028E+00   -0.313062E+00   -0.310082E+00   -0.306333E+00   -0.301891E+00   -0.296767E+00   -0.290960E+00   -0.284502E+00   -0.277451E+00   -0.269868E+00   -0.261810E+00   -0.253333E+00   -0.244496E+00   -0.235359E+00   -0.225977E+00   -0.216406E+00   -0.206713E+00   -0.196930E+00   -0.187163E+00   -0.177481E+00   -0.167874E+00   -0.158387E+00   -0.149080E+00   -0.139990E+00   -0.131146E+00   -0.122577E+00   -0.114307E+00   -0.106357E+00   -0.987426E-01   -0.914752E-01   -0.845631E-01   -0.780110E-01   -0.718200E-01   -0.659887E-01   -0.605128E-01   -0.553857E-01   -0.505989E-01   -0.461421E-01   -0.420036E-01   -0.381705E-01   -0.346293E-01   -0.313655E-01   -0.283645E-01   -0.256113E-01   -0.230909E-01   -0.207886E-01   -0.186896E-01   -0.167799E-01   -0.150455E-01   -0.134733E-01   -0.120505E-01   -0.107652E-01   -0.960599E-02   -0.856206E-02   -0.762339E-02   -0.678059E-02   -0.602492E-02   -0.534827E-02   -0.474317E-02   -0.420273E-02   -0.372060E-02   -0.329099E-02   -0.290861E-02   -0.256862E-02   -0.226663E-02   -0.199866E-02   -0.176111E-02   -0.155070E-02   -0.136452E-02   -0.119991E-02   -0.105450E-02   -0.926144E-03   -0.812932E-03   -0.713146E-03   -0.625257E-03   -0.547904E-03   -0.479863E-03   -0.420054E-03   -0.367527E-03   -0.321411E-03   -0.280938E-03   -0.245452E-03   -0.214352E-03   -0.187105E-03   -0.163248E-03   -0.142373E-03   -0.124133E-03   -0.108200E-03   -0.942762E-04   -0.821162E-04   -0.715186E-04   -0.622636E-04   -0.541650E-04   -0.471019E-04   -0.409410E-04   -0.355945E-04   -0.309679E-04   -0.269117E-04   -0.233668E-04   -0.203025E-04   -0.176514E-04   -0.153232E-04   -0.132668E-04   -0.114652E-04   -0.990197E-05   -0.858666E-05   -0.750755E-05   -0.657503E-05   -0.574872E-05   -0.495997E-05   -0.425290E-05   -0.365134E-05   -0.316995E-05   -0.275030E-05   -0.241386E-05   -0.211714E-05   -0.181348E-05   -0.151020E-05   -0.125212E-05   -0.109924E-05   -0.101567E-05   -0.923766E-06   -0.800592E-06   -0.659055E-06   -0.563865E-06   -0.547127E-06   -0.607095E-06   -0.599770E-06   -0.501492E-06   -0.382448E-06   -0.248864E-06   -0.120767E-06    0.143295E-07    0.521366E-07   -0.853719E-07   -0.276198E-06   -0.390417E-06   -0.365009E-06   -0.172720E-06    0.607951E-07    0.200407E-06    0.205880E-06    0.121420E-06    0.230854E-07   -0.561785E-07   -0.160441E-06   -0.284954E-06   -0.341376E-06   -0.280063E-06   -0.874860E-07    0.860711E-07    0.165057E-06    0.163471E-06    0.724113E-07   -0.385884E-07   -0.649621E-07   -0.744428E-07   -0.140320E-06   -0.201436E-06   -0.121007E-06    0.140203E-07    0.493684E-07    0.896907E-08   -0.465355E-07   -0.396778E-07   -0.294461E-07   -0.620629E-07   -0.849923E-07   -0.135178E-07    0.332284E-07   -0.551766E-08   -0.847366E-07   -0.736400E-07   -0.262724E-08    0.416055E-07   -0.439332E-07   -0.162361E-06   -0.246960E-06   -0.262768E-06   -0.188281E-06   -0.983997E-07   -0.181027E-07    0.590619E-07    0.169370E-06    0.216881E-06    0.120618E-06   -0.292823E-07   -0.150621E-06   -0.218713E-06   -0.196034E-06   -0.112602E-06    0.404000E-07    0.173452E-06    0.120122E-06   -0.161206E-07   -0.111344E-06   -0.172712E-06   -0.178257E-06   -0.797215E-07    0.428559E-08    0.291667E-07    0.121652E-08   -0.450905E-07   -0.490066E-07   -0.640241E-07   -0.453675E-07   -0.103633E-07    0.400694E-07    0.107394E-06    0.141114E-06    0.104180E-06   -0.376312E-07   -0.158066E-06   -0.196057E-06   -0.108702E-06   -0.136346E-08    0.629010E-07    0.888952E-07    0.198508E-07   -0.963591E-07   -0.179076E-06   -0.221706E-06   -0.196567E-06   -0.133490E-06   -0.237550E-07    0.603392E-07    0.873948E-07    0.175689E-07   -0.729129E-07   -0.387151E-07    0.935526E-07    0.171287E-06    0.810731E-07   -0.726993E-07   -0.214405E-06   -0.229693E-06   -0.130410E-06   -0.593380E-08    0.714865E-07    0.233043E-07   -0.173834E-07    0.157161E-07    0.258325E-07   -0.301636E-07   -0.121584E-06   -0.168509E-06   -0.149967E-06   -0.771106E-07\r\n"
     ]
    }
   ],
   "source": [
    "!head output/v_00003"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this next and final singularity example, we get around the problem of needing to port MPI by using the same MPI that's in the container to launch the containers. The trick is this code, which comes at the end of the .bashrc. What it does is effectively replace our shell on the machine with an execution of bash inside the singularity image.\n",
    "\n",
    "<pre>\n",
    "# Put the full path to a singularity image in the file $HOME/sing.txt.\n",
    "if [ -r $HOME/work/sing.txt ]\n",
    "then\n",
    "    IMAGE=$(cat $HOME/work/sing.txt)\n",
    "fi\n",
    "if [ \"$IMAGE\" != \"\" ]\n",
    "then\n",
    "    if [ -r \"$IMAGE\" ]\n",
    "    then\n",
    "        # If the SINGULARITY_CONTAINER variable is set,\n",
    "        # then we are already in the container\n",
    "        if [ \"$SINGULARITY_CONTAINER\" = \"\" ]\n",
    "        then\n",
    "            # Switch to running inside singularity\n",
    "            exec singularity exec $IMAGE bash --login\n",
    "        fi\n",
    "    else\n",
    "        echo Could not read image file $IMAGE\n",
    "    fi\n",
    "fi\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!echo /home/jovyan/singu/funwave-tvd.img > ~/work/sing.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's no need to call singularity explicitly, as it's called by each invocation of .bashrc. Note that the funwave-tvd we are executing is the one from inside the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading input.txt...\n",
      "######################################################################## 100.0%\n",
      "REMOTE_COMMAND=rm -fr output && mpirun -np 2 /home/install/FUNWAVE-TVD/src/funwave_vessel && tar cvzf singout.tgz output\n",
      "REQUESTBIN_URL=https://requestbin.agaveapi.co/1131s6o1\n",
      "\n",
      " ** QUERY STRING FOR REQUESTBIN **\n",
      "https://requestbin.agaveapi.co/1131s6o1?inspect\n",
      "\n",
      "INPUTS={\"datafile\":\"agave://nectar-storage-stevenrbrandt/input.txt\"}\n",
      "JOB_FILE=job-remote-19597.txt\n",
      "Writing file `job-remote-19597.txt'\n",
      "OUTPUT=Successfully submitted job 7597476498183155225-242ac11b-0001-007\n",
      "JOB_ID=7597476498183155225-242ac11b-0001-007\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PENDING\n",
      "STAT=PROCESSING_INPUTS\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=STAGED\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=SUBMITTING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=RUNNING\n",
      "STAT=FINISHED\n",
      "CMD=jobs-output-get 7597476498183155225-242ac11b-0001-007 fork-command-1.out\n",
      "All done! Output follows.\n",
      "Reading file `fork-command-1.out'\n",
      "======================================================================\n",
      " BATHY_CORRECTION DOES NOT EXIST. USE DEFAULT VALUE\n",
      " INI_UVZ DOES NOT EXIST. USE DEFAULT VALUE\n",
      " WaterLevel DOES NOT EXIST. USE DEFAULT VALUE\n",
      " ETA_LIMITER DOES NOT EXIST. USE DEFAULT VALUE\n",
      " EqualEnergy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " PERIODIC DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DIFFUSION_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DIRECT_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " FRICTION_SPONGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OBSTACLE_FILE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " BREAKWATER_FILE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " DISPERSION DOES NOT EXIST. USE DEFAULT VALUE\n",
      "         DISPERSION Default:  DISPERSION\n",
      " Gamma1 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "        Gamma1 Default:  1.0: DISPERSION\n",
      " Gamma2 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "              Gamma2 Default:  1.0: Full nonlinear\n",
      " Beta_ref DOES NOT EXIST. USE DEFAULT VALUE\n",
      "               Beta_ref Default:  -0.531\n",
      " Gamma3 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                      Gamma3 Default:  1.0: NOT fully linear\n",
      " VISCOSITY_BREAKING DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                   VISCOSITY_BREAKING Default:  SWE Breaking\n",
      " SWE_ETA_DEP DOES NOT EXIST. USE DEFAULT VALUE\n",
      "               SWE_ETA_DEP Default:  0.8\n",
      " Friction_Matrix DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             Friction_Matrix Default:  constant Cd\n",
      " Cd DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                  Cd_fixed Default:  0.0\n",
      " Time_Scheme DOES NOT EXIST. USE DEFAULT VALUE\n",
      "       Time_Scheme Default:  Runge_Kutta\n",
      " CONSTRUCTION DOES NOT EXIST. USE DEFAULT VALUE\n",
      " HIGH_ORDER DOES NOT EXIST. USE DEFAULT VALUE\n",
      "  HIGH_ORDER                     NOT DEFINED, USE FOURTH-ORDER\n",
      " CFL DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                       CFL Default:  0.5\n",
      " DT_fixed DOES NOT EXIST. USE DEFAULT VALUE\n",
      " FroudeCap DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                 FroudeCap Default:  3.0\n",
      " MinDepth DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                MinDepth Default:  0.1 m\n",
      " MinDepthFrc DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             MinDepthFrc Default:  0.1 m\n",
      " SHOW_BREAKING DOES NOT EXIST. USE DEFAULT VALUE\n",
      "            SHOW_BREAKING Default:  TRUE\n",
      " Cbrk1 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    Cbrk1 Default:  0.65\n",
      " Cbrk2 DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    Cbrk2 Default:  0.35\n",
      " WAVEMAKER_Cbrk DOES NOT EXIST. USE DEFAULT VALUE\n",
      "            WAVEMAKER_Cbrk Default:  1.0\n",
      " WAVEMAKER_VIS DOES NOT EXIST. USE DEFAULT VALUE\n",
      "           WAVEMAKER_VIS Default:  FALSE\n",
      " T_INTV_mean DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             T_INTV_mean Default:  LARGE\n",
      " STEADY_TIME DOES NOT EXIST. USE DEFAULT VALUE\n",
      "             STEADY_TIME Default:  LARGE\n",
      " C_smg DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                     C_smg Default:  0.0\n",
      " nu_bkg DOES NOT EXIST. USE DEFAULT VALUE\n",
      "                    nu_bkg Default:  0.0\n",
      " FIELD_IO_TYPE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " NumberStations DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OUTPUT_RES DOES NOT EXIST. USE DEFAULT VALUE\n",
      "         OUTPUT_RES NOT FOUND, OUTPUT_RES=1: full resolution\n",
      " DEPTH_OUT DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Hmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Hmin DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Umax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MFmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " VORmax DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MASK DOES NOT EXIST. USE DEFAULT VALUE\n",
      " MASK9 DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Umean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Vmean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " ETAmean DOES NOT EXIST. USE DEFAULT VALUE\n",
      " WaveHeight DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SXL DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SXR DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SYL DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SYR DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SourceX DOES NOT EXIST. USE DEFAULT VALUE\n",
      " SourceY DOES NOT EXIST. USE DEFAULT VALUE\n",
      " P DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Q DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Fx DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Fy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Gx DOES NOT EXIST. USE DEFAULT VALUE\n",
      " Gy DOES NOT EXIST. USE DEFAULT VALUE\n",
      " AGE DOES NOT EXIST. USE DEFAULT VALUE\n",
      " OUT_NU DOES NOT EXIST. USE DEFAULT VALUE\n",
      " TMP DOES NOT EXIST. USE DEFAULT VALUE\n",
      "      EtaBlowVal Default:  100xmax_depth\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.4428E-01  0.4428E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1937E+04  0.1247E+08  0.2996E+01 -0.4596E-06  0.4050E-01  0.4050E-01\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.4050E-01  0.1066E+02  0.3800E-02\n",
      "   PRINTING FILE NO.     1  TIME/TOTAL:        0.044 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.1017E+01  0.4414E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1927E+04  0.1246E+08  0.1900E+01 -0.1727E-05  0.7329E+00  0.7325E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.7338E+00  0.1050E+02  0.6987E-01\n",
      "   PRINTING FILE NO.     2  TIME/TOTAL:        1.017 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.2037E+01  0.4473E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1906E+04  0.1246E+08  0.7393E+00 -0.2109E-05  0.9186E+00  0.9148E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.9215E+00  0.1024E+02  0.8998E-01\n",
      "   PRINTING FILE NO.     3  TIME/TOTAL:        2.037 /       3.000\n",
      " ----------------- STATISTICS ----------------\n",
      "  TIME        DT\n",
      "  0.3032E+01  0.4566E-01\n",
      "  MassVolume  Energy      MaxEta      MinEta      Max U       Max V \n",
      "  0.1890E+04  0.1246E+08  0.7188E+00 -0.9413E+00  0.7352E+00  0.6932E+00\n",
      "  MaxTotalU   PhaseS      Froude \n",
      "  0.7378E+00  0.1019E+02  0.7239E-01\n",
      "   PRINTING FILE NO.     4  TIME/TOTAL:        3.032 /       3.000\n",
      " Simulation takes   15.642075522999221      seconds\n",
      " Normal Termination!\n",
      "output/\n",
      "output/eta_00001\n",
      "output/v_00003\n",
      "output/u_00001\n",
      "output/eta_00003\n",
      "output/v_00001\n",
      "output/u_00002\n",
      "output/eta_00004\n",
      "output/eta_00002\n",
      "output/u_00003\n",
      "output/v_00004\n",
      "output/v_00002\n",
      "output/u_00004\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!files-upload -F input.txt -S ${AGAVE_STORAGE_SYSTEM_ID} ./\n",
    "r.runagavecmd(\n",
    "    \"rm -fr output && \"+\n",
    "    \"mpirun -np 2 /home/install/FUNWAVE-TVD/src/funwave_vessel && \"+\n",
    "    \"tar cvzf singout.tgz output\",\n",
    "    \"agave://${AGAVE_STORAGE_SYSTEM_ID}/input.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "######################################################################## 100.0%\n",
      "eta_00001  eta_00003  u_00001  u_00003\tv_00001  v_00003\n",
      "eta_00002  eta_00004  u_00002  u_00004\tv_00002  v_00004\n"
     ]
    }
   ],
   "source": [
    "!rm -fr output singout.tgz\n",
    "!jobs-output-get ${JOB_ID} singout.tgz\n",
    "!tar xzf singout.tgz\n",
    "!ls output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clean up so that we don't boot into the singularity image without intending to\n",
    "!rm -f ~/work/sing.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}