Hum0B0t3000_BertModel.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Hum0B0t3000 BertModel",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dataSci-rigo/hum0rB0T3000/blob/master/Hum0B0t3000_BertModel.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meO7ZaISZfZ1",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IhdjgZWAZ60C",
        "colab_type": "text"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wHQH4OCHZ9bq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkTLZ3I4_7c_",
        "colab_type": "text"
      },
      "source": [
        "# BERT End to End (Fine-tuning + Predicting) in 5 minutes with Cloud TPU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1wtjs1QDb3DX",
        "colab_type": "text"
      },
      "source": [
        "## Overview\n",
        "\n",
        "**BERT**, or **B**idirectional **E**mbedding **R**epresentations from **T**ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The academic paper can be found here: https://arxiv.org/abs/1810.04805.\n",
        "\n",
        "This Colab demonstates using a free Colab Cloud TPU to fine-tune sentence and sentence-pair classification tasks built on top of pretrained BERT models and \n",
        "run predictions on tuned model. The colab demonsrates loading pretrained BERT models from both [TF Hub](https://www.tensorflow.org/hub) and checkpoints.\n",
        "\n",
        "**Note:**  You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud \n",
        "Storage) bucket for this Colab to run.\n",
        "\n",
        "Please follow the [Google Cloud TPU quickstart](https://cloud.google.com/tpu/docs/quickstart) for how to create GCP account and GCS bucket. You have [$300 free credit](https://cloud.google.com/free/) to get started with any GCP product. You can learn more about Cloud TPU at https://cloud.google.com/tpu/docs.\n",
        "\n",
        "This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select **File > View on GitHub**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UjiKRZ_faLRy",
        "colab_type": "text"
      },
      "source": [
        "## Learning objectives\n",
        "\n",
        "In this notebook, you will learn how to train and evaluate a BERT model using TPU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ld-JXlueIuPH",
        "colab_type": "text"
      },
      "source": [
        "## Instructions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POkof5uHaQ_c",
        "colab_type": "text"
      },
      "source": [
        "<h3><a href=\"https://cloud.google.com/tpu/\"><img valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"></a>  &nbsp;&nbsp;Train on TPU</h3>\n",
        "\n",
        "   1. Create a Cloud Storage bucket for your TensorBoard logs at http://console.cloud.google.com/storage and fill in the BUCKET parameter in the \"Parameters\" section below.\n",
        " \n",
        "   1. On the main menu, click Runtime and select **Change runtime type**. Set \"TPU\" as the hardware accelerator.\n",
        "   1. Click Runtime again and select **Runtime > Run All** (Watch out: the \"Colab-only auth for this notebook and the TPU\" cell requires user input). You can also run the cells manually with Shift-ENTER."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdMmwCJFaT8F",
        "colab_type": "text"
      },
      "source": [
        "### Set up your TPU environment\n",
        "\n",
        "In this section, you perform the following tasks:\n",
        "\n",
        "*   Set up a Colab TPU running environment\n",
        "*   Verify that you are connected to a TPU device\n",
        "*   Upload your credentials to TPU to access your GCS bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "191zq3ZErihP",
        "colab_type": "code",
        "outputId": "cdafc937-8aaa-4e8d-f5c3-17d79883631b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 420
        }
      },
      "source": [
        "import datetime\n",
        "import json\n",
        "import os\n",
        "import pprint\n",
        "import random\n",
        "import string\n",
        "import sys\n",
        "import tensorflow as tf\n",
        "\n",
        "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n",
        "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "print('TPU address is', TPU_ADDRESS)\n",
        "\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "with tf.Session(TPU_ADDRESS) as session:\n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(session.list_devices())\n",
        "\n",
        "  # Upload credentials to TPU.\n",
        "  with open('/content/adc.json', 'r') as f:\n",
        "    auth_info = json.load(f)\n",
        "  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "  # Now credentials are set for all future sessions on this TPU.\n",
        "import pandas as pd\n"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "TPU address is grpc://10.6.80.130:8470\n",
            "TPU devices:\n",
            "[_DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:CPU:0, CPU, -1, 17864271032720771166),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 8628967968841390208),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 2631560272620451248),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 4452277740090177287),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 4235196487869766127),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 4432635597286159955),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 9120617361511650660),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 5462997468622014564),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 15039751751893603303),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 7025692466618791957),\n",
            " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 8589934592, 4291325289929364655)]\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZwvtRbOaQwbE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "reddit = pd.read_json('https://query.data.world/s/wtye65og2jr3slglflm5m2q6weu2kg')\n",
        "stupid_stuff = pd.read_json('https://query.data.world/s/3zcjma2uidkf6tzt6olm2o2cxf75x5')\n",
        "wocka = pd.read_json('https://query.data.world/s/4bamgqs4m5i7mlnz4la5wljzpxnok5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUBP35oCDmbF",
        "colab_type": "text"
      },
      "source": [
        "### Prepare and import BERT modules\n",
        "​\n",
        "With your environment configured, you can now prepare and import the BERT modules. The following step clones the source code from GitHub and import the modules from the source. Alternatively, you can install BERT using pip (!pip install bert-tensorflow)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7wzwke0sxS6W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "\n",
        "!test -d bert_repo || git clone https://github.com/google-research/bert bert_repo\n",
        "if not 'bert_repo' in sys.path:\n",
        "  sys.path += ['bert_repo']\n",
        "\n",
        "# import python modules defined by BERT\n",
        "import modeling\n",
        "import optimization\n",
        "import run_classifier\n",
        "import run_classifier_with_tfhub\n",
        "import tokenization\n",
        "import numpy as np\n",
        "\n",
        "# import tfhub \n",
        "import tensorflow_hub as hub"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRu1aKO1D7-Z",
        "colab_type": "text"
      },
      "source": [
        "### Prepare for training\n",
        "\n",
        "This next section of code performs the following tasks:\n",
        "\n",
        "*  Specify task and download training data.\n",
        "*  Specify BERT pretrained model\n",
        "*  Specify GS bucket, create output directory for model checkpoints and eval results.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tYkaAlJNfhul",
        "colab_type": "code",
        "outputId": "888f1284-87ed-4b73-edb3-205616454774",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "TASK = 'Jokes' #@param {type:\"string\"}\n",
        "assert TASK in ('MRPC', 'CoLA',\"Jokes\"), 'Only (MRPC, CoLA) are demonstrated here.'\n",
        "\n",
        "if TASK != \"Jokes\":\n",
        "\n",
        "  # Download glue data.\n",
        "  ! test -d download_glue_repo || git clone https://gist.github.com/60c2bdb54d156a41194446737ce03e2e.git download_glue_repo\n",
        "  !python download_glue_repo/download_glue_data.py --data_dir='glue_data' --tasks=$TASK\n",
        "\n",
        "  TASK_DATA_DIR = 'glue_data/' + TASK\n",
        "  print('***** Task data directory: {} *****'.format(TASK_DATA_DIR))\n",
        "  !ls $TASK_DATA_DIR\n",
        "\n",
        "BUCKET = 'humorbot3000' #@param {type:\"string\"}\n",
        "assert BUCKET, 'Must specify an existing GCS bucket name'\n",
        "OUTPUT_DIR = 'gs://{}/hum0rb0t/3000/{}'.format(BUCKET, TASK)\n",
        "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
        "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n",
        "\n",
        "# Available pretrained model checkpoints:\n",
        "#   uncased_L-12_H-768_A-12: uncased BERT base model\n",
        "#   uncased_L-24_H-1024_A-16: uncased BERT large model\n",
        "#   cased_L-12_H-768_A-12: cased BERT large model\n",
        "BERT_MODEL = 'uncased_L-24_H-1024_A-16' #@param {type:\"string\"}\n",
        "BERT_MODEL_HUB = 'https://tfhub.dev/google/bert_' + BERT_MODEL + '/1'"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***** Model output directory: gs://humorbot3000/hum0rb0t/3000/Jokes *****\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hcpfl4N2EdOk",
        "colab_type": "text"
      },
      "source": [
        "Now let's load tokenizer module from TF Hub and play with it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TylDOYd-6AH-",
        "colab_type": "code",
        "outputId": "693bf126-60ca-49ea-b54e-46b7c7d34cce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        }
      },
      "source": [
        "tokenizer = run_classifier_with_tfhub.create_tokenizer_from_hub_module(BERT_MODEL_HUB)\n",
        "tokenizer.tokenize(\"This here's an example of using the BERT tokenizer\")"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/tokenization.py:125: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/tokenization.py:125: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this',\n",
              " 'here',\n",
              " \"'\",\n",
              " 's',\n",
              " 'an',\n",
              " 'example',\n",
              " 'of',\n",
              " 'using',\n",
              " 'the',\n",
              " 'bert',\n",
              " 'token',\n",
              " '##izer']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zaANGk3aEKXa",
        "colab_type": "code",
        "outputId": "f8d0cdd7-a048-4da2-e0f2-13caa9e133f2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "print(reddit.shape)\n",
        "print(reddit.columns)\n",
        "print(reddit['title'].head(5))\n",
        "print(reddit['body'].head(5))\n",
        "print(reddit['score'].max())\n",
        "print(reddit['score'].mean())\n",
        "print(reddit['score'].median())\n",
        "print(len(reddit['score']))\n",
        "print(np.unique(reddit['score'],return_counts=True))\n",
        "print(len(reddit[reddit['score']>10]))\n",
        "print(len(reddit[reddit['score']>15])/len(reddit['score']))\n",
        "\n"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(194553, 4)\n",
            "Index(['body', 'id', 'score', 'title'], dtype='object')\n",
            "0     I hate how you cant even say black paint anymore\n",
            "1    What's the difference between a Jew in Nazi Ge...\n",
            "2                       I recently went to America....\n",
            "3    Brian raises his hand and says, “He’s in Heaven.”\n",
            "4    You hear about the University book store worke...\n",
            "Name: title, dtype: object\n",
            "0    Now I have to say \"Leroy can you please paint ...\n",
            "1    Pizza doesn't scream when you put it in the ov...\n",
            "2    ...and being there really helped me learn abou...\n",
            "3    A Sunday school teacher is concerned that his ...\n",
            "4    He got caught trying to sell the two books to ...\n",
            "Name: body, dtype: object\n",
            "48526\n",
            "118.22325535972203\n",
            "3.0\n",
            "194553\n",
            "(array([    0,     1,     2, ..., 39570, 45500, 48526]), array([61561, 19347, 13684, ...,     1,     1,     1]))\n",
            "58927\n",
            "0.2540438852137978\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bn0CSRn8RLHb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "red=reddit.copy(deep=True)\n",
        "reddit['score']=pd.np.digitize( reddit.score, bins = [15])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ugJQ_8iURVaF",
        "colab_type": "code",
        "outputId": "f7f575fa-8b5a-4992-e4a3-d4692ffffa8d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "pos=reddit[reddit['score']==1]\n",
        "neg=reddit[reddit['score']==0]\n",
        "neg=neg.sample(frac=0.333,random_state=200)\n",
        "print(len(neg),len(pos))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "47806 50990\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f9w2I5iNSJAy",
        "colab_type": "code",
        "outputId": "06a99d50-5299-4251-b296-3fa20b93d10e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#DataFrame.append(self, other, ignore_index=False, verify_integrity=False, sort=False) → 'DataFrame\n",
        "\n",
        "reddit=neg.append(pos,ignore_index=True,sort=True)\n",
        "np.unique(reddit['score'],return_counts=True)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([0, 1]), array([47806, 50990]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKQe2ObUWJRq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F3NQDGoyEB3n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class JokeProcessor(run_classifier.DataProcessor):\n",
        "  \"\"\"Processor for the Quora Question pair data set.\"\"\"\n",
        "\n",
        "  def get_train_examples(self, data_dir):\n",
        "    \"\"\"Reading train.tsv and converting to list of InputExample\"\"\"\n",
        "    return self._create_examples(data_dir[0], 'train')\n",
        "\n",
        "  def get_dev_examples(self, data_dir):\n",
        "    \"\"\"Reading dev.tsv and converting to list of InputExample\"\"\"\n",
        "    return self._create_examples(data_dir[1], 'dev')\n",
        "  \n",
        "  def get_test_examples(self, data_dir):\n",
        "    \"\"\"Reading train.tsv and converting to list of InputExample\"\"\"\n",
        "    return self._create_examples(\n",
        "        self._read_tsv(data_dir+'test.tsv'), 'test')\n",
        "  \n",
        "  def convert_examples(self, sentence_pairs):\n",
        "    \"\"\"Given question pairs, conevrting to list of InputExample\"\"\"\n",
        "    examples = []\n",
        "    for (i, qpair) in enumerate(sentence_pairs):\n",
        "      guid = \"predict-%d\" % (i)\n",
        "      # converting questions to utf-8 and creating InputExamples\n",
        "      text_a = tokenization.convert_to_unicode(qpair[3])\n",
        "      text_b = tokenization.convert_to_unicode(qpair[0])\n",
        "      label = int(qpair[2])\n",
        "      # We will add label  as 0, because None is not supported in converting to features\n",
        "      examples.append(\n",
        "          run_classifier.InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))\n",
        "    return examples\n",
        "\n",
        "  def get_predict_examples(self, sentence_pairs):\n",
        "    \"\"\"Given question pairs, conevrting to list of InputExample\"\"\"\n",
        "    examples = []\n",
        "    for (i, qpair) in enumerate(sentence_pairs):\n",
        "      guid = \"predict-%d\" % (i)\n",
        "      # converting questions to utf-8 and creating InputExamples\n",
        "      text_a = tokenization.convert_to_unicode(qpair[3])\n",
        "      text_b = tokenization.convert_to_unicode(qpair[0])\n",
        "      # We will add label  as 0, because None is not supported in converting to features\n",
        "      examples.append(\n",
        "          run_classifier.InputExample(guid=guid, text_a=text_a, text_b=text_b, label=0))\n",
        "    return examples\n",
        "  \n",
        "  def _create_examples(self, lines, set_type):\n",
        "    \"\"\"Creates examples for the training, dev and test sets.\"\"\"\n",
        "    examples = []\n",
        "    for (i, line) in lines.iterrows():\n",
        "      guid = \"%s-%d\" % (set_type, i)\n",
        "      if set_type=='test':\n",
        "        # removing header and invalid data\n",
        "        if i == 0 or len(line)!=4:\n",
        "          print(guid, line)\n",
        "          continue\n",
        "        text_a = tokenization.convert_to_unicode(line[3])\n",
        "        text_b = tokenization.convert_to_unicode(line[0])\n",
        "        label = int(line[2]) # We will use zero for test as convert_example_to_features doesn't support None\n",
        "      else:\n",
        "        # removing header and invalid data\n",
        "        if i == 0 or len(line)!=4:\n",
        "          print(guid, line)\n",
        "          continue\n",
        "        text_a = tokenization.convert_to_unicode(line[3])\n",
        "        text_b = tokenization.convert_to_unicode(line[0])\n",
        "        label = int(line[2])\n",
        "      examples.append(\n",
        "          run_classifier.InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))\n",
        "    return examples\n",
        "\n",
        "  def get_labels(self):\n",
        "    \"return class labels\"\n",
        "    return [0,1]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oqQ_LHyzcoYW",
        "colab_type": "text"
      },
      "source": [
        "Also we initilize our hyperprams, prepare the training data and initialize TPU config."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pYVYULZiKvUi",
        "colab_type": "code",
        "outputId": "ae9f6703-3621-4d72-c963-5dffc9051959",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "TRAIN_BATCH_SIZE = 32\n",
        "EVAL_BATCH_SIZE = 8\n",
        "PREDICT_BATCH_SIZE = 8\n",
        "LEARNING_RATE = 2e-5\n",
        "NUM_TRAIN_EPOCHS = 5.0\n",
        "MAX_SEQ_LENGTH = 128\n",
        "# Warmup is a period of time where hte learning rate \n",
        "# is small and gradually increases--usually helps training.\n",
        "WARMUP_PROPORTION = 0.1\n",
        "# Model configs\n",
        "SAVE_CHECKPOINTS_STEPS = 1000\n",
        "SAVE_SUMMARY_STEPS = 500\n",
        "\n",
        "processors = {\n",
        "  \"cola\": run_classifier.ColaProcessor,\n",
        "  \"jokes\":JokeProcessor,\n",
        "  \"mnli\": run_classifier.MnliProcessor,\n",
        "  \"mrpc\": run_classifier.MrpcProcessor,\n",
        "}\n",
        "processor = processors[TASK.lower()]()\n",
        "label_list = processor.get_labels()\n",
        "if TASK == \"Jokes\":\n",
        "  train=reddit.sample(frac=0.8,random_state=200) #random state is a seed value\n",
        "  test=reddit.drop(train.index)\n",
        "  print(\"length of train set \",len(train),\"length of test set \",len(test))\n",
        "  #train_test_examples = train_test_split(reddit, test_size=0.2)\n",
        "  TASK_DATA_DIR =[train,test]\n",
        "# Compute number of train and warmup steps from batch size\n",
        "train_examples = processor.get_train_examples(TASK_DATA_DIR)\n",
        "print(\"length of train examples set\",len(train_examples))\n",
        "num_train_steps = int(len(train_examples) / TRAIN_BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
        "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)\n",
        "\n",
        "# Setup TPU related config\n",
        "tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n",
        "NUM_TPU_CORES = 8\n",
        "ITERATIONS_PER_LOOP = 1000\n",
        "\n",
        "def get_run_config(output_dir):\n",
        "  return tf.contrib.tpu.RunConfig(\n",
        "    cluster=tpu_cluster_resolver,\n",
        "    model_dir=output_dir,\n",
        "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,\n",
        "    tpu_config=tf.contrib.tpu.TPUConfig(\n",
        "        iterations_per_loop=ITERATIONS_PER_LOOP,\n",
        "        num_shards=NUM_TPU_CORES,\n",
        "        per_host_input_for_training=tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2))\n"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "length of train set  79037 length of test set  19759\n",
            "train-0 body                                       Addadictomy.\n",
            "id                                               3un5n5\n",
            "score                                                 0\n",
            "title    What is called when a woman gets a sex change?\n",
            "Name: 0, dtype: object\n",
            "length of train examples set 79036\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3iFMeqLaSll",
        "colab_type": "text"
      },
      "source": [
        "# Fine-tune and Run Predictions on a pretrained BERT Model from TF Hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXyJRc0OIHEU",
        "colab_type": "text"
      },
      "source": [
        "This section demonstrates fine-tuning from a pre-trained BERT TF Hub module and running predictions.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nwcsdbLuIX2I",
        "colab_type": "code",
        "outputId": "2468afe6-c46c-4694-f779-4afe975a149e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        }
      },
      "source": [
        "# Force TF Hub writes to the GS bucket we provide.\n",
        "os.environ['TFHUB_CACHE_DIR'] = OUTPUT_DIR\n",
        "\n",
        "model_fn = run_classifier_with_tfhub.model_fn_builder(\n",
        "  num_labels=len(label_list),\n",
        "  learning_rate=LEARNING_RATE,\n",
        "  num_train_steps=num_train_steps,\n",
        "  num_warmup_steps=num_warmup_steps,\n",
        "  use_tpu=True,\n",
        "  bert_hub_module_handle=BERT_MODEL_HUB\n",
        ")\n",
        "\n",
        "estimator_from_tfhub = tf.contrib.tpu.TPUEstimator(\n",
        "  use_tpu=True,\n",
        "  model_fn=model_fn,\n",
        "  config=get_run_config(OUTPUT_DIR),\n",
        "  train_batch_size=TRAIN_BATCH_SIZE,\n",
        "  eval_batch_size=EVAL_BATCH_SIZE,\n",
        "  predict_batch_size=PREDICT_BATCH_SIZE,\n",
        ")\n"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fcb68cc1d08>) includes params argument, but params are not passed to Estimator.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fcb68cc1d08>) includes params argument, but params are not passed to Estimator.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://humorbot3000/hum0rb0t/3000/Jokes', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.6.80.130:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "isolate_session_state: true\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb6fb87da0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.6.80.130:8470', '_evaluation_master': 'grpc://10.6.80.130:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fcb6e99fc50>}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://humorbot3000/hum0rb0t/3000/Jokes', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.6.80.130:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "isolate_session_state: true\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb6fb87da0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.6.80.130:8470', '_evaluation_master': 'grpc://10.6.80.130:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fcb6e99fc50>}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MGVYj_rlcDhc",
        "colab_type": "text"
      },
      "source": [
        "At this point, you can now fine-tune the model, evaluate it, and run predictions on it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5U_c8s2AvhgL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Train the model\n",
        "def model_train(estimator):\n",
        "  print('MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...')\n",
        "  # We'll set sequences to be at most 128 tokens long.\n",
        "  train_features = run_classifier.convert_examples_to_features(\n",
        "      train_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  print('***** Started training at {} *****'.format(datetime.datetime.now()))\n",
        "  print('  Num examples = {}'.format(len(train_examples)))\n",
        "  print('  Batch size = {}'.format(TRAIN_BATCH_SIZE))\n",
        "  tf.logging.info(\"  Num steps = %d\", num_train_steps)\n",
        "  train_input_fn = run_classifier.input_fn_builder(\n",
        "      features=train_features,\n",
        "      seq_length=MAX_SEQ_LENGTH,\n",
        "      is_training=True,\n",
        "      drop_remainder=True)\n",
        "  estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
        "  print('***** Finished training at {} *****'.format(datetime.datetime.now()))\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q_OYLJpXvZFd",
        "colab_type": "code",
        "outputId": "05d31c11-c44e-4615-dada-12b39b0c2554",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        }
      },
      "source": [
        "# Train the model\n",
        "def model_train(estimator):\n",
        "  print('MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...')\n",
        "  # We'll set sequences to be at most 128 tokens long.\n",
        "  train_features = run_classifier.convert_examples_to_features(\n",
        "      train_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  print('***** Started training at {} *****'.format(datetime.datetime.now()))\n",
        "  print('  Num examples = {}'.format(len(train_examples)))\n",
        "  print('  Batch size = {}'.format(TRAIN_BATCH_SIZE))\n",
        "  tf.logging.info(\"  Num steps = %d\", num_train_steps)\n",
        "  train_input_fn = run_classifier.input_fn_builder(\n",
        "      features=train_features,\n",
        "      seq_length=MAX_SEQ_LENGTH,\n",
        "      is_training=True,\n",
        "      drop_remainder=True)\n",
        "  estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
        "  print('***** Finished training at {} *****'.format(datetime.datetime.now()))\n",
        "\n",
        "estimator_from_tfhub = tf.contrib.tpu.TPUEstimator(\n",
        "  use_tpu=True,\n",
        "  model_fn=model_fn,\n",
        "  config=get_run_config(OUTPUT_DIR),\n",
        "  train_batch_size=TRAIN_BATCH_SIZE,\n",
        "  eval_batch_size=EVAL_BATCH_SIZE,\n",
        "  predict_batch_size=PREDICT_BATCH_SIZE,\n",
        ")\n"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fcb68cc1d08>) includes params argument, but params are not passed to Estimator.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fcb68cc1d08>) includes params argument, but params are not passed to Estimator.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://humorbot3000/hum0rb0t/3000/Jokes', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.6.80.130:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "isolate_session_state: true\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb6e8ed3c8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.6.80.130:8470', '_evaluation_master': 'grpc://10.6.80.130:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fcb6e99fc50>}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://humorbot3000/hum0rb0t/3000/Jokes', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.6.80.130:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "isolate_session_state: true\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb6e8ed3c8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.6.80.130:8470', '_evaluation_master': 'grpc://10.6.80.130:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fcb6e99fc50>}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TRhzyk4_WD2n",
        "colab_type": "code",
        "outputId": "a8963d83-4e36-4397-da0b-091bbc737057",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "model_train(estimator_from_tfhub)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...\n",
            "WARNING:tensorflow:From bert_repo/run_classifier.py:774: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier.py:774: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-30010\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-30010\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] pokemon go down south . [SEP] i ' ve never been happier to live in the bible belt . [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] pokemon go down south . [SEP] i ' ve never been happier to live in the bible belt . [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 20421 2175 2091 2148 1012 102 1045 1005 2310 2196 2042 19366 2000 2444 1999 1996 6331 5583 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 20421 2175 2091 2148 1012 102 1045 1005 2310 2196 2042 19366 2000 2444 1999 1996 6331 5583 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-39512\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-39512\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] noun : son of a b * * * h ! [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] noun : son of a b * * * h ! [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 15156 1024 2365 1997 1037 1038 1008 1008 1008 1044 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 15156 1024 2365 1997 1037 1038 1008 1008 1008 1044 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-49677\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-49677\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] what are the strongest days of the week ? [SEP] saturday and sunday , the rest are weekdays . i know , i know . . . even i ' m ashamed of myself for posting this ! [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] what are the strongest days of the week ? [SEP] saturday and sunday , the rest are weekdays . i know , i know . . . even i ' m ashamed of myself for posting this ! [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2054 2024 1996 10473 2420 1997 1996 2733 1029 102 5095 1998 4465 1010 1996 2717 2024 19759 1012 1045 2113 1010 1045 2113 1012 1012 1012 2130 1045 1005 1049 14984 1997 2870 2005 14739 2023 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2054 2024 1996 10473 2420 1997 1996 2733 1029 102 5095 1998 4465 1010 1996 2717 2024 19759 1012 1045 2113 1010 1045 2113 1012 1012 1012 2130 1045 1005 1049 14984 1997 2870 2005 14739 2023 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-86330\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-86330\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] what gets easier to pick up the heavier it gets ? [SEP] women . [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] what gets easier to pick up the heavier it gets ? [SEP] women . [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2054 4152 6082 2000 4060 2039 1996 11907 2009 4152 1029 102 2308 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2054 4152 6082 2000 4060 2039 1996 11907 2009 4152 1029 102 2308 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-30945\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: train-30945\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] the lord of a castle makes a deal with one of his nobles . [SEP] he tells him for three days whatever the lord hunts in the wilderness will be the noble ' s . whatever the noble can find is the lords . the deal is made , and they are off to work for the next three days . at night , the lord ' s wife starts to watch the noble and gains interest in him . she persuade ##s him to give her a kiss that night . the first day , the lord returns with a sc ##rum ##pt ##ious deer . the noble pays him back with a kiss . the lord laughs and says , \" where did you [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] the lord of a castle makes a deal with one of his nobles . [SEP] he tells him for three days whatever the lord hunts in the wilderness will be the noble ' s . whatever the noble can find is the lords . the deal is made , and they are off to work for the next three days . at night , the lord ' s wife starts to watch the noble and gains interest in him . she persuade ##s him to give her a kiss that night . the first day , the lord returns with a sc ##rum ##pt ##ious deer . the noble pays him back with a kiss . the lord laughs and says , \" where did you [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1996 2935 1997 1037 3317 3084 1037 3066 2007 2028 1997 2010 13969 1012 102 2002 4136 2032 2005 2093 2420 3649 1996 2935 28526 1999 1996 9917 2097 2022 1996 7015 1005 1055 1012 3649 1996 7015 2064 2424 2003 1996 8140 1012 1996 3066 2003 2081 1010 1998 2027 2024 2125 2000 2147 2005 1996 2279 2093 2420 1012 2012 2305 1010 1996 2935 1005 1055 2564 4627 2000 3422 1996 7015 1998 12154 3037 1999 2032 1012 2016 13984 2015 2032 2000 2507 2014 1037 3610 2008 2305 1012 1996 2034 2154 1010 1996 2935 5651 2007 1037 8040 6824 13876 6313 8448 1012 1996 7015 12778 2032 2067 2007 1037 3610 1012 1996 2935 11680 1998 2758 1010 1000 2073 2106 2017 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1996 2935 1997 1037 3317 3084 1037 3066 2007 2028 1997 2010 13969 1012 102 2002 4136 2032 2005 2093 2420 3649 1996 2935 28526 1999 1996 9917 2097 2022 1996 7015 1005 1055 1012 3649 1996 7015 2064 2424 2003 1996 8140 1012 1996 3066 2003 2081 1010 1998 2027 2024 2125 2000 2147 2005 1996 2279 2093 2420 1012 2012 2305 1010 1996 2935 1005 1055 2564 4627 2000 3422 1996 7015 1998 12154 3037 1999 2032 1012 2016 13984 2015 2032 2000 2507 2014 1037 3610 2008 2305 1012 1996 2034 2154 1010 1996 2935 5651 2007 1037 8040 6824 13876 6313 8448 1012 1996 7015 12778 2032 2067 2007 1037 3610 1012 1996 2935 11680 1998 2758 1010 1000 2073 2106 2017 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 10000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 10000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 20000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 20000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 30000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 30000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 40000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 40000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 50000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 50000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 60000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 60000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 70000 of 79036\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 70000 of 79036\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "***** Started training at 2020-02-04 18:54:39.734063 *****\n",
            "  Num examples = 79036\n",
            "  Batch size = 32\n",
            "INFO:tensorflow:  Num steps = 12349\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  Num steps = 12349\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Querying Tensorflow master (grpc://10.6.80.130:8470) for TPU system metadata.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Querying Tensorflow master (grpc://10.6.80.130:8470) for TPU system metadata.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 17864271032720771166)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 17864271032720771166)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 2631560272620451248)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 2631560272620451248)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 4452277740090177287)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 4452277740090177287)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 4235196487869766127)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 4235196487869766127)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 4432635597286159955)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 4432635597286159955)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 9120617361511650660)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 9120617361511650660)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 5462997468622014564)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 5462997468622014564)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 15039751751893603303)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 15039751751893603303)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 7025692466618791957)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 7025692466618791957)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 8589934592, 4291325289929364655)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 8589934592, 4291325289929364655)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 8628967968841390208)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 8628967968841390208)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Features ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Features ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = input_ids, shape = (4, 128)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = input_ids, shape = (4, 128)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = input_mask, shape = (4, 128)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = input_mask, shape = (4, 128)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = label_ids, shape = (4,)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = label_ids, shape = (4,)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = segment_ids, shape = (4, 128)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:  name = segment_ids, shape = (4, 128)\n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/embeddings/word_embeddings) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/embeddings/token_type_embeddings) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/embeddings/position_embeddings) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/embeddings/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/embeddings/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_0/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_1/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_2/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_3/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_4/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_5/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_6/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_7/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_8/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_9/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_10/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_11/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_12/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_13/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_14/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_15/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_16/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_17/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_18/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_19/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_20/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_21/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_22/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/self/query/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/self/query/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/self/key/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/self/key/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/self/value/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/self/value/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/attention/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/intermediate/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/intermediate/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/output/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/output/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/output/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/encoder/layer_23/output/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/pooler/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/bert/pooler/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/cls/predictions/transform/dense/kernel) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/cls/predictions/transform/dense/bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/cls/predictions/transform/LayerNorm/beta) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/cls/predictions/transform/LayerNorm/gamma) is not supported on the TPU. Execution will fail if this op is used in the graph. \n",
            "ERROR:absl:Operation of type Placeholder (module_apply_tokens/cls/predictions/output_bias) is not supported on the TPU. Execution will fail if this op is used in the graph. \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:62: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:62: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:69: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:69: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:72: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:72: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/optimization.py:27: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/optimization.py:27: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/optimization.py:32: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/optimization.py:32: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/optimization.py:70: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From bert_repo/optimization.py:70: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1375: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1375: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
            "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:TPU job name worker\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:TPU job name worker\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from gs://humorbot3000/hum0rb0t/3000/Jokes/model.ckpt-0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from gs://humorbot3000/hum0rb0t/3000/Jokes/model.ckpt-0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file utilities to get mtimes.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file utilities to get mtimes.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eoXRtSPZvdiS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def model_eval(estimator):\n",
        "  # Eval the model.\n",
        "  eval_examples = processor.get_dev_examples(TASK_DATA_DIR)\n",
        "  eval_features = run_classifier.convert_examples_to_features(\n",
        "      eval_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  print('***** Started evaluation at {} *****'.format(datetime.datetime.now()))\n",
        "  print('  Num examples = {}'.format(len(eval_examples)))\n",
        "  print('  Batch size = {}'.format(EVAL_BATCH_SIZE))\n",
        "\n",
        "  # Eval will be slightly WRONG on the TPU because it will truncate\n",
        "  # the last batch.\n",
        "  eval_steps = int(len(eval_examples) / EVAL_BATCH_SIZE)\n",
        "  eval_input_fn = run_classifier.input_fn_builder(\n",
        "      features=eval_features,\n",
        "      seq_length=MAX_SEQ_LENGTH,\n",
        "      is_training=False,\n",
        "      drop_remainder=True)\n",
        "  result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps)\n",
        "  print('***** Finished evaluation at {} *****'.format(datetime.datetime.now()))\n",
        "  output_eval_file = os.path.join(OUTPUT_DIR, \"eval_results.txt\")\n",
        "  with tf.gfile.GFile(output_eval_file, \"w\") as writer:\n",
        "    print(\"***** Eval results *****\")\n",
        "    for key in sorted(result.keys()):\n",
        "      print('  {} = {}'.format(key, str(result[key])))\n",
        "      writer.write(\"%s = %s\\n\" % (key, str(result[key])))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lLQehBr4WHjj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_eval(estimator_from_tfhub)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w7VEW5Y-bTkp",
        "colab_type": "text"
      },
      "source": [
        "## Prediction and Analysis:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h39vzCX5bQlp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "examples=[[ 'They played it on my flight home and there were only two walkouts.','4','1',\n",
        "\"I don't know why everyone is saying Cats (the movie) was bad.\"]]\n",
        "def model_predict_from(estimator,examples):\n",
        "  # Make predictions on a subset of eval examples\n",
        "  prediction_examples = processor.convert_examples(examples)\n",
        "  input_features = run_classifier.convert_examples_to_features(prediction_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=True)\n",
        "  predictions = estimator.predict(predict_input_fn)\n",
        "  rows=[]\n",
        "  for example, prediction in zip(prediction_examples, predictions):\n",
        "    row={'q1': str(example.text_a),'q2': str(example.text_b),'lbl': str(example.label),'pred': str(prediction['probabilities'])}\n",
        "    rows.append(row)\n",
        "  return rows \n",
        "def model_predict(estimator):\n",
        "  # Make predictions on a subset of eval examples\n",
        "  prediction_examples = processor.get_dev_examples(TASK_DATA_DIR)\n",
        "  input_features = run_classifier.convert_examples_to_features(prediction_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=True)\n",
        "  predictions = estimator.predict(predict_input_fn)\n",
        "  rows=[]\n",
        "  for example, prediction in zip(prediction_examples, predictions):\n",
        "    row={'q1': str(example.text_a),'q2': str(example.text_b),'lbl': str(example.label),'pred': str(prediction['probabilities'])}\n",
        "    rows.append(row)\n",
        "  return rows \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fsYT3Du3kCiu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data2 = model_predict_from(estimator_from_tfhub,examples)\n",
        "data2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Hguu936k73V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data=model_predict(estimator)\n",
        "SUB_TASK='jokes'\n",
        "postprocessing(data,[])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RT9fEaf9k-yW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9KeXwK73a3FG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "'''current_tasks=[\"QQP\"]\n",
        "current_tasks=[\"StackExchange\",'StackCola']\n",
        "predict_results=[]\n",
        "predict_setups=[]\n",
        "for SUB_TASK in current_tasks:\n",
        "  !gsutil -m cp gs://{BUCKET}/Final_Data/{SUB_TASK}/dev.tsv /content/Final_Data/{SUB_TASK}/dev.tsv\n",
        "  \n",
        "  TASK_PREDICT=\"QQP\" if SUB_TASK in [\"QQP\",'StackExchange','StackCola'] else \"PAWS\"\n",
        "  TASK_PREDICT_DIR = 'Final_Data/'+SUB_TASK+'/'\n",
        "  #TASK_PREDICT_DIR='/content'\n",
        "  processor = processors[TASK_PREDICT.lower()]()\n",
        "  label_list = processor.get_labels()\n",
        "  predict_result =model_predict(estimator_from_tfhub,TASK_PREDICT_DIR,processor)\n",
        "  predict_setup={'TASK_PREDICT':TASK_PREDICT+\"_\"+SUB_TASK}\n",
        "  predict_results.append(predict_result)\n",
        "  predict_setups.append(predict_setup)'''"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4WldFd2Fa8N4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def postprocessing(data,predict_set_up):\n",
        "  df = pd.DataFrame(data)\n",
        "  df['pred_lbl']=0\n",
        "  df['type_error']=\"None\"\n",
        "\n",
        "  df['lbl']=df['lbl'].astype(int)\n",
        "  preds=[]\n",
        "  for table in df.pred.values:\n",
        "\n",
        "    pred_row=json.loads(table.replace(\" \",',',1))\n",
        "    preds.append(pred_row)\n",
        "  preds=np.array(preds)\n",
        "  print(preds.shape)\n",
        "  df['first']=preds[:,0].astype(float)\n",
        "  df['second']=preds[:,1].astype(float)\n",
        "  df.loc[ ((df['first'])< (df['second'])),['pred_lbl']]= 1\n",
        "  df.loc[((df['lbl']==1) & (df['pred_lbl']==1)),['type_error']]=\"TP\" \n",
        "  df.loc[((df['lbl']==0) & (df['pred_lbl']==0)),['type_error']]=\"TN\" \n",
        "  df.loc[((df['lbl']==0) & (df['pred_lbl']==1)),['type_error']]=\"FP\" \n",
        "  df.loc[((df['lbl']==1) & (df['pred_lbl']==0)),['type_error']]=\"FN\" \n",
        "  filename=\"prediction_\"+BERT_MODEL+TASK+\"_\"+SUB_TASK+\".csv\"\n",
        "  df.to_csv(filename)\n",
        "  !gsutil  -m cp /content/{filename} gs://{BUCKET}/prediction_results/{filename}\n",
        "  error_acronyms=['TN','TP','FN','FP']\n",
        "  met_dict={error_type:df[df['type_error']==error_type] for error_type in error_acronyms}\n",
        "  met={error_type:met_dict[error_type].shape[0] for error_type in error_acronyms}\n",
        "\n",
        "  #Precision = TP/TP+FP\n",
        "  precision=met['TP']/(met['TP']+met['FP'])\n",
        "  #Recall = TP/TP+FN\n",
        "  recall=met['TP']/(met['TP']+met['FN'])\n",
        "  #F1 Score = 2*(Recall * Precision) / (Recall + Precision)\n",
        "  f1_score=2*(recall * precision) / (recall + precision)\n",
        "\n",
        "\n",
        "  accuracy=(met['TP']+met['TN'])/sum([met[m] for m in error_acronyms])\n",
        "  score_names=('accuracy','precision','recall', 'f1_score')\n",
        "  sc=(accuracy,precision,recall, f1_score)\n",
        "  scores={}\n",
        "  for name,score in zip(score_names,sc):\n",
        "    print(name, score)\n",
        "    scores={name:score}\n",
        "  out={**predict_set_up,**scores,**met}\n",
        "\n",
        "  e_a=['TN','TP','FN','FP']\n",
        "  errors=['True Negatives','True positives','False Negatives','False positives']\n",
        "  for i, error_type in zip(e_a,errors):\n",
        "    print('The number of '+error_type+' is: ',met_dict[i].shape[0])\n",
        "  print('using ',TASK, SUB_TASK)\n",
        "  print('Analysis of ',TASK, SUB_TASK)\n",
        "  print('Examples of False positives:')\n",
        "  FP=met_dict['FP'][['q1','q2']].head(15).to_numpy()\n",
        "  FN=met_dict['FN'][['q1','q2']].head(15).to_numpy()\n",
        "  for q in range(1,10):\n",
        "    if q<=FP.shape[0]:\n",
        "      print('Example: ',q)\n",
        "      print('Statement 1:',FP[q,0])\n",
        "      print('Statement 2:',FP[q,1])\n",
        "\n",
        "  print()  \n",
        "  print('Examples of False Negatives:')\n",
        "  for q in range(1,10):\n",
        "    if q<=FN.shape[0]:\n",
        "      print('Example: ',q)\n",
        "      print('Statement 1:',FN[q,0])\n",
        "      print('Statement 2:',FN[q,1])\n",
        "      print()\n",
        "  return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5-xDieNZa_a4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "postprocessing(predict_results[0],predict_setups[0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-ilQqS8bCbm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "import json\n",
        "import pandas as pd\n",
        "\n",
        "df = pd.DataFrame(data)\n",
        "df['pred_lbl']=0\n",
        "df['type_error']=\"None\"\n",
        "\n",
        "df['lbl']=df['lbl'].astype(int)\n",
        "preds=[]\n",
        "for table in df.pred.values:\n",
        "\n",
        "  pred_row=json.loads(table.replace(\" \",',',1))\n",
        "  preds.append(pred_row)\n",
        "preds=np.array(preds)\n",
        "print(preds.shape)\n",
        "df['first']=preds[:,0].astype(float)\n",
        "df['second']=preds[:,1].astype(float)\n",
        "df.loc[ ((df['first'])< (df['second'])),['pred_lbl']]= 1\n",
        "df.loc[((df['lbl']==1) & (df['pred_lbl']==1)),['type_error']]=\"TP\" \n",
        "df.loc[((df['lbl']==0) & (df['pred_lbl']==0)),['type_error']]=\"TN\" \n",
        "df.loc[((df['lbl']==0) & (df['pred_lbl']==1)),['type_error']]=\"FP\" \n",
        "df.loc[((df['lbl']==1) & (df['pred_lbl']==0)),['type_error']]=\"FN\" \n",
        "filename=\"prediction_\"+BERT_MODEL+\"_label_wiki.csv\"\n",
        "df.to_csv(filename)\n",
        "!gsutil  -m cp /content/{filename} gs://{BUCKET}/prediction_results/{filename}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RG8KQkGWbH6n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "FN=df[df['type_error']==\"FN\"]\n",
        "FP=df[df['type_error']==\"FP\"]\n",
        "print('The number of False positives is: ',FP[['q1','q2']].shape[0])\n",
        "print('The number of False Negatives is: ',FN[['q1','q2']].shape[0])\n",
        "\n",
        "print('Analysis of PAWS WIKI')\n",
        "print('Examples of False positives:')\n",
        "FP=FP[['q1','q2']].head(15).to_numpy()\n",
        "FN=FN[['q1','q2']].head(15).to_numpy()\n",
        "for q in range(1,10):\n",
        "  print('Example: ',q)\n",
        "  print('Statement 1:',FP[q,0])\n",
        "  print('Statement 2:',FP[q,1])\n",
        "\n",
        "print()  \n",
        "print('Examples of False Negatives:')\n",
        "for q in range(1,10):\n",
        "  print('Example: ',q)\n",
        "  print('Statement 1:',FN[q,0])\n",
        "  print('Statement 2:',FN[q,1])\n",
        "  print()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mml5ELUaajmp",
        "colab_type": "text"
      },
      "source": [
        "# Fine-tune and run predictions on a pre-trained BERT model from checkpoints"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xtgLSuh8IdGP",
        "colab_type": "text"
      },
      "source": [
        "Alternatively, you can also load pre-trained BERT models from saved checkpoints."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uu2dQ_TId-uH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Setup task specific model and TPU running config.\n",
        "BERT_PRETRAINED_DIR = 'gs://cloud-tpu-checkpoints/bert/' + BERT_MODEL \n",
        "print('***** BERT pretrained directory: {} *****'.format(BERT_PRETRAINED_DIR))\n",
        "!gsutil ls $BERT_PRETRAINED_DIR\n",
        "\n",
        "CONFIG_FILE = os.path.join(BERT_PRETRAINED_DIR, 'bert_config.json')\n",
        "INIT_CHECKPOINT = os.path.join(BERT_PRETRAINED_DIR, 'bert_model.ckpt')\n",
        "\n",
        "model_fn = run_classifier.model_fn_builder(\n",
        "  bert_config=modeling.BertConfig.from_json_file(CONFIG_FILE),\n",
        "  num_labels=len(label_list),\n",
        "  init_checkpoint=INIT_CHECKPOINT,\n",
        "  learning_rate=LEARNING_RATE,\n",
        "  num_train_steps=num_train_steps,\n",
        "  num_warmup_steps=num_warmup_steps,\n",
        "  use_tpu=True,\n",
        "  use_one_hot_embeddings=True\n",
        ")\n",
        "\n",
        "OUTPUT_DIR = OUTPUT_DIR.replace('bert-tfhub', 'bert-checkpoints')\n",
        "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
        "\n",
        "estimator_from_checkpoints = tf.contrib.tpu.TPUEstimator(\n",
        "  use_tpu=True,\n",
        "  model_fn=model_fn,\n",
        "  config=get_run_config(OUTPUT_DIR),\n",
        "  train_batch_size=TRAIN_BATCH_SIZE,\n",
        "  eval_batch_size=EVAL_BATCH_SIZE,\n",
        "  predict_batch_size=PREDICT_BATCH_SIZE,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yd61zFHgW2aN",
        "colab_type": "text"
      },
      "source": [
        "Now, you can repeat the training, evaluation, and prediction steps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bZp2_qt4VvED",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_train(estimator_from_checkpoints)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QIdXkglNVxM5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_eval(estimator_from_checkpoints)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0yamCRHcV-nQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_predict(estimator_from_checkpoints)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHjWhqAodM8a",
        "colab_type": "text"
      },
      "source": [
        "## What's next\n",
        "\n",
        "* Learn about [Cloud TPUs](https://cloud.google.com/tpu/docs) that Google designed and optimized specifically to speed up and scale up ML workloads for training and inference and to enable ML engineers and researchers to iterate more quickly.\n",
        "* Explore the range of [Cloud TPU tutorials and Colabs](https://cloud.google.com/tpu/docs/tutorials) to find other examples that can be used when implementing your ML project.\n",
        "\n",
        "On Google Cloud Platform, in addition to GPUs and TPUs available on pre-configured [deep learning VMs](https://cloud.google.com/deep-learning-vm/),  you will find [AutoML](https://cloud.google.com/automl/)*(beta)* for training custom models without writing code and [Cloud ML Engine](https://cloud.google.com/ml-engine/docs/) which will allows you to run parallel trainings and hyperparameter tuning of your custom models on powerful distributed hardware.\n"
      ]
    }
  ]
}