validate_and_analyze_corpus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we go through some basic validation and analysis of data in ELAN files. The same can be adapted to other files and data sources – once we have it in a dictionary or data frame the things are similar. There is also a data format called \"feather\" that can be used to save data in a tabular structure that can be read to both Python and R data frames.\n",
    "\n",
    "First we load some libraries. Pympi can be installed with `pip install pympi-ling`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import pympi\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example removes a bunch of tiers from an ELAN file. We do not check if they have content or not, so this is not the most delicate appoach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing unknown version of ELAN spec... This could result in errors...\n"
     ]
    }
   ],
   "source": [
    "elan_path = \"corpus/kpv_csys19570000-291_1a-Mezador.eaf\"\n",
    "\n",
    "tier_regex = r\".?(UPA|CYR|LAT|IPA|word|morph|ft-deu|ft-nob|ft-swe|ft-ling|ft-fin|lemma|pos|lg\\(word\\)|note\\(word\\))@.?\"\n",
    "\n",
    "elan = pympi.Elan.Eaf(elan_path)\n",
    "\n",
    "tiers = elan.get_tier_names()\n",
    "\n",
    "for tier in list(tiers):\n",
    "        \n",
    "    if re.findall(tier_regex, tier):\n",
    "\n",
    "        elan.remove_tier(tier)\n",
    "        \n",
    "elan.to_file(elan_path.replace(\"corpus\", \"corpus_clean\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we create a list that has all ELAN files in our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "elan_corpus = []\n",
    "\n",
    "for file in Path(\"corpus\").glob(\"*eaf\"):\n",
    "    \n",
    "    elan_corpus.append(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('corpus/kpv_uvyc19570000-290_3bz-Pop.eaf'),\n",
       " PosixPath('corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf'),\n",
       " PosixPath('corpus/kpv_csys19570000-291_1a-Mezador.eaf'),\n",
       " PosixPath('corpus/kpv_vym19570000-290_3a-Soska.eaf'),\n",
       " PosixPath('corpus/kpv_csys19611213-1329_2az-Kunib.eaf'),\n",
       " PosixPath('corpus/kpv_izva19570000-290_3bz-Bakur.eaf')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elan_corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We loop through those files, and read every tier of type `orthT`. We collect this data into a dictionary, and pick all other information that we have in the filename and participant naming conventions. Here the filenames contain a language and a dialect tag, i.e. kpv_udo means Udora dialect of Komi. And the names are of type NTP-M-1986, meaning Niko Tapio Partanen, male, born in 1986."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing unknown version of ELAN spec... This could result in errors...\n",
      "Parsing unknown version of ELAN spec... This could result in errors...\n",
      "Parsing unknown version of ELAN spec... This could result in errors...\n",
      "Parsing unknown version of ELAN spec... This could result in errors...\n"
     ]
    }
   ],
   "source": [
    "elan_data = []\n",
    "\n",
    "for file in elan_corpus:\n",
    "    \n",
    "    elan = pympi.Elan.Eaf(file)\n",
    "    \n",
    "    tiers = elan.get_tier_ids_for_linguistic_type(\"orthT\")\n",
    "    \n",
    "    for tier in tiers:\n",
    "        \n",
    "        annotations = elan.get_annotation_data_for_tier(tier)\n",
    "        \n",
    "        for annotation in annotations:\n",
    "            \n",
    "            a = {}\n",
    "            \n",
    "            a['start_ms'] = annotation[0]\n",
    "            a['end_ms'] = annotation[1]\n",
    "            a['utterance'] = annotation[2]\n",
    "            a['reference'] = annotation[3]\n",
    "            a['participant'] = tier.split(\"@\")[1]\n",
    "            a['gender'] = a['participant'].split(\"-\")[1]\n",
    "            a['birthyear'] = a['participant'].split(\"-\")[2]\n",
    "            a['dialect'] = re.search(r\"(?<=kpv_)(izva|csys|izva|udo|uvyc|vym)\", str(file)).group(0)\n",
    "            a['filename'] = str(file)\n",
    "            \n",
    "            elan_data.append(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'start_ms': 1765,\n",
       " 'end_ms': 8825,\n",
       " 'utterance': 'Ме чужи Удора районын, Усть-Вачерга сиктын, коді сулалӧ Вашка ю дорын.',\n",
       " 'reference': 'kpv_udo19570000-290_3a-01',\n",
       " 'participant': 'XUV-F-1920',\n",
       " 'gender': 'F',\n",
       " 'birthyear': '1920',\n",
       " 'dialect': 'udo',\n",
       " 'filename': 'corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elan_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is how the annotations look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1765 8825 Ме чужи Удора районын, Усть-Вачерга сиктын, коді сулалӧ Вашка ю дорын.\n",
      "17551 21030 Миян зэв, природаыс миян зэв мича.\n",
      "21416 23873 Гӧгӧр сулалӧны яг.\n",
      "24275 26053 Сиктсӧ ягӧн гӧгӧртӧма.\n",
      "27708 31646 И юыс миян сэтшӧм визюв мый\n",
      "32136 37646 ю вылас кӧ пыжӧн сынан сразу пыр ӧтнад кӧ сынан ӧтнад он вермы мыйкӧ керны, сразу нуыштас кытчӧ кӧ.\n",
      "38270 40203 И если кӧ,\n",
      "41586 44076 на пример ӧтчыд мӧдӧдчим\n",
      "46886 51081 Йилемъяскӧд мӧдӧдчим пужӧн льӧм вотны.\n",
      "51081 53745 Мунім, кыытім, кыытім, юӧдіс\n",
      "53745 56833 и друг сэтшӧм виам воис\n",
      "56833 60810 мый миянлысь пыжнымӧс бергӧдіс и ставным усим юас.\n",
      "60810 64316 А оказывайтся абу вӧлӧма йир и\n",
      "64316 68660 ми гортӧдз эта берег дорӧдзыс\n",
      "68881 71573 йылан котырыс ӧдва и доберитчим.\n",
      "72596 73573 Сэсся\n",
      "75170 81305 миян колӧ мунны районнӧй чентрсяньыс Усть-Вачерга сиктӧдзыс квайтымын километр.\n",
      "81305 82626 квайтымын верс.\n",
      "82835 86938 И сэті туйыс зэв лёк, а ми велӧдчим\n",
      "87321 91741 Кослан, Косланын, помалім семилеткасӧ и велӧдчим\n",
      "91838 93505 Косланас, районнӧй чентрас.\n",
      "93685 95973 И мӧдам вӧлі и,\n",
      "96600 101615 кыз шок кодь ныдын эськама туйсӧ кыдз абуджыка прӧдитім.\n",
      "101615 104728 Мӧдӧдча гӧг-, туйыс зэв няйтӧсь.\n",
      "104728 109426 И пыр пӧшти вӧлнас оз ёна ветлӧ.\n",
      "109426 113691 И ветлӧдлам унджыкысӧ ветлӧдлім подӧн, мый ӧд\n",
      "113906 116553 йилӧмъясыдлӧн миян кокным ӧд ён.\n",
      "116553 118088 Ӧдйӧ вермам мунны.\n",
      "118483 119196 Сэсся,\n",
      "120585 127425 ӧтпыр ми мунім, мунім, да друг сэтшӧм пемыд лоис и мый нинӧм оз тыдав.\n",
      "127425 135338 Няйтыс, няйтас вӧлі, сэтшӧм няйтӧсь вӧлі мый подӧн кок вывті воӧ и.\n",
      "696 3168 Täällä on Keski-Sysolan murre.\n",
      "4155 4825 Кхм.\n",
      "6656 9356 Ме чужлі Сысола районса Межадор сиктын.\n",
      "9356 11291 Менӧ Конанова Александраӧн.\n",
      "11780 14796 Миян Межадор сикт сулалӧ Сыктыв ю дорын.\n",
      "15908 19870 Ме ӧні висьтала видз вылын уджала ӧтик лун йылысь.\n",
      "20703 23125 Миян колхозлӧн видзьяс ылынӧсь.\n",
      "23596 25181 Верст дас вит сайын.\n",
      "25863 29631 Асывнас медым мунны видзьяс вылӧ катӧны водз.\n",
      "29861 32116 Машинаяс мунӧны вит часын.\n",
      "32845 34795 Лэдзыны зэв гажа.\n",
      "35323 36871 Нывъяс сьылӧны.\n",
      "37110 38956 Машинаяс пу улъяс,\n",
      "38956 42603 машинаясӧ пу улъяс сідз и ӧтӧны юръясад.\n",
      "43796 48780 Асывнас воан видз вылӧ и уджалысьяс ытшкӧны нин.\n",
      "49741 54398 Сідз жӧ посни челядь уджалӧны видзьяс вылын.\n",
      "54526 56540 Кыскалӧны юръяс.\n",
      "57426 59116 Зэв гажа.\n",
      "60626 65465 Асылӧдз, асылӧдз петкӧдліс мича лун.\n",
      "65465 67811 Асывсянь петіс шонді.\n",
      "69105 71786 Коис югыръяссӧ.\n",
      "74976 76546 Пажым кадӧ\n",
      "77628 81860 вӧлі вывті жар и ми мунім купайтчыны.\n",
      "82290 85303 Воӧм бӧрын пӧрысьяс кутісны виччыны.\n",
      "85505 87503 Пыр пӧ тіянӧс жар босьтӧма.\n",
      "89471 92295 Видзӧдам да кыпӧтчӧ зэв сьӧд кымӧр.\n",
      "94386 97343 Регыд кымӧр каис небеста шӧрӧ,\n",
      "97343 101733 и заводитіс ывла-, ывлаӧ пемдыны.\n",
      "101941 105601 Регыд кымӧр тюпкис ставсӧ небестасӧ.\n",
      "106546 108628 Кутіс чорыда гымавны.\n",
      "109120 111016 Ёна югъявны.\n",
      "111811 114376 Ми пӧдругакӧд вӧлі ытшкам.\n",
      "114643 118795 Бригадир кутіс дравзыны, медым шыбитім косаяс.\n",
      "119000 121980 Тіянлы пӧ жаль кӧсаясыд, асьым пӧ абу жаль леса.\n",
      "122820 124693 Ми сэк косаяс жыбитім.\n",
      "124693 127446 И мӧдӧдчим котӧртны тшӧтш йӧз бӧрсяыс.\n",
      "128133 132200 Кыпӧдчи зэл ыджыд тӧл, заводитіс ставсӧ бергӧдчыны.\n",
      "132515 134203 Турунсӧ нӧбалӧ\n",
      "134508 135906 кытчӧ веськалӧ.\n",
      "136060 139723 Уна юръяс лэдзӧдіс ваӧдзӧ.\n",
      "141595 145860 Сэтшӧма кутіс зэрны мый ми пасьтӧг кӧтасим.\n",
      "146285 148876 Рытнас ӧдва воим гортӧдз.\n",
      "1406 3706 Ме быдми Шошка сиктын.\n",
      "4105 7066 Сіе карсяньыс кызь верст сайын.\n",
      "7665 9513 Аслам карсянь, Сыктывкарсянь.\n",
      "10775 18191 Ичӧтдырйи, школаын велӧдчигӧн на, нёльӧд клас бӧрын, витӧд клас бӧрын.\n",
      "18311 21458 Ми челядь пианыс вӧлі мӧс видзам гожӧмнас.\n",
      "21458 25157 Мӧскыс быдӧнлӧн эм миян сиктас, быд керкаын мӧскыс эм.\n",
      "25510 27865 Ми вӧлі ассьыным мӧснымӧс видзам.\n",
      "28398 31851 Мӧс видзанныс вӧлі нёль верс сайын кымын.\n",
      "32868 33642 Вот.\n",
      "33642 38190 Ми ӧтчыд мунім, сэтчӧ мӧсъяссӧ нуим.\n",
      "38731 41884 Асьным бӧр туй вылас каям, вӧдӧдіс.\n",
      "42140 51320 Восьлім ружъе, сьӧрысь, и туйӧдыс каям, мир туй вылас, паськыд туйсӧ кыдзи миян шуӧны, мир туйӧн.\n",
      "52003 56473 Каям, веськыд ладор бокас эм йӧр, кушин.\n",
      "57753 61605 Ме бара ружъеаыслы тулкниті бокас, видзӧдлі мися кӧчыс.\n",
      "61765 64375 Пукалӧ, зэв лӧсьыда, тыдалӧ, кӧч пукалӧ.\n",
      "64630 66071 Настояшшӧй кӧч и эм.\n",
      "66293 70763 Некытчӧ оз бергӧдчы, нинӧм, зэв лӧсьыда пукалӧ сэтӧн и кӧть бось да лый.\n",
      "71243 74533 Сійӧ водіс, метитчи зэв дыр.\n",
      "74948 79190 Метитчим, метитчим, ме сылы тулкеда бодас, мися лыи нин, лыи нин.\n",
      "79671 83815 И – эз лыи, абу навернӧ, шуӧ кӧч.\n",
      "84386 89291 Мунім неуна водзӧджык, видзӧдім, вӧлӧм ке мыр.\n",
      "89291 90851 Мыръяс муртса ловъяи.\n",
      "91211 96976 И сійӧ мырйыс дыркодь сэтӧн сулаліс, и век вӧлі вспоминайтам тайӧ случайсэ.\n",
      "97505 100205 Мӧдыс бара ю дорынӧсь вӧлім.\n",
      "100205 102360 Ю мӧдарас видзим мӧсъяссӧ.\n",
      "102725 104638 И рытланьыс нин.\n",
      "105388 108180 Сэтӧн и юыслӧн берегыс крут.\n",
      "108765 110625 Метраӧн дзыньйӧн судта кымын.\n",
      "111340 113646 Сэтшӧм крут, берегыс.\n",
      "113646 116870 И юсӧ кутіс ош вуджны.\n",
      "117973 122360 Ошкыс вуджис, кайнысӧ - сэті эськӧ некод оз вермы кайнысӧ да –\n",
      "122360 126694 первойсӧ заводитліс кайнысӧ берегӧдыс веськыда, вылъяс.\n",
      "127088 129543 И эз вермы, бузгысис бӧр баас усис.\n",
      "130195 133195 Сэсся ӧдзӧдчис да кайис.\n",
      "133195 136873 Мӧсъясыс миян ӧтилаӧ ставыс чукӧртчӧмаӧсь, баксӧны, горзӧны.\n",
      "136873 143625 А сійӧ на вылӧ эз и внимание обратит, немтор, прӧстӧ босьтіс да, уйкнитіс муніс вӧрас да и.\n",
      "1040 1380 Мм.\n",
      "4602 7423 Ме чужи Куниб деревняын.\n",
      "8058 12000 И, сійӧ деревняас зэв уна эмӧсь гӧраяс.\n",
      "12500 17500 Гӧраяс вывсяньыс мийӧ зэв частӧ иславлім.\n",
      "17500 20897 иславлім ми даддьӧн, лыжйӧн.\n",
      "22730 25423 Лыжйӧн зэв лӧсьыд иславны,\n",
      "26272 29333 эшшо кӧ гӧраыс дзудзыдджык.\n",
      "31000 33852 Дерт ми сэні лымнад ворслывлім.\n",
      "34743 37243 И унаторъяс эшшо вӧчлывлім.\n",
      "46640 49628 Иславны ме заводиті ичӧтсянь.\n",
      "49628 51923 И первоя классъяссянь жӧ\n",
      "52615 56294 ме вӧлі пӧчти первой местаясын асланым школаын.\n",
      "56448 58846 Участвуйтлім быд [?] вылын.\n",
      "59025 63397 И занима-, и босьтлывлімӧ местасӧ и [?]\n",
      "63666 66961 Ӧні мыйта мӧдӧд [?] лыжи кузя.\n",
      "67740 72653 Участвуйтлі сідз жӧ [?] [?] вылын.\n",
      "75205 77384 Кыдзи гожӧмын сідз и тӧлын.\n",
      "82282 87192 Ислалімӧ первӧйсӧ асланым районса вӧчӧм лыжъяс вылын.\n",
      "87192 90217 А сы бӧрын кодыр нин [?].\n",
      "90410 94380 То клинскӧй лыжъясӧн, кленӧй лыжъяс вылын.\n",
      "107113 108293 Эгӧ.\n",
      "109223 111151 Тоже дойдьнас иславлі вы.\n",
      "111840 112794 Ми эгӧ.\n",
      "178988 180565 А гожӧмнас\n",
      "181330 184488 уна, особенно мячӧн ми ворслывлім.\n",
      "185538 187388 Группӧвӧй играясӧн.\n",
      "190435 193692 Мийӧ ветлімӧ каколь босьтны совхозӧ.\n",
      "195166 201660 Миян бригадаа веськалісны уна, уна том йӧз.\n",
      "203720 207000 Сэні мийӧ быдӧн каколь босьтімӧ, выпоняйтім нормаяс.\n",
      "208282 210717 Какольсӧ босьтім вилаӧн.\n",
      "218080 223666 Зэв гажа вӧлі, особенно том йӧз дырйи.\n",
      "223666 230000 А сы бӧрын сылан и шойччан.\n",
      "230000 231487 Ас…\n",
      "231487 234820 Поводдяыс таво какольтӧ босьтны вӧлі зэв бур.\n",
      "43273 46345 \n",
      "79716 81290 \n",
      "95121 97778 \n",
      "98353 103273 \n",
      "103678 105088 \n",
      "105088 107318 А дядьнас, дойдьӧн ин излавлӧй?\n",
      "107905 109775 Ми ӧні дойдьӧн иславлімӧ.\n",
      "112330 113293 \n",
      "216717 217653 Гажа вӧлі?\n",
      "230065 231589 Поводдяыс бур вӧлі?\n",
      "23500 24420 Важӧн\n",
      "25060 27100 до, дореволучионнӧй воясын,\n",
      "27840 28680 Изьва районысь\n",
      "29160 31340 уна нужнӧй войтыр ветлісныс\n",
      "31760 34320 Печора кыытӧдӧ рӧбитны завӧдын.\n",
      "35280 39320 Печора кыытӧдын сэк завӧдыс вӧлі ӧтік часнӧй фирмалӧн.\n",
      "41160 47840 Тысяча ӧкмыссё даскыкӧт воын сы завӧдас рӧбитны муніс и менам ае.\n",
      "48690 51960 Сыа муніс ёрткӧд тӧлын.\n",
      "52900 54220 Муніныс ныа вӧлӧн.\n",
      "55200 62460 Тулыс сія жӧ тысяча ӧкмыс сё даскыкӧт воын, менӧ маме ыстіс ае дорӧ.\n",
      "63600 65480 Мен сэк вӧлі дасӧтік ар.\n",
      "66700 68740 Ме завӧдас муні пароходен.\n",
      "69600 71900 Пароход вылас пукси Изьва пристаньын.\n",
      "74600 81640 Пароход Изьва кузя кыытіс Печера вылӧдз, а сэсся Печера кузя завӧдӧдз.\n",
      "82860 83820 Завӧдас вӧлі…\n",
      "84900 87880 Завӧдас воӧм бӧрын ме аддзи аес.\n",
      "88720 92200 Сыа мукӧд рӧбӧчӧйяскед оліс баракын.\n",
      "93480 101580 Сэк ме медводз на аддзи, кудз олӧныс вӧлі сыа воясас, царскӧй влась дырйи, рӧбӧчӧйяс.\n",
      "102480 105240 Рӧбӧчӧйяс олӧныс вӧлі зэй лёка.\n",
      "106440 109980 Баракъясас вӧлі кык сода наръяс.\n",
      "111380 114420 Баракъясас вӧліныс гразнӧйӧсь зэй.\n",
      "115020 116260 Зэй уна вӧлі лудук.\n",
      "116880 126240 Даже сьӧкыд рӧбитӧм бӧрын, рӧбӧчӧйяс бура из вермыныс шойччыны-узьны войнас.\n",
      "130420 138580 Кор ме лои ыджыд нин, сыа жӧ завӧдас приве- приведитчис ветлыны рӧбитны и менум аслум.\n",
      "139900 141800 Сыа вӧлі нин револуция бӧрын.\n",
      "143460 148860 Рӧбӧчӧйяслӧн условие вӧлі уна бурджык, чем вӧлі важӧн.\n",
      "149860 155060 Рӧбӧчӧйяс олӧныс бур ыджыд керкаясын.\n",
      "155960 163300 Быд рӧбӧчӧйлы сетӧныс вӧлі койка, вольпась, и вӧлі зэй чистӧ оланінас.\n",
      "1320 4020 Täällä meillä on Semjaškin Kindei Marković,\n",
      "4560 7160 Bakur kylästä Ižmasta.\n"
     ]
    }
   ],
   "source": [
    "for annotation in elan_data:\n",
    "\n",
    "    print(annotation['start_ms'], annotation['end_ms'], annotation['utterance'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we test which transcriptions have non-allowed characters (from the point of view of Komi)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus/kpv_csys19570000-291_1a-Mezador.eaf 696 3168 Täällä on Keski-Sysolan murre.\n",
      "corpus/kpv_izva19570000-290_3bz-Bakur.eaf 1320 4020 Täällä meillä on Semjaškin Kindei Marković,\n",
      "corpus/kpv_izva19570000-290_3bz-Bakur.eaf 4560 7160 Bakur kylästä Ižmasta.\n"
     ]
    }
   ],
   "source": [
    "for annotation in elan_data:\n",
    "    \n",
    "    if re.match(r\"[^А-ЯЁӦІа-яёӧі,.!?…]\", annotation['utterance']):\n",
    "\n",
    "        print(annotation['filename'], annotation['start_ms'], annotation['end_ms'], annotation['utterance'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a list of empty annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus/kpv_csys19611213-1329_2az-Kunib.eaf 43273 46345\n",
      "corpus/kpv_csys19611213-1329_2az-Kunib.eaf 79716 81290\n",
      "corpus/kpv_csys19611213-1329_2az-Kunib.eaf 95121 97778\n",
      "corpus/kpv_csys19611213-1329_2az-Kunib.eaf 98353 103273\n",
      "corpus/kpv_csys19611213-1329_2az-Kunib.eaf 103678 105088\n",
      "corpus/kpv_csys19611213-1329_2az-Kunib.eaf 112330 113293\n"
     ]
    }
   ],
   "source": [
    "for annotation in elan_data:\n",
    "    \n",
    "    if not annotation['utterance']:\n",
    "\n",
    "        print(annotation['filename'], annotation['start_ms'], annotation['end_ms'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also visualize the material in this point. First we `pandas` package to convert the dictionary into a data frame, which is a different tabular structure. And we load also package seaborn, which is one of the many visualization packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "\n",
    "elan_df = pd.DataFrame.from_dict(elan_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_ms</th>\n",
       "      <th>end_ms</th>\n",
       "      <th>utterance</th>\n",
       "      <th>reference</th>\n",
       "      <th>participant</th>\n",
       "      <th>gender</th>\n",
       "      <th>birthyear</th>\n",
       "      <th>dialect</th>\n",
       "      <th>filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1765</td>\n",
       "      <td>8825</td>\n",
       "      <td>Ме чужи Удора районын, Усть-Вачерга сиктын, ко...</td>\n",
       "      <td>kpv_udo19570000-290_3a-01</td>\n",
       "      <td>XUV-F-1920</td>\n",
       "      <td>F</td>\n",
       "      <td>1920</td>\n",
       "      <td>udo</td>\n",
       "      <td>corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>17551</td>\n",
       "      <td>21030</td>\n",
       "      <td>Миян зэв, природаыс миян зэв мича.</td>\n",
       "      <td>kpv_udo19570000-290_3a-02</td>\n",
       "      <td>XUV-F-1920</td>\n",
       "      <td>F</td>\n",
       "      <td>1920</td>\n",
       "      <td>udo</td>\n",
       "      <td>corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21416</td>\n",
       "      <td>23873</td>\n",
       "      <td>Гӧгӧр сулалӧны яг.</td>\n",
       "      <td>kpv_udo19570000-290_3a-03</td>\n",
       "      <td>XUV-F-1920</td>\n",
       "      <td>F</td>\n",
       "      <td>1920</td>\n",
       "      <td>udo</td>\n",
       "      <td>corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24275</td>\n",
       "      <td>26053</td>\n",
       "      <td>Сиктсӧ ягӧн гӧгӧртӧма.</td>\n",
       "      <td>kpv_udo19570000-290_3a-04</td>\n",
       "      <td>XUV-F-1920</td>\n",
       "      <td>F</td>\n",
       "      <td>1920</td>\n",
       "      <td>udo</td>\n",
       "      <td>corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27708</td>\n",
       "      <td>31646</td>\n",
       "      <td>И юыс миян сэтшӧм визюв мый</td>\n",
       "      <td>kpv_udo19570000-290_3a-05</td>\n",
       "      <td>XUV-F-1920</td>\n",
       "      <td>F</td>\n",
       "      <td>1920</td>\n",
       "      <td>udo</td>\n",
       "      <td>corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>143460</td>\n",
       "      <td>148860</td>\n",
       "      <td>Рӧбӧчӧйяслӧн условие вӧлі уна бурджык, чем вӧл...</td>\n",
       "      <td>kpv_izva19570000-290_3bz-28</td>\n",
       "      <td>XXV-M-19XX</td>\n",
       "      <td>M</td>\n",
       "      <td>19XX</td>\n",
       "      <td>izva</td>\n",
       "      <td>corpus/kpv_izva19570000-290_3bz-Bakur.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>149860</td>\n",
       "      <td>155060</td>\n",
       "      <td>Рӧбӧчӧйяс олӧныс бур ыджыд керкаясын.</td>\n",
       "      <td>kpv_izva19570000-290_3bz-29</td>\n",
       "      <td>XXV-M-19XX</td>\n",
       "      <td>M</td>\n",
       "      <td>19XX</td>\n",
       "      <td>izva</td>\n",
       "      <td>corpus/kpv_izva19570000-290_3bz-Bakur.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>155960</td>\n",
       "      <td>163300</td>\n",
       "      <td>Быд рӧбӧчӧйлы сетӧныс вӧлі койка, вольпась, и ...</td>\n",
       "      <td>kpv_izva19570000-290_3bz-30</td>\n",
       "      <td>XXV-M-19XX</td>\n",
       "      <td>M</td>\n",
       "      <td>19XX</td>\n",
       "      <td>izva</td>\n",
       "      <td>corpus/kpv_izva19570000-290_3bz-Bakur.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>1320</td>\n",
       "      <td>4020</td>\n",
       "      <td>Täällä meillä on Semjaškin Kindei Marković,</td>\n",
       "      <td>kpv_izva19570000-290_3bz-01</td>\n",
       "      <td>EEI-M-1913</td>\n",
       "      <td>M</td>\n",
       "      <td>1913</td>\n",
       "      <td>izva</td>\n",
       "      <td>corpus/kpv_izva19570000-290_3bz-Bakur.eaf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>4560</td>\n",
       "      <td>7160</td>\n",
       "      <td>Bakur kylästä Ižmasta.</td>\n",
       "      <td>kpv_izva19570000-290_3bz-02</td>\n",
       "      <td>EEI-M-1913</td>\n",
       "      <td>M</td>\n",
       "      <td>1913</td>\n",
       "      <td>izva</td>\n",
       "      <td>corpus/kpv_izva19570000-290_3bz-Bakur.eaf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>183 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     start_ms  end_ms                                          utterance  \\\n",
       "0        1765    8825  Ме чужи Удора районын, Усть-Вачерга сиктын, ко...   \n",
       "1       17551   21030                 Миян зэв, природаыс миян зэв мича.   \n",
       "2       21416   23873                                 Гӧгӧр сулалӧны яг.   \n",
       "3       24275   26053                             Сиктсӧ ягӧн гӧгӧртӧма.   \n",
       "4       27708   31646                        И юыс миян сэтшӧм визюв мый   \n",
       "..        ...     ...                                                ...   \n",
       "178    143460  148860  Рӧбӧчӧйяслӧн условие вӧлі уна бурджык, чем вӧл...   \n",
       "179    149860  155060              Рӧбӧчӧйяс олӧныс бур ыджыд керкаясын.   \n",
       "180    155960  163300  Быд рӧбӧчӧйлы сетӧныс вӧлі койка, вольпась, и ...   \n",
       "181      1320    4020        Täällä meillä on Semjaškin Kindei Marković,   \n",
       "182      4560    7160                             Bakur kylästä Ižmasta.   \n",
       "\n",
       "                       reference participant gender birthyear dialect  \\\n",
       "0      kpv_udo19570000-290_3a-01  XUV-F-1920      F      1920     udo   \n",
       "1      kpv_udo19570000-290_3a-02  XUV-F-1920      F      1920     udo   \n",
       "2      kpv_udo19570000-290_3a-03  XUV-F-1920      F      1920     udo   \n",
       "3      kpv_udo19570000-290_3a-04  XUV-F-1920      F      1920     udo   \n",
       "4      kpv_udo19570000-290_3a-05  XUV-F-1920      F      1920     udo   \n",
       "..                           ...         ...    ...       ...     ...   \n",
       "178  kpv_izva19570000-290_3bz-28  XXV-M-19XX      M      19XX    izva   \n",
       "179  kpv_izva19570000-290_3bz-29  XXV-M-19XX      M      19XX    izva   \n",
       "180  kpv_izva19570000-290_3bz-30  XXV-M-19XX      M      19XX    izva   \n",
       "181  kpv_izva19570000-290_3bz-01  EEI-M-1913      M      1913    izva   \n",
       "182  kpv_izva19570000-290_3bz-02  EEI-M-1913      M      1913    izva   \n",
       "\n",
       "                                          filename  \n",
       "0    corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf  \n",
       "1    corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf  \n",
       "2    corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf  \n",
       "3    corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf  \n",
       "4    corpus/kpv_udo19570000-290_3a-Ust-Vacerga.eaf  \n",
       "..                                             ...  \n",
       "178      corpus/kpv_izva19570000-290_3bz-Bakur.eaf  \n",
       "179      corpus/kpv_izva19570000-290_3bz-Bakur.eaf  \n",
       "180      corpus/kpv_izva19570000-290_3bz-Bakur.eaf  \n",
       "181      corpus/kpv_izva19570000-290_3bz-Bakur.eaf  \n",
       "182      corpus/kpv_izva19570000-290_3bz-Bakur.eaf  \n",
       "\n",
       "[183 rows x 9 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elan_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEJCAYAAAByupuRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXAklEQVR4nO3de1BU993H8c8CCyJqFQWrqXXGW63iZcKoJSrEMUQMMiZA6yWpMfWSBLXWabxUTdp4jcYUHY1NbBzsY/LUiAIKYfBSja3gJZJWQ0xCOhEbnQQ0ihrEZYF9/rDuE6PCohwW/L1ff4Wzu2e/e8bse8/Z3bM2l8vlEgDASD7eHgAA4D1EAAAMRgQAwGBEAAAMRgQAwGB+3h7AU9XV1SorK5PdbpfNZvP2OADQJLhcLjmdTgUFBcnH59bX/U0mAmVlZSosLPT2GADQJPXo0UMtW7a8ZXmTiYDdbpd0/YH4+/t7eRoAaBoqKipUWFjofg79viYTgRuHgPz9/RUQEODlaQCgabnTYXTeGAYAgxEBADAYEQAAgxEBADAYEQAAgxEBADAYEQAAgxEB3FF1pdPbIzQabAvcr5rMl8XQ8Hz87MpfOdnbYzQK4XPe8vYIgCXYEwAAgxEBADAYEQAAgxEBADAYEQAAgxEBADAYEQAAgxEBADAYEQAAg1kagR07dig2NlaxsbFasWKFJOmTTz5RQkKCRowYoQULFqiystLKEQAANbAsAuXl5Vq6dKk2b96sHTt26NixY8rLy9Ps2bP14osvateuXXK5XNq6datVIwAAamFZBKqqqlRdXa3y8nJVVlaqsrJSfn5+unbtmvr37y9Jio+PV05OjlUjAABqYdkJ5Fq0aKGZM2dq5MiRatasmQYOHCi73a6QkBD3dUJCQlRcXGzVCACAWlgWgU8//VTbt2/X/v371bJlS73wwgvKzc295Xo2m61O6y0oKKivEVGL8PBwb4/QqOTn53t7BKDeWRaBgwcPKiIiQm3btpV0/dDPxo0bdf78efd1zp07p9DQ0DqtNywsTAEBAfU6K+AJooimyOFw1Pji2bL3BHr27Km8vDxdvXpVLpdL+/bt08CBAxUQEOB+RZWRkaHIyEirRgAA1MKyPYEhQ4bo5MmTio+Pl91uV58+fTR16lRFR0dr4cKFKisrU69evTRhwgSrRgAA1MLSXxabOnWqpk6detOynj17atu2bVbeLQDAQ3xjGAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMZmkE9u3bp/j4eMXExGjJkiWSpLy8PMXFxenRRx9VcnKylXcPAKiFZRH48ssv9fvf/17r169XZmamTp48qQMHDmj+/Plav369srOzVVBQoAMHDlg1AgCgFpZFYM+ePXrsscf0wx/+UHa7XcnJyQoMDFTnzp3VqVMn+fn5KS4uTjk5OVaNAACohZ9VKz59+rTsdrsmTZqkc+fOadiwYerevbtCQkLc1wkNDVVxcbFVIwAAamFZBKqqqnTs2DFt3rxZzZs3V1JSkgIDA2+5ns1mq9N6CwoK6mtE1CI8PNzbIzQq+fn53h4BqHeWRaBdu3aKiIhQcHCwJGn48OHKycmRr6+v+zolJSUKDQ2t03rDwsIUEBBQr7MCniCKaIocDkeNL54te09g2LBhOnjwoC5fvqyqqir94x//UExMjE6dOqXTp0+rqqpKWVlZioyMtGoEAEAtLNsT6NevnyZPnqzx48fL6XRq8ODBGjdunLp06aIZM2bI4XAoKipKMTExVo0AAKiFZRGQpMTERCUmJt60LCIiQjt37rTybgEAHuIbwwBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAYjAgBgMCIAAAbzKALFxcW3LPv3v/9d78MAABpWjREoLS1VaWmppkyZokuXLrn/Pn/+vJKSkhpqRgCARWr8ofnf/va3ys3NlSQNGjTo/2/k56dHHnnE2skAAJarMQIbN26UJP3ud7/T8uXLG2QgAEDDqTECNyxfvlxnz57VpUuX5HK53Mt79+5t2WAAAOt5FIFVq1Zp8+bNatu2rXuZzWbT3/72N8sGAwBYz6MIZGdna/fu3Wrfvr3V8wAAGpBHHxHt0KEDAQCA+5BHewIRERFauXKlhg8frmbNmrmX854AADRtHkUgLS1NkpSTk+NexnsCAND0eRSBffv2WT0HAMALPIpASkrKbZc/88wz9ToMAKBheRSBwsJC939XVFQoPz//pm8QAwCaJo+/LPZdFy5c0Jw5cywZCADQcO7qVNLBwcE6e/Zsfc8CAGhgdX5PwOVyqaCg4KZvDwMAmqY6vycgXf/yGIeDAKDpq9N7AmfPnlVlZaU6d+5s6VAAgIbhUQROnz6tpKQklZSUqLq6Wm3atNGbb76prl27Wj0fAMBCHr0xvGjRIk2ePFkffPCB8vPz9fzzz+vll1+2ejYAgMU8isA333yjJ554wv13QkKCLl68aNlQAICG4VEEqqqqVFpa6v77woULHt/BihUrNG/ePEnSJ598ooSEBI0YMUILFixQZWVl3aYFANQrjyLw1FNPacyYMVq9erVWr16tcePGady4cbXe7tChQ0pPT3f/PXv2bL344ovatWuXXC6Xtm7deveTAwDumUcRiIqKkiQ5nU598cUXKi4uVnR0dI23KS0tVXJysp577jlJ1z9ZdO3aNfXv31+SFB8ff9NZSQEADc+jTwfNmzdPTz75pCZMmCCHw6G//vWvmj9/vv785z/f8TYvvfSSZs2apa+++kqSVFJSopCQEPflISEhKi4uvsfxAQD3wqMIXLx4URMmTJAkBQQEaOLEicrIyLjj9VNTU9WhQwdFRES4f4vguz9Qf4PNZqvzwAUFBXW+De5OeHi4t0doVPLz8709AlDvPIpAVVWViouL3T8xef78+ds+qd+QnZ2tc+fOafTo0bp06ZKuXr0qm82m8+fPu69z7tw5hYaG1nngsLAwBQQE1Pl2wL0iimiKHA5HjS+ePYrAxIkT9fjjj2vo0KGy2WzKy8ur8bQR3z3XUFpamo4eParly5dr1KhRys/PV3h4uDIyMhQZGVmHhwIAqG8eRSAxMVFhYWE6fPiwfH19NWnSJPXo0aPOd7Zq1SotXLhQZWVl6tWrl/sQEwDAOzyKgCT17NlTPXv2rPMdxMfHKz4+3r2Obdu21XkdAABr3NXvCQAA7g9EAAAMRgQAwGBEAAAMRgQAwGBEAGggFZVOb4/QaLAtGg+PPyIK4N74+9k1MWWmt8doFDY9s8bbI+C/2BMAAIMRAQAwGBEAAIMRAQAwGBEAAIMRAQAwGBEAAIMRAQAwGBEAAIMRAQAw2H0VgQpnlbdHaDTYFgA8cV+dO8jf7qvxc97x9hiNwv+ufNLbIwBoAu6rPQEA5qiq4EykN9zLtriv9gQAmMPX367sCc94e4xG4bH/Sbnr27InAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDAiAAAGIwIAYDBLI7Bu3TrFxsYqNjZWK1eulCTl5eUpLi5Ojz76qJKTk628ewBALSyLQF5eng4ePKj09HRlZGTo448/VlZWlubPn6/169crOztbBQUFOnDggFUjAABqYVkEQkJCNG/ePPn7+8tut6tr164qKipS586d1alTJ/n5+SkuLk45OTlWjQAAqIVlEejevbv69+8vSSoqKlJ2drZsNptCQkLc1wkNDVVxcbFVIwAAauFn9R18/vnnevbZZzV37lz5+fnp1KlTN11us9nqtL6CgoI7XhYeHn5XM96v8vPz7+n2bM+bsT3rF9uzft3t9rQ0Avn5+fr1r3+t+fPnKzY2VkePHtX58+fdl5eUlCg0NLRO6wwLC1NAQEB9j3pf4n+S+sX2rF9sz/p1p+3pcDhqfPFs2eGgr776StOmTdOqVasUGxsrSerXr59OnTql06dPq6qqSllZWYqMjLRqBABALSzbE9i4caMcDodeeeUV97KxY8fqlVde0YwZM+RwOBQVFaWYmBirRgAA1MKyCCxcuFALFy687WU7d+606m4BAHXAN4YBwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAM5pUIZGZm6rHHHlN0dLTeeecdb4wAAJDk19B3WFxcrOTkZKWlpcnf319jx47VoEGD1K1bt4YeBQCM1+ARyMvL089+9jO1bt1akjRixAjl5ORo+vTpNd7O5XJJkioqKmq8Xqvm9nqZs6lzOBz1s6JmLetnPU1cfW3PlvagellPU1df29OnJf8+pZq3543nzBvPod9nc93pEou8+eabunr1qmbNmiVJSk1N1YkTJ7R48eIab3flyhUVFhY2xIgAcN/p0aOHWt4mmg2+J3C75thstlpvFxQUpB49eshut3t0fQDA9edcp9OpoKDb74U2eATat2+vY8eOuf8uKSlRaGhorbfz8fG5bcUAADVr1qzZHS9r8E8HPfTQQzp06JAuXLig8vJy7d69W5GRkQ09BgBAXtoTmDVrliZMmCCn06nExET17du3occAAMgLbwwDABoPvjEMAAYjAgBgMCIAAAYjAgBgMCJggbVr12rt2rXeHgPAPfjoo4+0YMECb49huQb/iCgANAV9+vRRnz59vD2G5YhAHR05ckTr1q3T5s2bJUnz5s3TwIEDdeHCBW3dulVt2rRRq1at3N992L9/v1avXq3q6mp16tRJixYtUrt27bz5EBoFl8ulVatWae/evfL19dWYMWNUXV2t9PR0+fj4qG/fvlq0aJHGjx+vpKQkDRkyRC6XSyNGjNDmzZu1adMm5ebmytfXV8OHD6/1BIT3u+nTp2vUqFGKiYmRJA0aNEhOp1PHjh2Tj4+Pjh49qg0bNmjKlCl644035HK59J///EcjRoxQy5YttXfvXknShg0b+Pf5X0eOHFFycrLKy8vdy86cOaPRo0erpKTkpu0dHx+vxYsXq6ysTMnJybp27ZouXbqk2bNna+TIkd56CB7hcFA9uHjxorZv36709HSlpKTo66+/liR98803eumll/T6668rMzNTDz74oBYtWuTlaRuHnJwcffjhh8rMzFRqaqpSU1O1atUqbd++XWlpabLZbCouLlZCQoJ27twpSTp27Jh+/OMfq7KyUn//+9+1c+dObdmyRUVFRfV31tQmavTo0crOzpYkFRUVqV27durVq5eOHDkiSUpPT1d8fLwk6fjx41q+fLnee+89bdmyRcHBwUpLS9NPfvITvffee157DI2R3W7Xjh07tGPHDr388stq166dpk+ffsv2djgc6t27t95++20tWbJE6enpWrp0qdavX+/lR1A7IlBPoqKiFBQUpObNm7tfHZw4cUJ9+/bVj370I0nSmDFjdPjwYW+O2Wh88MEHGjlypPz9/RUUFKSsrCxFRkYqMTFR69at05NPPqn27dtr5MiRysvLU3l5ufuJrH379goICNDYsWO1adMm/eY3v1FAQIC3H5JXRUVF6V//+pe+/fZbZWVlKS4uzh3Q8vJyHT58WI888oik62eT7NChgwIDA9WmTRtFRERIkjp27KjLly9782E0WsXFxXrhhReUnJys4ODg225vSXr11Vf1+eef6/XXX1dKSorKysq8PHntiEAd2Wy2m86E6nQ6ZbPZVF1d7V7m53f9KNt3l0nXD4FUVlY2zKCN3I1tdMOZM2f02muv6Q9/+INcLpcmT56so0ePqnnz5oqMjFROTo77iczPz0+pqamaOXOmSktLNXbsWJ06dcpLj6Rx8Pf318MPP6x9+/YpJydHcXFxiomJUW5urnbt2qXIyEj5+/tLuv7q9rt8fX29MXKT4XA4lJSUpBkzZqhXr16Sbr+9JWn8+PE6ceKEwsLC9Nxzz3lzbI8RgTpq06aNvvzySzkcDpWWlio/P18+Pj56//33deXKFTkcDu3Zs0eS1K9fPx0/flxnzpyRJL377rsaNGiQN8dvNAYMGKA9e/bI6XSqvLxcEydO1IMPPqgePXpo5syZGjx4sD777DNJUkJCgpKTkzV06FD5+/vr5MmTeuqppzRgwADNnTtXXbt2NT4C0vVDQikpKfrBD36gBx54QIGBgYqMjNQf//hH96Eg1N38+fM1YMAAjR49+qbl39/epaWlKioq0syZMxUVFaXc3FxVVVV5aWrP8cZwHXXv3l1RUVGKjY3VAw88oPDwcLVq1UpPP/20EhMT1apVK3Xs2FGS1K5dOy1atEjTp0+X0+lUx44dtXTpUi8/gsYhOjpaBQUFio+PV3V1tX71q1+poqJCiYmJCgwMVIcOHfTEE09IksLDw2Wz2ZSQkCBJ6tWrl/r3769Ro0YpMDBQP/3pTzkTra5vpytXrmjs2LHuZbGxsfrwww/Vr18/L07WdP3zn/+U0+lUWFiYHn/8cblcLnXr1k2vvfbaLdu7devW+vnPf67Y2Fi1aNFC/fv317Vr13T16lU1b97cy4/kzjiBHBo1l8ulwsJCzZ07VxkZGd4ep0mpqqpScnKy2rZtq2eeecbb46CRYk8Ajdpf/vIXvfXWW1qzZo23R2lyEhIS1KZNG/3pT3/y9ihoxNgTAACD8cYwABiMCACAwYgAABiMCAC3kZOTo1/+8pdas2ZNrZ9KSktL07PPPnvX97Vu3Tr3uXuAhsang4AazJw50/L7OHLkiLp162b5/QC3QwSA/1qzZo0yMzPVunVrde7cWdL1s8R2795dkyZN0rZt2/Tuu+/K6XTq0qVLmjJlisaPH3/TOq5cuaKlS5eqsLBQTqdTERERmjNnjvz8/HT8+HEtWbJE5eXlstvtmjNnjr744gsVFBRo5cqV8vX1VXR0tDceOgzG4SBA0t69e7V7925lZGRoy5Yt+vbbb2+6vKysTKmpqdqwYYMyMjKUnJysV1999Zb1LFu2TL1791ZaWpoyMjJ08eJFpaSkyOl0atq0aZo2bZqysrK0ePFiLVu2TOPGjVNYWJjmzJlDAOAV7AkAkg4dOqTo6Gi1aNFC0vUvWt34zQhJCgoK0htvvKEDBw6oqKhIn376qa5evXrLet5//3199NFH2rZtmyTp2rVrkqTCwkL5+Pjo4YcfliSFhYUpMzPT4kcF1I4IALr17LDfP7Pm119/rTFjxugXv/iFwsPDFRMTo/3799+ynurqaq1Zs0Zdu3aVJF2+fFk2m01nz56VzWa76bqFhYXq0qWLBY8G8ByHgwBJQ4cOVU5Oji5fvqzq6mrt2LHjpssLCgoUHByspKQkDR061B2A758lcsiQIdq0aZNcLpcqKir0/PPP6+2331aXLl1ks9mUm5srSfr444/19NNPq7q6Wr6+vpxiHF7DngCg6z/K8tlnnykhIUGtWrVSz549dfHiRfflgwcP1rZt2xQTE6PAwED17dtXwcHBOn369E3rWbBggZYuXaq4uDg5nU499NBDmjx5sux2u9auXatly5Zp5cqV7r/9/f01bNgwrVixQk6n033mVKChcO4gADAYh4MAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAMRgQAwGBEAAAM9n+hjLCHfgcuigAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.set_theme(style=\"whitegrid\")\n",
    "ax = sns.countplot(x=\"dialect\", data=elan_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEJCAYAAAB/pOvWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAATEUlEQVR4nO3de2xT9f/H8VdZtyIDgmA7iZIlIjBxCsmMOi9bEAcbc0E2vKITI6KixHgBcQyNt4BkOkwMifol/IGowTnHJaOgogtSiNAYsTpFIyyykHUTQRjQdVt/fwD9iZ8xukt7yng+EuJ62p7znmn67DlnbW2hUCgkAAD+pZ/VAwAA4g9xAAAYiAMAwEAcAAAG4gAAMNitHqCn2tvb1dzcrMTERNlsNqvHAYDzQigUUjAYVHJysvr1M/cTzvs4NDc3a8+ePVaPAQDnpdGjR2vQoEHG8vM+DomJiZJO/oJJSUkWTwMA54eWlhbt2bMn/Bz6X+d9HE4fSkpKSpLD4bB4GgA4v5ztcDwnpAEABuIAADAQBwCAgTgAAAzEAQBgIA4AAANxAAAYiMMpLcE2q0dAHOJxgQvVef8muN6SlJig++evtnoMxJmPls6wegTAEuw5AAAMxAEAYCAOAAADcQAAGIgDAMBAHAAAhqjG4ejRo7rjjju0f/9+SZLH41FBQYEmTZqk8vLy8O1qa2tVVFSkyZMna+HChWptbY3mWACAc4haHH744Qfdd9992rdvnyTpxIkTKikp0fLly1VdXS2fz6eamhpJ0rx587Ro0SJt2rRJoVBIa9asidZYAIAIRC0Oa9as0csvvyyXyyVJ2r17t1JTUzVixAjZ7XYVFBTI7Xarvr5eJ06c0Pjx4yVJhYWFcrvd0RoLABCBqL1D+o033jjjst/vl9PpDF92uVxqaGgwljudTjU0NERrLABABGL28RmhUMhYZrPZzrq8q3w+X7fmOi0jI6NH90ff5fV6rR4BiLmYxSElJUVNTU3hy36/Xy6Xy1je2NgYPhTVFenp6XI4HL0yK/BvvHBAXxQIBDp9UR2zP2UdN26c9u7dq7q6OrW1tWnDhg3KysrSZZddJofDEX51VlVVpaysrFiNBQDoQMz2HBwOh5YsWaK5c+cqEAgoOztbubm5kqSysjKVlpaqublZY8eOVXFxcazGAgB0IOpx2LJlS/jnzMxMrVu3zrhNWlqaKioqoj0KACBCvEMaAGAgDgAAA3EAABiIAwDAQBwAAAbiAAAwEAcAgIE4AAAMxAEAYCAOAAADcQAAGIgDAMBAHAAABuIAADAQBwCAgTgAAAzEAQBgIA4AAANxAAAYiAMAwEAcAAAG4gAAMBAHAICBOAAADMQBAGAgDgAAA3EAABiIAwDAQBwAAAbiAAAwWBKHtWvXKj8/X/n5+XrzzTclSbW1tSoqKtLkyZO1cOFCtba2WjEaAEAWxOH48eN64403tGrVKq1du1a7du2Sx+PRvHnztGjRIm3atEmhUEhr1qyJ9WgAgFNiHoe2tja1t7fr+PHjam1tVWtrq+x2u06cOKHx48dLkgoLC+V2u2M9GgDgFHusNzhw4EA9/fTTysvLU//+/XX99dcrMTFRTqczfBun06mGhoZYjwYAOCXmcfjll1/02Wef6euvv9agQYP0/PPPa9u2bcbtbDZbl9br8/l6NFdGRkaP7o++y+v1Wj0CEHMxj8O3336rzMxMDRs2TNLJQ0grVqxQU1NT+DaNjY1yuVxdWm96erocDkevzgpIvHBA3xQIBDp9UR3zcw5paWnyeDw6duyYQqGQtmzZouuvv14OhyP8Cq2qqkpZWVmxHg0AcErM9xxuueUW/fzzzyosLFRiYqKuueYazZ49Wzk5OSotLVVzc7PGjh2r4uLiWI8GADgl5nGQpNmzZ2v27NlnLEtLS1NFRYUV4wAA/oN3SAMADMQBiHPtrUGrR0AcivbjwpLDSgAi18+eKO/SWVaPgTiTMf9/UV0/ew4AAANxAAAYiAMAwEAcAAAG4gAAMBAHAICBOAAADMQBAGAgDgAAA3EAABiIAwDAQBwAAAbiAAAwEAcAgIE4AAAMxAEAYCAOAAADcQAAGIgDAMBAHAAABuIAADAQBwCAgTgAAAzEAQBgIA4AAANxAAAYiAMAwBBRHBoaGoxlv//+e7c3umXLFhUWFio3N1evv/66JMnj8aigoECTJk1SeXl5t9cNAOi5TuNw6NAhHTp0SI8++qgOHz4cvtzU1KQ5c+Z0a4N//vmnXn75ZS1fvlzr16/Xzz//rJqaGpWUlGj58uWqrq6Wz+dTTU1Nt9YPAOg5e2dXPvfcc9q2bZsk6YYbbvj/O9ntuv3227u1wS+++EJTpkzRpZdeKkkqLy9XXV2dUlNTNWLECElSQUGB3G63srOzu7UNAEDPdBqHFStWSJJefPFFLV68uFc2WFdXp8TERD3yyCNqbGzUhAkTNGrUKDmdzvBtXC5Xh4eyAACx0WkcTlu8eLHq6+t1+PBhhUKh8PKrr766yxtsa2vTrl27tGrVKg0YMEBz5szRRRddZNzOZrN1ab0+n6/Ls/xbRkZGj+6Pvsvr9Vq6fR6bOJtoPjYjikNZWZlWrVqlYcOGhZfZbDZ99dVXXd7gJZdcoszMTA0dOlSSNHHiRLndbiUkJIRv4/f75XK5urTe9PR0ORyOLs8DnAtPzohXPXlsBgKBTl9URxSH6upqbd68WSkpKd0e5LQJEybohRde0D///KPk5GRt3bpVubm5ev/991VXV6fLL79cGzZsUFFRUY+3BQDonojiMHz48F4JgySNGzdOs2bN0v33369gMKibb75Z9913n6644grNnTtXgUBA2dnZys3N7ZXtAQC6LqI4ZGZmaunSpZo4caL69+8fXt6dcw6SNH36dE2fPt3Yxrp167q1PgBA74ooDpWVlZIkt9sdXtbdcw4AgPgXURy2bNkS7TkAAHEkojisXLmyw+UPP/xwrw4DAIgPEcVhz5494Z9bWlrk9XrPeMc0AKBvifhNcP928OBBzZ8/PyoDAQCs162P7B46dKjq6+t7exYAQJzo8jmHUCgkn893xrulAQB9S5fPOUgn3xTHYSUA6Lu6dM6hvr5era2tSk1NjepQAABrRRSHuro6zZkzR36/X+3t7br44ov13nvvaeTIkdGeDwBggYhOSL/66quaNWuWdu7cKa/XqyeeeEKvvPJKtGcDAFgkojj89ddfmjZtWvhyUVGR/v7776gNBQCwVkRxaGtr06FDh8KXDx48GK15AABxIKJzDg888IDuuece5eXlSZI2btyohx56KKqDAQCsE9GeQ3Z2tiQpGAzqjz/+UENDg3JycqI6GADAOhHtOSxYsEAzZsxQcXGxAoGAPv74Y5WUlOiDDz6I9nwAAAtEtOfw999/q7i4WJLkcDg0c+ZMNTY2RnUwAIB1Ij4h3dDQEL7c1NSkUCgUtaEAANaK6LDSzJkzdeedd+rWW2+VzWaTx+Ph4zMAoA+LKA7Tp09Xenq6duzYoYSEBD3yyCMaPXp0tGcDAFgkojhIUlpamtLS0qI5CwAgTnTr+xwAAH0bcQAAGIgDAMBAHAAABuIAADAQBwCAgTgAAAzEAQBgIA4AAINlcXjzzTe1YMECSVJtba2Kioo0efJkLVy4UK2trVaNBQCQRXHYvn27Pv/88/DlefPmadGiRdq0aZNCoZDWrFljxVgAgFNiHodDhw6pvLxcjz/+uCSpvr5eJ06c0Pjx4yVJhYWFcrvdsR4LAPAvMY/DSy+9pGeeeUaDBw+WJPn9fjmdzvD1TqfzjO+OAADEXsSfytobPv30Uw0fPlyZmZmqrKyUpA6/NMhms3V53T6fr0ezZWRk9Oj+6Lu8Xq+l2+exibOJ5mMzpnGorq5WY2Ojpk6dqsOHD+vYsWOy2WxqamoK36axsVEul6vL605PT5fD4ejNcQFJPDkjfvXksRkIBDp9UR3TOKxcuTL8c2Vlpb777jstXrxYd9xxh7xerzIyMlRVVaWsrKxYjgUA+I+YxuFsysrKVFpaqubmZo0dO1bFxcVWjwQAFzTL4lBYWKjCwkJJJ79lrqKiwqpRAAD/wTukAQAG4gAAMBAHAICBOAAADMQBAGAgDgAAA3EAABiIAwDAQBwAAAbiAAAwEAcAgIE4AAAMxAEAYCAOAAADcQAAGIgDAMBAHAAABuIAADAQBwCAgTgAAAzEAQBgIA4AAANxAAAYiAMAwEAcAAAG4gAAMBAHAICBOAAADMQBAGAgDgAAA3EAABgsicO7776r/Px85efna+nSpZIkj8ejgoICTZo0SeXl5VaMBQA4JeZx8Hg8+vbbb/X555+rqqpKP/30kzZs2KCSkhItX75c1dXV8vl8qqmpifVoAIBTYh4Hp9OpBQsWKCkpSYmJiRo5cqT27dun1NRUjRgxQna7XQUFBXK73bEeDQBwij3WGxw1alT453379qm6uloPPvignE5neLnL5VJDQ0OX1uvz+Xo0V0ZGRo/uj77L6/Vaun0emzibaD42Yx6H03777Tc99thjeuGFF2S327V3794zrrfZbF1aX3p6uhwOR2+OCEjiyRnxqyePzUAg0OmLaktOSHu9Xs2cOVPPPfecpk2bppSUFDU1NYWv9/v9crlcVowGAJAFcThw4ICefPJJlZWVKT8/X5I0btw47d27V3V1dWpra9OGDRuUlZUV69EAAKfE/LDSihUrFAgEtGTJkvCye++9V0uWLNHcuXMVCASUnZ2t3NzcWI8GADgl5nEoLS1VaWlph9etW7cuxtMAADrCO6QBAAbiAAAwEAcAgIE4AAAMxAEAYCAOAAADcQAAGIgDAMBAHAAABuIAADAQBwCAgTgAAAzEAQBgIA4AAANxAAAYiAMAwEAcAAAG4gAAMBAHAICBOAAADMQBAGAgDgAAA3EAABiIAwDAQBwAAAbiAAAwEAcAgIE4AAAMxAEAYCAOAABDXMVh/fr1mjJlinJycrR69WqrxwGAC5bd6gFOa2hoUHl5uSorK5WUlKR7771XN9xwg6688kqrRwOAC07cxMHj8ejGG2/UkCFDJEmTJ0+W2+3WU0891en9QqGQJKmlpaXHMwwekNjjdaBvCQQCVo9wUv9BVk+AONPTx+bp58zTz6H/FTdx8Pv9cjqd4csul0u7d+8+5/2CwaAkac+ePT2e4dGCkT1eB/oWn89n9Qgn3fyA1RMgzvTWYzMYDKp///7G8riJQ0f1stls57xfcnKyRo8ercTExIhuDwA4+ZwbDAaVnJzc4fVxE4eUlBTt2rUrfNnv98vlcp3zfv369dOgQexyA0BXdbTHcFrc/LXSTTfdpO3bt+vgwYM6fvy4Nm/erKysLKvHAoALUlztOTzzzDMqLi5WMBjU9OnTde2111o9FgBckGyhs52qBgBcsOLmsBIAIH4QBwCAgTgAAAzEAQBgIA6QJO3fv1/p6emaOnXqGf8OHDhg9Wi4gO3fv19jxozRSy+9dMby2tpajRkzRpWVlRZN1vfFzZ+ywnoul0tr1661egzgDEOGDNHWrVvV1tamhIQESVJ1dbWGDh1q8WR9G3sOAOJacnKyrrrqKu3cuTO8bNu2bbrpppssnKrvY88BYX6/X1OnTg1fLigo0KxZsyycCDgpLy9PmzZt0o033qjdu3drzJgxZ/00UfQO4oAwDishXk2YMEHLli1Te3u7Nm7cqLy8PFVXV1s9Vp/GYSUAcW/gwIFKS0uT1+vVjh07OKQUA8QBwHkhLy9Pb731ltLT02W3c9Aj2ogDgPPChAkTVFtbqylTplg9ygWBD94DABjYcwAAGIgDAMBAHAAABuIAADAQBwCAgTgAMeZ2u/Xggw9aPQbQKeIAADDwNkPgLN5//31VVFQoOTlZ1113nb766iu53W6VlZVp586damtr09ixY1VaWqqBAwfqtttu07Rp07R9+3YdOHBAeXl5mj9/viTpnXfe0fr16zVkyBClpqaGt9HS0tLp+q699lr9+uuvevbZZ5WTk2PV/wpcgNhzADqwdetWVVZWqqKiQpWVlWpubpZ0MhgJCQmqrKzUunXr5HK5VFZWFr7fsWPH9NFHH+mTTz7Rhx9+qD///FNffvmlNm/erKqqKn3yySc6evRo+PbnWt+oUaO0ceNGwoCYY88B6EBNTY1yc3M1ePBgSdKMGTO0Y8cOffPNNzpy5Ig8Ho8kKRgMatiwYeH7TZw4UZKUkpKiYcOG6fDhw9q+fbtycnI0cOBASVJRUZFWrVolSedc33XXXRf9XxboAHEAOmC328/4voDT30DW3t6ukpISZWdnS5Kam5sVCATCt3M4HOGfbTabQqFQ+L//XVck6xswYEAv/2ZAZDisBHQgOztbmzdv1pEjRyRJFRUVkqRbbrlFq1evVktLi9rb27Vo0SK9/fbbna7r1ltvldvt1j///KP29vYzvjOjO+sDYoE4AB3IzMzU3XffrXvuuUeFhYU6cuSILrroIs2ZM0eXXXaZpk2bpilTpigUCmnBggWdris7O1tFRUUqKirSXXfdpUGDBoWv6876gFjgU1mBDvz444/6/vvvVVxcLElauXKlfvjhBy1btszawYAYIQ5AB44ePaqSkhL98ccfstlsGj58uF577TWlpKRYPRoQE8QBAGDgnAMAwEAcAAAG4gAAMBAHAICBOAAADMQBAGD4PxDoU9J9SFRKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = sns.countplot(x=\"gender\", data=elan_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "birthyear\n",
       "19XX    64\n",
       "193X    41\n",
       "1941    34\n",
       "1920    31\n",
       "1933    10\n",
       "1913     3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elan_df.value_counts(\"birthyear\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:kielipankki]",
   "language": "python",
   "name": "conda-env-kielipankki-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}