samples/caso-uso-2-pydatajson-xlsx-justicia-no-valido.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Caso de uso 2 - Validación, transformación y harvesting con el catálogo del Ministerio de Justicia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caso 2:  catálogo no válido por no tener completos los campos \"title\" de la clase dataset.\n",
    "\n",
    "En esta prueba se realiza el proceso completo de validación, transformación y harvesting a partir de un archivo xlsx que contiene los metadatos pertenecientes al catálogo del Ministerio de Justicia. \n",
    "\n",
    "**Nota**: Se trata de un catálogo conocido y originalmente válido en cuanto a su estructura y metadatos, al que se le han borrado los valores de la propiedad \"dataset_title\". Archivo utilizado: catalogo-justicia-con-error-datasets.xlsx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importación de metodos y clases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import arrow\n",
    "import os, sys\n",
    "sys.path.insert(0, os.path.abspath(\"..\"))\n",
    "from pydatajson import DataJson #lib y clase\n",
    "from pydatajson.readers import read_catalog # lib, modulo ... metodo Lle el catalogo -json o xlsx o (local o url) dicc- y lo transforma en un diccionario de python\n",
    "from pydatajson.writers import write_json_catalog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Declaración de variables y paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#completar con lo que corresponda\n",
    "ORGANISMO = 'justicia'\n",
    "catalogo_xlsx = os.path.join(\"archivos-tests\", \"excel-no-validos\", \"catalogo-justicia-con-error-datasets.xlsx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#NO MODIFICAR\n",
    "\n",
    "#Creo la estructura de directorios necesaria si no existe\n",
    "if not os.path.isdir(\"archivos-generados\"):\n",
    "    os.mkdir(\"archivos-generados\")\n",
    "\n",
    "for directorio in [\"jsons\", \"reportes\", \"configuracion\"]:\n",
    "    path = os.path.join(\"archivos-generados\", directorio)\n",
    "    if not os.path.isdir(path):\n",
    "        os.mkdir(path)\n",
    "\n",
    "# Declaro algunas variables de interés\n",
    "HOY = arrow.now().format('YYYY-MM-DD-HH_mm')\n",
    "catalogo_a_json = os.path.join(\"archivos-generados\",\"jsons\",\"catalogo-{}-{}.json\".format(ORGANISMO, HOY))\n",
    "reporte_datasets = os.path.join(\"archivos-generados\", \"reportes\", \"reporte-catalogo-{}-{}.xlsx\".format(ORGANISMO, HOY))\n",
    "archivo_config_sin_reporte = os.path.join(\"archivos-generados\", \"configuracion\", \"archivo-config_-{}-{}-sinr.csv\".format(ORGANISMO, HOY))\n",
    "archivo_config_con_reporte = os.path.join(\"archivos-generados\", \"configuracion\", \"archivo-config-{}-{}-conr.csv\".format(ORGANISMO, HOY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validación del archivo xlsx y transformación a json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validación del catálogo en xlsx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "catalogo = read_catalog(catalogo_xlsx)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformación del catálogo, de xlsx a json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "write_json_catalog(catalogo, catalogo_a_json)\n",
    "##write_json_catalog(catalog, target_file)  escribe un dicc a un archivo json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validación del catalogo en json y harvesting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validación del catálogo en json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "source": [
    "Instanciación de la clase DataJson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dj = DataJson()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Validación -V/F- del catálogo en json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.is_valid_catalog(catalogo)\n",
    "# obtenemos FALSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validación detallada del catálogo en json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'error': {u'catalog': {u'errors': [],\n",
       "   u'status': u'OK',\n",
       "   u'title': u'Datos Justicia Argentina'},\n",
       "  u'dataset': [{u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 0],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 1],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 2],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 3],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 4],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 5],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 6],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 7],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 8],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 9],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 10],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 11],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 12],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 13],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 14],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'title' is a required property\",\n",
       "      u'path': [u'dataset', 15],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': None}]},\n",
       " u'status': u'ERROR'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.validate_catalog(catalogo)\n",
    "# el mensaje de Error indica que \"title\" es una propiedad requerida"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reporte al área correspondiente"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En este caso debe hacerse el reporte del error al área correspondiente para que puedan completarse las omisiones. Recién luego, se estará en condiciones de seguir con el proceso."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "toc": {
   "nav_menu": {
    "height": "233px",
    "width": "253px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}