Files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading and writing files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to play back an audio file, type Esc-o to hide player\n",
    "from IPython.display import Audio\n",
    "Audio(\"media/fl-intro.mp3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Entering input via the keyboard and reading the output from a console obviously isn't a very convenient way of dealing with large amounts of data. Most of the times, we will want to operate on files (eg TXT, CSV or JSON files to mention a few). Most commonly, files are used to save data entered by the user (think of a .doc file). However, files can be downloaded from some online API or database; they can also be used to store intermediate results of computation for further processing, and to store large amounts of output during batch processing. Dealing with files is luckily straightforward in Python.\n",
    "\n",
    "Note that since the files we will work with sit in the same directory (and on the same machine) as the notebook, you may need to view and edit them through the Jupyter Notebook interface rather than through your local text editor, according to where you are running."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text vs Binary files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are essentially two types of data files: *text* and *binary*. \n",
    "\n",
    "**Text files** can be opened and modified with a plain text editor such as gedit, emacs or notepad. They are not necessarily written in plan English: a Python program, an [HTML](https://en.wikipedia.org/wiki/HTML) page,  a [JSON](https://en.wikipedia.org/wiki/JSON) file, an [XML](https://en.wikipedia.org/wiki/XML) file, a [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) file, a [FASTA](https://en.wikipedia.org/wiki/FASTA_format) file or a [PDB](https://en.wikipedia.org/wiki/Protein_Data_Bank_(file_format)) file are all text files. While not always meant for human consumption, text files are generally somewhat human-readable and are portable across different operating systems and editors, with very minor niggles. The disadvantage is that they take up a lot of space on the disk.\n",
    "\n",
    "**Binary files** contain data in the internal machine representation. They are less portable and are generally used either to talk directly to the machine, or to store large amounts of data efficiently, or to protect intellectual property. Python bytecode (.pyc) files, compressed (.zip) files, Microsoft Word (.doc) files, executable files (.exe on Windows systems), image files (.jpg, .gif), and music files (.wav, .mp3) are all examples of binary files. An attempt at opening them in a text editor will only show gibberish on the screen.\n",
    "\n",
    "Happily many of the file types used in web programming, bioinformatics and other fields are text files with open format specifications, so we will not worry about binary files here (specialised libraries exist for the most common file types anyway). It is important however to remember, in your future professional life, that sometimes what you really need is not a bigger disk, but  only a more compact data representation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unless they just contain plain text meant for humans (eg a README file), text files normally follow detailed specifications that describe how the information they contain is formatted, and make it possible to retrieve and process it automatically. One of the simplest text file specifications is the [FASTA](https://en.wikipedia.org/wiki/FASTA_format) format, extremely common in Bioinformatics. It is used to store the sequence of proteins (ie, their chemical composition). Essentially, a FASTA file looks like this:\n",
    "```\n",
    "> Accession code | Name of protein\n",
    "THEENTIRESEQUENCEARRANGEDONSEVERALLI\n",
    "NESWITHOUTSPACESANDBROKENUPWITHNEWLI\n",
    "NESINARBITRARYPLACESTOMAKEITLOOKTIDY\n",
    "```\n",
    "To see an actual example, run the cell below and click on the link it creates. If your browser won't open it, go to File->Open in the Jupyter interface or use a text editor of your choice (note: this example requires the file to exist and to be in the same directory as this notebook)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is just an IPython trick that creates a link to the file below.\n",
    "# Click on the link to view the file\n",
    "from IPython.display import FileLink\n",
    "FileLink('P04637.fas')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the ```>``` sign that marks the header line and the ```|``` sign that separates the accession code from the name of the protein. The challenge is to read the file retrieving the accession code, the protein name and the entire sequence as one string, discarding the newlines that are just pretty-printing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading this file in Python requires first a call to ```open()```, that returns a *file handle*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FASTA=open(\"P04637.fas\", \"r\")\n",
    "print(FASTA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the ```\"r\"``` indicates we are opening this file for *reading*. The *handle* that's returned is a convenience object used by Python to keep track of all data relative to the file (location, type, position, etc). Just treat it as you would any other Python object and assign it to a variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using ```readlines()```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An easy way to read all the file is now to use ```readlines()```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this won't work twice in a row, see below\n",
    "everything=FASTA.readlines()\n",
    "print(everything)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, ```readlines()``` returns a list of strings, each corresponding to a line in the file. This exhausts the contents of the file. All that remains to do now is to close the file to free the associated system resources:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FASTA.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you cannot run ```readlines()``` on the same file twice without \"rewinding\" it (for example by closing it and reopening)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-readlines.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using ```readline()```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using ```readlines()``` is less than ideal as it slurps the entire file up in one go, which is somewhat inconvenient as the file may contain different types of data (in this case the header and the protein proper) or may be too large to fit entirely in the memory. In general, it is better to process it one line at a time, using repeated calls to ```readline()```. In this case, this also gives us the chance to assemble the protein into a single string. Here is how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FASTA=open(\"P04637.fas\", \"r\")\n",
    "header=FASTA.readline()\n",
    "protein=\"\" # build up the sequence here\n",
    "while True:\n",
    "    ll=FASTA.readline()\n",
    "    if ll==\"\": break\n",
    "    protein+=ll.rstrip() # remove trailing '\\n'\n",
    "FASTA.close()\n",
    "# Done. This is just pretty-printing\n",
    "(code, name)= header.split('|')\n",
    "print(\"Accession code:\")\n",
    "print(code)\n",
    "print(\"\\nName:\")\n",
    "print(name)\n",
    "print(\"Protein:\")\n",
    "print(protein)\n",
    "print(\"\\nNumber of residues:\")\n",
    "print(len(protein))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-readline.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using a ```for``` loop:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a more \"Pythonic\" way of doing this that uses the fact that a file is an *iterable* and therefore can be read directly using a ```for``` loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FASTA=open(\"P04637.fas\", \"r\")\n",
    "header=FASTA.readline()\n",
    "protein=\"\" \n",
    "for ll in FASTA: # couldn't be easier!\n",
    "    protein+=ll.rstrip()\n",
    "FASTA.close()\n",
    "# Done. This is just pretty-printing\n",
    "(code, name)= header.split('|')\n",
    "print(\"Accession code:\")\n",
    "print(code)\n",
    "print(\"\\nName:\")\n",
    "print(name)\n",
    "print(\"Protein:\")\n",
    "print(protein)\n",
    "print(\"\\nNumber of residues:\")\n",
    "print(len(protein))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-forloop.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading other text file formats obviously requires a knowledge of the relevant format specification. Common formats such as JSON and CSV (see below) are supported by Python through the standard library (see for example [here](https://docs.python.org/3/library/json.html)), others are supported through third party libraries (eg [Biopython](https://biopython.org/)). In the other cases, the general strategies described above coupled with some experimentation should hopefully see you through."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing files is not much more of a hassle than reading them. In fact, the main steps are the same:\n",
    "* open a file\n",
    "* write the content\n",
    "* close it\n",
    "\n",
    "The only differences are that the file must be opened in write mode, and that the ```write()``` method must be used to actually write data to it.\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTF=open(\"greetings.txt\", \"wt\")\n",
    "# strings can be written directly\n",
    "OUTF.write(\"Hello World!\\n\") # \\n means newline\n",
    "value=('The Answer', 42) # other stuff needs to be converted to a string\n",
    "OUTF.write(str(value))\n",
    "OUTF.close() # makes sure the buffer gets flushed to the disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the usual IPython trick that creates a link to the file below.\n",
    "# Click on the link to view the file\n",
    "from IPython.display import FileLink\n",
    "FileLink('greetings.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try running the example above twice, and check the *greetings.txt* file each time. You will see that the content gets overwritten. If you do not like that behaviour, change ```\"wt\"``` to ```\"at\"``` in the ```open``` statement to open the file for *appending*. See for yourself what that does. Appending to a non-existing file just creates it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-theanswer.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So for instance, writing your imaginary protein out in FASTA format can be done this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your imaginary protein\n",
    "accession=\"PXXXX\"\n",
    "description=\"My candidate protein - Homo programmaticus (Programmer)\" \n",
    "sequence=\"\"\"QIKDLLVSSSTDLDTTLVLVNAIYFKGMWKTAFNAEDTREMPFHVTKQESKPVQMMCMNNSFNVATLPAEKMK\n",
    "    ILELPFASGDLSMLVLLPDEVSDLERIEKTINFEKLTEWTNPNTMEKRRVKVYLPQMKIEEKYNLTS\n",
    "    VLMALGMTDLFIPSANLTGISSAESLKISQAVHGAFMELSEDGIEMAGSTGVIEDIKHSPESEQFRADHP\n",
    "\tFLFLIKHNPTNTIVYFGRYWSP\"\"\" # whatever\n",
    "# get rid of tabs, newlines and spaces in the above string\n",
    "sequence=sequence.replace(' ','')\n",
    "sequence=sequence.replace('\\t', '')\n",
    "sequence=sequence.replace('\\n', '')\n",
    "\n",
    "# Ok, now let's write it\n",
    "OUTF=open(accession+\".fas\", \"w\")\n",
    "# first the header\n",
    "header=\"> \" + accession + \" | \" + description+\"\\n\"\n",
    "OUTF.write(header)\n",
    "# output in 60 char lines for convenience\n",
    "linew=60\n",
    "pos=0\n",
    "while sequence[pos:pos+linew]!='':\n",
    "        OUTF.write(sequence[pos:pos+linew]+\"\\n\")\n",
    "        pos+=linew\n",
    "OUTF.close() # Done!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ...the same old trick... check the file out!\n",
    "from IPython.display import FileLink\n",
    "FileLink(accession+\".fas\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-hprogrammaticus.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSV Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [CSV (Comma-Separated Values)](https://en.wikipedia.org/wiki/Comma-separated_values) file format is a popular text file format that lists each record on a separate line. Data fields for the same entry are separated by commas (or occasionaly semicolons or tabs). Many popular packages can output data as CSV files, among others Excel. Reading and writing csv files in Python is easy, either directly or via the ```csv``` module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example - the file *marksheet.csv* contains the following text:\n",
    "\n",
    "```\n",
    "Name, Surname, Mark\n",
    "John, Smith, 50\n",
    "Anne, Larsson, 65\n",
    "Emiliano, Zapata, 95\n",
    "Donald, Duck, 40\n",
    "```\n",
    "\n",
    "When you click on a link to this file, the browser is likely to suggest opening it as a spreadsheet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ...and again\n",
    "from IPython.display import FileLink\n",
    "FileLink('marksheet.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can, however, treat this as a normal text file. Suppose that we want to compute the average mark:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILE=open(\"marksheet.csv\",\"r\")\n",
    "FILE.readline() # skip header\n",
    "total=0.0\n",
    "students=0\n",
    "for line in FILE:\n",
    "    entries=line.split(',')\n",
    "    total+=float(entries[2])\n",
    "    students+=1\n",
    "FILE.close()\n",
    "\n",
    "print(\"Average: \", total/students)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-csvdiy.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```csv``` module offers a slightly more convenient way of accessing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv # import the csv module\n",
    "FILE=open(\"marksheet.csv\",\"r\")\n",
    "\n",
    "total=0.0\n",
    "students=0\n",
    "marksheet=csv.reader(FILE)\n",
    "next(marksheet) # skip first line\n",
    "for line in marksheet: # already split for us\n",
    "    total+=float(line[2])\n",
    "    students+=1\n",
    "FILE.close()\n",
    "\n",
    "print(\"Average: \", total/students)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For such a simple file it's hardly worth the trouble; however, the ```csv``` module can handle several types of files (essentially different separators) and automatically guess what software might have generated the file, see the [online help](https://docs.python.org/3/library/csv.html) for details.\n",
    "\n",
    "The ```csv``` module also provides a ```writer``` object. However, remember that writing a ```csv``` file is essentially only a matter of putting the commas in the right places."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-csvwcsv.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The ```with``` context manager"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python provides a context management control structure, introduced by the ```with``` keyword. The context manager takes care of initialising and mopping up resources your program is using - in this case, a file. In practice, instead of writing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "INFILE=open(\"marksheet.csv\",\"r\")\n",
    "for line in INFILE:\n",
    "    print(line, end='') # newline is already included in line\n",
    "INFILE.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can simply write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"marksheet.csv\",\"r\") as INFILE:\n",
    "    for line in INFILE:\n",
    "        print(line, end='')\n",
    "        \n",
    "# the context manager automatically closes the file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file is closed automatically for you at the end of the ```with``` block, when the context manager exits. The same happens if the file is opened for writing.\n",
    "\n",
    "This is a very Pythonic way of doing things, and it is very effective at fixing a common problem - i.e. files being left open when they are no longer needed. Context managers have applications also in other areas, that however are mostly beyond the scope of this module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this cell to show a video, use slider to resize it, type Esc-o to hide it\n",
    "from IPython.display import Video; from ipywidgets import interactive, IntSlider\n",
    "def _play(resize): display(Video(filename=\"media/fl-oldvsnew.webm\",data=\"\",width=resize))\n",
    "interactive(_play, resize=IntSlider(min=150, max=900, step=50, value=600, continuous_update=False, readout=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2014,2020 Fabrizio Smeraldi** ([f.smeraldi@qmul.ac.uk](mailto:f.smeraldi@qmul.ac.uk) - [web](http://www.eecs.qmul.ac.uk/~fabri/)), all rights reserved. In: \"Computer Programming\", School of Electronic Engineering and Computer Science, Queen Mary University of London."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}