Commiting before working on idx

book/_build/html/_sources/data.ipynb

@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","metadata":{"id":"cbqX9TMrgYpE"},"source":["# Working with data"]}],"metadata":{"kernelspec":{"name":"python3","language":"python","display_name":"Python 3"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}

book/_build/html/_sources/data_advanced_databases.ipynb

@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","metadata":{"id":"0zHXSDQI6FyT"},"source":["<a href=\"https://colab.research.google.com/github/smart-stats/ds4bio_book/blob/main/book/data_advanced_databases.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n","\n","# HDF5\n","\n","You've probably already learned about some variation of databases\n","either sql, nosql, spark, a cloud db, ... We covered sqlite last chapter.  Often,\n","the backend of these databases can be quite complicated, while the\n","front end requires SQL querries or something similar. We'll look at a\n","non-relational database format that is specifically useful for\n","scientific computing called hdf5. HDF5 has implementations in many\n","languages, but we'll look at python. This is a hierarchical data\n","format specifically useful for large array calculations.\n","\n","Let's create a basic h5py file. First, let's load our stuff."]},{"cell_type":"code","metadata":{"id":"9ZtaYNrj6Fyb"},"source":["import numpy as np\n","import h5py"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"p5yrVspO6Fye"},"source":["Now, let's create an empty hdf5 file. Here's the basic code; the\n","option `w` is open for writing. There's also `w-`, `r`, `r+`, `a` for\n","write protected, read only, read/write, read/write and create.  The\n","first time I ran it I used:"]},{"cell_type":"code","metadata":{"id":"C49xsIVf6Fyf"},"source":["f = h5py.File('sensor.hdf5', 'w')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"39jxCNSm6Fyf"},"source":["Then, subsequently"]},{"cell_type":"code","metadata":{"id":"EtYpCqUU6Fyg"},"source":["#| eval: false\n","f = h5py.File('sensor.hdf5', 'r+')"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZjxFh5J76Fyh"},"source":["Now let's populate it with some data. The hdf5 file works almost like\n","a directory where we can store hierarchical data. For example, suppose\n","that we want sensors stored in a superstructure called `sensors` and\n","want to fill in the data for `sensor1` and `sensor1`."]},{"cell_type":"code","metadata":{"id":"FvFPY3oU6Fyh"},"source":["f['sensors/sensor1'] = np.random.normal(size = 1024)\n","f['sensors/sensor2'] = np.random.normal(size = 1024)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VjI8JM416Fyi"},"source":["Now we can do normal `np` stuff on this sensor. However, hdf5 is only\n","bringing in the part that we are using into memory. This allows us to\n","work with very large files. Also, as we show here, you can name the\n","data to a variable since that's more convenient."]},{"cell_type":"code","metadata":{"id":"sqLuWexJ6Fyj"},"source":["s1 = f['sensors/sensor1']\n","s2 = f['sensors/sensor2']"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hxbwpN256Fyk"},"source":["## Blockwise basic statistical calculations\n","Now, consider taking the mean of both variables. Imagine that the time\n","series is so long it's not feasible to load into memory. So, we want\n","to read it in blocks. You want your blocks to be as big as possible,\n","since that's fastest. In our case, of course, none of this is\n","necessary.\n","\n","Our goal in this section is to do the following: calculate the\n","empirical mean and variance for each sensor, center and scale each\n","sensor, and write those changes to those variables, calculate the\n","sample correlation then calculate the residual for sensor1 given\n","sensor2. (I think typically you wouldn't want to overwrite the\n","original data; but, this is for pedagogical purposes.) We want our\n","data organized so sensors are stored in a hierarchical \"folder\" called\n","sensors and processed data is in a different folder.\n","\n","We're just simulating iid standard normals. So, we have a rough idea\n","of the answers we should get, since the the data are theoretically\n","mean 0, variance 1 and uncorrelated. After our calculations, they will\n","have empirical mean 0 and variance 1 and the empirical correlation\n","between the residual and sensor 2 will be 0.\n","\n","Let's consider a block variation of the inner product.\n","\n","$$\n","<a, b> = \\sum_{i=0}^{n-1} a_i b_i = \\sum_{i=0}^{n/B} \\sum_{j=0}^{B-1} a_{j + i B} b_{j + i B}\n","$$\n","\n","(if $n$ is divisible by $B$. Otherwise you have to figure out what to do with the final block, which isn't hard but makes the notation messier.) So, for example, the (sample) mean is then $<x, J>/n$ where $J$ is a vector of ones.\n","\n","Let's calculate the mean using blockwise calculations."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Jwe-4O4S6Fyk","executionInfo":{"status":"ok","timestamp":1713271323038,"user_tz":240,"elapsed":138,"user":{"displayName":"Brian Caffo","userId":"07979705296072332292"}},"outputId":"cc10a0ce-0dcc-455d-8af3-600582a336ba"},"source":["n = s1.shape[0]\n","B = 32\n","## mean center the blocks\n","mean1 = 0\n","mean2 = 0\n","for i in range(int(n/B)):\n","    block_indices = np.array(range(B)) + i * B\n","    mean1 += s1[block_indices].sum() / n\n","    mean2 += s2[block_indices].sum() / n\n","\n","[mean1, mean2]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[-0.03648099576614891, -0.01236374986065932]"]},"metadata":{},"execution_count":13}]},{"cell_type":"markdown","metadata":{"id":"r7F1ZOfZ6Fyl"},"source":["Let's now center our time series."]},{"cell_type":"code","metadata":{"id":"ptxnou_86Fyl"},"source":["for i in range(int(n/B)):\n","    block_indices = np.array(range(B)) + i * B\n","    s1[block_indices] -= mean1\n","    s2[block_indices] -= mean2"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NIi1UXGz6Fym"},"source":["Now the (unbiased, sample) variance of centered vector $a$ is simply $<a, a>/(n-1)$."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ESUei8em6Fym","executionInfo":{"status":"ok","timestamp":1713271327058,"user_tz":240,"elapsed":14,"user":{"displayName":"Brian Caffo","userId":"07979705296072332292"}},"outputId":"13a0f842-f849-44bc-bf5d-6a5ae4ef5f9d"},"source":["v1, v2 = 0, 0\n","for i in range(int(n/B)):\n","    block_indices = np.array(range(B)) + i * B\n","    v1 += np.sum(s1[block_indices] ** 2) / (n - 1)\n","    v2 += np.sum(s2[block_indices] ** 2) / (n - 1)\n","[v1, v2]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[0.9796337756284553, 0.9316733839552979]"]},"metadata":{},"execution_count":15}]},{"cell_type":"markdown","metadata":{"id":"vmNqIcEm6Fym"},"source":["Now let's scale our vectors as"]},{"cell_type":"code","metadata":{"id":"8ViKBZuJ6Fyn"},"source":["sd1 = np.sqrt(v1)\n","sd2 = np.sqrt(v2)\n","for i in range(int(n/B)):\n","    block_indices = np.array(range(B)) + i * B\n","    s1[block_indices] /= v1\n","    s2[block_indices] /= v2"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"z7st-8Cr6Fyn"},"source":["Now that our vectors are centered and scaled, the empirical correlation is simply $<a, b>/(n-1)$. Let's do that"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"bbUmONqb6Fyo","executionInfo":{"status":"ok","timestamp":1713271330791,"user_tz":240,"elapsed":192,"user":{"displayName":"Brian Caffo","userId":"07979705296072332292"}},"outputId":"96654807-2113-4e4d-fd02-31b9c5177680"},"source":["cor = 0\n","for i in range(int(n/B)):\n","    block_indices = np.array(range(B)) + i * B\n","    cor += np.sum(s1[block_indices] * s2[block_indices]) / (n-1)\n","cor"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["-0.020708540657507272"]},"metadata":{},"execution_count":17}]},{"cell_type":"markdown","metadata":{"id":"OkhQ_8zB6Fyo"},"source":["Finally, we want to \"regress out\" s2 from s1. Since we normalized our series, the correlation is slope coefficient from linear regression (regardless of the outcome and dependent variable) and the intercept is zero (since we centered). Thus, the residual we want is $e_{12} = s_1 - \\rho s_2$ where $\\rho$ is the correlation."]},{"cell_type":"code","metadata":{"id":"a7pGkt_V6Fyp"},"source":["f['processed/resid_s1_s2'] = np.empty(n)\n","e12 = f['processed/resid_s1_s2']\n","for i in range(int(n/B)):\n","    block_indices = np.array(range(B)) + i * B\n","    e12[block_indices] += s1[block_indices] - cor * s2[block_indices]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vU9ZFzqn6Fyq"},"source":["Now we have our new processed data stored in a vector. To close our\n","database simply do:"]},{"cell_type":"code","metadata":{"id":"FbDa-Pgd6Fyq"},"source":["f.close()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"j361azRJ6Fyq"},"source":["Now our processed data is stored on disk."]},{"cell_type":"code","metadata":{"id":"2mxaoPSP6Fyr"},"source":["f = h5py.File('sensor.hdf5', 'r')\n","f['processed/resid_s1_s2']"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["We can close the database with the method `.close()` as follows."],"metadata":{"id":"B0pncZs19l3U"}},{"cell_type":"code","metadata":{"id":"m9pI9lOT6Fyr"},"source":["f.close()"],"execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"name":"python3","language":"python","display_name":"Python 3"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}

book/_build/html/_sources/graphics.ipynb

@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","metadata":{"id":"uEeRdv7HGf8d"},"source":["# Graphics"]}],"metadata":{"kernelspec":{"name":"python3","language":"python","display_name":"Python 3"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}