Skip to content

Commit

Permalink
deploy: 3da6755
Browse files Browse the repository at this point in the history
  • Loading branch information
@tmieslinger
tmieslinger committed May 15, 2024
1 parent 8b72b38 commit 4876889
Show file tree
Hide file tree
Showing 60 changed files with 545,283 additions and 1,611 deletions.
Binary file added _images/16a359de01afe0ad27b4695e8a277fef46d501774d653e5135a0c998bf93c921.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed _images/16bcbc205d8f573b853600c891e900aefa8997a92408ce7c83f39fc1bc58bfc9.png
View file
Binary file not shown.
1,234 changes: 1,211 additions & 23 deletions _sources/bacardi.ipynb
View file

Large diffs are not rendered by default.

2,512 changes: 2,486 additions & 26 deletions _sources/bco_cloudradar.ipynb
View file

Large diffs are not rendered by default.

9,913 changes: 9,876 additions & 37 deletions _sources/botany_dales.ipynb
View file

Large diffs are not rendered by default.

761 changes: 738 additions & 23 deletions _sources/c3ontext.ipynb
View file

Large diffs are not rendered by default.

177 changes: 133 additions & 44 deletions _sources/cloud_stat_tools.ipynb
View file

Large diffs are not rendered by default.

245 changes: 177 additions & 68 deletions _sources/cloudmasks.ipynb
View file

Large diffs are not rendered by default.

214 changes: 168 additions & 46 deletions _sources/data_on_ipfs.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "43e852b4",
"id": "84a08083",
"metadata": {},
"source": [
"# Data on IPFS\n",
Expand All @@ -16,10 +16,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "0ce006e1",
"execution_count": 1,
"id": "a2f9032f",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"['ATR', 'Atalante', 'BOREAL', 'CU-RAAVEN', 'Caravela']"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import eurec4a\n",
"cat = eurec4a.get_intake_catalog(use_ipfs=False)\n",
Expand All @@ -28,7 +39,7 @@
},
{
"cell_type": "markdown",
"id": "d2d1146d",
"id": "e536b7b4",
"metadata": {},
"source": [
"This above command returns the first five entries in the [eurec4a-intake catalog](https://github.com/eurec4a/eurec4a-intake/#readme). Most of the data references by this catalog is accessed via the [OPeNDAP](https://www.opendap.org/) protocol from several servers in use by the EUREC4A community.\n",
Expand All @@ -37,18 +48,29 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "9ddf2911",
"execution_count": 2,
"id": "1ce95a9c",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"['ATR', 'Atalante', 'BOREAL', 'CU-RAAVEN', 'Caravela']"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat = eurec4a.get_intake_catalog(use_ipfs=True)\n",
"list(sorted(cat))[:5]"
]
},
{
"cell_type": "markdown",
"id": "651d67d2",
"id": "ad66f9d0",
"metadata": {},
"source": [
"... we get the same result 😃. Ideally, the resulting intake catalog should behave just as the \"classical\" counterpart. Apart from a few edge-cases, which often are related to {doc}`netcdf_datatypes`, you should be able to use both catalogs interchangeably, although almost everything behind the scenes works differently.\n",
Expand Down Expand Up @@ -120,28 +142,50 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "78b49e9f",
"execution_count": 3,
"id": "48a471aa",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"'d3751d33f9cd5049c4af2b462735457e4d3baf130bcbb87f389e349fbaeb20b9'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from hashlib import sha256\n",
"sha256(b\"hallo\").hexdigest()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "04f9dc28",
"execution_count": 4,
"id": "c0e9b2c3",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"'2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sha256(b\"hello\").hexdigest()"
]
},
{
"cell_type": "markdown",
"id": "9ec61e4b",
"id": "75a1848c",
"metadata": {},
"source": [
"Two similar (but different) input byte-sequences return completely different hash values.\n",
Expand All @@ -150,36 +194,58 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "17317dba",
"execution_count": 5,
"id": "4a94ede7",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"'702746827e553786bb026ac120cb58745fef3d3f554c33891809001cc37639f0'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"sha256(np.arange(1000)).hexdigest()"
]
},
{
"cell_type": "markdown",
"id": "7326dfb8",
"id": "7b1c05db",
"metadata": {},
"source": [
"And if we redo a hash-computation of the same input, we'll get exactly the same output as before:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "83f96c6e",
"execution_count": 6,
"id": "b7e82163",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"'2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sha256(b\"hello\").hexdigest()"
]
},
{
"cell_type": "markdown",
"id": "0a44970f",
"id": "f659bf44",
"metadata": {},
"source": [
"Furthermore, if you do this hash computation on your own machine, you'll obtain exactly the same value as shown here.\n",
Expand All @@ -204,8 +270,8 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "831ba5b7",
"execution_count": 7,
"id": "c75d1650",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -222,7 +288,7 @@
},
{
"cell_type": "markdown",
"id": "357aff3a",
"id": "206549f1",
"metadata": {},
"source": [
"The store is just a wrapper around a dictionary, with the crucial change that a `put` doesn't have the `key` as an argument, but as a return value.\n",
Expand All @@ -231,8 +297,8 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "4da8177f",
"execution_count": 8,
"id": "4adffa69",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -243,43 +309,65 @@
},
{
"cell_type": "markdown",
"id": "bb390cdf",
"id": "6a59a5bd",
"metadata": {},
"source": [
"As we know the key of `hello` from before, we can now use that to ask our store for that value:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2547e9f9",
"execution_count": 9,
"id": "4f10d043",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"b'hello'"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"store.get('2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824')"
]
},
{
"cell_type": "markdown",
"id": "6e74c4ea",
"id": "be5f2f2e",
"metadata": {},
"source": [
"Note that the modified version of `hello` with a capital `H` is also in the store, but of course under a different key:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2de43004",
"execution_count": 10,
"id": "20be05d9",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"b'Hello'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"store.get(k_Hello)"
]
},
{
"cell_type": "markdown",
"id": "b2269efb",
"id": "233ee63a",
"metadata": {},
"source": [
"Finding out the value of `k_Hello` is left as an exercise to the reader 😃.\n",
Expand All @@ -290,10 +378,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "08b329d3",
"execution_count": 11,
"id": "1d43fcc5",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"item count before put: 2\n",
"item count after put: 2\n"
]
}
],
"source": [
"print(f\"item count before put: {len(store._store)}\")\n",
"store.put(b\"hello\")\n",
Expand All @@ -302,7 +399,7 @@
},
{
"cell_type": "markdown",
"id": "5dc26185",
"id": "af668539",
"metadata": {},
"source": [
"Because every item is stored under it's own hash, identical items will only be stored once.\n",
Expand All @@ -320,21 +417,34 @@
},
{
"cell_type": "code",
"execution_count": null,
"id": "960e8d4b",
"execution_count": 12,
"id": "ac4f501d",
"metadata": {
"tags": [
"raises-exception"
]
},
"outputs": [],
"outputs": [
{
"ename": "KeyError",
"evalue": "'d3751d33f9cd5049c4af2b462735457e4d3baf130bcbb87f389e349fbaeb20b9'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[12], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mstore\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43md3751d33f9cd5049c4af2b462735457e4d3baf130bcbb87f389e349fbaeb20b9\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# the key of b\"hallo\"\u001b[39;00m\n",
"Cell \u001b[0;32mIn[7], line 9\u001b[0m, in \u001b[0;36mCAStore.get\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget\u001b[39m(\u001b[38;5;28mself\u001b[39m, key):\n\u001b[0;32m----> 9\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_store\u001b[49m\u001b[43m[\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m]\u001b[49m\n",
"\u001b[0;31mKeyError\u001b[0m: 'd3751d33f9cd5049c4af2b462735457e4d3baf130bcbb87f389e349fbaeb20b9'"
]
}
],
"source": [
"store.get('d3751d33f9cd5049c4af2b462735457e4d3baf130bcbb87f389e349fbaeb20b9') # the key of b\"hallo\""
]
},
{
"cell_type": "markdown",
"id": "8eb1fa04",
"id": "882ae7e9",
"metadata": {},
"source": [
"If there would be a method to find a store on another machine which does contain the missing key, this would open the door for a global and fault tolerant content distribution system.\n",
Expand Down Expand Up @@ -441,6 +551,18 @@
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
},
"source_map": [
12,
22,
Expand Down
Loading

0 comments on commit 4876889

Please sign in to comment.