Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup and improve text for RIKEN tutorial #34

Merged
merged 8 commits into from
Apr 13, 2024
Merged

Cleanup and improve text for RIKEN tutorial #34

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8db9e51
Fixes some URLs
ilumsden Apr 13, 2024
b956726
Updates notebook "goals"
ilumsden Apr 13, 2024
798fb57
Updates description of job throughput fig and adds proper figure numb…
ilumsden Apr 13, 2024
9772d79
Adds proper figure numbers and captions to module 3
ilumsden Apr 13, 2024
3b6a339
Adds fig numbers, captions, and footnotes to every figure in tutorial
ilumsden Apr 13, 2024
bdea7f8
Updates refs to figures in text to correspond to figure numbering
ilumsden Apr 13, 2024
5088251
Updates verb tenses, fixes a DYAD figure, and adds a link to survey
ilumsden Apr 13, 2024
3b1195c
Grammar
ilumsden Apr 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions 2024-RIKEN-AWS/JupyterNotebook/tutorial/notebook/01_flux_tutorial.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
" \n",
"Flux is a flexible framework for resource management, built for your site. The framework consists of a suite of projects, tools, services, and libraries which may be used to build site-custom resource managers for High Performance Computing centers. Flux is a next-generation resource manager and scheduler with many transformative capabilities like hierarchical scheduling and resource management (you can think of it as \"fractal scheduling\") and directed-graph based resource representations.\n",
"\n",
"To provide some brief, added background on Flux and a bit more motivation for our tutorial, start by watching our YouTube video:"
"To provide some brief, added background on Flux and a bit more motivation for our tutorial, let's start by watching our YouTube video:"
]
},
{
Expand Down Expand Up @@ -65,7 +65,7 @@
"\n",
"When run as a job in another resource manager, Flux is started like an MPI program, e.g., under Slurm we might run `srun [OPTIONS] flux start [SCRIPT]`. Flux is unique in that a test instance which mimics a multi-node instance can be started locally with simply `flux start --test-size=N`. This offers users to a way to learn and test interfaces and commands without access to an HPC cluster.\n",
"\n",
"To start a Flux session with 4 brokers in your container, run:"
"To start a Flux session with 4 brokers in your container, let's run:"
]
},
{
Expand All @@ -87,7 +87,7 @@
"\n",
"In this module, we introduced Flux and showed how to get started with Flux, particularly on systems using a different system-wide scheduler.\n",
"\n",
"Next, we use Flux for both traditional batch scheduling and hierarchical scheduling. To continue, open [Module 2](./02_flux_scheduling.ipynb)."
"To continue, open [Module 2](./02_flux_scheduling.ipynb)."
]
},
{
Expand Down
76 changes: 38 additions & 38 deletions 2024-RIKEN-AWS/JupyterNotebook/tutorial/notebook/02_flux_scheduling.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@
"<figure>\n",
"<img src=\"img/single-submit.png\">\n",
"<figcaption>\n",
"<i>Image created by Vanessa Sochat for Flux Framework Components documentation</i></figcaption>\n",
"Figure 1: Traditional batch schedulers use a centralized design<sup>1</sup>\n",
"</figcaption>\n",
"</figure>\n",
"\n",
"Traditional schedulers provide 3 main operations:\n",
Expand Down Expand Up @@ -135,9 +136,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Similar to Slurm's `salloc`, users can submit interactive jobs using `flux alloc`. When launching an interactive job, you can request resources using the same flags that you would pass to `flux batch` (e.g., `-N` for requesting a number of nodes).\n",
"Similar to Slurm's `salloc`, users can submit interactive jobs using `flux alloc`. When launching an interactive job, users can request resources using the same flags that you would pass to `flux batch` (e.g., `-N` for requesting a number of nodes).\n",
"\n",
"Due to Jupyter's lack of a pseudo-terminal, we cannot show `flux alloc` in this notebook. So, we will open a terminal in Jupyter. To do so, click on `FILE -> NEW -> TERMINAL`. Then, copy and paste the following commands into the terminal:\n",
"Due to Jupyter's lack of a pseudo-terminal, we cannot show `flux alloc` in this notebook. So, we open a terminal in Jupyter. To do so, click on `FILE -> NEW -> TERMINAL`. Then, copy and paste the following commands into the terminal:\n",
"\n",
"```bash\n",
"$ flux alloc --nodes=2 --nslots=2 --cores-per-slot=1\n",
Expand All @@ -161,24 +162,6 @@
"Code(filename='hello-batch.sh', language='bash')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Optional: connecting to an existing Flux instance using flux proxy\n",
"\n",
"TODO check if this text or original example should be put in supplement\n",
"\n",
"One cool feature that Flux provides is the ability to connect to an existing Flux instance/allocation from any other node of the system using `flux proxy`. To use this command, we first need to get the ID of the Flux instance we want to connect to. Assuming the interactive job we just launched is still running, we can get the job ID (which is the same as the instance ID) using `flux jobs`. Once we have that ID, we can run `flux proxy <ID>` to connect to that Flux instance.\n",
"\n",
"Once we're connected to the interactive allocation, we can run the following job in one terminal:\n",
"```bash\n",
"$ flux run --nodes=1 sleep inf\n",
"```\n",
"\n",
"Then, in the other terminal, we should be able to see that `flux run` by running `flux jobs`."
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -233,7 +216,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, this script runs 4 different `flux submit` commands, each of which prints a message to a different file. If this script were to use `flux run`, these commands would run one after the other. Instead, by using `flux submit` instead of `flux run`, Flux can run all of these `echo` programs in parallel (assuming there are enough resources to do so). This means the job that runs this script can (theoretically) complete **4 times faster** than it could using `flux run`.\n",
"As you can see, this script runs 4 different `flux submit` commands, each of which prints a message to a different file. If this script were to use `flux run`, these commands would run one after the other. Instead, by using `flux submit` instead of `flux run`, Flux runs all of these `echo` programs in parallel (assuming there are enough resources to do so). As a result, the job that runs this script can (theoretically) complete **4 times faster** than it could using `flux run`.\n",
"\n",
"Because `flux submit` does not wait for jobs, batch scripts that use this command must use another approach for waiting on job completion. To help with this scenario, Flux provides the `flux job wait` command, which waits for the specified job/program (or all of them if the `--all` flag is provided) to complete. *Note that, to use `flux job wait`, you must pass the `--flags=waitable` flag to your Flux command.*\n",
"\n",
Expand Down Expand Up @@ -280,7 +263,7 @@
"source": [
"The flags provided to `flux bulksubmit` tell it to print the output of each job to the terminal and wait for all the jobs to finish before returning.\n",
"\n",
"Second, there is the `-cc` flag to `flux submit`. This flag tells Flux to spawn multiple copies of a single command with different job IDs. Unlike `flux bulksubmit`, you cannot substitute arbitrary values into the command. Instead, when using the `-cc` flag, you can only substitute the job ID using `{cc}`.\n",
"Second, there is the `-cc` flag to `flux submit`. This flag tells Flux to spawn multiple copies of a single command with different job IDs. Unlike `flux bulksubmit`, users cannot substitute arbitrary values into the command. Instead, when using the `-cc` flag, users can only substitute the job ID using `{cc}`.\n",
"\n",
"Let's run a simple example of `flux submit`."
]
Expand Down Expand Up @@ -322,7 +305,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"To see the status of all pending, running, or completed jobs, we will run `flux jobs`."
"To see the status of all pending, running, or completed jobs, we run `flux jobs`."
]
},
{
Expand Down Expand Up @@ -421,31 +404,34 @@
"source": [
"## Hierarchical scheduling with Flux\n",
"\n",
"With traditional batch schedulers (e.g., Slurm), all job requests from all users are submitted to one centralized service. In this case, the maximum job throughput is one job per second.\n",
"With traditional batch schedulers (e.g., Slurm), all users submit all their job requests to one centralized service. In this case, the maximum job throughput is one job per second.\n",
"\n",
"<figure>\n",
"<img src=\"img/single-submit.png\">\n",
"<figcaption>\n",
"<i>Image created by Vanessa Sochat for Flux Framework Components documentation</i></figcaption>\n",
"Figure 2: Traditional batch schedulers use a centralized design<sup>1</sup>\n",
"</figcaption>\n",
"</figure>\n",
"\n",
"The throughput of this approach is limited by the scheduler's ability to process a single job. To improve throughput, Flux introduces the ability to launch multiple Flux instances within an existing Flux instance. This creates a hierarchy of Flux instances across which job requests can be distributed. For example, let's say we create a Flux instance that has control of some number of nodes. We then create 3 child instances (each with its own scheduler and queue). By scheduling across this hierarchy of instances, we get a throughput of 1x3, or 3 jobs per second.\n",
"The scheduler's ability to process a single job limits the throughput of this approach. To improve throughput, Flux introduces the ability to launch multiple Flux instances within an existing Flux instance. This creates a hierarchy of Flux instances across which job requests are distributed. For example, let's say we create a Flux instance that has control of some number of nodes. We then create 3 child instances (each with its own scheduler and queue). By scheduling across this hierarchy of instances, we get a throughput of 1x3, or 3 jobs per second.\n",
"\n",
"<figure>\n",
"<img src=\"img/instance-submit.png\">\n",
"<figcaption>\n",
"<i>Image created by Vanessa Sochat for Flux Framework Components documentation</i></figcaption>\n",
"Figure 3: Hierarchical scheduling improve throughput by dividing work across multiple schedulers<sup>1</sup>\n",
"</figcaption>\n",
"</figure>\n",
"\n",
"By leveraging a hierarchy of Flux instances to achieve a divide-and-conquer approach to scheduling, we can exponentially increase throughput. The figure below (from our [learning guide](https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html#fully-hierarchical-resource-management-techniques)) shows this exponential increase in an actual experiment. We submit 500 jobs/second using only a three-level hierarchy, whereas a centralized scheduler (1-Level in the figure) achieves only one 1 job/second.\n",
"By leveraging a hierarchy of Flux instances to achieve a divide-and-conquer approach to scheduling, we exponentially increase throughput. Figure 4 (from our [learning guide](https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html#fully-hierarchical-resource-management-techniques)) shows this exponential increase in an actual experiment. We submit a 4096 job workload to three different scheduler configurations managing resources ranging from 1 core to 1152 cores on LLNL's [Quartz](https://hpc.llnl.gov/hardware/compute-platforms/quartz) supercomputer. When running with a three-level hierarchy, we scale to a maximum scheduling throughput of over 500 jobs/second. On the other hand, a centralized scheduler (1-Level in the figure) achieves only a constant throughput of 1 job/second. \n",
"\n",
"<figure>\n",
"<img src=\"img/scaled-submit.png\">\n",
"<figcaption>\n",
"<i>Image from <a href=\"https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html#fully-hierarchical-resource-management-techniques\">Flux learning guide</a></i></figcaption>\n",
"Figure 4: Hierarchical scheduling enables much higher throughput than traditional, centralized scheduling, even for small hierarchies<sup>2</sup>\n",
"</figcaption>\n",
"</figure>\n",
"\n",
"There are different ways to create hierarchies of Flux instances. In this tutorial, we will focus on 2 of them:\n",
"There are different ways to create hierarchies of Flux instances. In this tutorial, we focus on 2 of them:\n",
"1. Nested invocations of `flux batch`\n",
"2. The `flux tree` command"
]
Expand Down Expand Up @@ -487,9 +473,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"When scheduled with `flux batch`, `sub_job1.sh` will run in a new Flux instance. It will then run `flux batch` again to run `sub_job2.sh`. Because the second `flux batch` command is within `sub_job1.sh`, the job request produced by the second `flux batch` command will go to the scheduler of the child Flux instance instead of the parent Flux instance.\n",
"When scheduled with `flux batch`, `sub_job1.sh` runs in a new Flux instance. It then runs `flux batch` again to run `sub_job2.sh`. Because the second `flux batch` command is within `sub_job1.sh`, the job request produced by the second `flux batch` command goes to the scheduler of the child Flux instance instead of the parent Flux instance.\n",
"\n",
"We can see this in action by running the cell below."
"Let's run the cell below to see this in action."
]
},
{
Expand All @@ -505,7 +491,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Once we have submitted `sub_job1.sh`, we can look at the hierarchy for all the jobs we've run using `flux pstree`. Normally, this command can be used to show jobs in a Flux instance. However, since we are running in a Jupyter notebook, this command will have limited functionality. So, instead of just running the single command, we will run `flux pstree -a` to look at **all** jobs. In a more complex environment with more jobs, this command would show a deeper nesting. You can see examples of more complex outputs [here](https://flux-framework.readthedocs.io/en/latest/jobs/hierarchies.html?h=pstree#flux-pstree-command)."
"Once we have submitted `sub_job1.sh`, we look at the hierarchy for all the jobs we've run using `flux pstree`. Normally, this command is used to show jobs in a Flux instance. However, since we are running in a Jupyter notebook, this command has limited functionality. So, instead of just running the single command, we run `flux pstree -a` to look at **all** jobs. In a more complex environment with more jobs, this command would show a deeper nesting. We can see examples of more complex outputs [here](https://flux-framework.readthedocs.io/en/latest/jobs/hierarchies.html?h=pstree#flux-pstree-command)."
]
},
{
Expand Down Expand Up @@ -554,7 +540,8 @@
"<figure>\n",
"<img src=\"img/flux-tree.png\">\n",
"<figcaption>\n",
"<i>Image created by Ian Lumsden based on images by Vanessa Sochat</i></figcaption>\n",
"Figure 5: The example of <code>flux tree</code> in this notebook produces this hierarchy<sup>3</sup>\n",
"</figcaption>\n",
"</figure>\n",
"\n",
"For this tutorial, we show `flux tree` with a relatively simple job (i.e., `hostname`). However, since this command accepts any valid jobspec that can be recognized by `flux submit`, it can be used to rapidly deploy much more complex scenarios, including scenarios where different programs are run on each leaf-level instance."
Expand All @@ -566,13 +553,26 @@
"source": [
"# This concludes Module 2.\n",
"\n",
"In this module, we demonstrated how to:\n",
"1. Use Flux for traditional batch scheduling similar to what is provided by other schedulers like Slurm\n",
"2. Use Flux for hierarchical scheduling to achieve greater scheduling throughput\n",
"In this module, we demonstrated:\n",
"1. Traditional batch scheduling with Flux (similar to what is provided by other schedulers like Slurm)\n",
"2. Hierarchical scheduling with Flux to achieve higher throughput (novel capability of Flux)\n",
"\n",
"To continue with the tutorial, open [Module 3](./03_flux_framework.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<u>Footnotes:</u>\n",
"\n",
"<sup>1. Image created by Vanessa Sochat for Flux Framework Components documentation</sup>\n",
"\n",
"<sup>2. Image from <a href=\"https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html#fully-hierarchical-resource-management-techniques\">Flux learning guide</a></sup>\n",
"\n",
"<sup>3. Image created by Ian Lumsden based on images by Vanessa Sochat</sup>"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down
Loading
Loading