add: flux riken tutorial 2024 #28
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds the new directory for the Flux Riken tutorial, with the following additions: 1. flux-tree was removed from flux-sched and is added here 2. tutorial files were kept in rse-ops, are now moved here 3. New tutorial content: flux tree and hierarchy section/examples 4. New tutorial content: flux archive (previously flux filemap) 5. images that show a dummy example of job throughout 6. update of names in login page / directory to be more general 7. automated builds updated for riken Signed-off-by: vsoch <[email protected]>
vsoch
force-pushed
the
add/2024-riken-aws
branch
from
23db79f to
0555ba6
Compare
garlick
reviewed
Apr 4, 2024
* Updates Dockerfile.spawn and requirements.txt to handle the newest version of DYAD and DLIO * Moves copy/move of tutorial materials into the root user block * Adds intro, tutorial code cells, and extra dependencies to DYAD portion of the tutorial * Adds cleanup for the data generation directory in DYAD portion of tutorial * DLIO version of the Docker file. * Starts to break the tutorial down into modules * Minor text changes to 02_flux_scheduling.ipynb * Adds a description of the scheduling policies used in flux tree example * Adds an image for flux tree * Adds attribution to images * Adds module labels throughout tutorial * Adds Flux logo to other notebooks * Makes a couple of fixes for DLIO use case * Last few bugfixes in DYAD notebook * Minor changes to Flux scheduling notebook to correct job waiting * Moves the YouTube video in the intro to a code cell so it will work in JupyterHub * Adds DYAD to LD_LIBRARY_PATH before launching DLIO * Renames dyad_dlio.ipynb to 04_dyad_dlio.ipynb * Updates Dockerfile.spawn to get the DLIO use case working * Adds a tutorial-specific copy of the DYAD Torch data loader * Minor bugfixes after moving DLIO extensions into the repo * Simplifies the DYAD Torch data loader * Adds reference to LC table and flux proxy optional section * Adds a step to docker-builds to remove unneeded stuff from runner * Adds module 3 * Small summary change * Finishes the DYAD notebook * Adds the supplement and updates the conclusions * Tweaks a figure width * Removes dead changes for testing DLIO * Removes the old DYAD notebook from previous years since it's been superceded by 04_dyad_dlio.ipynb --------- Co-authored-by: Hariharan Devarajan <[email protected]>
Problem: we need to test deployment tomorrow with riken containers Solution: update aws configs to use the TBA builds Signed-off-by: vsoch <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the new directory for the Flux Riken tutorial, with the following additions:
I have the basic images here if they need to be edited (likely!) let me know and I can add you to it. https://docs.google.com/presentation/d/18FjFiSxonwAwjg4k-2pgRPBoSb_4ra4V5bslcyJhong/edit?usp=sharing