remote Jupyter Lab on Frontera #40

Aangniu · 2024-05-10T14:50:31Z

1 It seems that the current tacc-apptainer module on Frontera does not support 'sigularity' as the command. But 'apptainer' serves the purpose.
2 It seems the current docker will automatically build a singularity container. I tested that it can also be used. So I add the following suggestion to the frontera.md
You can also use the automatically generated container after pulling the docker container

module load tacc-apptainer
apptainer pull -F docker://seissol/training:latest
apptainer run training_latest.sif

3 I change the name of 'northridge/northridge_resampled.srf' to 'northridge/northridge_resampled.nrf'.

Aangniu · 2024-05-10T14:52:00Z

I will test the generated docker and get back to confirm the changes are valid.

sebwolf-de · 2024-05-10T15:13:49Z

northridge/northridge_resampled.nrf

Please don't rename this. An srf file is an ASCII file defined here: http://equake-rc.info/static/paper/SRF-Description-Graves_2.0.pdf.
An nrf file is our SeisSol specific netcdf (binary) converted version of an srf file.

Uh, OK. I will than just rename the parameter file:

Before it was still asking for .nrf file

&SourceType Type = 42 ! 42: finite source in netcdf format FileName = './northridge_resampled.srf' ! input file. /

there is an intermediate step that convert srf to nrf (see rconv in readthedocs).
So you do not need to change the parameter file.

Oh, sorry. Maybe it means I just need to convert that srf to nrf after pulling the files.

yes, search for rconv in https://github.com/SeisSol/Training/blob/main/northridge/Northridge.ipynb

Found it in the python notebook. I will test it on Frontera again then.

!rconv -i northridge_resampled.srf -o northridge_resampled.nrf -x visualization.xdmf -m "+proj=tmerc +datum=WGS84 +k=0.9996 +lon_0=-118.5150 +lat_0=34.3440 +axis=enu"

I checked that I can use the above command to properly generate the .nrf file.

1 I removed my previous rename of the -srf file. So with the latest commit, the only difference between the main branch is in the README file.
2 I add the following instructions on how to run the Northridge senario to the README.

To run the northridge scenario, you should:

cd seissol-training/northridge mpirun apptainer run ~/my-training.sif pumgen -s msh2 mesh_northridge.msh mpirun apptainer run ~/my-training.sif rconv -i northridge_resampled.srf -o northridge_resampled.nrf -x visualization.xdmf -m "+proj=tmerc +datum=WGS84 +k=0.9996 +lon_0=-118.5150 +lat_0=34.3440 +axis=enu" OMP_NUM_THREADS=28 mpirun -n 2 apptainer run ~/my-training.sif seissol parameters.par

You can change seissol to SeisSol_Release_dhsw_4_viscoelastic2 if you want to run visco-elastic simulation instead of the default elastic one.

But rconv is not MPI parallel, so you can omit the mpirun there.

Thanks for pointing out. I removed the mpirun for rconv.

wangyinz · 2024-05-10T16:30:25Z

The apptainer module still has the singularity command and it is simply a symbolic link to the apptainer binary, so it is fine to call any of those singularity commands. Still, I think it is less confusing to update that and use apptainer directly.

$ ls -l /opt/apps/tacc-apptainer/1.1.8/bin
total 50892
-rwxr-xr-x 1 root root 52107416 May  2  2023 apptainer
-rwxr-xr-x 1 root root     1455 May  2  2023 run-singularity
lrwxrwxrwx 1 root root        9 May 10  2023 singularity -> apptainer

Aangniu · 2024-05-10T20:41:14Z

Thanks for commenting on this @wangyinz !
I checked again. You are right, the 'singularity' command also works.

sebwolf-de · 2024-05-13T07:22:29Z

Apptainer is the new name, so I'd like to use apptainer wherever possible, such that we have forward compability.

…your local machin.

Aangniu · 2024-05-20T19:08:52Z

I added the job.jupyter for accessing jupyter notebook on Frontera with local machine.

Aangniu · 2024-05-20T19:11:33Z

The environment for visualization is installed while creating the jupyter environment in job.jupyter with

pip install vtk pyvista

frontera.md

Thomas-Ulrich · 2024-05-21T07:32:20Z

frontera.md

+```
+Step 6: Paste the link to your local browser, you will have access to the Frontera environment on your local machine.
+```
+https://frontera.tacc.utexas.edu:60320/?token=2e0fade1f8b1ce00b303a7e97dd962c5cd10c17f03a245e8c761ca7e1d5e1597


Works, but then I get:

also note that OMP_NUM_THREADS=4 could be increased
(could add a note on that)

ok I found from slack that you have to use, e.g.
!mpirun singularity run /work2/09160/ulrich/frontera/Training/my_training.sif -3 tpv13_training.geo

Here is my solution (and to be honest meta-llama's):

# Define variables depending on the environment is_remote = True # Set to True if running remotely on Frontera, False otherwise singularity_image = '/work2/09160/ulrich/frontera/Training/training_pr-40.sif' # Define a function to run a command with an optional Singularity wrapper def run_command(cmd, singularity=False): if singularity: return f"mpirun singularity run {singularity_image} {cmd}" else: return f"{cmd}

and then:

gmsh_cmd = run_command("gmsh -3 tpv13_training.geo", is_remote) !{gmsh_cmd}

and also:

pumgen_cmd = run_command("pumgen -s msh2 tpv13_training.msh", is_remote) seissol_cmd = run_command("OMP_NUM_THREADS=28 seissol parameters.par", is_remote)

Yep, you are right. That was because it is not running in the container environment, but just on Frontera. So we will need to run with something like mpirun apptainer run <container> pumgen ...

Alice suggested in the slack we'd better have an additional session in the notebook for running remotely on Frontera.

my option allows not duplicating the notebook.
@davschneller @sebwolf-de what do you think? How should we proceed?

other option is

!pumgen -s msh2 tpv13_training.msh # on frontera with singularity # !mpirun singularity run $singularity_image pumgen -s msh2 tpv13_training.msh

Yes, i think this option is clear for both remote and frontera users

frontera.md

Thomas-Ulrich · 2024-05-21T07:46:22Z

frontera.md

 ```
+You can change `seissol` to `SeisSol_Release_dhsw_4_viscoelastic2` if you want to run visco-elastic simulation instead of the default elastic one.


if you want to account for attenuation (https://seissol.readthedocs.io/en/latest/attenuation.html) instead of assuming a fully elastic rheology?

Thanks, updated in the latest commit.

Thomas-Ulrich · 2024-05-21T07:49:32Z

frontera.md

 ```

 You can abort the jupyter lab with Ctrl-C, confirm with `y`.
 Now you should see a directory `seissol-training`.
 This folder should contain four directories for different scenarios.

+You can also open the jupyter notebook that runs on Frontera on your local machine with the following steps:
+
+Step 1: change `SHARED_PATH="/your/path/to/container/"` in line 75 of `job.jupyter` to the path where your sigularity container is built.


could we have SHARED_PATH as an argument and run e.g.
sbatch job.jupyter /work2/09160/ulrich/frontera/Training/my_training.sif

and then in job.jupyter
my_container=$1?
(I mean I don't know if that works with sbatch)

frontera.md

AliceGabriel · 2024-05-22T00:22:35Z

@sebwolf-de we would like to merge this one before the training tomorrow

update frontera.md change file name in northridge/

b7fd0ae

Aangniu requested a review from Thomas-Ulrich May 10, 2024 14:50

Thomas-Ulrich requested a review from wangyinz May 10, 2024 14:54

sebwolf-de requested changes May 10, 2024

View reviewed changes

bring back the .srf file in northridge/

04cb3eb

Aangniu added 2 commits May 10, 2024 22:51

add explanations to how to run the Northridge example

26c945c

n to N

fbf3af4

Aangniu requested a review from sebwolf-de May 10, 2024 21:09

Aangniu added 5 commits May 13, 2024 10:43

remove mpirun for rconv

a50c3d2

add job.jupyter for accessing notebook on Frontera locally

2dea390

add documentation on how to access jupyter notebook in Frontera with …

16383c3

…your local machin.

correct format errors

1375a99

correct format errors 2

e76da1c

Thomas-Ulrich reviewed May 21, 2024

View reviewed changes

Thomas-Ulrich changed the title ~~update frontera.md change file name in northridge/~~ remote Jupyter Lab on Frontera May 21, 2024

Aangniu added 3 commits May 21, 2024 13:15

update frontera readme with suggestion from Thomas

892623f

update to tail -f

93fa2c6

update format

5fc0b49

Thomas-Ulrich reviewed May 21, 2024

View reviewed changes

frontera.md Show resolved Hide resolved

Thomas-Ulrich reviewed May 21, 2024

View reviewed changes

frontera.md Outdated Show resolved Hide resolved

Aangniu added 4 commits May 21, 2024 15:27

ln -s ./training_latest.sif ~/my-training.sif

9b943b9

OMP_NUM_THREADS=26 mpirun -n 2 apptainer

f2c92e6

ln -s /absolute/path/to/training_latest.sif ~/my-training.sif

6cb666c

also use 26 threads for tpv13

8d62caa

Thomas-Ulrich and others added 5 commits May 21, 2024 22:15

add instructions for Frontera

e524b35

remove outdated parameters

9bb8185

Merge pull request #45 from SeisSol/thomas/zihua_testing

a29aa82

update container to hps-2024-frontera

b5568ff

update to Thomas changes

b5564d5

AliceGabriel approved these changes May 22, 2024

View reviewed changes

jwjeremy approved these changes May 22, 2024

View reviewed changes

sebwolf-de approved these changes May 22, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into zihua/testing

b1ba9b0

Thomas-Ulrich merged commit e31b2b2 into main May 22, 2024
1 check passed

Thomas-Ulrich deleted the zihua/testing branch May 22, 2024 09:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remote Jupyter Lab on Frontera #40

remote Jupyter Lab on Frontera #40

Aangniu commented May 10, 2024

Aangniu commented May 10, 2024

sebwolf-de May 10, 2024 •

edited

Loading

Aangniu May 10, 2024

Aangniu May 10, 2024

Thomas-Ulrich May 10, 2024

Aangniu May 10, 2024

Thomas-Ulrich May 10, 2024

Aangniu May 10, 2024

Aangniu May 10, 2024

sebwolf-de May 13, 2024

Aangniu May 13, 2024

wangyinz commented May 10, 2024

Aangniu commented May 10, 2024

sebwolf-de commented May 13, 2024

Aangniu commented May 20, 2024

Aangniu commented May 20, 2024

Thomas-Ulrich May 21, 2024

Thomas-Ulrich May 21, 2024

Thomas-Ulrich May 21, 2024

Thomas-Ulrich May 21, 2024

Thomas-Ulrich May 21, 2024

Aangniu May 21, 2024

Aangniu May 21, 2024

Thomas-Ulrich May 21, 2024

Thomas-Ulrich May 21, 2024 •

edited

Loading

jwjeremy May 21, 2024

Thomas-Ulrich May 21, 2024

Aangniu May 21, 2024

Thomas-Ulrich May 21, 2024

AliceGabriel commented May 22, 2024

		```
		You can change `seissol` to `SeisSol_Release_dhsw_4_viscoelastic2` if you want to run visco-elastic simulation instead of the default elastic one.

remote Jupyter Lab on Frontera #40

remote Jupyter Lab on Frontera #40

Conversation

Aangniu commented May 10, 2024

Aangniu commented May 10, 2024

sebwolf-de May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyinz commented May 10, 2024

Aangniu commented May 10, 2024

sebwolf-de commented May 13, 2024

Aangniu commented May 20, 2024

Aangniu commented May 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thomas-Ulrich May 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AliceGabriel commented May 22, 2024

sebwolf-de May 10, 2024 •

edited

Loading

Thomas-Ulrich May 21, 2024 •

edited

Loading