Moving Data

This page provides an introduction to moving data around Gaea and PPAN, as well as transferring data to a local laptop or workstation.

f5<->f6

At this moment, f5 (f6) cannot directly access f6 (f5). Instead, users will need to transfer data through LDTN or RDTN nodes. Below is a quick example of how to submit an interaction job and transfer data between F5 and F6:

Yi-cheng.Teng@gaea55:~> salloc --partition=ldtn_c5
Yi-cheng.Teng@dtn57:~>module load gcp
Yi-cheng.Teng@dtn69:~> gcp -r /gpfs/f5/cefi/world-shared/nep_xml/ /gpfs/f6/ira-cefi/world-shared/upload/
Yi-cheng.Teng@dtn69:~> ls /gpfs/f6/ira-cefi/world-shared/upload/
nep_xml

Keep in mind that the interactive method only works on C5 currently. If users want to transfer data between F5 and F6 on C6, they will need to submit a job through the Slurm Queueing System. Below is an example script:

#!/usr/bin/sh -e
#SBATCH --cluster=es
#SBATCH --partition=ldtn_c5
#SBATCH --nodes=1
#SBATCH --time=04:00:00

# How to use:
# from c6 login node:
#sbatch f56.sh /gpfs/f5/... /gpfs/f6/...

# Check if the correct number of arguments is provided
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <source_directory> <destination_directory>"
    exit 1
fi

# Assign command-line arguments to variables
source_directory="$1" #like /gpfs/f6/cefi/scratch/Yi-cheng.Teng/github/ci_data/nwa12_input
destination_directory="$2" #like /gpfs/f6/ira-cefi/scratch/Yi-cheng.Teng/

/ncrc/usw/gcp/local/opt/gcp/2.3.33/gcp -cd -r gaea:"$source_directory" "$destination_directory"

let cp_status=$?+256
if [ $cp_status -ne 256 ]
then
    # An exit status greater than 256 will indicate the copy
    # failed.  Any exit status less than 256 will be a failure
    # from something else
    exit $cp_status
fi

Keep in mind that you will need to load the gcp module before submitting the script.

Gaea <-> GFDL

Users can transfer data between GFDL and Gaea filesystems with GCP. This can be done on the login nodes and rdtn’s only. Users can interactively run gcp commands from a login node or submit gcp calls in scripts to run in the rdtn queue.

module load gcp
gcp gaea:/gpfs/f5/<project>/scratch/$USER/file gfdl:/gfdl/specific/path/file
gcp gfdl:/gfdl/specific/path/file gaea:/gpfs/f5/<project>/scratch/$USER/path/file

PPAN <-> External (Untrusted) Data Transfers

To transfer data from PPAN/archive to your local laptop or computer, you can use Globus.. First, install Globus Connect Persona and create an endpoint on your local device. Once set up, use the Globus web interface to transfer data between NOAA RDHPCS UDTN (e.g., on PPAN, the transferred data will be placed under /collab1/data_untrusted/$USER) and your local computer.