Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restrict ebuser and let it submit jobs #58

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Restrict ebuser and let it submit jobs #58

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 41 additions & 17 deletions eb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ unset CPATH
unset LIBRARY_PATH
unset LD_LIBRARY_PATH
export PATH=/usr/bin
if [[ $HOSTNAME =~ archimedes.c3.ca$ ]]; then
if [[ $(hostname) =~ ^login..int.archimedes.c3.ca$ ]]; then
export PATH=$PATH:/opt/software/slurm/bin
export EASYBUILD_JOB_BACKEND='Slurm'
export SBATCH_MEM_PER_CPU='3500m'
Expand Down Expand Up @@ -126,9 +126,14 @@ export RSNT_EASYBUILD_MAGIC_COOKIE=263ca73bb634185aab1d1b41627fdbba

# argument validation
NEW_ARGS=()
for argument in "$@"; do
if [[ "$CURRENT_USER" == "ebuser" ]]; then
if [[ $argument =~ --inject-checksums* || $argument =~ --robot* || "$argument" == "-r" ]]; then
if [[ "$CURRENT_USER" == "ebuser" ]]; then
if ! [[ $(hostname) =~ ^login..archimedes.c3.ca$ ]]; then
echo "Please only submit as ebuser on login nodes"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why restrict it to login node only ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you want compute node sessions to send recursive jobs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say the following is valid workflow:

  1. salloc to get a dedicated node
  2. eb as myself to test and develop
  3. sudo ebuser eb

i.e. to use compute nodes as development nodes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but for 3. should it spin off a job to a second compute node or stay on that node? Compile for both avx2 and avx512 or avx2 only?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say it should behave just as if it was done on the login node, i.e. launch a new job and build for both avx2 and avx512

exit 1
fi
ec=""
for argument in "$@"; do
if [[ $argument =~ --inject-checksums* || $argument =~ --robot* || "$argument" == "-r" || "$argument" == "--job" ]]; then
echo "Please do not use $argument as $CURRENT_USER"
exit 1
fi
Expand All @@ -138,10 +143,6 @@ for argument in "$@"; do
exit 1
fi
fi
if [[ $argument =~ ^--job$ ]]; then
echo "Output of job will be in /cvmfs/local/var/log/%x-%j.out"
cd /cvmfs/local/var/log
fi
if [[ $argument =~ .*.eb$ ]]; then
if [[ $argument =~ ^/cvmfs/soft.computecanada.ca/easybuild/software/.* ]]; then
NEW_ARGS+=("$argument")
Expand All @@ -159,22 +160,45 @@ for argument in "$@"; do
letter=${letter,} # lowercase first letter
name=${argument%%-*}
# some names contain - (e.g. HOOMD-blue) so we need pathname expansion
ec=$(find $EASYBUILD_ROOT/easyconfigs/$letter/$name* -name $argument 2>/dev/null)
custom_ec=$(find $EASYBUILD_ROOT/site-packages/custom-easyconfigs/easybuild/easyconfigs/$letter/$name* -name $argument 2>/dev/null)
if [[ -n "$ec" && -f $ec && -z "$custom_ec" ]]; then
NEW_ARGS+=("$ec")
else # if no corresponding easyconfig was found in easyconfigs folder, we get it found normally
NEW_ARGS+=("$argument")
if [[ -n "$custom_ec" && -f "$custom_ec" ]]; then
# custom easyconfigs have priority
ec="$custom_ec"
else
ec=$(find $EASYBUILD_ROOT/easyconfigs/$letter/$name* -name $argument 2>/dev/null)
if [[ -z "$ec" || ! -f "$ec" ]]; then
echo "Easyconfig $argument not found"
exit 1
fi
fi
NEW_ARGS+=("$ec")
fi
else
NEW_ARGS+=("$argument")
fi
done
if [[ -z "$ec" ]]; then
echo "No easyconfig specified, exiting"
exit 1
fi
done

if [[ "$CURRENT_USER" == "ebuser" ]]; then
exec $EB --configfiles=$LOCAL_EASYBUILD_CONFIGFILES "${NEW_ARGS[@]}"
cd /cvmfs/local/var/log
$EB --fetch --configfiles=$LOCAL_EASYBUILD_CONFIGFILES "${NEW_ARGS[@]}"
export RSNT_ARCH=avx2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This essentially forbids building for avx or sse3. Even though we don't do it often, I don't think it should be forbidden ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, we need to allow that

$EB --job --configfiles=$LOCAL_EASYBUILD_CONFIGFILES "${NEW_ARGS[@]}"
python <<EOF
import sys
from easybuild.framework.easyconfig.parser import EasyConfigParser
ecp = EasyConfigParser("$ec")
name = ecp.get_config_dict()['toolchain']['name']
sys.exit(name == 'system' or name == 'dummy')
mboisson marked this conversation as resolved.
Show resolved Hide resolved
EOF
if [[ $? == 0 ]]; then
export RSNT_ARCH=avx512
$EB --job --configfiles=$LOCAL_EASYBUILD_CONFIGFILES "${NEW_ARGS[@]}"
else
echo "Not building for x86-64 (SYSTEM toolchain)"
fi
echo "Output of job(s) will be in /cvmfs/local/var/log/%x-%j.out"
else
exec $EB --configfiles=$LOCAL_EASYBUILD_CONFIGFILES ${1+"$@"}
fi