Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cmor and logging and yaml #326

Draft
wants to merge 25 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
9ecc707
add --no-wait and --pause options to fre pp run, dinking around with …
ilaflott Jan 17, 2025
5197ba9
add verbose flag to base fre command and define fre_logger in fre/fre…
ilaflott Jan 17, 2025
3416118
print --> logger.info or logger.warn
ilaflott Jan 17, 2025
1e5a69f
loggers... loggers... EVERYWHERE!!!!
ilaflott Jan 17, 2025
ed23206
put new log settings in pytest.ini. set format of logging in fre.py a…
ilaflott Jan 17, 2025
f837840
see if this keeps the pipeline from being grumpy... maybe TOO much lo…
ilaflott Jan 17, 2025
03acb95
update git ignore
ilaflott Jan 22, 2025
4ce479a
prelim cmor yaml and cmor_yamler, very rough, but lets get back into …
ilaflott Jan 31, 2025
190e513
merge with recent version of main
ilaflott Jan 31, 2025
086b925
unify logging module usage, at least partially. subtools are still al…
ilaflott Jan 31, 2025
ce9e8a6
touch up the fre cmor yaml routine bits- bones and some other things.…
ilaflott Jan 31, 2025
508d574
ok fine i am happy with that for now.
ilaflott Jan 31, 2025
8db16df
update with cracking open the yaml file
ilaflott Jan 31, 2025
5638047
spot check- tests and basic yaml to slowly-transition to smth else.
ilaflott Jan 31, 2025
78767d6
proper unit test- plugged in. naive yaml case works fine.
ilaflott Jan 31, 2025
b7df165
catch up and push- show me what i am working with
ilaflott Jan 31, 2025
df0899d
test yaml deets to test routine
ilaflott Feb 11, 2025
290bacf
add quick/dirty scratch work script im using to check on yaml things
ilaflott Feb 11, 2025
22aa889
oh, yaml doesn't hate whitespace quite as much as i thought, have yam…
ilaflott Feb 11, 2025
7d81c86
Merge branch 'main' into cmor-and-logging-and-yaml
ilaflott Feb 11, 2025
2141987
bump cmor cmip6 table submodule commit
ilaflott Feb 11, 2025
4588744
tweak git ignore
ilaflott Feb 11, 2025
e0a157b
lets go with something like this for the yaml.... coalescing... tweak…
ilaflott Feb 12, 2025
eab37a9
yamlcheck edits
ilaflott Feb 13, 2025
d476680
rename yaml
ilaflott Feb 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add --no-wait and --pause options to fre pp run, dinking around with …
…logger and where to first make one. fre.py?
ilaflott committed Jan 17, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 9ecc7079b5f4ff9a68bd1fac2061fa21696f608e
2 changes: 2 additions & 0 deletions fre/fre.py
Original file line number Diff line number Diff line change
@@ -50,6 +50,8 @@

def fre():
''' entry point function to subgroup functions '''
print('hello from fre.py fre()')

if __name__ == '__main__':
print('hello from fre.py __main__')
fre()
10 changes: 8 additions & 2 deletions fre/pp/frepp.py
Original file line number Diff line number Diff line change
@@ -45,9 +45,15 @@ def status(experiment, platform, target):
@click.option("-t", "--target", type=str,
help="Target name",
required=True)
def run(experiment, platform, target):
@click.option("--pause", is_flag=True, default=False,
help="Pause the workflow immediately on start up",
required=False)
@click.option("--no_wait", is_flag=True, default=False,
help="after submission, do not wait to ping the scheduler and confirm success",
required=False)
def run(experiment, platform, target, pause, no_wait):
""" - Run PP configuration"""
run_script.pp_run_subtool(experiment, platform, target)
run_script.pp_run_subtool(experiment, platform, target, pause, no_wait)

# fre pp validate
@pp_cli.command()
22 changes: 18 additions & 4 deletions fre/pp/run_script.py
Original file line number Diff line number Diff line change
@@ -3,7 +3,8 @@
import subprocess
import time

def pp_run_subtool(experiment = None, platform = None, target = None):
def pp_run_subtool(experiment = None, platform = None, target = None,
pause = False, no_wait = False):
"""
Start or restart the Cylc workflow identified by:
<experiment>__<platform>__<target>
@@ -21,16 +22,29 @@ def pp_run_subtool(experiment = None, platform = None, target = None):
return

# If not running, start it
cmd = f"cylc play {name}"
cmd = "cylc play"
if pause:
cmd+=" --pause"
cmd +=f" {name}"
subprocess.run(cmd, shell=True, check=True)

# wait 30 seconds for the scheduler to come up; then confirm
# not interested in the confirmation? gb2work now
if no_wait:
return

# give the scheduler 30 seconds of peace before we hound it
print("Workflow started; waiting 30 seconds to confirm")
time.sleep(30)
result = subprocess.run(['cylc', 'scan', '--name', f"^{name}$"], capture_output=True).stdout.decode('utf-8')

# confirm the scheduler came up. note the regex surrounding {name} for start/end of a string to avoid glob matches
result = subprocess.run(
['cylc', 'scan', '--name', f"^{name}$"],
capture_output = True ).stdout.decode('utf-8')

if not len(result):
raise Exception('Cylc scheduler was started without error but is not running after 30 seconds')

print(result)

if __name__ == "__main__":
pp_run_subtool()