Fix the ufs-weather-model package at spack

[edwardhartnett]

Fix the ufs-weather-model package at spack

[edwardhartnett]

#374 is related to this issue.

Basically, we need outside users to be able to build this as well as users with spack-stack installed.

This will mean adding this yaml file to the main spack repo, if it's not already there.

[AlexanderRichert-NOAA]

I'm not sure I understand the role of the yaml file there-- Theoretically this should be taken care of by getting the dependencies right for the ufs-weather-model package, no?

[ulmononian]

@edwardhartnett #374 is meant for the ufs short-range weather application, not for the general ufs weather model. i did raise an issue (#361) for getting the ufs weather model template updated to match develop, but i am not sure if this is desired. i thought it would be useful to have a spack.yaml like this for the develop branch of the weather model: https://github.com/ulmononian/spack-stack/blob/feature/ufs-wm-dev/configs/templates/ufs-wm-dev/spack.yaml, instead of one that just contains the "ufs-weather-model-env" spec.

if you meant to get the yaml file from the ufs-srw-dev template into the main spack repo (as is done with the ufs-weather-model-env here https://github.com/NOAA-EMC/spack-stack/blob/develop/configs/templates/ufs-weather-model/spack.yaml), then that would be useful. however, having a fixed environment in the main spack repo for either of these develop versions (ufs-srw-dev or ufs-wm-dev) seems like it could be hard to keep up to date? or at least harder than having templates in the emc spack-stack repo that are regularly updated with the latest spec versions from the related ufs app develop branches.

[AlexanderRichert-NOAA]

@climbfuji I'm working on an update for the ufs-weather-model package in main spack which will allow users to install UFS WM based on the develop branch. The only packages not currently in main spack that will be needed are mapl, yafyaml, and gftl-shared. Would you consider those ready for prime time? If so, I can include them in my upcoming PR for main spack once I work through one or two residual issues with the UFS build. For that PR I'm not planning to worry about including static libs; that can always happen later.

[climbfuji]

@climbfuji I'm working on an update for the ufs-weather-model package in main spack which will allow users to install UFS WM based on the develop branch. The only packages not currently in main spack that will be needed are mapl, yafyaml, and gftl-shared. Would you consider those ready for prime time? If so, I can include them in my upcoming PR for main spack once I work through one or two residual issues with the UFS build. For that PR I'm not planning to worry about including static libs; that can always happen later.

Please check with @mathomp4 on those three packages. Matt has a draft PR that he's been working on for a while on the various geos dependencies, best to coordinate when it is a good time to move the packages from the jcsda-emc "repo" (i.e. directory in our spack fork) to the "builtin" repo and then push them upstream.

[mathomp4]

@climbfuji I'm working on an update for the ufs-weather-model package in main spack which will allow users to install UFS WM based on the develop branch. The only packages not currently in main spack that will be needed are mapl, yafyaml, and gftl-shared. Would you consider those ready for prime time? If so, I can include them in my upcoming PR for main spack once I work through one or two residual issues with the UFS build. For that PR I'm not planning to worry about including static libs; that can always happen later.

Please check with @mathomp4 on those three packages. Matt has a draft PR that he's been working on for a while on the various geos dependencies, best to coordinate when it is a good time to move the packages from the jcsda-emc "repo" (i.e. directory in our spack fork) to the "builtin" repo and then push them upstream.

Yes, I have a draft PR here:

JCSDA/spack#174

and to be "safe" I put all the packages I was working on in the jcsda-emc repo. But, there is no reason I couldn't move them to builtin.

But I might need @climbfuji to help me here. I just tried moving them and:

$ spack spec -IL [email protected]+extdata2g+flap+pflogger+pfunit+fargparse
==> Error: Package 'gftl' not found.
You may need to run 'spack clean -m'.

I have run spack clean -m and even spack clean -a and spack still is not happy I've moved some things around. @climbfuji do you know if there is any way to have spack be happy when I move things between repos?

[climbfuji]

Have you rerun the concretization step? Not sure if that makes any difference. I can try to test your code changes if you get stuck.

…

On Nov 30, 2022, at 6:54 AM, Matthew Thompson ***@***.***> wrote: @climbfuji <https://github.com/climbfuji> I'm working on an update for the ufs-weather-model package in main spack which will allow users to install UFS WM based on the develop branch. The only packages not currently in main spack that will be needed are mapl, yafyaml, and gftl-shared. Would you consider those ready for prime time? If so, I can include them in my upcoming PR for main spack once I work through one or two residual issues with the UFS build. For that PR I'm not planning to worry about including static libs; that can always happen later. Please check with @mathomp4 <https://github.com/mathomp4> on those three packages. Matt has a draft PR that he's been working on for a while on the various geos dependencies, best to coordinate when it is a good time to move the packages from the jcsda-emc "repo" (i.e. directory in our spack fork) to the "builtin" repo and then push them upstream. Yes, I have a draft PR here: JCSDA/spack#174 <JCSDA/spack#174> and to be "safe" I put all the packages I was working on in the jcsda-emc repo. But, there is no reason I couldn't move them to builtin. But I might need @climbfuji <https://github.com/climbfuji> to help me here. I just tried moving them and: $ spack spec -IL ***@***.***+extdata2g+flap+pflogger+pfunit+fargparse ==> Error: Package 'gftl' not found. You may need to run 'spack clean -m'. I have run spack clean -m and even spack clean -a and spack still is not happy I've moved some things around. @climbfuji <https://github.com/climbfuji> do you know if there is any way to have spack be happy when I move things between repos? — Reply to this email directly, view it on GitHub <#377 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RPZDSSG4GCV4JBC5WDWK5MCXANCNFSM6AAAAAAQ7ZYJO4>. You are receiving this because you were mentioned.

[mathomp4]

@climbfuji I just tried:

spack concretize -Uf

and still the same error.

I'm guessing there is some "memory" somewhere? I also just tried spack install gftl and it installed again (even though I'd already done it), and it still won't see it.

I'm going to nuke my env and start from scratch. Back in...a couple hours!

[mathomp4]

Also, fair warning to @climbfuji I'm not exactly pure in my testing because I'm not as good as you at all this.

For example, since I'm not too smart at spack-stack yet, I did:

spack stack create env --site discover --template empty --name geosgcm.discover

and then in my envs/geosgcm.discover/common/packages.yaml I made some hacks to support the versions I wanted (removed yaFyaml, moved ESMF to 8.4.0, etc.). spack kept wanting to use older versions for things and this was the only way I knew how to fix it. I'm sure you can help me do things The Right Way™ in the future. 😄

[AlexanderRichert-NOAA]

Updated ufs-weather-model package.py pushed to spack-stack by JCSDA/spack#217
I'll close this issue once it's in main spack, which will happen after MAPL is added.

[mathomp4]

@AlexanderRichert-NOAA Sounds like you'd like MAPL in mainline spack sooner rather than later? Okay. I can work on that. I'll try and get a "good" package soon that you can test.

[AlexanderRichert-NOAA]

@mathomp4 Awesome thank you!

[climbfuji]

@AlexanderRichert-NOAA Where are we with this issue, have you updated/fixed the ufs-weather-model package in the authoritative repository?

[AlexanderRichert-NOAA]

Just waiting on MAPL to be in main repo, then I'll put in the PR.

[climbfuji]

@AlexanderRichert-NOAA What are we going to with this issue?

[AlexanderRichert-NOAA]

It's almost in main spack (spack/spack#39265), just waiting on final review.

[AlexanderRichert-NOAA]

spack/spack#39265 merged