plot new SEEPS and SEEPS_MPR statistics

[TatianaBurek]

Describe the Enhancement

MET development version 11.0.0-beta3 made several changes to the "stat" output.

• Issue Enhance Point-Stat to compute SEEPS for point observations and write new SEEPS and SEEPS_MPR STAT line types. MET#1942 enhances Point-Stat to write new SEEPS and SEEPS_MPR line types. Both of these new line type should be added to the schema. Note that Enhance Point-Stat to compute SEEPS for point observations and write new SEEPS and SEEPS_MPR STAT line types. MET#1942 will enhance Grid-Stat to also compute the SEEPS line type, but should make no additional changes to the schema.

Need to add plottiing of SEEPS and SEEPS_MPR statistics
dtcenter/METdataio#119
dtcenter/METcalcpy#216

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2799991

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[TatianaBurek]

@JohnHalleyGotway I have a few questions:

1. Which stats should we plot for SEEPS line type? So far I added SEEPS_MEAN_FCST, SEEPS_MEAN_OBS, SEEPS. There other columns like s{number} and pf{number}, total. Should I plot them also or ignore?
2. Which stats if any should we calculate using agg logic? Where can I find the formulas? MET issue comments that "There is no way to mathematically aggregate SEEPS".
3. METviewer doesn't plot lat/lon line types like MPR. Should I still add SEEPS_MPR stats? How?

[TatianaBurek]

Need a scientist to become the "local expert" on the computation and plotting of SEEPS.
Oct. 24,2022
John:
I think we need to identify a METplus scientist to become the "local expert" on the computation and plotting of SEEPS. Right now, Howard has done the development and I'm familiar with the technical details. Rachel North from the MetOffice is definitely the scientist who knows the most about it and has directed that development. But I feel like it's quickly going to become a tenuous situation. Providing user support and training relying exclusively on Rachel isn't realistic.

Tara:
we don't really have the resources for having an in-house SEEPS expert at this time, especially with a 50% cut to our funding from the Met Office. I think this is where we need to develop closer ties to Met Office so they can provide that SME capability "in-kind" and we learn how to lean on our partners. I will bring it up with Marion, but maybe we need to have a discussion at the next Met Office - METplus meeting.

[JohnHalleyGotway]

@RachelNorth, we're looking for some guidance on the plotting of the newly created SEEPS data by MET. The SEEPS and SEEPS_MPR line types will now be loaded into a database and @TatianaBurek is working on how/which of these statistics to plot.

For the SEEPS line type, is it sufficient to plot column 38 (SEEPS) or should METviewer also be able to plot the "S" counts and "PF/PV" marginal probabilities? The SEEPS_MPR data is similar in kind to the MPR (matched pair) line type. While that can be loaded into the database I don't think that METviewer is actually plotting it yet.

@hsoh-u and @RachelNorth, we also need to advise on aggregation logic. Let's say you have a SEEPS output line for each of 30 days. Is the 30-day "aggregation" just the weighted average of each of the 30 days... where the weight is the number of matched pairs in each (i.e. TOTAL column)?

As of right now, the Stat-Analysis tool just errors out when you try to "aggregate" the SEEPS lines:

stat_analysis -job aggregate -line_type SEEPS -lookin point_stat/point_stat_NCMET_NAM_NDAS_SEEPS_360000L_20120410_120000V_seeps.txt 
ERROR  : do_job_aggr() -> the "-line_type" option must be set to one of:
ERROR  : 	FHO, CTC, MCTC,
ERROR  : 	SL1L2, SAL1L2, VL1L2, VAL1L2,
ERROR  : 	PCT, NBRCTC, NBRCNT, GRAD, ISC,
ERROR  : 	ECNT, RPS, RHIST, PHIST, RELP, SSVAR

What should the behavior be?

[TaraJensen]

@mpm-meto - can you provide input before Rachel gets back from leave?

[RachelNorth]

@JohnHalleyGotway, yes it is sufficient to plot column 38. I've mainly plotted this as either a timeseries, or with forecast lead time and kept an eye on the TOTAL value without explicitly plotting it. I would suggest taking a look at Figure 6 in Haiden et al 2012 as a way of presenting the 'S' values for a particular forecast lead time. That should be sufficient for now. I haven't actively plotted the marginal probabilities to date, but I don't think we would want to preclude someone from doing so.

Regarding the aggregation, it depends what you are wanting to aggregate. For output per station, yes you would want just a temporal average (using TOTAL). If you want to aggregate over time and an area then you should be applying a density-weighted average (which isn't yet implemented?) over the stations first. I've not tried to aggregate station-based scores without applying the density weighting detailed in Rodwell et al (2010); I wouldn't recommend trying to do this.

One point, I've just been to the linked documentation for the SEEPS linetype and I think columns 26-31 are described incorrectly. The scoring matrix components should be named according the observed and forecast categories with s12 indicating OBS_CAT 1, FCST_CAT 2. The documentation indicates the opposite. Could you double check what is happening in the code, please?

I'll send through an email with some example plot types.
I could do with some clarification on the expected workflow for the different types of output, if that's okay.

I'm hoping you don't need me to respond to the Stat-Analysis error question?

[TatianaBurek]

Following simple statistics were included in METviewer. Sum or agg logic is not available
SEEPS_MEAN_FCST
SEEPS_MEAN_OBS
SEEPS
SEEPS_S12
SEEPS_S13
SEEPS_S21
SEEPS_S23
SEEPS_S31
SEEPS_S32
SEEPS_PF1
SEEPS_PF2
SEEPS_PF3
SEEPS_PV1
SEEPS_PV2
SEEPS_PV3

[TatianaBurek]

@JohnHalleyGotway I added SEEPS stats without the aggregation option.
Is this sufficient?

[JohnHalleyGotway]

@TatianaBurek, I'd say that makes sense for now. I don't see that @hsoh-u has added any logic to Stat-Analysis yet to aggregate multiple SEEPS line types together or logic to aggregate multiple SEEPS_MPR lines together into a SEEPS output line.

That's logic that we'd want to emulate in METcalcpy and METviewer.

Once that logic is added to Stat-Analysis, we could write up a separate issue to support that aggregation across the METplus Analysis tools as well.