fix: handle empty HRRR data files via linear imputation

[danielolsen]

Pull Request doc

Purpose

When generating wind power profiles from HRRR data, gracefully handle any missing data via linear interpolation. Closes #244.

What the code is doing

The impute module is moved from prereise.gather.winddata.rap.impute to prereise.gather.winddata.impute, and a new linear interpolation method is added which should perform well on small data gaps.

Within prereise.gather.winddata.hrrr.calculations, calculate_pout is refactored to first build an array of all wind speed magnitudes obtained from the NOAA grib files (filling in NA when files are empty), then impute missing values as necessary, and finally convert wind speeds to wind powers.

Testing

Unit tests still pass, and this has been tested end-to-end when generating 2020 wind power profiles for the HIFLD grid (see #227 (comment)). When downloading the 2020 data, there were four files which downloaded empty, even after several attempts, suggesting that the data are missing from the NOAA server.

Usage Example/Visuals

from datetime import datetime
from powersimdata import Grid
from prereise.gather.winddata.hrrr.hrrr import retrieve_data
from prereise.gather.winddata.hrrr.calculations import calculate_pout

start_dt = datetime.fromisoformat("2020-01-01")
end_dt = datetime.fromisoformat("2021-01-01")
directory = "./"

grid = Grid("USA", "hifld")
wind_farms = grid.plant.query("type == 'wind' or type == 'wind_offshore'").copy()
wind_farms["state_abv"] = wind_farms.zone_id.map(grid.model_immutables.zones["id2abv"])
retrieve_data(start_dt=start_dt, end_dt=end_dt, directory=directory)
df = calculate_pout(wind_farms=wind_farms, start_dt=start_dt, end_dt=end_dt, directory=directory)

Time estimate

15-30 minutes.

[jenhagg]

Makes sense

[rouille]

Do we need the enumerate here? It seems we just need the length of the dts

[danielolsen]

enumerate() vs. range(len()), which is more pythonic?

[jenhagg]

for i in tqdm(dts)?

[BainanXia]

@jon-hagg I think we will need the index rather than the element of dts here.

[jenhagg]

tqdm(dts, total=len(dts)) would work, but not sure if that's better than enumerate/range. It's unclear to me if any is the most pythonic, but I think enumerate is fine

[danielolsen]

We could also refactor how we build wind_speed_data--as a dataframe instead of a numpy array--and then we could build wind_power_data using apply calls instead of list comprehensions.

[rouille]

See #221 (comment).

[danielolsen]

Aha. Maybe I'll leave this alone for now then.

EDIT: Or maybe the pandas version can be made a little more transparent by instantiating the dataframe with index= and columns=...

[danielolsen]

Done. I tried to strike a good balance between compactness and readability.

[BainanXia]

Good catch. Thanks!