[WIP] Add new spatially dependent lorenz-forcing calibration comparison between methods

[odunbar]

Purpose

Close #424

To-do

• Add outer for loop to run different ensemble sizes for the problem
• Add testing different random seeds for each of the algorithms
• Note: results might be different than my python results because EKI and ETKI don't have the prior included as regularization within the cost function yet

Content

• Included spin-up period in the statistics function: (i) Look at some trajectories of x(t) from prior, how long is the time-to-steady-state (make the spin up for each ensemble > 2*max of this) (ii) optional create a heatmap of these
• Initialized approach (for initial state condition) - (may want to change the method later).
• Included RMSE calculation within the iteration loop: get_error(ekiobj) , apply sqrt and /sqrt(length(y))
• introduced break/continue in loop when RMSE < 1.

---

• I have read and checked the items on the review checklist.

[rgjini]

Work in progress (WIP)

[rgjini]

The spin-up period in the statistics function is working properly:

• I checked the trajectory of x(t) and it seems like the initial spinup is a little less than 2 total time steps. Multiplied by 2, gives 4 time units for spinup
• I looked at heat maps in python, attached is an example of a trajectory
  

[rgjini]

I finished up the example and everything seems to be working! I fixed the random seed and ensemble size for loops, so those are both working. I kept the amount of different ensemble sizes and random seeds low for now, the code does take a while to run (maybe 10 minutes). I also have a few plots I made at the end. Two plots are of the final optimized solution compared to the true parameters for different ensemble sizes and the last one is supposed to be how many iterations each algorithm took for each ensemble size. For some reason, the last plot isn't looking how I would expect, but it might be related to an issue I ran into that I will describe in another comment. Here are some of the plots I made!:

[rgjini]

I ran into an issue running through the different algorithms for different ensemble sizes and random seeds. After the first iteration, I think UKI doesn't get reinitialized and starts at the same place it ended from the last time the algorithm was run. Every time after the first time UKI runs, the algorithm's RMSE is low for its initial ensemble and then doesn't even run through the algorithm. I think I can try to change where the UKI method gets initialized, but I wanted to bring it up here because I'm not sure if that is the expected behavior when all the other algorithms seem to be reinitialized when running the different random seeds and ensemble sizes.