Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle similar model criterion values #34

Open
dilpath opened this issue Jan 25, 2023 · 0 comments
Open

Handle similar model criterion values #34

dilpath opened this issue Jan 25, 2023 · 0 comments

Comments

@dilpath
Copy link
Member

dilpath commented Jan 25, 2023

Models can have indistinguishable criterion values, such that choosing one over the other is determined by numerical noise.

For example, in one case, during a forward search, two models with the same number of parameters were calibrated to have the same likelihood to several decimal places. Repeating the forward search 100 times resulted in 2 different trajectories through model space, occurring approximately 50/50. This is because numerical noise determined which of the two similar models were chosen at this point during the forward search.

Options to handle:

  • emit a warning that it is unclear which model to select, when models are very similar
  • create branches in the model selection, when encountering models that are within some epsilon criterion of each other or the best model so far
  • allow users to restart model selection at specific points, to explore trajectories that might have been chosen given different numerical noise, or were within some epsilon criterion of the best model so far
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant