understand why using opt to get MAP estimate in vignette "ode-solve-example.Rmd" fails?

[njtierney]

I can run the example separately here:

library(greta.dynamics)
#> Loading required package: greta
#> 
#> Attaching package: 'greta'
#> The following objects are masked from 'package:stats':
#> 
#>     binomial, cov2cor, poisson
#> The following objects are masked from 'package:base':
#> 
#>     %*%, apply, backsolve, beta, chol2inv, colMeans, colSums, diag,
#>     eigen, forwardsolve, gamma, identity, rowMeans, rowSums, sweep,
#>     tapply
# priors for the parameters
rIng <- uniform(0, 2) # /day, rate of ingestion
#> ℹ Initialising python and checking dependencies, this may take a moment.
#> ✔ Initialising python and checking dependencies ... done!
#> 
rGrow <- uniform(0, 3) # /day, growth rate of prey
rMort <- uniform(0, 1) # /day, mortality rate of predator
assEff <- uniform(0, 1) # -, assimilation efficiency
K <- uniform(0, 30) # mmol/m3, carrying capacity
obs_sd <- uniform(0, 1)
# build the model (takes a few seconds to define the tensorflow graph)
m <- model(rIng, rGrow, rMort, assEff, K, obs_sd)

# compute MAP estimate
o <- opt(
  model = m,
  initial_values = initials(
    rIng = 0.2,
    rGrow = 1.0,
    rMort = 0.2,
    assEff = 0.5,
    K = 10.0
  )
)
o
#> $par
#> $par$rIng
#> [1] 1
#> 
#> $par$rGrow
#> [1] 1.5
#> 
#> $par$rMort
#> [1] 0.5
#> 
#> $par$assEff
#> [1] 0.5
#> 
#> $par$K
#> [1] 15
#> 
#> $par$obs_sd
#> [1] 0.5
#> 
#> 
#> $value
#> [1] 8.317766
#> 
#> $iterations
#> [1] 5
#> 
#> $convergence
#> [1] 0

^{Created on 2024-11-14 with reprex v2.1.1}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.2 (2024-10-31)
#>  os       macOS Sonoma 14.5
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Australia/Hobart
#>  date     2024-11-14
#>  pandoc   3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package        * version  date (UTC) lib source
#>  abind            1.4-8    2024-09-12 [1] CRAN (R 4.4.1)
#>  backports        1.5.0    2024-05-23 [1] CRAN (R 4.4.0)
#>  base64enc        0.1-3    2015-07-28 [1] CRAN (R 4.4.0)
#>  callr            3.7.6    2024-03-25 [1] CRAN (R 4.4.0)
#>  cli              3.6.3    2024-06-21 [1] CRAN (R 4.4.0)
#>  coda             0.19-4.1 2024-01-31 [1] CRAN (R 4.4.0)
#>  codetools        0.2-20   2024-03-31 [2] CRAN (R 4.4.2)
#>  crayon           1.5.3    2024-06-20 [1] CRAN (R 4.4.0)
#>  digest           0.6.37   2024-08-19 [1] CRAN (R 4.4.1)
#>  evaluate         1.0.1    2024-10-10 [1] CRAN (R 4.4.1)
#>  fastmap          1.2.0    2024-05-15 [1] CRAN (R 4.4.0)
#>  fs               1.6.5    2024-10-30 [1] CRAN (R 4.4.1)
#>  future           1.34.0   2024-07-29 [1] CRAN (R 4.4.0)
#>  globals          0.16.3   2024-03-08 [1] CRAN (R 4.4.0)
#>  glue             1.8.0    2024-09-30 [1] CRAN (R 4.4.1)
#>  greta          * 0.5.0    2024-11-06 [1] local
#>  greta.dynamics * 0.2.2    2024-11-13 [1] local
#>  hms              1.1.3    2023-03-21 [1] CRAN (R 4.4.0)
#>  htmltools        0.5.8.1  2024-04-04 [1] CRAN (R 4.4.0)
#>  jsonlite         1.8.9    2024-09-20 [1] CRAN (R 4.4.1)
#>  knitr            1.48     2024-07-07 [1] CRAN (R 4.4.0)
#>  lattice          0.22-6   2024-03-20 [2] CRAN (R 4.4.2)
#>  lifecycle        1.0.4    2023-11-07 [1] CRAN (R 4.4.0)
#>  listenv          0.9.1    2024-01-29 [1] CRAN (R 4.4.0)
#>  magrittr         2.0.3    2022-03-30 [1] CRAN (R 4.4.0)
#>  Matrix           1.7-1    2024-10-18 [2] CRAN (R 4.4.2)
#>  parallelly       1.38.0   2024-07-27 [1] CRAN (R 4.4.0)
#>  pkgconfig        2.0.3    2019-09-22 [1] CRAN (R 4.4.0)
#>  png              0.1-8    2022-11-29 [1] CRAN (R 4.4.0)
#>  prettyunits      1.2.0    2023-09-24 [1] CRAN (R 4.4.0)
#>  processx         3.8.4    2024-03-16 [1] CRAN (R 4.4.0)
#>  progress         1.2.3    2023-12-06 [1] CRAN (R 4.4.0)
#>  ps               1.8.1    2024-10-28 [1] CRAN (R 4.4.1)
#>  R6               2.5.1    2021-08-19 [1] CRAN (R 4.4.0)
#>  Rcpp             1.0.13-1 2024-11-02 [1] CRAN (R 4.4.1)
#>  reprex           2.1.1    2024-07-06 [1] CRAN (R 4.4.0)
#>  reticulate       1.39.0   2024-09-05 [1] CRAN (R 4.4.1)
#>  rlang            1.1.4    2024-06-04 [1] CRAN (R 4.4.0)
#>  rmarkdown        2.29     2024-11-04 [1] CRAN (R 4.4.1)
#>  rstudioapi       0.17.1   2024-10-22 [1] CRAN (R 4.4.1)
#>  sessioninfo      1.2.2    2021-12-06 [1] CRAN (R 4.4.0)
#>  tensorflow       2.16.0   2024-04-15 [1] CRAN (R 4.4.0)
#>  tfautograph      0.3.2    2021-09-17 [1] CRAN (R 4.4.0)
#>  tfruns           1.5.3    2024-04-19 [1] CRAN (R 4.4.0)
#>  vctrs            0.6.5    2023-12-01 [1] CRAN (R 4.4.0)
#>  whisker          0.4.1    2022-12-05 [1] CRAN (R 4.4.0)
#>  withr            3.0.2    2024-10-28 [1] CRAN (R 4.4.1)
#>  xfun             0.49     2024-10-31 [1] CRAN (R 4.4.1)
#>  yaml             2.3.10   2024-07-26 [1] CRAN (R 4.4.0)
#> 
#>  [1] /Users/nick/Library/R/arm64/4.4/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
#> 
#> ─ Python configuration ───────────────────────────────────────────────────────
#>  python:         /Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2/bin/python
#>  libpython:      /Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2/lib/libpython3.10.dylib
#>  pythonhome:     /Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2:/Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2
#>  version:        3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:51:49) [Clang 16.0.6 ]
#>  numpy:          /Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2/lib/python3.10/site-packages/numpy
#>  numpy_version:  1.26.4
#>  tensorflow:     /Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2/lib/python3.10/site-packages/tensorflow
#>  
#>  NOTE: Python version was forced by use_python() function
#> 
#> ──────────────────────────────────────────────────────────────────────────────

But running it inside of the vignette breaks and gives this error:

Error in py_call_impl(callable, call_args$unnamed, call_args$named) : 
  TypeError: <tf.Tensor 'chain_of_reshape_5_of_chain_of_shift_5_of_scale_5_of_sigmoid_5/forward/reshape_5/forward/Reshape:0' shape=(1, 1, 1) dtype=float64> is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it.
Please see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information.
<...truncated...>in _create_op_internal
    File "/Users/nick/Library/r-miniconda-arm64/envs/greta-env-tf2/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 1160, in from_node_def

The tensor <tf.Tensor 'chain_of_reshape_5_of_chain_of_shift_5_of_scale_5_of_sigmoid_5/forward/reshape_5/forward/Reshape:0' shape=(1, 1, 1) dtype=float64> cannot be accessed from here, because it was defined in FuncGraph(name=fn, id=15414436784), which is out of scope.

[njtierney]

Currently the workaround is to redefine the parameters and the model and run opt on it, but just noting this here.

[njtierney]

One of the errors above linked to this page, just for reference: https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values