Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: ensemble calibrate error #6002

Closed
shawnyama opened this issue Jan 10, 2025 · 1 comment
Closed

[BUG]: ensemble calibrate error #6002

shawnyama opened this issue Jan 10, 2025 · 1 comment
Assignees
@Tom-Szendrey
Labels
bug Something isn't working Q&A Quality Assurance

Comments

@shawnyama
Copy link
Contributor

Describe the issue
Simulation/calibration fails in pyciemss

pyciemss-worker

14:53:01 default: execute.run(EnsembleCalibrate(engine='ciemss', user_id='not_provided', model_configs=[M..., job_id='5ee3705d-5501-44a3-bdb3-1e598684e184') (5ee3705d-5501-44a3-bdb3-1e598684e184)
DEBUG:root:STARTED 5ee3705d-5501-44a3-bdb3-1e598684e184 (user_id: not_provided)
DEBUG:root:Updating simulation `http://hmi-server:3000/simulations/5ee3705d-5501-44a3-bdb3-1e598684e184` -- running start: True; finish: False; result_files: []
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "GET /simulations/5ee3705d-5501-44a3-bdb3-1e598684e184 HTTP/11" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "PUT /simulations/5ee3705d-5501-44a3-bdb3-1e598684e184 HTTP/11" 200 None
DEBUG:root:Fetching model f3410917-29e5-4933-8f87-c6d866eed930
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "GET /model-configurations/as-configured-model/f3410917-29e5-4933-8f87-c6d866eed930 HTTP/11" 200 None
DEBUG:root:Fetching model bb8ba8f1-27a0-43cc-ab64-c391e69b934d
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "GET /model-configurations/as-configured-model/bb8ba8f1-27a0-43cc-ab64-c391e69b934d HTTP/11" 200 None
DEBUG:root:Fetching dataset f22df112-8457-41c4-a1b7-d11d10f86c67
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "GET /datasets/f22df112-8457-41c4-a1b7-d11d10f86c67/download-url?filename=ensemble_calibrate_test.csv HTTP/11" 200 None
DEBUG:root:Overwriting timepoint_id
DEBUG:root:
DEBUG:pika.adapters.select_connection:Using EPollPoller
DEBUG:pika.adapters.base_connection:Created default connection workflow <pika.adapters.utils.connection_workflow.AMQPConnectionWorkflow object at 0x7fcba7c65210>
DEBUG:pika.adapters.utils.connection_workflow:Starting AMQP Connection workflow asynchronously.
DEBUG:pika.adapters.select_connection:call_later: added timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c65810> with deadline=5606078.812327032 and callback=functools.partial(<bound method AMQPConnectionWorkflow._start_new_cycle_async of <pika.adapters.utils.connection_workflow.AMQPConnectionWorkflow object at 0x7fcba7c65210>>, first=True); now=5606078.812327032; delay=0
DEBUG:pika.adapters.utils.connection_workflow:Beginning a new AMQP connection workflow cycle; attempts remaining after this: 0
DEBUG:pika.adapters.select_connection:call_later: added timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c65ea0> with deadline=5606078.812576807 and callback=<bound method AMQPConnectionWorkflow._try_next_config_async of <pika.adapters.utils.connection_workflow.AMQPConnectionWorkflow object at 0x7fcba7c65210>>; now=5606078.812576807; delay=0
DEBUG:pika.adapters.utils.connection_workflow:_try_next_config_async: 'rabbitmq.staging.terarium.ai':5672
DEBUG:pika.adapters.select_connection:add_callback_threadsafe: added callback=<bound method _AddressResolver._dispatch_result of <pika.adapters.utils.selector_ioloop_adapter._AddressResolver object at 0x7fcba7c65360>>
DEBUG:pika.adapters.select_connection:process_timeouts: invoking callback=<bound method _AddressResolver._dispatch_result of <pika.adapters.utils.selector_ioloop_adapter._AddressResolver object at 0x7fcba7c65360>>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:Invoking asynchronous getaddrinfo() completion callback; host='rabbitmq.staging.terarium.ai'
DEBUG:pika.adapters.utils.connection_workflow:getaddrinfo returned 1 records
DEBUG:pika.adapters.utils.connection_workflow:Attempting to connect using address record (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.16.40.21', 5672))
INFO:pika.adapters.utils.connection_workflow:Pika version 1.3.2 connecting to ('172.16.40.21', 5672)
DEBUG:pika.adapters.select_connection:add_callback_threadsafe: added callback=<bound method _AsyncSocketConnector._start_async of <pika.adapters.utils.io_services_utils._AsyncSocketConnector object at 0x7fcba7c65c30>>
DEBUG:pika.adapters.select_connection:call_later: added timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c65bd0> with deadline=5606088.819745723 and callback=<bound method AMQPConnector._on_tcp_connection_timeout of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>>; now=5606078.819745723; delay=10.0
DEBUG:pika.adapters.select_connection:call_later: added timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c65ed0> with deadline=5606093.819786222 and callback=<bound method AMQPConnector._on_overall_timeout of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>>; now=5606078.819786222; delay=15.0
DEBUG:pika.adapters.select_connection:process_timeouts: invoking callback=<bound method _AsyncSocketConnector._start_async of <pika.adapters.utils.io_services_utils._AsyncSocketConnector object at 0x7fcba7c65c30>>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.set_writer(8, <bound method _AsyncSocketConnector._on_writable of <pika.adapters.utils.io_services_utils._AsyncSocketConnector object at 0x7fcba7c65c30>>)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:set_writer(8, _) added handler Wr
DEBUG:pika.adapters.utils.io_services_utils:Connection-establishment is in progress for <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888)>.
INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.io_services_utils:_AsyncSocketConnector._report_completion(None); <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_writer(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_writer(8) removed handler
DEBUG:pika.adapters.select_connection:remove_timeout: removing timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c65bd0> with deadline=5606088.819745723 and callback=<bound method AMQPConnector._on_tcp_connection_timeout of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>>
DEBUG:pika.adapters.utils.connection_workflow:TCP connection to broker established: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>.
DEBUG:pika.adapters.utils.io_services_utils:_AsyncStreamConnector.start(); <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.select_connection:add_callback_threadsafe: added callback=<bound method _AsyncStreamConnector._start_async of <pika.adapters.utils.io_services_utils._AsyncStreamConnector object at 0x7fcba7c65960>>
DEBUG:pika.adapters.select_connection:process_timeouts: invoking callback=<bound method _AsyncStreamConnector._start_async of <pika.adapters.utils.io_services_utils._AsyncStreamConnector object at 0x7fcba7c65960>>
DEBUG:pika.adapters.utils.io_services_utils:_AsyncStreamConnector._start_async(); <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.io_services_utils:_AsyncStreamConnector._linkup()
DEBUG:pika.connection:New Connection state: CLOSED (prev=CLOSED)
DEBUG:pika.callback:Added: {'callback': <bound method Connection._on_connection_start of <SelectConnection CLOSED transport=None params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1}
DEBUG:pika.callback:Added: {'callback': <bound method Connection._on_connection_close_from_broker of <SelectConnection CLOSED transport=None params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1}
DEBUG:pika.callback:Added: {'callback': <bound method Connection._default_on_connection_error of <SelectConnection CLOSED transport=None params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': False, 'only': None, 'arguments': None}
DEBUG:pika.connection:New Connection state: INIT (prev=CLOSED)
DEBUG:pika.connection:Using external connection workflow.
DEBUG:pika.adapters.utils.io_services_utils:_AsyncTransportBase.__init__: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.set_reader(8, <bound method _AsyncPlaintextTransport._on_socket_readable of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:set_reader(8, _) added handler Rd
DEBUG:pika.adapters.utils.io_services_utils:_linkup(): created transport <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>
DEBUG:pika.connection:New Connection state: PROTOCOL (prev=INIT)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.set_writer(8, <bound method _AsyncPlaintextTransport._on_socket_writable of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:set_writer(8, _) updated handler RdWr
DEBUG:pika.adapters.utils.io_services_utils:Turned on writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.io_services_utils:_linkup(): introduced transport to protocol <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>; _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>
DEBUG:pika.adapters.utils.io_services_utils:_AsyncStreamConnector._report_completion((<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>)); <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
INFO:pika.adapters.utils.connection_workflow:Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>).
DEBUG:pika.callback:Removing callback #0: {'callback': <bound method Connection._default_on_connection_error of <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': False, 'only': None, 'arguments': None}
DEBUG:pika.callback:Added: {'callback': <bound method AMQPConnector._on_amqp_handshake_done of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>>, 'one_shot': False, 'only': None, 'arguments': None}
DEBUG:pika.callback:Added: {'callback': <bound method AMQPConnector._on_amqp_handshake_done of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>>, 'one_shot': False, 'only': None, 'arguments': None}
DEBUG:pika.adapters.utils.io_services_utils:_AsyncStreamConnector._cleanup(False)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_writer(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_writer(8) updated handler Rd
DEBUG:pika.adapters.utils.io_services_utils:Turned off writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.callback:Processing 0:Connection.Start
DEBUG:pika.callback:Processing use of oneshot callback
DEBUG:pika.callback:0 registered uses left
DEBUG:pika.callback:Removing callback #0: {'callback': <bound method Connection._on_connection_start of <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0}
DEBUG:pika.callback:Calling <bound method Connection._on_connection_start of <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>> for "0:Connection.Start"
DEBUG:pika.connection:New Connection state: START (prev=PROTOCOL)
DEBUG:pika.callback:Added: {'callback': <bound method Connection._on_connection_tune of <SelectConnection START transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1}
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.set_writer(8, <bound method _AsyncPlaintextTransport._on_socket_writable of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:set_writer(8, _) updated handler RdWr
DEBUG:pika.adapters.utils.io_services_utils:Turned on writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.io_services_utils:Recv would block on <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_writer(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_writer(8) updated handler Rd
DEBUG:pika.adapters.utils.io_services_utils:Turned off writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.callback:Processing 0:Connection.Tune
DEBUG:pika.callback:Processing use of oneshot callback
DEBUG:pika.callback:0 registered uses left
DEBUG:pika.callback:Removing callback #0: {'callback': <bound method Connection._on_connection_tune of <SelectConnection START transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0}
DEBUG:pika.callback:Calling <bound method Connection._on_connection_tune of <SelectConnection START transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>> for "0:Connection.Tune"
DEBUG:pika.connection:New Connection state: TUNE (prev=START)
DEBUG:pika.connection:Creating a HeartbeatChecker: 60
DEBUG:pika.heartbeat:timeout: 60.000000 send_interval: 30.000000 check_interval: 65.000000
DEBUG:pika.adapters.select_connection:call_later: added timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c64d00> with deadline=5606108.825394792 and callback=<bound method HeartbeatChecker._send_heartbeat of <pika.heartbeat.HeartbeatChecker object at 0x7fcba7c65e70>>; now=5606078.825394792; delay=30.0
DEBUG:pika.adapters.select_connection:call_later: added timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c64760> with deadline=5606143.825426067 and callback=<bound method HeartbeatChecker._check_heartbeat of <pika.heartbeat.HeartbeatChecker object at 0x7fcba7c65e70>>; now=5606078.825426067; delay=65
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.set_writer(8, <bound method _AsyncPlaintextTransport._on_socket_writable of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:set_writer(8, _) updated handler RdWr
DEBUG:pika.adapters.utils.io_services_utils:Turned on writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.callback:Added: {'callback': <bound method Connection._on_connection_open_ok of <SelectConnection TUNE transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1}
DEBUG:pika.adapters.utils.io_services_utils:Recv would block on <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_writer(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_writer(8) updated handler Rd
DEBUG:pika.adapters.utils.io_services_utils:Turned off writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.callback:Processing 0:Connection.OpenOk
DEBUG:pika.callback:Processing use of oneshot callback
DEBUG:pika.callback:0 registered uses left
DEBUG:pika.callback:Removing callback #0: {'callback': <bound method Connection._on_connection_open_ok of <SelectConnection TUNE transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0}
DEBUG:pika.callback:Calling <bound method Connection._on_connection_open_ok of <SelectConnection TUNE transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>> for "0:Connection.OpenOk"
DEBUG:pika.connection:New Connection state: OPEN (prev=TUNE)
DEBUG:pika.callback:Processing 0:_on_connection_open_ok
DEBUG:pika.callback:Calling <bound method AMQPConnector._on_amqp_handshake_done of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>> for "0:_on_connection_open_ok"
DEBUG:pika.adapters.utils.connection_workflow:AMQPConnector: AMQP handshake attempt completed; state=3; error=None; 'rabbitmq.staging.terarium.ai'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.16.40.21', 5672))
DEBUG:pika.adapters.utils.connection_workflow:AMQPConnector: AMQP connection established for 'rabbitmq.staging.terarium.ai'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.16.40.21', 5672)): <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>
INFO:pika.adapters.utils.connection_workflow:AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>
DEBUG:pika.adapters.select_connection:remove_timeout: removing timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c65ed0> with deadline=5606093.819786222 and callback=<bound method AMQPConnector._on_overall_timeout of <pika.adapters.utils.connection_workflow.AMQPConnector object at 0x7fcba7c65c90>>
DEBUG:pika.adapters.utils.connection_workflow:Connection attempt completed with <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>
INFO:pika.adapters.utils.connection_workflow:AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>
DEBUG:pika.adapters.utils.io_services_utils:Recv would block on <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
INFO:pika.adapters.blocking_connection:Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>
DEBUG:pika.callback:Added: {'callback': <bound method _CallbackResult.set_value_once of <pika.adapters.blocking_connection._CallbackResult object at 0x7fcba9d03f40>>, 'one_shot': False, 'only': None, 'arguments': None}
INFO:pika.adapters.blocking_connection:Closing connection (200): Normal shutdown
DEBUG:pika.connection:New Connection state: CLOSING (prev=OPEN)
INFO:pika.connection:Closing connection (200): 'Normal shutdown'
DEBUG:pika.callback:Added: {'callback': <bound method Connection._on_connection_close_ok of <SelectConnection CLOSING transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1}
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.set_writer(8, <bound method _AsyncPlaintextTransport._on_socket_writable of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:set_writer(8, _) updated handler RdWr
DEBUG:pika.adapters.utils.io_services_utils:Turned on writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_writer(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_writer(8) updated handler Rd
DEBUG:pika.adapters.utils.io_services_utils:Turned off writability watcher: <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.callback:Processing 0:Connection.CloseOk
DEBUG:pika.callback:Processing use of oneshot callback
DEBUG:pika.callback:0 registered uses left
DEBUG:pika.callback:Removing callback #0: {'callback': <bound method Connection._on_connection_close_ok of <SelectConnection CLOSING transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 0}
DEBUG:pika.callback:Calling <bound method Connection._on_connection_close_ok of <SelectConnection CLOSING transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0> params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>> for "0:Connection.CloseOk"
DEBUG:pika.connection:_on_connection_close_ok: frame=<METHOD(['channel_number=0', 'frame_type=1', 'method=<Connection.CloseOk>'])>
DEBUG:pika.heartbeat:Removing timer for next heartbeat send interval
DEBUG:pika.adapters.select_connection:remove_timeout: removing timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c64d00> with deadline=5606108.825394792 and callback=<bound method HeartbeatChecker._send_heartbeat of <pika.heartbeat.HeartbeatChecker object at 0x7fcba7c65e70>>
DEBUG:pika.heartbeat:Removing timer for next heartbeat check interval
DEBUG:pika.adapters.select_connection:remove_timeout: removing timeout <pika.adapters.select_connection._Timeout object at 0x7fcba7c64760> with deadline=5606143.825426067 and callback=<bound method HeartbeatChecker._check_heartbeat of <pika.heartbeat.HeartbeatChecker object at 0x7fcba7c65e70>>
INFO:pika.adapters.utils.io_services_utils:Aborting transport connection: state=1; <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
INFO:pika.adapters.utils.io_services_utils:_AsyncTransportBase._initate_abort(): Initiating abrupt asynchronous transport shutdown: state=1; error=None; <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
INFO:pika.adapters.utils.io_services_utils:Deactivating transport: state=1; <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_reader(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_reader(8) removed handler
DEBUG:pika.adapters.utils.selector_ioloop_adapter:SelectorIOServicesAdapter.remove_writer(8)
DEBUG:pika.adapters.utils.selector_ioloop_adapter:remove_writer(8) neither was set.
DEBUG:pika.adapters.select_connection:add_callback_threadsafe: added callback=functools.partial(<bound method _AsyncTransportBase._connection_lost_notify_async of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>, None)
DEBUG:pika.adapters.utils.io_services_utils:Leaving Plaintext consumer due to inactive state: state=3; <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
DEBUG:pika.adapters.select_connection:process_timeouts: invoking callback=functools.partial(<bound method _AsyncTransportBase._connection_lost_notify_async of <pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7fcba7c656f0>>, None)
DEBUG:pika.adapters.utils.io_services_utils:Concluding transport shutdown: state=3; error=None
DEBUG:pika.adapters.base_connection:connection_lost: None
INFO:pika.connection:AMQP stack terminated, failed to connect, or aborted: opened=True, error-arg=None; pending-error=ConnectionClosedByClient: (200) 'Normal shutdown'
DEBUG:pika.connection:New Connection state: CLOSED (prev=CLOSING)
INFO:pika.connection:Stack terminated due to ConnectionClosedByClient: (200) 'Normal shutdown'
DEBUG:pika.callback:Processing 0:_on_connection_closed
DEBUG:pika.callback:Calling <bound method _CallbackResult.set_value_once of <pika.adapters.blocking_connection._CallbackResult object at 0x7fcba9d03f40>> for "0:_on_connection_closed"
DEBUG:pika.connection:New Connection state: CLOSED (prev=CLOSED)
DEBUG:pika.callback:Added: {'callback': <bound method Connection._on_connection_start of <SelectConnection CLOSED transport=None params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 1}
DEBUG:pika.callback:Incremented callback reference counter: {'callback': <bound method Connection._on_connection_close_from_broker of <SelectConnection CLOSED transport=None params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>>, 'one_shot': True, 'only': None, 'arguments': None, 'calls': 2}
INFO:pika.adapters.utils.io_services_utils:Closing transport socket and unlinking: state=3; <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.244.3.160', 41888), raddr=('172.16.40.21', 5672)>
INFO:pika.adapters.blocking_connection:User-initiated close: result=BlockingConnection__OnClosedArgs(connection=<SelectConnection CLOSED transport=None params=<ConnectionParameters host=rabbitmq.staging.terarium.ai port=5672 virtual_host=/ ssl=False>>, error=ConnectionClosedByClient: (200) 'Normal shutdown')
INFO:root:5ee3705d-5501-44a3-bdb3-1e598684e184 started with the following args: {'model_paths_or_jsons': ['/tmp/5ee3705d-5501-44a3-bdb3-1e598684e184/./f3410917-29e5-4933-8f87-c6d866eed930.json', '/tmp/5ee3705d-5501-44a3-bdb3-1e598684e184/./bb8ba8f1-27a0-43cc-ab64-c391e69b934d.json'], 'solution_mappings': [<function convert_to_solution_mapping.<locals>.solution_mapping at 0x7fcc8cb7c700>, <function convert_to_solution_mapping.<locals>.solution_mapping at 0x7fcc8cb7c1f0>], 'data_path': '/tmp/5ee3705d-5501-44a3-bdb3-1e598684e184/./temp.json', 'start_time': 0.0, 'dirichlet_alpha': tensor([1., 1.]), 'progress_hook': <function gen_calibrate_rabbitmq_hook.<locals>.hook at 0x7fcba7d00b80>, 'noise_model': 'normal', 'noise_model_kwargs': {'scale': 0.1}, 'solver_method': 'dopri5', 'solver_options': {}, 'num_iterations': 100, 'lr': 0.03, 'verbose': False, 'num_particles': 10, 'deterministic_learnable_parameters': []}
/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/observation.py:49: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  if not data_df.applymap(lambda x: isinstance(x, (int, float))).all().all():
ERROR:root:
                ###############################

                There was an exception in pyciemss

                Error occured in function: ensemble_calibrate

                Function docs : 
    Infer parameters for an ensemble of DynamicalSystem models conditional on data.
    This uses variational inference with a mean-field variational family to infer the parameters of the model.

    Args:
    model_paths_or_jsons: List[Union[str, Dict]]
        - A list of paths to AMR model files or JSONs containing models in AMR form.
    solution_mappings: List[Callable[[Dict[str, torch.Tensor]], Dict[str, torch.Tensor]]]
        - A list of functions that map the solution of each model to a common solution space.
        - Each function takes in a dictionary of the form {state_variable_name: value}
            and returns a dictionary of the same form.
    data_path: str
        - A path to the data file.
    dirichlet_alpha: Optional[torch.Tensor]
        - A tensor of shape (num_models,) containing the Dirichlet alpha values for the ensemble.
            - A higher proportion of alpha values will result in higher weights for the corresponding models.
            - A larger total alpha values will result in more certain priors.
            - e.g. torch.tensor([1, 1, 1]) will result in a uniform prior over vectors of length 3 that sum to 1.
            - e.g. torch.tensor([1, 2, 3]) will result in a prior that is biased towards the third model.
        - If not provided, we will use a uniform Dirichlet prior.
    data_mapping: Dict[str, str]
        - A mapping from column names in the data file to state variable names in the model.
            - keys: str name of column in dataset
            - values: str name of state/observable in model
        - If not provided, we will assume that the column names in the data file match the state variable names.
        - Note: This mapping must match output of `solution_mappings`.
    noise_model: str
        - The noise model to use for the data.
        - Currently we only support the normal distribution.
    noise_model_kwargs: Dict[str, Any]
        - Keyword arguments to pass to the noise model.
        - Currently we only support the `scale` keyword argument for the normal distribution.
    solver_method: str
        - The method to use for solving the ODE. See torchdiffeq's `odeint` method for more details.
        - If performance is incredibly slow, we suggest using `euler` to debug.
            If using `euler` results in faster simulation, the issue is likely that the model is stiff.
    solver_options: Dict[str, Any]
        - Options to pass to the solver (including atol and rtol). See torchdiffeq' `odeint` method for more details.
    start_time: float
        - The start time of the model. This is used to align the `start_state` from the
            AMR model with the simulation timepoints.
        - By default we set the `start_time` to be 0.
    num_iterations: int
        - The number of iterations to run the inference algorithm for.
    lr: float
        - The learning rate to use for the inference algorithm.
    verbose: bool
        - Whether to print out the loss at each iteration.
    num_particles: int
        - The number of particles to use for the inference algorithm.
    deterministic_learnable_parameters: List[str]
        - A list of parameter names that should be learned deterministically.
        - By default, all parameters are learned probabilistically.
    progress_hook: Callable[[int, float], None]
        - A function that takes in the current iteration and the current loss.
        - This is called at the beginning of each iteration.
        - By default, this is a no-op.
        - This can be used to implement custom progress bars.

    Returns:
        result: Dict[str, Any]
            - Dictionary with the following key-value pairs.
                - inferred_parameters: pyro.nn.PyroModule
                    - A Pyro module that contains the inferred parameters of the model.
                    - This can be passed to `ensemble_sample` to sample from the model conditional on the data.
                - loss: float
                    - The final loss value of the approximate ELBO loss.
    

                ################################
            
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 10, in wrapped
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 339, in ensemble_calibrate
    inferred_parameters = autoguide(wrapped_model)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 305, in autoguide
    mvn_guide._setup_prototype()
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 1002, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 636, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 157, in _setup_prototype
    self.prototype_trace = poutine.block(poutine.trace(model).get_trace)(
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 198, in get_trace
    self(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 180, in __call__
    raise exc from e
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.
                  Trace Shapes:    
                   Param Sites:    
                  Sample Sites:    
             model_weights dist | 2
                          value | 2
      model_0/persistent_β dist |  
                          value |  
      model_0/persistent_γ dist |  
                          value |  
   model_1/persistent_beta dist |  
                          value |  
  model_1/persistent_gamma dist |  
                          value |  
  model_1/persistent_delta dist |  
                          value |  
  model_1/persistent_alpha dist |  
                          value |  
model_1/persistent_epsilon dist |  
                          value |  
   model_1/persistent_zeta dist |  
                          value |  
 model_1/persistent_lambda dist |  
                          value |  
    model_1/persistent_eta dist |  
                          value |  
    model_1/persistent_rho dist |  
                          value |  
  model_1/persistent_theta dist |  
                          value |  
  model_1/persistent_kappa dist |  
                          value |  
     model_1/persistent_mu dist |  
                          value |  
     model_1/persistent_nu dist |  
                          value |  
     model_1/persistent_xi dist |  
                          value |  
    model_1/persistent_tau dist |  
                          value |  
  model_1/persistent_sigma dist |  
                          value |  
DEBUG:rq.job:Running failure callbacks for 5ee3705d-5501-44a3-bdb3-1e598684e184
DEBUG:root:Updating simulation `http://hmi-server:3000/simulations/5ee3705d-5501-44a3-bdb3-1e598684e184` -- error start: False; finish: False; result_files: []
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "GET /simulations/5ee3705d-5501-44a3-bdb3-1e598684e184 HTTP/11" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "PUT /simulations/5ee3705d-5501-44a3-bdb3-1e598684e184 HTTP/11" 200 None
ERROR:root:
        ###############################

        There was an exception in CIEMSS Service

        job: 5ee3705d-5501-44a3-bdb3-1e598684e184
        <class 'ValueError'>: All solution mappings must return the same keys for each model.
                  Trace Shapes:    
                   Param Sites:    
                  Sample Sites:    
             model_weights dist | 2
                          value | 2
      model_0/persistent_β dist |  
                          value |  
      model_0/persistent_γ dist |  
                          value |  
   model_1/persistent_beta dist |  
                          value |  
  model_1/persistent_gamma dist |  
                          value |  
  model_1/persistent_delta dist |  
                          value |  
  model_1/persistent_alpha dist |  
                          value |  
model_1/persistent_epsilon dist |  
                          value |  
   model_1/persistent_zeta dist |  
                          value |  
 model_1/persistent_lambda dist |  
                          value |  
    model_1/persistent_eta dist |  
                          value |  
    model_1/persistent_rho dist |  
                          value |  
  model_1/persistent_theta dist |  
                          value |  
  model_1/persistent_kappa dist |  
                          value |  
     model_1/persistent_mu dist |  
                          value |  
     model_1/persistent_nu dist |  
                          value |  
     model_1/persistent_xi dist |  
                          value |  
    model_1/persistent_tau dist |  
                          value |  
  model_1/persistent_sigma dist |  
                          value |  
        ################################
    
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/rq/worker.py", line 1430, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1280, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1317, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/service/./execute.py", line 33, in run
    output = eval(operation_name)(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 29, in wrapped
    raise e
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 10, in wrapped
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 339, in ensemble_calibrate
    inferred_parameters = autoguide(wrapped_model)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 305, in autoguide
    mvn_guide._setup_prototype()
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 1002, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 636, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 157, in _setup_prototype
    self.prototype_trace = poutine.block(poutine.trace(model).get_trace)(
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 198, in get_trace
    self(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 180, in __call__
    raise exc from e
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.
                  Trace Shapes:    
                   Param Sites:    
                  Sample Sites:    
             model_weights dist | 2
                          value | 2
      model_0/persistent_β dist |  
                          value |  
      model_0/persistent_γ dist |  
                          value |  
   model_1/persistent_beta dist |  
                          value |  
  model_1/persistent_gamma dist |  
                          value |  
  model_1/persistent_delta dist |  
                          value |  
  model_1/persistent_alpha dist |  
                          value |  
model_1/persistent_epsilon dist |  
                          value |  
   model_1/persistent_zeta dist |  
                          value |  
 model_1/persistent_lambda dist |  
                          value |  
    model_1/persistent_eta dist |  
                          value |  
    model_1/persistent_rho dist |  
                          value |  
  model_1/persistent_theta dist |  
                          value |  
  model_1/persistent_kappa dist |  
                          value |  
     model_1/persistent_mu dist |  
                          value |  
     model_1/persistent_nu dist |  
                          value |  
     model_1/persistent_xi dist |  
                          value |  
    model_1/persistent_tau dist |  
                          value |  
  model_1/persistent_sigma dist |  
                          value |  
14:53:08 [Job 5ee3705d-5501-44a3-bdb3-1e598684e184]: exception raised while executing (execute.run)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/rq/worker.py", line 1430, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1280, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1317, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/service/./execute.py", line 33, in run
    output = eval(operation_name)(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 29, in wrapped
    raise e
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 10, in wrapped
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 339, in ensemble_calibrate
    inferred_parameters = autoguide(wrapped_model)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 305, in autoguide
    mvn_guide._setup_prototype()
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 1002, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 636, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 157, in _setup_prototype
    self.prototype_trace = poutine.block(poutine.trace(model).get_trace)(
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 198, in get_trace
    self(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 180, in __call__
    raise exc from e
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.
                  Trace Shapes:    
                   Param Sites:    
                  Sample Sites:    
             model_weights dist | 2
                          value | 2
      model_0/persistent_β dist |  
                          value |  
      model_0/persistent_γ dist |  
                          value |  
   model_1/persistent_beta dist |  
                          value |  
  model_1/persistent_gamma dist |  
                          value |  
  model_1/persistent_delta dist |  
                          value |  
  model_1/persistent_alpha dist |  
                          value |  
model_1/persistent_epsilon dist |  
                          value |  
   model_1/persistent_zeta dist |  
                          value |  
 model_1/persistent_lambda dist |  
                          value |  
    model_1/persistent_eta dist |  
                          value |  
    model_1/persistent_rho dist |  
                          value |  
  model_1/persistent_theta dist |  
                          value |  
  model_1/persistent_kappa dist |  
                          value |  
     model_1/persistent_mu dist |  
                          value |  
     model_1/persistent_nu dist |  
                          value |  
     model_1/persistent_xi dist |  
                          value |  
    model_1/persistent_tau dist |  
                          value |  
  model_1/persistent_sigma dist |  
                          value |  

ERROR:rq.worker:[Job 5ee3705d-5501-44a3-bdb3-1e598684e184]: exception raised while executing (execute.run)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/rq/worker.py", line 1430, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1280, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1317, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/service/./execute.py", line 33, in run
    output = eval(operation_name)(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 29, in wrapped
    raise e
  File "/usr/local/lib/python3.10/site-packages/pyciemss/integration_utils/custom_decorators.py", line 10, in wrapped
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 339, in ensemble_calibrate
    inferred_parameters = autoguide(wrapped_model)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 305, in autoguide
    mvn_guide._setup_prototype()
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 1002, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 636, in _setup_prototype
    super()._setup_prototype(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py", line 157, in _setup_prototype
    self.prototype_trace = poutine.block(poutine.trace(model).get_trace)(
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 198, in get_trace
    self(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 180, in __call__
    raise exc from e
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py", line 174, in __call__
    ret = self.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyro/poutine/messenger.py", line 12, in _context_wrap
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/interfaces.py", line 330, in wrapped_model
    solution = model(
  File "/usr/local/lib/python3.10/site-packages/pyro/nn/module.py", line 449, in __call__
    result = super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyciemss/ensemble/compiled_dynamics.py", line 49, in forward
    raise ValueError(
ValueError: All solution mappings must return the same keys for each model.
                  Trace Shapes:    
                   Param Sites:    
                  Sample Sites:    
             model_weights dist | 2
                          value | 2
      model_0/persistent_β dist |  
                          value |  
      model_0/persistent_γ dist |  
                          value |  
   model_1/persistent_beta dist |  
                          value |  
  model_1/persistent_gamma dist |  
                          value |  
  model_1/persistent_delta dist |  
                          value |  
  model_1/persistent_alpha dist |  
                          value |  
model_1/persistent_epsilon dist |  
                          value |  
   model_1/persistent_zeta dist |  
                          value |  
 model_1/persistent_lambda dist |  
                          value |  
    model_1/persistent_eta dist |  
                          value |  
    model_1/persistent_rho dist |  
                          value |  
  model_1/persistent_theta dist |  
                          value |  
  model_1/persistent_kappa dist |  
                          value |  
     model_1/persistent_mu dist |  
                          value |  
     model_1/persistent_nu dist |  
                          value |  
     model_1/persistent_xi dist |  
                          value |  
    model_1/persistent_tau dist |  
                          value |  
  model_1/persistent_sigma dist |  
                          value |

pyciemss-api

INFO:     10.244.8.0:25677 - "GET /ping HTTP/1.1" 404 Not Found
INFO:root:{'name': 'e1bb99f6-8a84-4a84-a0d7-52960447ad68', 'execution_payload': {'engine': 'ciemss', 'user_id': 'not_provided', 'model_configs': [{'id': 'f3410917-29e5-4933-8f87-c6d866eed930', 'solution_mappings': {'S_state': 'S', 'I_state': 'I', 'R_state': 'R', 'Diagnosed_observable_state': 'Diagnosed'}, 'weight': 1.0}, {'id': 'bb8ba8f1-27a0-43cc-ab64-c391e69b934d', 'solution_mappings': {'S_state': 'Susceptible', 'I_state': 'Infected', 'R_state': 'Recognized', 'Diagnosed_observable_state': 'Diagnosed'}, 'weight': 1.0}], 'timespan': {'start': 0.0, 'end': 100.0}, 'dataset': {'id': 'f22df112-8457-41c4-a1b7-d11d10f86c67', 'filename': 'ensemble_calibrate_test.csv', 'mappings': {'timepoint_id': 'timestamp', 'S_state': 'S_state', 'I_state': 'I_state', 'R_state': 'R_state', 'Diagnosed_observable_state': 'Diagnosed_observable_state'}}, 'step_size': 1.0, 'extra': {'noise_model': 'normal', 'noise_model_kwargs': {'scale': 0.1}, 'solver_method': 'dopri5', 'solver_options': {}, 'num_iterations': 100, 'lr': 0.03, 'verbose': False, 'num_particles': 10, 'deterministic_learnable_parameters': []}}, 'result_files': [], 'type': 'ensemble-calibrate', 'status': 'queued', 'engine': 'ciemss', 'workflow_id': 'e1bb99f6-8a84-4a84-a0d7-52960447ad68'}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): hmi-server:3000
DEBUG:urllib3.connectionpool:http://hmi-server:3000 "POST /simulations HTTP/11" 201 None
INFO:root:{'id': '848816c1-5251-4d3a-93e6-034fdb04a704', 'created_on': '2025-01-10T14:49:08.735+00:00', 'updated_on': '2025-01-10T14:49:08.735+00:00', 'name': 'e1bb99f6-8a84-4a84-a0d7-52960447ad68', 'file_names': [], 'temporary': False, 'public_asset': False, 'execution_payload': {'engine': 'ciemss', 'user_id': 'not_provided', 'model_configs': [{'id': 'f3410917-29e5-4933-8f87-c6d866eed930', 'solution_mappings': {'S_state': 'S', 'I_state': 'I', 'R_state': 'R', 'Diagnosed_observable_state': 'Diagnosed'}, 'weight': 1.0}, {'id': 'bb8ba8f1-27a0-43cc-ab64-c391e69b934d', 'solution_mappings': {'S_state': 'Susceptible', 'I_state': 'Infected', 'R_state': 'Recognized', 'Diagnosed_observable_state': 'Diagnosed'}, 'weight': 1.0}], 'timespan': {'start': 0.0, 'end': 100.0}, 'dataset': {'id': 'f22df112-8457-41c4-a1b7-d11d10f86c67', 'filename': 'ensemble_calibrate_test.csv', 'mappings': {'timepoint_id': 'timestamp', 'S_state': 'S_state', 'I_state': 'I_state', 'R_state': 'R_state', 'Diagnosed_observable_state': 'Diagnosed_observable_state'}}, 'step_size': 1.0, 'extra': {'noise_model': 'normal', 'noise_model_kwargs': {'scale': 0.1}, 'solver_method': 'dopri5', 'solver_options': {}, 'num_iterations': 100, 'lr': 0.03, 'verbose': False, 'num_particles': 10, 'deterministic_learnable_parameters': []}}, 'result_files': [], 'type': 'ENSEMBLE', 'status': 'QUEUED', 'progress': 0.0, 'engine': 'CIEMSS', 'updates': []}
DEBUG:rq.queue:Pushed job 848816c1-5251-4d3a-93e6-034fdb04a704 into default
INFO:     10.244.7.104:55548 - "POST /ensemble-calibrate HTTP/1.1" 200 OK
INFO:     10.244.7.0:39916 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     10.244.18.1:4935 - "GET /ping HTTP/1.1" 404 Not Found

Image
@shawnyama shawnyama added bug Something isn't working Q&A Quality Assurance labels Jan 10, 2025
@Tom-Szendrey
Copy link
Contributor

It appears as though this is due to mapping observables
I will look into fixing this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Tom-Szendrey Tom-Szendrey

Labels
bug Something isn't working Q&A Quality Assurance
Projects
None yet
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

refs/heads/TS/6002-calibrate-ensemble-with-obs DARPA-ASKEM/terarium
3 participants
@mwdchang @Tom-Szendrey @shawnyama