Handling policy for unhandled agent exceptions

[BenConstable9]

What happened?

If a custom agent raises an exception, there is an error in the processing of the message and the agent carries on.

ERROR:autogen_core:Error processing publish message
Traceback (most recent call last):
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_core/_single_threaded_agent_runtime.py", line 410, in _process_publish
    await asyncio.gather(*responses)
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_core/_single_threaded_agent_runtime.py", line 402, in _on_message
    return await agent.on_message(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_core/_base_agent.py", line 113, in on_message
    return await self.on_message_impl(message, ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_agentchat/teams/_group_chat/_sequential_routed_agent.py", line 48, in on_message_impl
    return await super().on_message_impl(message, ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_core/_routed_agent.py", line 485, in on_message_impl
    return await h(self, message, ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_core/_routed_agent.py", line 268, in wrapper
    return_value = await func(self, message, ctx)  # type: ignore
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/autogen_agentchat/teams/_group_chat/_chat_agent_container.py", line 53, in handle_request
    async for msg in self._agent.on_messages_stream(self._message_buffer, ctx.cancellation_token):
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/text_2_sql/autogen/src/autogen_text_2_sql/custom_agents/sql_schema_selection_agent.py", line 71, in on_messages_stream
    entity_results = await asyncio.gather(*entity_tasks)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/text_2_sql/text_2_sql_core/src/text_2_sql_core/connectors/open_ai.py", line 49, in run_completion_request
    response = await open_ai_client.beta.chat.completions.parse(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/openai/resources/beta/chat/completions.py", line 427, in parse
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1843, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1537, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/dstoolkit-text2sql-and-imageprocessing/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1638, in _request
    raise self._make_status_error_from_response(err.response) from None

What did you expect to happen?

Option for all exceptions to be raised and for the group chat to stop further message rounds.

This exception stems from a bad API request, so if this agent fails, the whole system should fail.

How can we reproduce it (as minimally and precisely as possible)?

Create a custom agent that throws an exception.
Watch how the group chat will fail to register the message and then carry on.

AutoGen version

0.4.dev.13

Which package was this bug in

AgentChat

Model used

No response

Python version

No response

Operating system

No response

Any additional info you think would be helpful for fixing this bug

No response

[jackgerrits]

I do agree, configurable policy for what to do upon an unhandled agent exception makes sense to me.

Here's what I'm thinking, we add a param to the single threaded agent runtime constructor that allows you to select what to do in the case of an unhandled exception. Probably just ignore or raise for now (it always gets logged).

Unfortunately, since the runtime is not accessible in the agentchat layer this is still not super helpful for your original issue. That is going to get fixed likely in 0.4.1, however, maybe we can expose this in the team constructor or run method?

@ekzhu

[ekzhu]

I agree let's focus on the Core exception handling first. For AgentChat we can configure this as part of the runtime, as we will make the runtime configurable for teams and agents.

[BenConstable9]

Thanks both. This is fairly critical and means we can't safely use AutoGen 0.4.0 yet with customers.

Is there any likely timeline or quick fix for now I can use?

[jackgerrits]

Not aware of a quick fix that will work well. We're going to do more regular releases moving forward, so hopefully 0.4.1 isn't too far away