Python: SK Multiagents - working for ChatCompletionAgents but not for AzureAssistantAgent

[zzulueta]

I have a multi-agent system that works with the ChatCompletionAgent but does not work for AzureAssistantAgent as shown below. If I switch my agents to an AzureAssistantAgent version, the system does not want to handoff to the next agent. Any idea why?

###ChatCompletionAgent WORKS###

Manager Assistant

manager_kernel = _create_kernel_with_chat_completion(MANAGER_NAME)
manager_agent = ChatCompletionAgent(
service_id=MANAGER_NAME,
kernel=manager_kernel,
name=MANAGER_NAME,
instructions=f"""
You count the number of conversations between {DALLE_NAME} and {VISION_NAME}.
An image generated by {DALLE_NAME} and an analysis by {VISION_NAME} is considered as a single conversation.
Once four (4) conversations are completed, you will provide a termination message: {TERMINATION_KEYWORD}
"""
)

DALLE Assistant

dalle_assistant_kernel = _create_kernel_with_chat_completion(DALLE_NAME)
dalle_assistant_kernel.add_plugin(GenerateImagePlugin(), plugin_name="GenerateImagePlugin")

settings = dalle_assistant_kernel.get_prompt_execution_settings_from_service_id(service_id=DALLE_NAME)
settings.function_choice_behavior = FunctionChoiceBehavior.Required()

dalle_assistant_agent = ChatCompletionAgent(
service_id=DALLE_NAME,
kernel=dalle_assistant_kernel,
name=DALLE_NAME,
instructions="""
As a premier AI specializing in image generation, you possess the expertise to craft precise visuals based on given prompts.
It is essential that you diligently generate the requested image, ensuring its accuracy and alignment with the user's specifications,
prior to delivering a response.
You will have access to the local file system to store the generated image.
You will generate an image based on the user's prompt and display it for review.
You will generate new images based on the feedback from the Vision Assistant.
""",
execution_settings=settings
)

Vision Assistant

vision_assistant_kernel = _create_kernel_with_chat_completion(VISION_NAME)
vision_assistant_kernel.add_plugin(AnalyzeImagePlugin(), plugin_name="AnalyzeImagePlugin")

settings = vision_assistant_kernel.get_prompt_execution_settings_from_service_id(service_id=VISION_NAME)
settings.function_choice_behavior = FunctionChoiceBehavior.Required()

vision_assistant_agent = ChatCompletionAgent(
service_id=VISION_NAME,
kernel=vision_assistant_kernel,
name=VISION_NAME,
instructions="""
As a leading AI expert in image analysis, you excel at scrutinizing and offering critiques to refine and improve images.
Your task is to thoroughly analyze an image, ensuring that all essential assessments are completed with precision
before you provide feedback to the user. You have access to the local file system where the image is stored.
You will analyze the image and provide a new prompt for Dall-e that enhances the image based on the criticism and analysis.
You will then instruct the Dall-e Assistant to generate a new image based on the new prompt.
""",
execution_settings=settings
)

selection_function = KernelFunctionFromPrompt(
function_name="selection",
prompt=f"""
Determine which participant takes the next turn in a conversation based on the most recent participant.
State only the name of the participant to take the next turn.
Choose only from these participants:
- {DALLE_NAME}
- {VISION_NAME}
- {MANAGER_NAME}

    You will follow this sequence:
    {DALLE_NAME} will generate an image based on the initial user prompt and display it for review.
    {VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
    {DALLE_NAME} will generate an image based on the {VISION_NAME} prompt and display it for review.
    {VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
    {DALLE_NAME} will generate an image based on the {VISION_NAME} prompt and display it for review.
    {VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
    
    {MANAGER_NAME} will count the number of conversations between {DALLE_NAME} and {VISION_NAME}.
    
    No participant should take more than one turn in a row.

    History:
    {{{{$history}}}}
    """,

)

termination_function = KernelFunctionFromPrompt(
function_name="termination",
prompt=f"""
Determine if the conversation should be terminated based on the number of conversations by the {MANAGER_NAME}.
If number of conversations is reached, respond with the termination keyword: {TERMINATION_KEYWORD}
RESPONSE:
{{{{$history}}}}"""
)

chat = AgentGroupChat(
agents=[dalle_assistant_agent, vision_assistant_agent, manager_agent],
selection_strategy=KernelFunctionSelectionStrategy(
function=selection_function,
kernel=_create_kernel_with_chat_completion("selection"),
result_parser=lambda result: str(result.value[0]) if result.value is not None else DALLE_NAME or VISION_NAME,
agent_variable_name="agents",
history_variable_name="history"
),
termination_strategy=KernelFunctionTerminationStrategy(
agents=[manager_agent],
function=termination_function,
kernel=_create_kernel_with_chat_completion("termination"),
result_parser=lambda result: TERMINATION_KEYWORD in str(result.value[0]).lower(),
history_variable_name="history",
maximum_iterations=10,
),
)

###AzureAssistantAgent Does not work###

Manager Assistant

manager_kernel = _create_kernel_with_chat_completion(MANAGER_NAME)

manager_agent = await AzureAssistantAgent.create(
service_id=MANAGER_NAME,
kernel=manager_kernel,
name=MANAGER_NAME,
instructions=f"""
An image generated by {DALLE_NAME} and an analysis by {VISION_NAME} is considered as a single conversation.
You will monitor the conversation between the {DALLE_NAME} and the {VISION_NAME} and count the number of conversations.
Once four (4) conversations are completed, you will provide a termination message: {TERMINATION_KEYWORD}
You will not provide any other input or output to the conversation other than the termination message.
"""
)

DALLE Assistant

dalle_assistant_kernel = _create_kernel_with_chat_completion(DALLE_NAME)
dalle_assistant_kernel.add_plugin(GenerateImagePlugin(), plugin_name="GenerateImagePlugin")

settings = dalle_assistant_kernel.get_prompt_execution_settings_from_service_id(service_id=DALLE_NAME)
settings.function_choice_behavior = FunctionChoiceBehavior.Required()

dalle_assistant_agent = await AzureAssistantAgent.create(
service_id=DALLE_NAME,
kernel=dalle_assistant_kernel,
name=DALLE_NAME,
instructions="""
As a premier AI specializing in image generation, you possess the expertise to craft precise visuals based on given prompts.
It is essential that you diligently generate the requested image, ensuring its accuracy and alignment with the user's specifications,
prior to delivering a response.
You will have access to the local file system to store the generated image.
You will generate an image based on the user's prompt and display it for review.
You will generate new images based on the feedback from the Vision Assistant.
""",
execution_settings=settings
)

Vision Assistant

vision_assistant_kernel = _create_kernel_with_chat_completion(VISION_NAME)
vision_assistant_kernel.add_plugin(AnalyzeImagePlugin(), plugin_name="AnalyzeImagePlugin")

settings = vision_assistant_kernel.get_prompt_execution_settings_from_service_id(service_id=VISION_NAME)
settings.function_choice_behavior = FunctionChoiceBehavior.Required()

vision_assistant_agent = await AzureAssistantAgent.create(
service_id=VISION_NAME,
kernel=vision_assistant_kernel,
name=VISION_NAME,
instructions="""
As a leading AI expert in image analysis, you excel at scrutinizing and offering critiques to refine and improve images.
Your task is to thoroughly analyze an image, ensuring that all essential assessments are completed with precision
before you provide feedback to the user. You have access to the local file system where the image is stored.
You will analyze the image and provide a new prompt for Dall-e that enhances the image based on the criticism and analysis.
You will then instruct the Dall-e Assistant to generate a new image based on the new prompt.
""",
execution_settings=settings
)

selection_function = KernelFunctionFromPrompt(
function_name="selection",
prompt=f"""
Determine which participant takes the next turn in a conversation based on the most recent participant.
State only the name of the participant to take the next turn.
Choose only from these participants:
- {DALLE_NAME}
- {VISION_NAME}
- {MANAGER_NAME}

    You will follow this sequence:
    {DALLE_NAME} will generate an image based on the initial user prompt and display it for review.
    {VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
    {MANAGER_NAME} will monitor the conversation between {DALLE_NAME} and {VISION_NAME} and count the number of conversations.
    {DALLE_NAME} will generate an image based on the {VISION_NAME} prompt and display it for review.
    {VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
    {MANAGER_NAME} will monitor the conversation between {DALLE_NAME} and {VISION_NAME} and count the number of conversations.
    {DALLE_NAME} will generate an image based on the {VISION_NAME} prompt and display it for review.
    {VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
    {MANAGER_NAME} will monitor the conversation between {DALLE_NAME} and {VISION_NAME} and count the number of conversations.
    
    No participant should take more than one turn in a row.

    History:
    {{{{$history}}}}
    """,

)

termination_function = KernelFunctionFromPrompt(
function_name="termination",
prompt=f"""
Determine if the conversation should be terminated based on the number of iterations.
If number of iterations is reached, respond with the termination keyword: {TERMINATION_KEYWORD}
RESPONSE:
{{{{$history}}}}"""
)

chat = AgentGroupChat(
agents=[dalle_assistant_agent, vision_assistant_agent, manager_agent],
selection_strategy=KernelFunctionSelectionStrategy(
function=selection_function,
kernel=_create_kernel_with_chat_completion("selection"),
result_parser=lambda result: str(result.value[0]) if result.value is not None else DALLE_NAME or VISION_NAME,
agent_variable_name="agents",
history_variable_name="history"
),
termination_strategy=KernelFunctionTerminationStrategy(
agents=[manager_agent],
function=termination_function,
kernel=_create_kernel_with_chat_completion("termination"),
result_parser=lambda result: TERMINATION_KEYWORD in str(result.value[0]).lower(),
history_variable_name="history",
maximum_iterations=10,
),
)

[moonbox3]

Hello @zzulueta could you give us some more information around:

If I switch my agents to an AzureAssistantAgent version, the system does not want to handoff to the next agent. Any idea why?

Is there a stack trace? An error? Which Azure OpenAI api version are you using? I need to look more closely through your code, but my initial hunch is that there is a difference with how Azure is handling the Required function choice behavior, versus OpenAI.

Can you turn on some warning or error level debugging in your script, please, if you aren't seeing any errors propagated right now? You can add:

import logging

logging.basicConfig(level=logging.DEBUG)

[zzulueta]

This is what came out:
INFO:semantic_kernel.agents.strategies.termination.termination_strategy:Evaluating termination criteria for c4211d81-885f-456a-bba5-ccfaeea5296e
INFO:semantic_kernel.agents.strategies.termination.termination_strategy:Agent c4211d81-885f-456a-bba5-ccfaeea5296e is out of scope
ERROR:semantic_kernel.agents.group_chat.agent_group_chat:Failed to select agent:

My Azure OpenAI api version is 2024-10-01-preview.

My code for ChatCompletionAgents and AzureAssistantAgent identical. The only difference is how the Agents were individually created. Two ChatCompletionAgents work. But when one is converted into an AzureAssistantAgent upon creation, it just fails.

[moonbox3]

Thanks for the additional context, @zzulueta. Let me spend some time on this, and I will get back to you ASAP.

[moonbox3]

I haven't had a chance to dig into this more. But looking at the code, we should get more of an exception or a stack trace if we fail to select an agent:

try:
    selected_agent = await self.selection_strategy.next(self.agents, self.history.messages)
except Exception as ex:
    logger.error(f"Failed to select agent: {ex}")
    raise AgentChatException("Failed to select agent") from ex

[zzulueta]

Sorry, where should I put this new code?