From 16c8b74780055a60009d226fae0989ec756d5bbb Mon Sep 17 00:00:00 2001 From: markwallace-microsoft <127216156+markwallace-microsoft@users.noreply.github.com> Date: Wed, 22 Jan 2025 16:31:29 +0000 Subject: [PATCH 1/6] Initial draft of the declarative agent schema for review --- .../0018-custom-prompt-template-formats.md | 285 --------------- .../NNNN-declarative-agent-schema.md | 332 ++++++++++++++++++ 2 files changed, 332 insertions(+), 285 deletions(-) delete mode 100644 docs/decisions/0018-custom-prompt-template-formats.md create mode 100644 docs/decisions/NNNN-declarative-agent-schema.md diff --git a/docs/decisions/0018-custom-prompt-template-formats.md b/docs/decisions/0018-custom-prompt-template-formats.md deleted file mode 100644 index 5cd1f7f90cb4..000000000000 --- a/docs/decisions/0018-custom-prompt-template-formats.md +++ /dev/null @@ -1,285 +0,0 @@ ---- -status: approved -contact: markwallace-microsoft -date: 2023-10-26 -deciders: matthewbolanos, markwallace-microsoft, SergeyMenshykh, RogerBarreto -consulted: dmytrostruk -informed: ---- - -# Custom Prompt Template Formats - -## Context and Problem Statement - -Semantic Kernel currently supports a custom prompt template language that allows for variable interpolation and function execution. -Semantic Kernel allows for custom prompt template formats to be integrated e.g., prompt templates using [Handlebars](https://handlebarsjs.com/) syntax. - -The purpose of this ADR is to describe how a custom prompt template formats will be supported in the Semantic Kernel. - -### Current Design - -By default the `Kernel` uses the `BasicPromptTemplateEngine` which supports the Semantic Kernel specific template format. - -#### Code Patterns - -Below is an expanded example of how to create a semantic function from a prompt template string which uses the built-in Semantic Kernel format: - -```csharp -IKernel kernel = Kernel.Builder - .WithPromptTemplateEngine(new BasicPromptTemplateEngine()) - .WithOpenAIChatCompletionService( - modelId: openAIModelId, - apiKey: openAIApiKey) - .Build(); - -kernel.ImportFunctions(new TimePlugin(), "time"); - -string templateString = "Today is: {{time.Date}} Is it weekend time (weekend/not weekend)?"; -var promptTemplateConfig = new PromptTemplateConfig(); -var promptTemplate = new PromptTemplate(templateString, promptTemplateConfig, kernel.PromptTemplateEngine); -var kindOfDay = kernel.RegisterSemanticFunction("KindOfDay", promptTemplateConfig, promptTemplate); - -var result = await kernel.RunAsync(kindOfDay); -Console.WriteLine(result.GetValue()); -``` - -We have an extension method `var kindOfDay = kernel.CreateSemanticFunction(promptTemplate);` to simplify the process to create and register a semantic function but the expanded format is shown above to highlight the dependency on `kernel.PromptTemplateEngine`. -Also the `BasicPromptTemplateEngine` is the default prompt template engine and will be loaded automatically if the package is available and no other prompt template engine is specified. - -Some issues with this: - -1. You need to have a `Kernel` instance to create a semantic function, which is contrary to one of the goals of allow semantic functions to be created once and reused across multiple `Kernel` instances. -1. `Kernel` only supports a single `IPromptTemplateEngine` so we cannot support using multiple prompt templates at the same time. -1. `IPromptTemplateEngine` is stateless and must perform a parse of the template for each render -1. Our semantic function extension methods rely on our implementation of `IPromptTemplate` (i.e., `PromptTemplate`) which stores the template string and uses the `IPromptTemplateEngine` to render it every time. Note implementations of `IPromptTemplate` are currently stateful as they also store the parameters. - -#### Performance - -The `BasicPromptTemplateEngine` uses the `TemplateTokenizer` to parse the template i.e. extract the blocks. -Then it renders the template i.e. inserts variables and executes functions. Some sample timings for these operations: - -| Operation | Ticks | Milliseconds | -| ---------------- | ------- | ------------ | -| Extract blocks | 1044427 | 103 | -| Render variables | 168 | 0 | - -Sample template used was: `"{{variable1}} {{variable2}} {{variable3}} {{variable4}} {{variable5}}"` - -**Note: We will use the sample implementation to support the f-string template format.** - -Using `HandlebarsDotNet` for the same use case results in the following timings: - -| Operation | Ticks | Milliseconds | -| ---------------- | ----- | ------------ | -| Compile template | 66277 | 6 | -| Render variables | 4173 | 0 | - -**By separating the extract blocks/compile from the render variables operation it will be possible to optimise performance by compiling templates just once.** - -#### Implementing a Custom Prompt Template Engine - -There are two interfaces provided: - -```csharp -public interface IPromptTemplateEngine -{ - Task RenderAsync(string templateText, SKContext context, CancellationToken cancellationToken = default); -} - -public interface IPromptTemplate -{ - IReadOnlyList Parameters { get; } - - public Task RenderAsync(SKContext executionContext, CancellationToken cancellationToken = default); -} -``` - -A prototype implementation of a handlebars prompt template engine could look something like this: - -```csharp -public class HandlebarsTemplateEngine : IPromptTemplateEngine -{ - private readonly ILoggerFactory _loggerFactory; - - public HandlebarsTemplateEngine(ILoggerFactory? loggerFactory = null) - { - this._loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - } - - public async Task RenderAsync(string templateText, SKContext context, CancellationToken cancellationToken = default) - { - var handlebars = HandlebarsDotNet.Handlebars.Create(); - - var functionViews = context.Functions.GetFunctionViews(); - foreach (FunctionView functionView in functionViews) - { - var skfunction = context.Functions.GetFunction(functionView.PluginName, functionView.Name); - handlebars.RegisterHelper($"{functionView.PluginName}_{functionView.Name}", async (writer, hcontext, parameters) => - { - var result = await skfunction.InvokeAsync(context).ConfigureAwait(true); - writer.WriteSafeString(result.GetValue()); - }); - } - - var template = handlebars.Compile(templateText); - - var prompt = template(context.Variables); - - return await Task.FromResult(prompt).ConfigureAwait(true); - } -} -``` - -**Note: This is just a prototype implementation for illustration purposes only.** - -Some issues: - -1. The `IPromptTemplate` interface is not used and causes confusion. -1. There is no way to allow developers to support multiple prompt template formats at the same time. - -There is one implementation of `IPromptTemplate` provided in the Semantic Kernel core package. -The `RenderAsync` implementation just delegates to the `IPromptTemplateEngine`. -The `Parameters` list get's populated with the parameters defined in the `PromptTemplateConfig` and any missing variables defined in the template. - -#### Handlebars Considerations - -Handlebars does not support dynamic binding of helpers. Consider the following snippet: - -```csharp -HandlebarsHelper link_to = (writer, context, parameters) => -{ - writer.WriteSafeString($"{context["text"]}"); -}; - -string source = @"Click here: {{link_to}}"; - -var data = new -{ - url = "https://github.com/rexm/handlebars.net", - text = "Handlebars.Net" -}; - -// Act -var handlebars = HandlebarsDotNet.Handlebars.Create(); -handlebars.RegisterHelper("link_to", link_to); -var template = handlebars1.Compile(source); -// handlebars.RegisterHelper("link_to", link_to); This also works -var result = template1(data); -``` - -Handlebars allows the helpers to be registered with the `Handlebars` instance either before or after a template is compiled. -The optimum would be to have a shared `Handlebars` instance for a specific collection of functions and register the helpers just once. -For use cases where the Kernel function collection may have been mutated we will be forced to create a `Handlebars` instance at render time -and then register the helpers. This means we cannot take advantage of the performance improvement provided by compiling the template. - -## Decision Drivers - -In no particular order: - -- Support creating a semantic function without a `IKernel`instance. -- Support late binding of functions i.e., having functions resolved when the prompt is rendered. -- Support allowing the prompt template to be parsed (compiled) just once to optimize performance if needed. -- Support using multiple prompt template formats with a single `Kernel` instance. -- Provide simple abstractions which allow third parties to implement support for custom prompt template formats. - -## Considered Options - -- Obsolete `IPromptTemplateEngine` and replace with `IPromptTemplateFactory`. - -### Obsolete `IPromptTemplateEngine` and replace with `IPromptTemplateFactory` - -ISKFunction class relationships - -Below is an expanded example of how to create a semantic function from a prompt template string which uses the built-in Semantic Kernel format: - -```csharp -// Semantic function can be created once -var promptTemplateFactory = new BasicPromptTemplateFactory(); -string templateString = "Today is: {{time.Date}} Is it weekend time (weekend/not weekend)?"; -var promptTemplate = promptTemplateFactory.CreatePromptTemplate(templateString, new PromptTemplateConfig()); -var kindOfDay = ISKFunction.CreateSemanticFunction("KindOfDay", promptTemplateConfig, promptTemplate) - -// Create Kernel after creating the semantic function -// Later we will support passing a function collection to the KernelBuilder -IKernel kernel = Kernel.Builder - .WithOpenAIChatCompletionService( - modelId: openAIModelId, - apiKey: openAIApiKey) - .Build(); - -kernel.ImportFunctions(new TimePlugin(), "time"); -// Optionally register the semantic function with the Kernel -// kernel.RegisterCustomFunction(kindOfDay); - -var result = await kernel.RunAsync(kindOfDay); -Console.WriteLine(result.GetValue()); -``` - -**Notes:** - -- `BasicPromptTemplateFactory` will be the default implementation and will be automatically provided in `KernelSemanticFunctionExtensions`. Developers will also be able to provide their own implementation. -- The factory uses the new `PromptTemplateConfig.TemplateFormat` to create the appropriate `IPromptTemplate` instance. -- We should look to remove `promptTemplateConfig` as a parameter to `CreateSemanticFunction`. That change is outside of the scope of this ADR. - -The `BasicPromptTemplateFactory` and `BasicPromptTemplate` implementations look as follows: - -```csharp -public sealed class BasicPromptTemplateFactory : IPromptTemplateFactory -{ - private readonly IPromptTemplateFactory _promptTemplateFactory; - private readonly ILoggerFactory _loggerFactory; - - public BasicPromptTemplateFactory(IPromptTemplateFactory promptTemplateFactory, ILoggerFactory? loggerFactory = null) - { - this._promptTemplateFactory = promptTemplateFactory; - this._loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - } - - public IPromptTemplate? CreatePromptTemplate(string templateString, PromptTemplateConfig promptTemplateConfig) - { - if (promptTemplateConfig.TemplateFormat.Equals(PromptTemplateConfig.SEMANTICKERNEL, System.StringComparison.Ordinal)) - { - return new BasicPromptTemplate(templateString, promptTemplateConfig, this._loggerFactory); - } - else if (this._promptTemplateFactory is not null) - { - return this._promptTemplateFactory.CreatePromptTemplate(templateString, promptTemplateConfig); - } - - throw new SKException($"Invalid prompt template format {promptTemplateConfig.TemplateFormat}"); - } -} - -public sealed class BasicPromptTemplate : IPromptTemplate -{ - public BasicPromptTemplate(string templateString, PromptTemplateConfig promptTemplateConfig, ILoggerFactory? loggerFactory = null) - { - this._loggerFactory = loggerFactory ?? NullLoggerFactory.Instance; - this._logger = this._loggerFactory.CreateLogger(typeof(BasicPromptTemplate)); - this._templateString = templateString; - this._promptTemplateConfig = promptTemplateConfig; - this._parameters = new(() => this.InitParameters()); - this._blocks = new(() => this.ExtractBlocks(this._templateString)); - this._tokenizer = new TemplateTokenizer(this._loggerFactory); - } - - public IReadOnlyList Parameters => this._parameters.Value; - - public async Task RenderAsync(SKContext executionContext, CancellationToken cancellationToken = default) - { - return await this.RenderAsync(this._blocks.Value, executionContext, cancellationToken).ConfigureAwait(false); - } - - // Not showing the implementation details -} -``` - -**Note:** - -- The call to `ExtractBlocks` is called lazily once for each prompt template -- The `RenderAsync` doesn't need to extract the blocks every time - -## Decision Outcome - -Chosen option: "Obsolete `IPromptTemplateEngine` and replace with `IPromptTemplateFactory`", because -addresses the requirements and provides good flexibility for the future. diff --git a/docs/decisions/NNNN-declarative-agent-schema.md b/docs/decisions/NNNN-declarative-agent-schema.md new file mode 100644 index 000000000000..34ad09d1826a --- /dev/null +++ b/docs/decisions/NNNN-declarative-agent-schema.md @@ -0,0 +1,332 @@ +--- +# These are optional elements. Feel free to remove any of them. +status: proposed +contact: markwallace-microsoft +date: 2025-01-17 +deciders: markwallace-microsoft, bentho, crickman +consulted: {list everyone whose opinions are sought (typically subject-matter experts); and with whom there is a two-way communication} +informed: {list everyone who is kept up-to-date on progress; and with whom there is a one-way communication} +--- + +# Schema for Declarative Agent Format + +## Context and Problem Statement + +This ADR describes a schema which can be used to define an Agent which can be loaded and executed using the Semantic Kernel Agent Framework. + +Currently the Agent Framework uses a code first approach to allow Agents to be defined and executed. +Using the schema defined by this ADR developers will be able to declaratively define an Agent and have the Semantic Kernel instantiate and execute the Agent. + +Here is some pseudo code to illustrate what we need to be able to do: + +```csharp +Kernel kernel = ... +string agentYaml = EmbeddedResource.Read("MyAgent.yaml"); +AgentFactory agentFactory = new AggregatorAgentFactory( + new ChatCompletionFactory(), + new OpenAIAssistantAgentFactory(), + new XXXAgentFactory()); +Agent agent = kernel.LoadAgentFromYaml(agentYaml); + +ChatHistory chatHistory = new(); +chatHistory.AddUserMessage(input); +await foreach (ChatMessageContent content in agent.InvokeAsync(chatHistory)) +{ + chatHistory.Add(content); +} +``` + +**Note:** + +1. The above pattern does is not supported at present. +2. We need to decide if the Agent Framework should define an abstraction to allow any Agent to be invoked. +3. If we are supporting an abstraction to allow any Agent to be invoked, what should the pattern look like. +4. We will support JSON also as an out-of-the-box option. + +Currently Semantic Kernel supports two Agent types and these have the following properties: + +1. [`ChatCompletionAgent`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel.agents.chatcompletionagent?view=semantic-kernel-dotnet): + - `Arguments`: Optional arguments for the agent. (Inherited from ChatHistoryKernelAgent) + - `Description`: The description of the agent (optional). (Inherited from Agent) + - `HistoryReducer`: (Inherited from ChatHistoryKernelAgent) + - `Id`: The identifier of the agent (optional). (Inherited from Agent) + - `Instructions`: The instructions of the agent (optional). (Inherited from KernelAgent) + - `Kernel`: The Kernel containing services, plugins, and filters for use throughout the agent lifetime. (Inherited from KernelAgent) + - `Logger`: The ILogger associated with this Agent. (Inherited from Agent) + - `LoggerFactory`: A ILoggerFactory for this Agent. (Inherited from Agent) + - `Name`: The name of the agent (optional). (Inherited from Agent) +2. ['OpenAIAssistantAgent'](https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel.agents.agent.description?view=semantic-kernel-dotnet#microsoft-semantickernel-agents-agent-description): + - `Arguments`: Optional arguments for the agent. + - `Definition`: The assistant definition. + - `Description`: The description of the agent (optional). (Inherited from Agent) + - `Id`: The identifier of the agent (optional). (Inherited from Agent) + - `Instructions`: The instructions of the agent (optional). (Inherited from KernelAgent) + - `IsDeleted`: Set when the assistant has been deleted via DeleteAsync(CancellationToken). An assistant removed by other means will result in an exception when invoked. + - `Kernel`: The Kernel containing services, plugins, and filters for use throughout the agent lifetime. (Inherited from KernelAgent) + - `Logger`: The ILogger associated with this Agent. (Inherited from Agent) + - `LoggerFactory`: A ILoggerFactory for this Agent. (Inherited from Agent) + - `Name`: The name of the agent (optional). (Inherited from Agent) + - `PollingOptions`: Defines polling behavior + +When executing an Agent that was declaratively defined some of the properties will be determined by the runtime: + +- `Kernel`: The runtime will be responsible for create the `Kernel` instance to be used by the Agent. This `Kernel` instance must be configured with the models and tools that the Agent requires. +- `Logger` or `LoggerFactory`: The runtime will be responsible for providing a correctly configured `Logger` or `LoggerFactory`. +- **Functions**: The runtime must be able to resolve any functions required by the Agent. E.g. the VSCode extension will provide a very basic runtime to allow developers to test Agents and it should eb able to resolve `KernelFunctions` defined in the current project. See later in the ADR for an example of this. + +For Agent properties that define behaviors e.g. `HistoryReducer` the Semantic Kernel **SHOULD**: + +- Provide implementations that can be configured declaratively i.e., for the most common scenarios we expect developers to encounter. +- Allow implementations to be resolved from the `Kernel` e.g., as required services or possibly `KernelFunction`'s. + +## Decision Drivers + +- Schema **MUST** allow model settings to be assigned to an Agent +- Schema **MUST** allow functions to be assigned to an Agent +- Schema **MUST** allow a Semantic Kernel prompt to be used to define the Agent instructions +- Schema **MUST** be extensible so that support for new Agent types can be added to Semantic Kernel +- Schema **MUST** allow third parties to contribute new Agent types to Semantic Kernel +- … + +## Considered Options + +- Use same semantics as the [Semantic Kernel Prompt Schema](https://learn.microsoft.com/en-us/semantic-kernel/concepts/prompts/yaml-schema#sample-yaml-prompt) +- {title of option 2} + +### Use Same Semantics as the Semantic Kernel Prompt Schema + +Consider the following use cases: + +1. `ChatCompletionAgent` +2. `ChatCompletionAgent` using Prompt Template +3. `ChatCompletionAgent` with Function Calling +4. `OpenAIAssistantAgent` with Function Calling +5. `OpenAIAssistantAgent` with Tools + +#### `ChatCompletionAgent` + +Code first approach: + +```csharp +ChatCompletionAgent agent = + new() + { + Name = "Parrot", + Instructions = "Repeat the user message in the voice of a pirate and then end with a parrot sound.", + Kernel = kernel, + }; +``` + +Declarative: + +```yml +name: Parrot +instructions: Repeat the user message in the voice of a pirate and then end with a parrot sound. +``` + +**Note**: `ChatCompletionAgent` could be the default agent type hence no explicit `type` property is required. + +#### `ChatCompletionAgent` using Prompt Template + +Code first approach: + +```csharp +string generateStoryYaml = EmbeddedResource.Read("GenerateStory.yaml"); +PromptTemplateConfig templateConfig = KernelFunctionYaml.ToPromptTemplateConfig(generateStoryYaml); + +ChatCompletionAgent agent = + new(templateConfig, new KernelPromptTemplateFactory()) + { + Kernel = this.CreateKernelWithChatCompletion(), + Arguments = new KernelArguments() + { + { "topic", "Dog" }, + { "length", "3" }, + } + }; +``` + +Declarative: + +```yml +name: GenerateStory +template: | + Tell a story about {{$topic}} that is {{$length}} sentences long. +template_format: semantic-kernel +description: A function that generates a story about a topic. +input_variables: + - name: topic + description: The topic of the story. + is_required: true + default: dog + - name: length + description: The number of sentences in the story. + is_required: true + default: 3 +``` + +**Note**: Only elements from the prompt template schema are needed. + +#### `ChatCompletionAgent` with Function Calling + +Code first approach: + +```csharp +ChatCompletionAgent agent = + new() + { + Instructions = "Answer questions about the menu.", + Name = "RestaurantHost", + Description = "This agent answers questions about the menu.", + Kernel = kernel, + Arguments = new KernelArguments(new OpenAIPromptExecutionSettings() { Temperature = 0.4, FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() }), + }; + +KernelPlugin plugin = KernelPluginFactory.CreateFromType(); +agent.Kernel.Plugins.Add(plugin); +``` + +Declarative: + +```yml +name: RestaurantHost +instructions: Answer questions about the menu. +description: This agent answers questions about the menu. +execution_settings: + default: + temperature: 0.4 + function_choice_behavior: + type: auto + functions: + - MenuPlugin.GetSpecials + - MenuPlugin.GetItemPrice +``` + +#### `OpenAIAssistantAgent` with Function Calling + +Code first approach: + +```csharp +OpenAIAssistantAgent agent = + await OpenAIAssistantAgent.CreateAsync( + clientProvider: this.GetClientProvider(), + definition: new OpenAIAssistantDefinition("gpt_4o") + { + Instructions = "Answer questions about the menu.", + Name = "RestaurantHost", + Metadata = new Dictionary { { AssistantSampleMetadataKey, bool.TrueString } }, + }, + kernel: new Kernel()); + +KernelPlugin plugin = KernelPluginFactory.CreateFromType(); +agent.Kernel.Plugins.Add(plugin); +``` + +Declarative: + +```yml +name: RestaurantHost +type: openai_assistant +instructions: Answer questions about the menu. +description: This agent answers questions about the menu. +execution_settings: + gpt_4o: + function_choice_behavior: + type: auto + functions: + - MenuPlugin.GetSpecials + - MenuPlugin.GetItemPrice + metadata: + sksample: true +``` + +**Note**: The `Kernel` instance used to create the Agent must have an instance of `OpenAIClientProvider` registered as a service. + +#### `OpenAIAssistantAgent` with Tools + +Code first approach: + +```csharp +OpenAIAssistantAgent agent = + await OpenAIAssistantAgent.CreateAsync( + clientProvider: this.GetClientProvider(), + definition: new(this.Model) + { + Instructions = "You are an Agent that can write and execute code to answer questions.", + Name = "Coder", + EnableCodeInterpreter = true, + EnableFileSearch = true, + Metadata = new Dictionary { { AssistantSampleMetadataKey, bool.TrueString } }, + }, + kernel: new Kernel()); +``` + +Declarative: + +```yml +name: Coder +type: openai_assistant +instructions: You are an Agent that can write and execute code to answer questions. +execution_settings: + default: + enable_code_interpreter: true + enable_file_search: true + metadata: + sksample: true +``` + +## Decision Outcome + +Chosen option: "{title of option 1}", because +{justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force {force} | … | comes out best (see below)}. + + + +### Consequences + +- Good, because {positive consequence, e.g., improvement of one or more desired qualities, …} +- Bad, because {negative consequence, e.g., compromising one or more desired qualities, …} +- … + + + +## Validation + +{describe how the implementation of/compliance with the ADR is validated. E.g., by a review or an ArchUnit test} + + + +## Pros and Cons of the Options + +### {title of option 1} + + + +{example | description | pointer to more information | …} + +- Good, because {argument a} +- Good, because {argument b} + +- Neutral, because {argument c} +- Bad, because {argument d} +- … + +### {title of other option} + +{example | description | pointer to more information | …} + +- Good, because {argument a} +- Good, because {argument b} +- Neutral, because {argument c} +- Bad, because {argument d} +- … + + + +## More Information + +{You might want to provide additional evidence/confidence for the decision outcome here and/or +document the team agreement on the decision and/or +define when this decision when and how the decision should be realized and if/when it should be re-visited and/or +how the decision is validated. +Links to other decisions and resources might appear here as well.} From d4b6ee14c3a6fd5cb113c28ae3cdbdfa0de8b3ed Mon Sep 17 00:00:00 2001 From: markwallace-microsoft <127216156+markwallace-microsoft@users.noreply.github.com> Date: Thu, 23 Jan 2025 16:21:59 +0000 Subject: [PATCH 2/6] Add M365 Declarative Agents as an option --- .../NNNN-declarative-agent-schema.md | 227 ++++++++++++------ 1 file changed, 151 insertions(+), 76 deletions(-) diff --git a/docs/decisions/NNNN-declarative-agent-schema.md b/docs/decisions/NNNN-declarative-agent-schema.md index 34ad09d1826a..1c27de7d36af 100644 --- a/docs/decisions/NNNN-declarative-agent-schema.md +++ b/docs/decisions/NNNN-declarative-agent-schema.md @@ -19,23 +19,35 @@ Using the schema defined by this ADR developers will be able to declaratively de Here is some pseudo code to illustrate what we need to be able to do: +```csharp +Kernel kernel = ... +string agentYaml = EmbeddedResource.Read("MyAgent.yaml"); +Agent agent = kernel.CreateAgentFromYaml(agentYaml); + +agent.InvokeAsync(input); +``` + +The above code represents the simplest case would work as follows: + +1. The Agent runtime is responsible for creating a `Kernel` instance which comes fully configured with all services, functions, etc. +2. The `CreateAgentFromYaml` will create one of the built-in Agent instances (currently just `ChatCompletionAgent`). +3. The new Agent instance is initialized with it's own `Kernel` instance configured the services and tools it requires and a default initial state. +4. The `Agent` abstraction contains a method to allow the Agent instance to be invoked with user input. + ```csharp Kernel kernel = ... string agentYaml = EmbeddedResource.Read("MyAgent.yaml"); AgentFactory agentFactory = new AggregatorAgentFactory( - new ChatCompletionFactory(), + new ChatCompletionAgentFactory(), new OpenAIAssistantAgentFactory(), new XXXAgentFactory()); -Agent agent = kernel.LoadAgentFromYaml(agentYaml); - -ChatHistory chatHistory = new(); -chatHistory.AddUserMessage(input); -await foreach (ChatMessageContent content in agent.InvokeAsync(chatHistory)) -{ - chatHistory.Add(content); -} +Agent agent = kernel.CreateAgentFromYaml(agentYaml, agentFactory, agentState); + +agent.InvokeAsync(input); ``` +The above example shows how different Agent types are supported and also how the initial Agent state can be specified. + **Note:** 1. The above pattern does is not supported at present. @@ -90,10 +102,69 @@ For Agent properties that define behaviors e.g. `HistoryReducer` the Semantic Ke ## Considered Options -- Use same semantics as the [Semantic Kernel Prompt Schema](https://learn.microsoft.com/en-us/semantic-kernel/concepts/prompts/yaml-schema#sample-yaml-prompt) -- {title of option 2} +- Use the [Declarative agent schema 1.2 for Microsoft 365 Copilot](https://learn.microsoft.com/en-us/microsoft-365-copilot/extensibility/declarative-agent-manifest-1.2) +- Extend the Declarative agent schema 1.2 for Microsoft 365 Copilot +- Extend the [Semantic Kernel prompt schema](https://learn.microsoft.com/en-us/semantic-kernel/concepts/prompts/yaml-schema#sample-yaml-prompt) + +## Pros and Cons of the Options + +### Use the Declarative agent schema 1.2 for Microsoft 365 Copilot + +Semantic Kernel already has support this, see the [declarative Agent concept sample](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Agents/DeclarativeAgents.cs). + +- Good, this is an existing standard adopted by the Microsoft 365 Copilot. +- Neutral, the schema splits tools into two properties i.e. `capabilities` which includes code interpreter and `actions` which specifies an API plugin manifest. +- Bad, because it does support different types of Agents. +- Bad, because it doesn't provide a way to specific and configure the AI Model to associate with the Agent. +- Bad, because it doesn't provide a way to use a Prompt Template for the Agent instructions. +- Bad, because `actions` property is focussed on calling REST API's and cater for native and semantic functions. + +### Extend the Declarative agent schema 1.2 for Microsoft 365 Copilot + +Some of the possible extensions include: -### Use Same Semantics as the Semantic Kernel Prompt Schema +1. Agent instructions can be created using a Prompt Template. +2. Agent Model settings can be specified including fallbacks based on the available models. +3. Better definition of functions e.g. support for native and semantic. + +- Good, because {argument a} +- Good, because {argument b} +- Neutral, because {argument c} +- Bad, because {argument d} +- … + +### Extend the Semantic Kernel Prompt Schema + +- Good, because {argument a} +- Good, because {argument b} +- Neutral, because {argument c} +- Bad, because {argument d} +- … + +## Decision Outcome + +Chosen option: "{title of option 1}", because +{justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force {force} | … | comes out best (see below)}. + + + +### Consequences + +- Good, because {positive consequence, e.g., improvement of one or more desired qualities, …} +- Bad, because {negative consequence, e.g., compromising one or more desired qualities, …} +- … + + + +## Validation + +{describe how the implementation of/compliance with the ADR is validated. E.g., by a review or an ArchUnit test} + + + +## More Information + +Below are examples showing the code first and equivalent declarative syntax fot creating different types of Agents. Consider the following use cases: @@ -117,14 +188,17 @@ ChatCompletionAgent agent = }; ``` -Declarative: +Declarative using M365 or Semantic Kernel schema: ```yml name: Parrot instructions: Repeat the user message in the voice of a pirate and then end with a parrot sound. ``` -**Note**: `ChatCompletionAgent` could be the default agent type hence no explicit `type` property is required. +**Note**: + +- Both M365 and Semantic Kernel schema would be identical +- `ChatCompletionAgent` could be the default agent type hence no explicit `type` property is required. #### `ChatCompletionAgent` using Prompt Template @@ -146,8 +220,16 @@ ChatCompletionAgent agent = }; ``` -Declarative: +Declarative using M365 Agent schema: + +```yml +name: GenerateStory +instructions: ${file['./GenerateStory.yaml']} +``` + +Agent YAML points to another file, the Declarative Agent implementation in Semantic Kernel already uses this technique to load a separate instructions file. +Prompt template which is used to define the instructions. ```yml name: GenerateStory template: | @@ -165,7 +247,7 @@ input_variables: default: 3 ``` -**Note**: Only elements from the prompt template schema are needed. +**Note**: Semantic Kernel could load this file directly. #### `ChatCompletionAgent` with Function Calling @@ -186,7 +268,23 @@ KernelPlugin plugin = KernelPluginFactory.CreateFromType(); agent.Kernel.Plugins.Add(plugin); ``` -Declarative: +Declarative using M365 Agent schema: + +```yml +name: RestaurantHost +instructions: Answer questions about the menu. +description: This agent answers questions about the menu. +actions: + - id: MenuPlugin +``` + +**Note**: + +- There is no way to specify `Temperature` +- There is no way to specify the function choice behavior (e.g. could not specify required) +- All functions in the Plugin would be used. + +Declarative using Semantic Kernel schema: ```yml name: RestaurantHost @@ -222,7 +320,22 @@ KernelPlugin plugin = KernelPluginFactory.CreateFromType(); agent.Kernel.Plugins.Add(plugin); ``` -Declarative: +Declarative using M365 Agent schema: + +```yml +name: RestaurantHost +type: openai_assistant +instructions: Answer questions about the menu. +description: This agent answers questions about the menu. +metadata: + sksample: true +actions: + - id: MenuPlugin +``` + +**Note**: `type` and `metadata` properties need to be added somehow + +Declarative using Semantic Kernel schema: ```yml name: RestaurantHost @@ -230,7 +343,7 @@ type: openai_assistant instructions: Answer questions about the menu. description: This agent answers questions about the menu. execution_settings: - gpt_4o: + default: function_choice_behavior: type: auto functions: @@ -261,7 +374,25 @@ OpenAIAssistantAgent agent = kernel: new Kernel()); ``` -Declarative: +Declarative using M365 Agent schema: + +```yml +name: RestaurantHost +type: openai_assistant +instructions: Answer questions about the menu. +description: This agent answers questions about the menu. +metadata: + sksample: true +capabilities: + - name: CodeInterpreter + - name: FileSearch +actions: + - id: MenuPlugin +``` + +**Note**: `FileSearch` capability needs to be added + +Declarative using Semantic Kernel: ```yml name: Coder @@ -274,59 +405,3 @@ execution_settings: metadata: sksample: true ``` - -## Decision Outcome - -Chosen option: "{title of option 1}", because -{justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force {force} | … | comes out best (see below)}. - - - -### Consequences - -- Good, because {positive consequence, e.g., improvement of one or more desired qualities, …} -- Bad, because {negative consequence, e.g., compromising one or more desired qualities, …} -- … - - - -## Validation - -{describe how the implementation of/compliance with the ADR is validated. E.g., by a review or an ArchUnit test} - - - -## Pros and Cons of the Options - -### {title of option 1} - - - -{example | description | pointer to more information | …} - -- Good, because {argument a} -- Good, because {argument b} - -- Neutral, because {argument c} -- Bad, because {argument d} -- … - -### {title of other option} - -{example | description | pointer to more information | …} - -- Good, because {argument a} -- Good, because {argument b} -- Neutral, because {argument c} -- Bad, because {argument d} -- … - - - -## More Information - -{You might want to provide additional evidence/confidence for the decision outcome here and/or -document the team agreement on the decision and/or -define when this decision when and how the decision should be realized and if/when it should be re-visited and/or -how the decision is validated. -Links to other decisions and resources might appear here as well.} From e800600344ba3c5083d40f045332754e6d599ebe Mon Sep 17 00:00:00 2001 From: markwallace-microsoft <127216156+markwallace-microsoft@users.noreply.github.com> Date: Tue, 28 Jan 2025 09:17:50 +0000 Subject: [PATCH 3/6] Fix type and update decision drivers --- docs/decisions/NNNN-declarative-agent-schema.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/decisions/NNNN-declarative-agent-schema.md b/docs/decisions/NNNN-declarative-agent-schema.md index 1c27de7d36af..28fe6ab30632 100644 --- a/docs/decisions/NNNN-declarative-agent-schema.md +++ b/docs/decisions/NNNN-declarative-agent-schema.md @@ -93,11 +93,12 @@ For Agent properties that define behaviors e.g. `HistoryReducer` the Semantic Ke ## Decision Drivers -- Schema **MUST** allow model settings to be assigned to an Agent -- Schema **MUST** allow functions to be assigned to an Agent -- Schema **MUST** allow a Semantic Kernel prompt to be used to define the Agent instructions -- Schema **MUST** be extensible so that support for new Agent types can be added to Semantic Kernel -- Schema **MUST** allow third parties to contribute new Agent types to Semantic Kernel +- Schema **MUST** be Agent Service agnostic i.e., will work with Agents targeting Azure, Open AI, Mistral AI, ... +- Schema **MUST** allow model settings to be assigned to an Agent. +- Schema **MUST** allow tools (e.g. functions, code interpreter, file search, ...) to be assigned to an Agent. +- Schema **MUST** allow a Semantic Kernel prompt (including Prompty format) to be used to define the Agent instructions. +- Schema **MUST** be extensible so that support for new Agent types with their own settings and tools, can be added to Semantic Kernel. +- Schema **MUST** allow third parties to contribute new Agent types to Semantic Kernel. - … ## Considered Options @@ -164,7 +165,7 @@ Chosen option: "{title of option 1}", because ## More Information -Below are examples showing the code first and equivalent declarative syntax fot creating different types of Agents. +Below are examples showing the code first and equivalent declarative syntax for creating different types of Agents. Consider the following use cases: From 59f2df5e47dc35daa59976d0a871388ba60509b2 Mon Sep 17 00:00:00 2001 From: markwallace-microsoft <127216156+markwallace-microsoft@users.noreply.github.com> Date: Wed, 29 Jan 2025 21:13:22 +0000 Subject: [PATCH 4/6] Update decision drivers --- .../NNNN-declarative-agent-schema.md | 27 ++++++++++++++----- 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/docs/decisions/NNNN-declarative-agent-schema.md b/docs/decisions/NNNN-declarative-agent-schema.md index 28fe6ab30632..6e35df0b6306 100644 --- a/docs/decisions/NNNN-declarative-agent-schema.md +++ b/docs/decisions/NNNN-declarative-agent-schema.md @@ -50,12 +50,11 @@ The above example shows how different Agent types are supported and also how the **Note:** -1. The above pattern does is not supported at present. +1. Providing Agent state is not supported in the Agent Framework at present. 2. We need to decide if the Agent Framework should define an abstraction to allow any Agent to be invoked. -3. If we are supporting an abstraction to allow any Agent to be invoked, what should the pattern look like. -4. We will support JSON also as an out-of-the-box option. +3. We will support JSON also as an out-of-the-box option. -Currently Semantic Kernel supports two Agent types and these have the following properties: +Currently Semantic Kernel supports three Agent types and these have the following properties: 1. [`ChatCompletionAgent`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel.agents.chatcompletionagent?view=semantic-kernel-dotnet): - `Arguments`: Optional arguments for the agent. (Inherited from ChatHistoryKernelAgent) @@ -67,7 +66,7 @@ Currently Semantic Kernel supports two Agent types and these have the following - `Logger`: The ILogger associated with this Agent. (Inherited from Agent) - `LoggerFactory`: A ILoggerFactory for this Agent. (Inherited from Agent) - `Name`: The name of the agent (optional). (Inherited from Agent) -2. ['OpenAIAssistantAgent'](https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel.agents.agent.description?view=semantic-kernel-dotnet#microsoft-semantickernel-agents-agent-description): +2. [`OpenAIAssistantAgent`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel.agents.agent.description?view=semantic-kernel-dotnet#microsoft-semantickernel-agents-agent-description): - `Arguments`: Optional arguments for the agent. - `Definition`: The assistant definition. - `Description`: The description of the agent (optional). (Inherited from Agent) @@ -79,8 +78,19 @@ Currently Semantic Kernel supports two Agent types and these have the following - `LoggerFactory`: A ILoggerFactory for this Agent. (Inherited from Agent) - `Name`: The name of the agent (optional). (Inherited from Agent) - `PollingOptions`: Defines polling behavior +3. [`AzureAIAgent`](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Agents/AzureAI/AzureAIAgent.cs) + - `Definition`: The assistant definition. + - `PollingOptions`: Defines polling behavior for run processing. + - `Description`: The description of the agent (optional). (Inherited from Agent) + - `Id`: The identifier of the agent (optional). (Inherited from Agent) + - `Instructions`: The instructions of the agent (optional). (Inherited from KernelAgent) + - `IsDeleted`: Set when the assistant has been deleted via DeleteAsync(CancellationToken). An assistant removed by other means will result in an exception when invoked. + - `Kernel`: The Kernel containing services, plugins, and filters for use throughout the agent lifetime. (Inherited from KernelAgent) + - `Logger`: The ILogger associated with this Agent. (Inherited from Agent) + - `LoggerFactory`: A ILoggerFactory for this Agent. (Inherited from Agent) + - `Name`: The name of the agent (optional). (Inherited from Agent) -When executing an Agent that was declaratively defined some of the properties will be determined by the runtime: +When executing an Agent that was defined declaratively some of the properties will be determined by the runtime: - `Kernel`: The runtime will be responsible for create the `Kernel` instance to be used by the Agent. This `Kernel` instance must be configured with the models and tools that the Agent requires. - `Logger` or `LoggerFactory`: The runtime will be responsible for providing a correctly configured `Logger` or `LoggerFactory`. @@ -96,11 +106,16 @@ For Agent properties that define behaviors e.g. `HistoryReducer` the Semantic Ke - Schema **MUST** be Agent Service agnostic i.e., will work with Agents targeting Azure, Open AI, Mistral AI, ... - Schema **MUST** allow model settings to be assigned to an Agent. - Schema **MUST** allow tools (e.g. functions, code interpreter, file search, ...) to be assigned to an Agent. +- Schema **MUST** allow new types of tools to be defined for an Agent to use. - Schema **MUST** allow a Semantic Kernel prompt (including Prompty format) to be used to define the Agent instructions. - Schema **MUST** be extensible so that support for new Agent types with their own settings and tools, can be added to Semantic Kernel. - Schema **MUST** allow third parties to contribute new Agent types to Semantic Kernel. - … +### Out of Scope + +- This ADR does not cover the multi-agent declarative format or the process declarative format + ## Considered Options - Use the [Declarative agent schema 1.2 for Microsoft 365 Copilot](https://learn.microsoft.com/en-us/microsoft-365-copilot/extensibility/declarative-agent-manifest-1.2) From a462b15afa88760fd1fa9a01e02d950a0c7c8e85 Mon Sep 17 00:00:00 2001 From: markwallace-microsoft <127216156+markwallace-microsoft@users.noreply.github.com> Date: Thu, 6 Feb 2025 18:09:09 +0000 Subject: [PATCH 5/6] Add use cases --- .../NNNN-declarative-agent-schema.md | 145 ++++++++++++++++++ 1 file changed, 145 insertions(+) diff --git a/docs/decisions/NNNN-declarative-agent-schema.md b/docs/decisions/NNNN-declarative-agent-schema.md index 6e35df0b6306..fa9d0e9cb7c8 100644 --- a/docs/decisions/NNNN-declarative-agent-schema.md +++ b/docs/decisions/NNNN-declarative-agent-schema.md @@ -112,6 +112,17 @@ For Agent properties that define behaviors e.g. `HistoryReducer` the Semantic Ke - Schema **MUST** allow third parties to contribute new Agent types to Semantic Kernel. - … +The document will describe the following use cases: + +1. Metadata about the agent and the file. +2. Creating an Agent with access to function tools and a set of instructions to guide it's behavior. +3. Allow templating of Agent instructions (and other properties). +4. Configuring the model and providing multiple model configurations. +5. Configuring data sources (context/knowledge) for the Agent to use. +6. Configuring additional tools for the Agent to use e.g. code interpreter, OpenAPI endpoints, . +7. Enabling additional modalities for the Agent e.g. speech. +8. Error conditions e.g. models or function tools not being available. + ### Out of Scope - This ADR does not cover the multi-agent declarative format or the process declarative format @@ -180,6 +191,8 @@ Chosen option: "{title of option 1}", because ## More Information +### Code First versus Declarative Format + Below are examples showing the code first and equivalent declarative syntax for creating different types of Agents. Consider the following use cases: @@ -353,6 +366,9 @@ actions: Declarative using Semantic Kernel schema: +Using the syntax below the assistant does not have the functions included in it's definition. +The functions must be added to the `Kernel` instance associated with the Agent and will be passed when the Agent is invoked. + ```yml name: RestaurantHost type: openai_assistant @@ -367,8 +383,28 @@ execution_settings: - MenuPlugin.GetItemPrice metadata: sksample: true +`` + +or the +```yml +name: RestaurantHost +type: openai_assistant +instructions: Answer questions about the menu. +description: This agent answers questions about the menu. +execution_settings: + default: + temperature: 0.4 +tools: + - type: function + name: MenuPlugin-GetSpecials + description: Provides a list of specials from the menu. + - type: function + name: MenuPlugin-GetItemPrice + description: Provides the price of the requested menu item. + parameters: '{"type":"object","properties":{"menuItem":{"type":"string","description":"The name of the menu item."}},"required":["menuItem"]}' ``` + **Note**: The `Kernel` instance used to create the Agent must have an instance of `OpenAIClientProvider` registered as a service. #### `OpenAIAssistantAgent` with Tools @@ -421,3 +457,112 @@ execution_settings: metadata: sksample: true ``` + +### Declarative Format Use Cases + +#### Metadata about the agent and the file + +```yaml +name: RestaurantHost +type: azureai_agent +description: This agent answers questions about the menu. +version: 0.0.1 +``` + +#### Creating an Agent with access to function tools and a set of instructions to guide it's behavior + +```yaml +name: RestaurantHost +type: azureai_agent +description: This agent answers questions about the menu. +version: 0.0.1 +instructions: Answer questions about the menu. +execution_settings: + default: + temperature: 0.4 + function_choice_behavior: + type: auto + functions: + - MenuPlugin.GetSpecials + - MenuPlugin.GetItemPrice +``` + +or + +```yml +name: RestaurantHost +type: azureai_agent +description: This agent answers questions about the menu. +instructions: Answer questions about the menu. +execution_settings: + default: + temperature: 0.4 +tools: + - type: function + name: MenuPlugin-GetSpecials + description: Provides a list of specials from the menu. + - type: function + name: MenuPlugin-GetItemPrice + description: Provides the price of the requested menu item. + parameters: '{"type":"object","properties":{"menuItem":{"type":"string","description":"The name of the menu item."}},"required":["menuItem"]}' +``` + +#### Allow templating of Agent instructions (and other properties) + +```yaml +``` + +#### Configuring the model and providing multiple model configurations + +```yaml +name: RestaurantHost +type: azureai_agent +description: This agent answers questions about the menu. +instructions: Answer questions about the menu. +execution_settings: + default: + temperature: 0.4 + gpt-4o: + temperature: 0.5 + gpt-4o-mini: + temperature: 0.5 +tools: + - type: function + name: MenuPlugin-GetSpecials + description: Provides a list of specials from the menu. + - type: function + name: MenuPlugin-GetItemPrice + description: Provides the price of the requested menu item. + parameters: '{"type":"object","properties":{"menuItem":{"type":"string","description":"The name of the menu item."}},"required":["menuItem"]}' +``` + +#### Configuring data sources (context/knowledge) for the Agent to use + +```yaml +``` + +#### Configuring additional tools for the Agent to use e.g. code interpreter, OpenAPI endpoints + +```yaml +name: Coder Agent +type: azureai_agent +description: This agent uses code to answer questions. +instructions: Use code to answer questions. +execution_settings: + default: + metadata: + sksample: true +tools: + - type: code_interpreter +``` + +```yaml +``` + +#### Enabling additional modalities for the Agent e.g. speech + +```yaml +``` + +#### Error conditions e.g. models or function tools not being available + From e15236a230bee6910effe14af97f3c0d0b5c8243 Mon Sep 17 00:00:00 2001 From: markwallace-microsoft <127216156+markwallace-microsoft@users.noreply.github.com> Date: Thu, 6 Feb 2025 18:26:24 +0000 Subject: [PATCH 6/6] Add some use case samples --- .../NNNN-declarative-agent-schema.md | 41 ++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/docs/decisions/NNNN-declarative-agent-schema.md b/docs/decisions/NNNN-declarative-agent-schema.md index fa9d0e9cb7c8..e49e884f6a66 100644 --- a/docs/decisions/NNNN-declarative-agent-schema.md +++ b/docs/decisions/NNNN-declarative-agent-schema.md @@ -475,7 +475,6 @@ version: 0.0.1 name: RestaurantHost type: azureai_agent description: This agent answers questions about the menu. -version: 0.0.1 instructions: Answer questions about the menu. execution_settings: default: @@ -510,6 +509,21 @@ tools: #### Allow templating of Agent instructions (and other properties) ```yaml +name: GenerateStory +description: An agent that generates a story about a topic. +template: | + Tell a story about {{$topic}} that is {{$length}} sentences long. +template_format: semantic-kernel +input_variables: + - name: topic + description: The topic of the story. + is_required: true + - name: length + description: The number of sentences in the story. + is_required: true +execution_settings: + default: + temperature: 0.6 ``` #### Configuring the model and providing multiple model configurations @@ -539,8 +553,20 @@ tools: #### Configuring data sources (context/knowledge) for the Agent to use ```yaml +name: Document FAQ Agent +type: azureai_agent +instructions: Use provide documents to answer questions. Politely decline to answer if the provided documents don't include an answer to the question. +description: This agent uses documents to answer questions. +execution_settings: + default: + metadata: + sksample: true +tools: + - type: file_search ``` +**TODO: How do we configure the documents to include?** + #### Configuring additional tools for the Agent to use e.g. code interpreter, OpenAPI endpoints ```yaml @@ -557,6 +583,19 @@ tools: ``` ```yaml +name: Country FAQ Agent +type: azureai_agent +instructions: Answer questions about countries. For all other questions politely decline to answer. +description: This agent answers question about different countries. +execution_settings: + default: + metadata: + sksample: true +tools: + - type: openapi + name: RestCountriesAPI + description: Web API version 3.1 for managing country items, based on previous implementations from restcountries.eu and restcountries.com. + schema: '{"openapi":"3.1.0","info":{"title":"Get Weather Data","description":"Retrieves current weather data for a location based on wttr.in.","version":"v1.0.0"},"servers":[{"url":"https://wttr.in"}],"auth":[],"paths":{"/{location}":{"get":{"description":"Get weather information for a specific location","operationId":"GetCurrentWeather","parameters":[{"name":"location","in":"path","description":"City or location to retrieve the weather for","required":true,"schema":{"type":"string"}},{"name":"format","in":"query","description":"Always use j1 value for this parameter","required":true,"schema":{"type":"string","default":"j1"}}],"responses":{"200":{"description":"Successful response","content":{"text/plain":{"schema":{"type":"string"}}}},"404":{"description":"Location not found"}},"deprecated":false}}},"components":{"schemes":{}}}' ``` #### Enabling additional modalities for the Agent e.g. speech