Azure OpenAI: 2.2.0-beta.1 (2025-01-01-preview label) (#329)

* adding gitattributes to handle line endings

* resquish after tooling re-merge

* post-merge regen of API, should diff more cleanly now

* retarget, retest, squish, disable unsupported things

* desquished complete

* add a minimal, explicit test for the latest o1 model

* improve AOAI parity spec generation (code no-op)

* squashed merge for retarget

* expose and test a distinct 'update' type for streamed response audio

* update readme for streaming accuracy and explanation

* 'responseaudio' to 'outputaudio', doc comments and cleaup

* PR feedback round

* Expose and require ContentModalities for Chat

* PR feedback

* merge, plus realtime vad create_response

* update to 2025-01-01-preview target

* minor changelog update; test assets repushed

* fix realtime conversation token usage

* remove duplicated file after merge

* tool updates for new spec peculiarities

* small spec accuracy improvements (no code change)

* a few more spec accuracy updates (also no code change)

* a few more spec accuracy updates (also no code change)

* several more versioning accuracy adjustments for the Azure spec

* scenario-specific default versioning affordance, test cleanup

* re-push assets; unclear why it didn't flush before, but it should work now

* no code change: update for spec docs preservation on extended scalar type 'ParallelToolCalls'

* update for latest 2025-01-01-preview functionality (files is back)

* fix developer message ordering to preserve int stability

* minor update, including gitattributes QoL merge from Chris's old PR

* prediction now works, so enable it!

* incorporate tool update for anonymous array element extraction

* merge, apply latest spec and tool updates

* PR feedback

* minor: flush replicated changes from azure-sdk-for-net stage

* PR feedback: new OutputPrediction instead of PredictedContent

* merge predicted output change and rename to SourceIP

* push assets again (stale file bug)

* PR feedback

* fix filemocks tests -- 'object': 'list' is required

* PR feedback

* test merge

* address constructor

* PR feedback, image test reliability for owls

---------

Co-authored-by: chschrae <[email protected]>

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/CHANGELOG.md

@@ -1,6 +1,21 @@
 # Release History
 
-## 2.1.0 (Unreleased)
+## 2.2.0-beta.1 (2025-02-04)
+
+This preview release aligns with the corresponding `2.2.0` beta of `OpenAI` and the `2025-01-01-Preview` Azure OpenAI Service API version.
+
+New features include since 2.1.0-beta.2 include:
+
+- Audio input for Chat Completions using `gpt-4o-audio-preview` or other compatible models: provide input audio via `ChatMessageContentPart.CreateInputAudioPart()`, set `AudioOptions` and `ResponseModalities` on `ChatCompletionOptions`, retrieve response audio via `OutputAudio` on `ChatCompletion`, and reference audio history from the assistant by using the `AssistantChatMessage(ChatCompletion)` constructor or using `ChatMessageContentPart.CreateAudioPart(string)`. For more information, refer to the examples in [the OpenAI README](https://github.com/openai/openai-dotnet/blob/main/README.md). 
+- Predicted outputs in Chat Completions: `ChatCompletionOptions` accepts an `OutputPrediction` property that can be used via `ChatOutputPrediction.CreateStaticContentPrediction()` with text content to optimize operation efficiency for scenarios like code completion. For more information, see [OpenAI's predicted outputs announcement](https://community.openai.com/t/introducing-predicted-outputs/1004502).
+- Chat Completions `o`-series model feature support: the new `developer` message role via `DeveloperChatMessage` (used just like `SystemChatMessage`), `ReasoningEffortLevel` on Chat Completion options
+- [AOAI exclusive] `UserSecurityContext` integration with [Defender for Cloud](https://learn.microsoft.com/azure/defender-for-cloud/gain-end-user-context-ai); add a `UserSecurityContext` instance to `ChatCompletionOptions` with `SetUserSecurityContext()`
+
+### Breaking Changes
+
+- **Batch**: files uploaded for batch operations (`UploadFile` with `FileUploadPurpose.Batch`) will now report a `status` of `processed`, matching expected behavior against OpenAI's `/v1` endpoint. This is a change from past behavior where such files would initially report `pending` and a later `processed`, `error`, or other status depending on operation progress. Batch input validation is instead consistently performed from the batch client.
+
+## 2.1.0 (2024-12-05)
 
 This GA library release aligns functionality with the latest `2024-10-21` stable service API label.
 
@@ -39,32 +54,21 @@ The `2024-10-21` service API label introduces stable support for batch chat comp
 
 This update brings compatibility with the Azure OpenAI `2024-10-01-preview` service API version as well as the `2.1.0-beta.2` release of the `OpenAI` library.
 
-### Breaking Changes
-
-- `[Experimental]` `ChatCitation` and `ChatRetrievedDocument` have each replaced the `Uri` property of type `System.Uri` with a `string` property named `Url`. This aligns with the REST specification and accounts for the wire value of `url` not always providing a valid RFC 3986 identifier [[azure-sdk-for-net \#46793](https://github.com/Azure/azure-sdk-for-net/issues/46793)]
-
 ### Features Added
 
 - The included update via `2024-09-01-preview` brings AOAI support for streaming token usage in chat completions; `Usage` is now automatically populated in `StreamingChatCompletionUpdate` instances.
   - Note 1: this feature is not yet compatible when using On Your Data features (after invoking the `.AddDataSource()` extension method on `ChatCompletionOptions`)
   - Note 2: this feature is not yet compatible when using image input (a `ChatMessageContentPart` of `Kind` `Image`)
 - `2024-10-01-preview` further adds support for ungrounded content detection in chat completion content filter results via the `UngroundedMaterial` property on `ResponseContentFilterResult`, as retrieved from a chat completion via the `GetResponseContentFilterResult()` extension method.
 
-Via `OpenAI 2.0.0-beta.2`:
+## Breaking Changes
 
-- Made improvements to the experimental Realtime API. Please note this features area is currently under rapid development and not all changes may be reflected here.
-  - Several types have been renamed for consistency and clarity.
-  - ConversationRateLimitsUpdate (previously ConversationRateLimitsUpdatedUpdate) now includes named RequestDetails and TokenDetails properties, mapping to the corresponding named items in the underlying rate_limits command payload.
+- `[Experimental]` `ChatCitation` and `ChatRetrievedDocument` have each replaced the `Uri` property of type `System.Uri` with a `string` property named `Url`. This aligns with the REST specification and accounts for the wire value of `url` not always providing a valid RFC 3986 identifier [[azure-sdk-for-net \#46793](https://github.com/Azure/azure-sdk-for-net/issues/46793)]
 
-### Bugs Fixed
+## Bugs Fixed
 
-- Addressed an HTTP 401 issue that caused certain connection retry attempts, such as those triggered for HTTP 429 rate limiting errors, to sometimes generate a malformed request with multiple `Authorization` headers that would then be rejected. [#46401](https://github.com/Azure/azure-sdk-for-net/pull/46401)
 - Addressed an issue that caused `ChatCitation` and `ChatRetrievedDocument` to sometimes throw on deserialization, specifically when a returned value in the `url` JSON field was not populated with an RFC 3986 compliant identifier for `System.Uri` [[azure-sdk-for-net \#46793](https://github.com/Azure/azure-sdk-for-net/issues/46793)]
 
-Via `OpenAI 2.0.0-beta.2`:
-
-- Fixed serialization and deserialization of ConversationToolChoice literal values (such as "required").
-
 ## 2.1.0-beta.1 (2024-10-01)
 
 Relative to the prior GA release, this update restores preview surfaces, retargeting to the latest `2024-08-01-preview` service `api-version` label. It also brings early support for the newly-announced `/realtime` capabilities with `gpt-4o-realtime-preview`. You can read more about Azure OpenAI support for `/realtime` in the annoucement post here: https://azure.microsoft.com/blog/announcing-new-products-and-features-for-azure-openai-service-including-gpt-4o-realtime-preview-with-audio-and-speech-capabilities/
@@ -634,7 +638,7 @@ management, and align with the Azure SDK guidelines.
   - In contrast to other capabilities, DALL-E image generation does not require explicit creation or specification of a deployment or model. Its surface as such does not include this concept.
 - Functions for chat completions are now supported: see [OpenAI's blog post on the topic](https://openai.com/blog/function-calling-and-other-api-updates) for much more detail.
   - A list of `FunctionDefinition` objects may be populated on `ChatCompletionsOptions` via its `Functions` property. These definitions include a name and description together with a serialized JSON Schema representation of its parameters; these parameters can be generated easily via `BinaryData.FromObjectAsJson` with dynamic objects -- see the README for example usage.
-  - **NOTE**: Chat Functions requires a minimum of the `-0613` model versions for `gpt-4` and `gpt-3.5-turbo`/`gpt-35-turbo`. Please ensure you're using these later model versions, as Functions are not supported with older model revisions. For Azure OpenAI, you can update a deployment's model version or create a new model deployment with an updated version via the Azure AI Studio interface, also accessible through Azure Portal.
+  - **NOTE**: Chat Functions requires a minimum of the `-0613` model versions for `gpt-4` and `gpt-3.5-turbo`/`gpt-35-turbo`. Please ensure you're using these later model versions, as Functions are not supported with older model revisions. For Azure OpenAI, you can update a deployment's model version or create a new model deployment with an updated version via the Azure AI Foundry interface, also accessible through Azure Portal.
 - (Azure OpenAI specific) Completions and Chat Completions responses now include embedded content filter annotations for prompts and responses
 - A new `Azure.AI.OpenAI.AzureOpenAIModelFactory` is now present for mocking.
 

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/Directory.Build.props

@@ -4,8 +4,8 @@
   <!-- Now configure the Azure OpenAI projects to reference the local OpenAI source code -->
   <PropertyGroup>
     <ExternalOpenAISource>$(RepoRoot)/../.dotnet/src/OpenAI.csproj</ExternalOpenAISource>
-    <GAServiceVersionLabel>2024_10_21</GAServiceVersionLabel>
     <!--
+    <GAServiceVersionLabel>2024_10_21</GAServiceVersionLabel>
     <SystemClientModelVersion>1.1.0-beta.5</SystemClientModelVersion>
     <ExternalOpenAILibrary>path/to/OpenAI.dll</ExternalOpenAILibrary>
     <ExternalAzureCoreSource>$(RepoRoot)/sdk/core/Azure.Core/src/Azure.Core.csproj</ExternalAzureCoreSource>

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/assets.json

@@ -2,5 +2,5 @@
   "AssetsRepo": "Azure/azure-sdk-assets",
   "AssetsRepoPrefixPath": "net",
   "TagPrefix": "dotnet.azure/openai/Azure.AI.OpenAI",
-  "Tag": "dotnet.azure/openai/Azure.AI.OpenAI_124bffd832"
+  "Tag": "dotnet.azure/openai/Azure.AI.OpenAI_9ba94506a0"
 }

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Assistants/AzureAssistantClient.cs

@@ -26,7 +26,7 @@ internal AzureAssistantClient(ClientPipeline pipeline, Uri endpoint, AzureOpenAI
         options ??= new();
 
         _endpoint = endpoint;
-        _apiVersion = options.Version;
+        _apiVersion = options.GetRawServiceApiValueForClient(this);
     }
 
     protected AzureAssistantClient()

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Audio/AzureAudioClient.cs

@@ -28,7 +28,7 @@ internal AzureAudioClient(ClientPipeline pipeline, string deploymentName, Uri en
 
         _deploymentName = deploymentName;
         _endpoint = endpoint;
-        _apiVersion = options.Version;
+        _apiVersion = options.GetRawServiceApiValueForClient(this);
     }
 
     protected AzureAudioClient()

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/AzureOpenAIClientOptions.cs

@@ -1,16 +1,20 @@
 // Copyright (c) Microsoft Corporation. All rights reserved.
 // Licensed under the MIT License.
 
+using Azure.AI.OpenAI.Files;
+using Azure.AI.OpenAI.RealtimeConversation;
 using System.ClientModel.Primitives;
 
 namespace Azure.AI.OpenAI;
 
+#pragma warning disable AOAI001
+
 /// <summary>
 /// Defines the scenario-independent, client-level options for the Azure-specific OpenAI client.
 /// </summary>
 public partial class AzureOpenAIClientOptions : ClientPipelineOptions
 {
-    internal string Version { get; }
+    internal ServiceVersion? ExplicitVersion { get; }
 
     /// <summary>
     /// The authorization audience to use when authenticating with Azure authentication tokens
@@ -45,27 +49,33 @@ public string UserAgentApplicationId
     private string _userAgentApplicationId;
 
     /// <summary>
-    /// Initializes a new instance of <see cref="AzureOpenAIClientOptions"/>
+    /// Initializes a new instance of <see cref="AzureOpenAIClientOptions"/>.
     /// </summary>
-    /// <param name="version"> The service API version to use with the client. </param>
-    /// <exception cref="NotSupportedException"> The provided service API version is not supported. </exception>
-    public AzureOpenAIClientOptions(ServiceVersion version = LatestVersion)
+    /// <remarks>
+    /// When using this constructor, the best matching service API version will automatically be selected. This is
+    /// typically the latest service API version available when the library is published. To specify a service API
+    /// version manually, use the <see cref="AzureOpenAIClientOptions(ServiceVersion)"/> overload, instead.
+    /// </remarks>
+    public AzureOpenAIClientOptions()
         : base()
     {
-        Version = version switch
-        {
-#if !AZURE_OPENAI_GA
-            ServiceVersion.V2024_08_01_Preview => "2024-08-01-preview",
-            ServiceVersion.V2024_09_01_Preview => "2024-09-01-preview",
-            ServiceVersion.V2024_10_01_Preview => "2024-10-01-preview",
-#endif
-            ServiceVersion.V2024_06_01 => "2024-06-01",
-            ServiceVersion.V2024_10_21 => "2024-10-21",
-            _ => throw new NotSupportedException()
-        };
         RetryPolicy = new RetryWithDelaysPolicy();
     }
 
+    /// <summary>
+    /// Initializes a new instance of <see cref="AzureOpenAIClientOptions"/>.
+    /// </summary>
+    /// <remarks>
+    /// This overload will attempt to use a specific service API version label that may differ from the preferred
+    /// default. Please note that operation behavior may differ when using non-default service API versions.
+    /// </remarks>
+    /// <param name="version"> The service API version to use with the client. </param>
+    public AzureOpenAIClientOptions(ServiceVersion version)
+        : this()
+    {
+        ExplicitVersion = version;
+    }
+
     /// <summary> The version of the service to use. </summary>
     public enum ServiceVersion
     {
@@ -76,6 +86,56 @@ public enum ServiceVersion
         V2024_10_01_Preview = 3,
 #endif
         V2024_10_21 = 4,
+#if !AZURE_OPENAI_GA
+        V2024_12_01_Preview = 5,
+        V2025_01_01_Preview = 6,
+#endif
+    }
+
+    private static string GetStringForVersion(ServiceVersion version)
+    {
+        return version switch
+        {
+            ServiceVersion.V2024_06_01 => "2024-06-01",
+#if !AZURE_OPENAI_GA
+            ServiceVersion.V2024_08_01_Preview => "2024-08-01-preview",
+            ServiceVersion.V2024_09_01_Preview => "2024-09-01-preview",
+            ServiceVersion.V2024_10_01_Preview => "2024-10-01-preview",
+#endif
+            ServiceVersion.V2024_10_21 => "2024-10-21",
+#if !AZURE_OPENAI_GA
+            ServiceVersion.V2024_12_01_Preview => "2024-12-01-preview",
+            ServiceVersion.V2025_01_01_Preview => "2025-01-01-preview",
+#endif
+            _ => throw new NotSupportedException($"The specified {nameof(ServiceVersion)} value ({version}) is not supported.")
+        };
+    }
+
+    internal string GetRawServiceApiValueForClient(object client)
+    {
+        if (ExplicitVersion.HasValue)
+        {
+            return GetStringForVersion(ExplicitVersion.Value);
+        }
+        else
+        {
+            ServiceVersion defaultVersion = client switch
+            {
+#if !AZURE_OPENAI_GA
+                // Realtime (preview only) is currently *only* supported on 2024-10-01-preview; override default
+                // version selection for optimal out-of-the-box support if it's not explicitly specified.
+                AzureRealtimeConversationClient _ => ServiceVersion.V2024_10_01_Preview,
+#endif
+#if !AZURE_OPENAI_GA
+                // Standard default for beta libraries: latest preview version
+                _ => ServiceVersion.V2025_01_01_Preview,
+#else
+                // Standard default for GA libraries: latest stable version
+                _ => ServiceVersion.V2024_10_21,
+#endif
+            };
+            return GetStringForVersion(defaultVersion);
+        }
     }
 
     internal class RetryWithDelaysPolicy : ClientRetryPolicy
@@ -107,7 +167,7 @@ protected override TimeSpan GetNextDelay(PipelineMessage message, int tryCount)
     }
 
 #if !AZURE_OPENAI_GA
-    private const ServiceVersion LatestVersion = ServiceVersion.V2024_10_01_Preview;
+    private const ServiceVersion LatestVersion = ServiceVersion.V2025_01_01_Preview;
 #else
     private const ServiceVersion LatestVersion = ServiceVersion.V2024_10_21;
 #endif

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Batch/AzureBatchClient.cs

@@ -27,7 +27,7 @@ internal AzureBatchClient(ClientPipeline pipeline, Uri endpoint, AzureOpenAIClie
         options ??= new();
 
         _endpoint = endpoint;
-        _apiVersion = options.Version;
+        _apiVersion = options.GetRawServiceApiValueForClient(this);
     }
 
     protected AzureBatchClient()

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Chat/AzureChatClient.cs

@@ -33,7 +33,7 @@ internal AzureChatClient(ClientPipeline pipeline, string deploymentName, Uri end
 
         _deploymentName = deploymentName;
         _endpoint = endpoint;
-        _apiVersion = options.Version;
+        _apiVersion = options.GetRawServiceApiValueForClient(this);
     }
 
     protected AzureChatClient()

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Chat/AzureChatExtensions.cs

@@ -106,4 +106,23 @@ public static ResponseContentFilterResult GetResponseContentFilterResult(this St
             chatUpdate?.Choices?.ElementAtOrDefault(0)?.SerializedAdditionalRawData,
             "content_filter_results");
     }
+
+    [Experimental("AOAI001")]
+    public static void SetUserSecurityContext(this ChatCompletionOptions options, UserSecurityContext userSecurityContext)
+    {
+        options.SerializedAdditionalRawData ??= new Dictionary<string, BinaryData>();
+
+        AdditionalPropertyHelpers.SetAdditionalProperty(
+            options.SerializedAdditionalRawData,
+            "user_security_context",
+            userSecurityContext);
+    }
+
+    [Experimental("AOAI001")]
+    public static UserSecurityContext GetUserSecurityContext(this ChatCompletionOptions options)
+    {
+        return AdditionalPropertyHelpers.GetAdditionalPropertyAsUserSecurityContext(
+            options.SerializedAdditionalRawData,
+            "user_security_context");
+    }
 }

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Chat/UserSecurityContext.cs

@@ -0,0 +1,27 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+using System;
+using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
+
+namespace Azure.AI.OpenAI;
+
+/// <summary> User security context contains several parameters that describe the application itself, and the end user that interacts with the application. These fields assist your security operations teams to investigate and mitigate security incidents by providing a comprehensive approach to protecting your AI applications. [Learn more](https://aka.ms/TP4AI/Documentation/EndUserContext) about protecting AI applications using Microsoft Defender for Cloud. </summary>
+[CodeGenModel("AzureUserSecurityContext")]
+[Experimental("AOAI001")]
+public partial class UserSecurityContext
+{
+    /// <summary> The name of the application. Sensitive personal information should not be included in this field. </summary>
+    public string ApplicationName { get; set;}
+    /// <summary> This identifier is the Microsoft Entra ID (formerly Azure Active Directory) user object ID used to authenticate end-users within the generative AI application. Sensitive personal information should not be included in this field. </summary>
+    public string EndUserId { get; set; }
+    /// <summary> The Microsoft 365 tenant ID the end user belongs to. It's required when the generative AI application is multitenant. </summary>
+    public string EndUserTenantId { get; set; }
+    /// <summary> Captures the original client's IP address. </summary>
+    [CodeGenMember("SourceIp")]
+    public string SourceIP { get; set; }
+
+    public UserSecurityContext()
+    { }
+}

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Common/AdditionalPropertyHelpers.cs

@@ -71,6 +71,11 @@ internal static IList<RequestContentFilterResult> GetAdditionalPropertyAsListOfR
         return GetAdditionalPropertyAsList(additionalProperties, additionalPropertyKey, RequestContentFilterResult.DeserializeRequestContentFilterResult);
     }
 
+    internal static UserSecurityContext GetAdditionalPropertyAsUserSecurityContext(IDictionary<string, BinaryData> additionalProperties, string additionalPropertyKey)
+    {
+        return GetAdditionalProperty(additionalProperties, additionalPropertyKey, UserSecurityContext.DeserializeUserSecurityContext);
+    }
+
     internal static void SetAdditionalProperty<T>(IDictionary<string, BinaryData> additionalProperties, string key, T value)
     {
         using MemoryStream stream = new();

.dotnet.azure/sdk/openai/Azure.AI.OpenAI/src/Custom/Common/Internal/GeneratorStubs.cs

@@ -9,7 +9,7 @@ namespace Azure.AI.OpenAI;
 [Experimental("AOAI001")][CodeGenModel("AzureContentFilterBlocklistIdResult")] internal partial class InternalAzureContentFilterBlocklistIdResult { }
 [Experimental("AOAI001")][CodeGenModel("AzureContentFilterBlocklistResultDetail")] internal partial class InternalAzureContentFilterBlocklistResultDetail { }
 [Experimental("AOAI001")][CodeGenModel("AzureContentFilterResultForPromptContentFilterResults")] internal partial class InternalAzureContentFilterResultForPromptContentFilterResults { }
-[Experimental("AOAI001")][CodeGenModel("AzureContentFilterResultForPromptContentFilterResultsError")] internal partial class InternalAzureContentFilterResultForPromptContentFilterResultsError { }
+[Experimental("AOAI001")][CodeGenModel("AzureContentFilterResultForChoiceError")] internal partial class InternalAzureContentFilterResultForChoiceError { }
 #if AZURE_OPENAI_GA
 [Experimental("AOAI001")][CodeGenModel("AzureContentFilterCompletionTextSpan")] internal partial class ContentFilterTextSpan { }
 #endif