.Net: Processes State Management Part 2 (microsoft#9481)

### Description - Moving `ProcessStateMetadata` logic to factory/extension to try to keep clean existing core components - Adding check of step name when adding new steps/processes to process - names must be unique so when applying saved state it propagates properly. Currently there are issues if multiple steps have the same name - State versioning initial support: - Processes with 1:1 step match previous and new version (step name change and/or step logic change) - minor changes on process steps and not in root process itself Fixing microsoft#9358 #### Out of scope but will be addressed in next PRs - Processes with N:M steps (process flow changed and potentially also step logic changed) - New samples showcasing how to make use of N:M mapping ### Contribution Checklist  - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Chris <[email protected]>
esttenorio · Nov 5, 2024 · a7a56e5 · a7a56e5
1 parent 76e904c
commit a7a56e5
Show file tree

Hide file tree

Showing 35 changed files with 683 additions and 186 deletions.
diff --git a/dotnet/samples/GettingStartedWithProcesses/README.md b/dotnet/samples/GettingStartedWithProcesses/README.md
@@ -271,6 +271,56 @@ flowchart RL
     end
 ```
 
+## Concepts
+
+### Components
+
+- **Process:** A sequence of steps designed to achieve a specific goal. These steps are interconnected in such a way that they can communicate by sending and receiving events. The connections between the steps are established during the process creation.
+- **Steps:** Individual activities within a process, each with defined inputs and outputs, contributing to the overall objective. Existing processes can be utilized as steps within another process. There are two main types of steps:
+    - _Stateless Steps_: These steps do not retain any information between executions. They operate independently without the need to store state data.
+    - _Stateful Steps_: These steps maintain a state that can be persisted, allowing the state to be reused and updated in subsequent runs of the process. The state of these steps can be stored and serialized.
+
+In general, both processes and steps are designed to be reusable across different processes.
+
+### Versioning
+
+Once stateful steps/processes have been deployed, versioning becomes a crucial aspect to understand. 
+It enables you to tweak and improve processes while maintaining the ability to read step states generated by previous versions of the steps.
+
+Stateful processes involve steps that maintain state information. 
+When these processes are updated, it's important to manage versioning effectively to ensure continuity and compatibility with previously saved states.
+
+There are two primary scenarios to consider when addressing process state versioning:
+
+1. **Minor SK Process Improvements/Changes:**
+
+    In this scenario, the root process remains conceptually the same, but with some modifications:
+
+    - _Step Renaming:_ Some step names may have been changed.
+    - _Step Version Updates:_ New versions of one or more steps used by the root process or any steps in a subprocess may be introduced.
+
+    **Considerations:**
+
+    - Ensure backward compatibility by mapping old step names to new step names.
+    - Validate that the new step versions can read and interpret the state data generated by previous versions.
+
+    **Related Samples:**
+
+    - `Step03a_FoodPreparation.cs/RunStatefulFriedFishV2ProcessWithLowStockV1StateFromFileAsync`
+    - `Step03a_FoodPreparation.cs/RunStatefulFishSandwichV2ProcessWithLowStockV1StateFromFileAsync`
+
+2. **Major SK Process Improvements/Changes:**
+
+    This scenario involves significant modifications to the root process, which may include:
+
+    - _Step Refactoring_: Multiple steps may be refactored and replaced. However, some properties of the replaced steps can be used to set properties of the new steps.
+    - _Custom Mappings:_ Custom equivalent mappings may be required to translate the previous stored state to the current process state.
+
+    **Considerations:**
+
+    - Develop a detailed mapping strategy to align old and new process states.
+    - Implement and test custom mappings to ensure data integrity and process continuity.
+
 ## Running Examples with Filters
 Examples may be explored and ran within _Visual Studio_ using _Test Explorer_.
 

diff --git a/dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FishAndChipsProcess.cs b/dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FishAndChipsProcess.cs
@@ -17,6 +17,7 @@ public static class ProcessEvents
     {
         public const string PrepareFishAndChips = nameof(PrepareFishAndChips);
         public const string FishAndChipsReady = nameof(FishAndChipsReady);
+        public const string FishAndChipsIngredientOutOfStock = nameof(FishAndChipsIngredientOutOfStock);
     }
 
     public static ProcessBuilder CreateProcess(string processName = "FishAndChipsProcess")
@@ -48,6 +49,35 @@ public static ProcessBuilder CreateProcess(string processName = "FishAndChipsPro
         return processBuilder;
     }
 
+    public static ProcessBuilder CreateProcessWithStatefulSteps(string processName = "FishAndChipsWithStatefulStepsProcess")
+    {
+        var processBuilder = new ProcessBuilder(processName);
+        var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulStepsV1());
+        var makePotatoFriesStep = processBuilder.AddStepFromProcess(PotatoFriesProcess.CreateProcessWithStatefulSteps());
+        var addCondimentsStep = processBuilder.AddStepFromType<AddFishAndChipsCondimentsStep>();
+        // An additional step that is the only one that emits an public event in a process can be added to maintain event names unique
+        var externalStep = processBuilder.AddStepFromType<ExternalFishAndChipsStep>();
+
+        processBuilder
+            .OnInputEvent(ProcessEvents.PrepareFishAndChips)
+            .SendEventTo(makeFriedFishStep.WhereInputEventIs(FriedFishProcess.ProcessEvents.PrepareFriedFish))
+            .SendEventTo(makePotatoFriesStep.WhereInputEventIs(PotatoFriesProcess.ProcessEvents.PreparePotatoFries));
+
+        makeFriedFishStep
+            .OnEvent(FriedFishProcess.ProcessEvents.FriedFishReady)
+            .SendEventTo(new ProcessFunctionTargetBuilder(addCondimentsStep, parameterName: "fishActions"));
+
+        makePotatoFriesStep
+            .OnEvent(PotatoFriesProcess.ProcessEvents.PotatoFriesReady)
+            .SendEventTo(new ProcessFunctionTargetBuilder(addCondimentsStep, parameterName: "potatoActions"));
+
+        addCondimentsStep
+            .OnEvent(AddFishAndChipsCondimentsStep.OutputEvents.CondimentsAdded)
+            .SendEventTo(new ProcessFunctionTargetBuilder(externalStep));
+
+        return processBuilder;
+    }
+
     private sealed class AddFishAndChipsCondimentsStep : KernelProcessStep
     {
         public static class Functions

diff --git a/dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FishSandwichProcess.cs b/dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FishSandwichProcess.cs
@@ -46,10 +46,38 @@ public static ProcessBuilder CreateProcess(string processName = "FishSandwichPro
         return processBuilder;
     }
 
-    public static ProcessBuilder CreateProcessWithStatefulSteps(string processName = "FishSandwichWithStatefulStepsProcess")
+    public static ProcessBuilder CreateProcessWithStatefulStepsV1(string processName = "FishSandwichWithStatefulStepsProcess")
     {
-        var processBuilder = new ProcessBuilder(processName);
-        var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulSteps());
+        var processBuilder = new ProcessBuilder(processName) { Version = "FishSandwich.V1" };
+        var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulStepsV1());
+        var addBunsStep = processBuilder.AddStepFromType<AddBunsStep>();
+        var addSpecialSauceStep = processBuilder.AddStepFromType<AddSpecialSauceStep>();
+        // An additional step that is the only one that emits an public event in a process can be added to maintain event names unique
+        var externalStep = processBuilder.AddStepFromType<ExternalFriedFishStep>();
+
+        processBuilder
+            .OnInputEvent(ProcessEvents.PrepareFishSandwich)
+            .SendEventTo(makeFriedFishStep.WhereInputEventIs(FriedFishProcess.ProcessEvents.PrepareFriedFish));
+
+        makeFriedFishStep
+            .OnEvent(FriedFishProcess.ProcessEvents.FriedFishReady)
+            .SendEventTo(new ProcessFunctionTargetBuilder(addBunsStep));
+
+        addBunsStep
+            .OnEvent(AddBunsStep.OutputEvents.BunsAdded)
+            .SendEventTo(new ProcessFunctionTargetBuilder(addSpecialSauceStep));
+
+        addSpecialSauceStep
+            .OnEvent(AddSpecialSauceStep.OutputEvents.SpecialSauceAdded)
+            .SendEventTo(new ProcessFunctionTargetBuilder(externalStep));
+
+        return processBuilder;
+    }
+
+    public static ProcessBuilder CreateProcessWithStatefulStepsV2(string processName = "FishSandwichWithStatefulStepsProcess")
+    {
+        var processBuilder = new ProcessBuilder(processName) { Version = "FishSandwich.V2" };
+        var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulStepsV2("FriedFishStep"), aliases: ["FriedFishWithStatefulStepsProcess"]);
         var addBunsStep = processBuilder.AddStepFromType<AddBunsStep>();
         var addSpecialSauceStep = processBuilder.AddStepFromType<AddSpecialSauceStep>();
         // An additional step that is the only one that emits an public event in a process can be added to maintain event names unique

diff --git a/dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FriedFishProcess.cs b/dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FriedFishProcess.cs
@@ -1,6 +1,7 @@
 // Copyright (c) Microsoft. All rights reserved.
 
 using Microsoft.SemanticKernel;
+using Microsoft.SemanticKernel.Process;
 using Step03.Models;
 using Step03.Steps;
 namespace Step03.Processes;
@@ -16,7 +17,7 @@ public static class ProcessEvents
         // When multiple processes use the same final step, the should event marked as public
         // so that the step event can be used as the output event of the process too.
         // In these samples both fried fish and potato fries end with FryStep success
-        public const string FriedFishReady = nameof(FryFoodStep.OutputEvents.FriedFoodReady);
+        public const string FriedFishReady = FryFoodStep.OutputEvents.FriedFoodReady;
     }
 
     /// <summary>
@@ -30,7 +31,7 @@ public static ProcessBuilder CreateProcess(string processName = "FriedFishProces
         var processBuilder = new ProcessBuilder(processName);
 
         var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsStep>();
-        var chopStep = processBuilder.AddStepFromType<CutFoodStep>("chopStep");
+        var chopStep = processBuilder.AddStepFromType<CutFoodStep>();
         var fryStep = processBuilder.AddStepFromType<FryFoodStep>();
 
         processBuilder
@@ -52,19 +53,48 @@ public static ProcessBuilder CreateProcess(string processName = "FriedFishProces
         return processBuilder;
     }
 
+    public static ProcessBuilder CreateProcessWithStatefulStepsV1(string processName = "FriedFishWithStatefulStepsProcess")
+    {
+        // It is recommended to specify process version in case this process is used as a step by another process
+        var processBuilder = new ProcessBuilder(processName) { Version = "FriedFishProcess.v1" }; ;
+
+        var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsWithStockStep>();
+        var chopStep = processBuilder.AddStepFromType<CutFoodStep>();
+        var fryStep = processBuilder.AddStepFromType<FryFoodStep>();
+
+        processBuilder
+            .OnInputEvent(ProcessEvents.PrepareFriedFish)
+            .SendEventTo(new ProcessFunctionTargetBuilder(gatherIngredientsStep));
+
+        gatherIngredientsStep
+            .OnEvent(GatherFriedFishIngredientsWithStockStep.OutputEvents.IngredientsGathered)
+            .SendEventTo(new ProcessFunctionTargetBuilder(chopStep, functionName: CutFoodWithSharpeningStep.Functions.ChopFood));
+
+        chopStep
+            .OnEvent(CutFoodWithSharpeningStep.OutputEvents.ChoppingReady)
+            .SendEventTo(new ProcessFunctionTargetBuilder(fryStep));
+
+        fryStep
+            .OnEvent(FryFoodStep.OutputEvents.FoodRuined)
+            .SendEventTo(new ProcessFunctionTargetBuilder(gatherIngredientsStep));
+
+        return processBuilder;
+    }
+
     /// <summary>
     /// For a visual reference of the FriedFishProcess with stateful steps check this
     /// <see href="https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/GettingStartedWithProcesses/README.md#fried-fish-preparation-with-knife-sharpening-and-ingredient-stock-process" >diagram</see>
     /// </summary>
     /// <param name="processName">name of the process</param>
     /// <returns><see cref="ProcessBuilder"/></returns>
-    public static ProcessBuilder CreateProcessWithStatefulSteps(string processName = "FriedFishWithStatefulStepsProcess")
+    public static ProcessBuilder CreateProcessWithStatefulStepsV2(string processName = "FriedFishWithStatefulStepsProcess")
     {
-        var processBuilder = new ProcessBuilder(processName);
+        // It is recommended to specify process version in case this process is used as a step by another process
+        var processBuilder = new ProcessBuilder(processName) { Version = "FriedFishProcess.v2" };
 
-        var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsWithStockStep>();
-        var chopStep = processBuilder.AddStepFromType<CutFoodWithSharpeningStep>("chopStep");
-        var fryStep = processBuilder.AddStepFromType<FryFoodStep>();
+        var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsWithStockStep>(name: "gatherFishIngredientStep", aliases: ["GatherFriedFishIngredientsWithStockStep"]);
+        var chopStep = processBuilder.AddStepFromType<CutFoodWithSharpeningStep>(name: "chopFishStep", aliases: ["CutFoodStep"]);
+        var fryStep = processBuilder.AddStepFromType<FryFoodStep>(name: "fryFishStep", aliases: ["FryFoodStep"]);
 
         processBuilder
             .OnInputEvent(ProcessEvents.PrepareFriedFish)
@@ -97,11 +127,13 @@ public static ProcessBuilder CreateProcessWithStatefulSteps(string processName =
         return processBuilder;
     }
 
+    [KernelProcessStepMetadata("GatherFishIngredient.V1")]
     private sealed class GatherFriedFishIngredientsStep : GatherIngredientsStep
     {
         public GatherFriedFishIngredientsStep() : base(FoodIngredients.Fish) { }
     }
 
+    [KernelProcessStepMetadata("GatherFishIngredient.V2")]
     private sealed class GatherFriedFishIngredientsWithStockStep : GatherIngredientsWithStockStep
     {
         public GatherFriedFishIngredientsWithStockStep() : base(FoodIngredients.Fish) { }

diff --git a/...s/GettingStartedWithProcesses/Step03/ProcessesStates/FishSandwichStateProcessSuccess.json b/...s/GettingStartedWithProcesses/Step03/ProcessesStates/FishSandwichStateProcessSuccess.json
@@ -4,39 +4,44 @@
       "stepsState": {
         "GatherFriedFishIngredientsWithStockStep": {
           "state": {
-            "IngredientsStock": 3
+            "IngredientsStock": 4
           },
-          "id": "cfc7d51f3bfa455c980d282dcdd85b13",
-          "name": "GatherFriedFishIngredientsWithStockStep"
+          "id": "32301d167a604f7bb1b69663d8feafe6",
+          "name": "GatherFriedFishIngredientsWithStockStep",
+          "versionInfo": "GatherFishIngredient.V2"
         },
-        "chopStep": {
-          "state": {
-            "KnifeSharpness": 2
-          },
-          "id": "daf4a782df094deeb111ae701431ddbb",
-          "name": "chopStep"
+        "CutFoodStep": {
+          "id": "3be68c614bc44bedb7bce0f002110968",
+          "name": "CutFoodStep",
+          "versionInfo": "CutFoodStep.V1"
         },
         "FryFoodStep": {
-          "id": "e508ffe1d4714097b9716db1fd48b587",
-          "name": "FryFoodStep"
+          "id": "5f80945fafec4a0bbeccf3ff23a8c1a1",
+          "name": "FryFoodStep",
+          "versionInfo": "FryFoodStep.V1"
         }
       },
-      "id": "a201334a1f534e9db9e1d46dce8345a8",
-      "name": "FriedFishWithStatefulStepsProcess"
+      "id": "c9d2e56dcc5a4b5ea73e05ae53635c90",
+      "name": "FriedFishWithStatefulStepsProcess",
+      "versionInfo": "FriedFishProcess.v1"
     },
     "AddBunsStep": {
-      "id": "8a9b2d66e0594ee898d1c94c8bc07d0e",
-      "name": "AddBunsStep"
+      "id": "ef113fc874b0473c9c275827effe8dd8",
+      "name": "AddBunsStep",
+      "versionInfo": "v1"
     },
     "AddSpecialSauceStep": {
-      "id": "6b0e92097cb74f5cbac2a71473a4e9c2",
-      "name": "AddSpecialSauceStep"
+      "id": "65a35f6dec6e45cab9e1b4df7d43a6bd",
+      "name": "AddSpecialSauceStep",
+      "versionInfo": "v1"
     },
     "ExternalFriedFishStep": {
-      "id": "59ebd4724684469ab19d86d281c205e3",
-      "name": "ExternalFriedFishStep"
+      "id": "ab186a81480a45249cfff9e0c56319b8",
+      "name": "ExternalFriedFishStep",
+      "versionInfo": "v1"
     }
   },
-  "id": "38e8f477-c022-41fc-89f1-3dd3509d0e83",
-  "name": "FishSandwichWithStatefulStepsProcess"
+  "id": "0de6d539-c1bf-44ad-816d-650231cd2034",
+  "name": "FishSandwichWithStatefulStepsProcess",
+  "versionInfo": "FishSandwich.V1"
 }
diff --git a/...gStartedWithProcesses/Step03/ProcessesStates/FishSandwichStateProcessSuccessLowStock.json b/...gStartedWithProcesses/Step03/ProcessesStates/FishSandwichStateProcessSuccessLowStock.json
@@ -6,37 +6,42 @@
           "state": {
             "IngredientsStock": 1
           },
-          "id": "cfc7d51f3bfa455c980d282dcdd85b13",
-          "name": "GatherFriedFishIngredientsWithStockStep"
+          "id": "32301d167a604f7bb1b69663d8feafe6",
+          "name": "GatherFriedFishIngredientsWithStockStep",
+          "versionInfo": "GatherFishIngredient.V2"
         },
-        "chopStep": {
-          "state": {
-            "KnifeSharpness": 2
-          },
-          "id": "daf4a782df094deeb111ae701431ddbb",
-          "name": "chopStep"
+        "CutFoodStep": {
+          "id": "3be68c614bc44bedb7bce0f002110968",
+          "name": "CutFoodStep",
+          "versionInfo": "CutFoodStep.V1"
         },
         "FryFoodStep": {
-          "id": "e508ffe1d4714097b9716db1fd48b587",
-          "name": "FryFoodStep"
+          "id": "5f80945fafec4a0bbeccf3ff23a8c1a1",
+          "name": "FryFoodStep",
+          "versionInfo": "FryFoodStep.V1"
         }
       },
-      "id": "a201334a1f534e9db9e1d46dce8345a8",
-      "name": "FriedFishWithStatefulStepsProcess"
+      "id": "c9d2e56dcc5a4b5ea73e05ae53635c90",
+      "name": "FriedFishWithStatefulStepsProcess",
+      "versionInfo": "FriedFishProcess.v1"
     },
     "AddBunsStep": {
-      "id": "8a9b2d66e0594ee898d1c94c8bc07d0e",
-      "name": "AddBunsStep"
+      "id": "ef113fc874b0473c9c275827effe8dd8",
+      "name": "AddBunsStep",
+      "versionInfo": "v1"
     },
     "AddSpecialSauceStep": {
-      "id": "6b0e92097cb74f5cbac2a71473a4e9c2",
-      "name": "AddSpecialSauceStep"
+      "id": "65a35f6dec6e45cab9e1b4df7d43a6bd",
+      "name": "AddSpecialSauceStep",
+      "versionInfo": "v1"
     },
     "ExternalFriedFishStep": {
-      "id": "59ebd4724684469ab19d86d281c205e3",
-      "name": "ExternalFriedFishStep"
+      "id": "ab186a81480a45249cfff9e0c56319b8",
+      "name": "ExternalFriedFishStep",
+      "versionInfo": "v1"
     }
   },
-  "id": "38e8f477-c022-41fc-89f1-3dd3509d0e83",
-  "name": "FishSandwichWithStatefulStepsProcess"
+  "id": "0de6d539-c1bf-44ad-816d-650231cd2034",
+  "name": "FishSandwichWithStatefulStepsProcess",
+  "versionInfo": "FishSandwich.V1"
 }