[email protected]

## [1.4.2] - 2023-12-05 - Issue where playmode tests were not recording test results correctly after the first run was fixed. (DSTR-984) - A timeout message is now printed when the test failures occurs before the test finishes. (DSTR-476) - Nested enumerators are now having their first step executed in the same frame as the parent enumerator. (DSTR-888) - Fixed an issue where test projects without PlayMode tests would give the error "No callbacks received." when running in batchmode on some Unity versions.
needle-mirror · Dec 5, 2023 · dfe4a2e · dfe4a2e
1 parent d7833c1
commit dfe4a2e
Show file tree

Hide file tree

Showing 38 changed files with 209 additions and 127 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,11 @@
 # Changelog
-## [1.4.1] - 2023-11-27
+## [1.4.2] - 2023-12-05
+- Issue where playmode tests were not recording test results correctly after the first run was fixed. (DSTR-984)
+- A timeout message is now printed when the test failures occurs before the test finishes. (DSTR-476)
+- Nested enumerators are now having their first step executed in the same frame as the parent enumerator. (DSTR-888)
+- Fixed an issue where test projects without PlayMode tests would give the error "No callbacks received." when running in batchmode on some Unity versions.
+
+## [1.4.1] - 2023-11-13
 - Multiple improvements to the UI, including better dropdowns, filtering, and a new test list view for Player.
 - Fixed uncategorized UI tests filtering for parameterized tests (DSTR-219).
 - In async tests, any failing logs will now first be evaluated after the async method has completed. (DSTR-839)

diff --git a/Documentation~/TableOfContents.md b/Documentation~/TableOfContents.md
@@ -1,10 +1,10 @@
 * [Unity Test Framework overview](./index.md)
+* [What's new](./whats-new.md)
 * [Edit Mode vs. Play Mode tests](./edit-mode-vs-play-mode-tests.md)
 * [Getting started with UTF](./getting-started.md)
   * [How to create a new test assembly](./workflow-create-test-assembly.md)
   * [How to create a test](./workflow-create-test.md)
   * [How to run a test](./workflow-run-test.md)
-  * [How to create a Play Mode test](./workflow-create-playmode-test.md)
   * [How to run a Play Mode test as standalone](./workflow-run-playmode-test-standalone.md)
 * [Extending UTF](./extending.md)
   * [How to split the build and run process for standalone Play Mode tests](./reference-attribute-testplayerbuildmodifier.md#split-build-and-run-for-player-mode-tests)
@@ -13,7 +13,7 @@
   * [How to retrieve the list of tests](./extension-retrieve-test-list.md)
 * [Reference](./manual.md#reference)
   * [Running tests from the command-line](./reference-command-line.md)
-    *[Test settings file](./reference-test-settings-file.md)
+    * [Test settings file](./reference-test-settings-file.md)
   * [UnityTest attribute](./reference-attribute-unitytest.md)
   * [Setup and cleanup at build time](./reference-setup-and-cleanup.md)
     * [IPrebuildSetup](./reference-setup-and-cleanup.md#iprebuildsetup)
@@ -25,7 +25,7 @@
     * [Domain Reloads](./reference-actions-outside-tests.md#domain-reloads)
   * [Custom attributes](./reference-custom-attributes.md)
     * [ConditionalIgnore attribute](./reference-attribute-conditionalignore.md)
-    * * [ParametrizedIgnore attribute](./reference-attribute-parameterizedignore.md)
+    * [ParameterizedIgnore attribute](./reference-attribute-parameterizedignore.md)
     * [PostBuildCleanup attribute](./reference-setup-and-cleanup.md#prebuildsetup-and-postbuildcleanup)
     * [PrebuildSetup attribute](./reference-setup-and-cleanup.md#prebuildsetup-and-postbuildcleanup)
     * [TestMustExpectAllLogs attribute](./reference-attribute-testmustexpectalllogs.md)

diff --git a/Documentation~/images/create-test-assembly-folder.png b/Documentation~/images/create-test-assembly-folder.png
diff --git a/Documentation~/images/editmode-create-test-script.png b/Documentation~/images/editmode-create-test-script.png
diff --git a/Documentation~/images/editmode-run-test.png b/Documentation~/images/editmode-run-test.png
diff --git a/Documentation~/images/editmode-run-tests.png b/Documentation~/images/editmode-run-tests.png
diff --git a/Documentation~/images/editmode-tab.png b/Documentation~/images/editmode-tab.png
diff --git a/Documentation~/images/new-test-script.png b/Documentation~/images/new-test-script.png
diff --git a/Documentation~/images/playmode-run-standalone.png b/Documentation~/images/playmode-run-standalone.png
diff --git a/Documentation~/images/run-playmode-standalone.png b/Documentation~/images/run-playmode-standalone.png
diff --git a/Documentation~/images/run-tests.png b/Documentation~/images/run-tests.png
diff --git a/Documentation~/images/sample-test-tree.png b/Documentation~/images/sample-test-tree.png
diff --git a/Documentation~/images/test-assembly-folder.png b/Documentation~/images/test-assembly-folder.png
diff --git a/Documentation~/images/test-templates.png b/Documentation~/images/test-templates.png
diff --git a/Documentation~/images/tests-folder-assembly.png b/Documentation~/images/tests-folder-assembly.png
diff --git a/Documentation~/images/updated-test-runner-window.png b/Documentation~/images/updated-test-runner-window.png
diff --git a/...reference-attribute-parametrizedignore.md → ...eference-attribute-parameterizedignore.md b/...reference-attribute-parametrizedignore.md → ...eference-attribute-parameterizedignore.md
diff --git a/Documentation~/reference-tests-parameterized.md b/Documentation~/reference-tests-parameterized.md
@@ -16,3 +16,6 @@ public IEnumerator MyTestWithMultipleValues([ValueSource("values")] int value)
 }
 ```
 
+## Ignore based on parameters
+
+You can selectively ignore tests based on the parameters supplied to the test method by using the [ParameterizedIgnoreAttribute](xref:UnityEngine.TestTools.ParameterizedIgnoreAttribute).
diff --git a/Documentation~/whats-new.md b/Documentation~/whats-new.md
@@ -0,0 +1,28 @@
+# What's new in version 1.4
+
+This page summarizes new features, improvements, and issues resolved in version 1.4 of Unity Test Framework. 
+
+## Added
+### API updates
+
+In this version, we introduced several API changes:
+* Added API for saving results
+* Added API for canceling test runs
+* Added overloads of LogAssert.Expect which allow users to expect a log message without specifying a severity, which will match logged messages of any severity.
+
+### Ignore tests based on arguments
+
+This version introduces the ParameterizedIgnoreAttribute which allows ignoring tests based on arguments which were passed to the test method of a [parameterized test](./reference-tests-parameterized.md).
+
+## Updated
+### Revised Test Runner UI
+This version includes a revised Test Runner window and several usability improvements:
+* Added a third tab to the Test Runner window, for running in a player explicitly. This makes it easier to run a subset of tests, as well as retaining the test results from the latest player run.
+* Moved the run and action buttons to the bottom of the window, to separate them from the filters.
+* The stack traces are clickable, and will open the relevant file in the external editor.
+* When searching, the parents of the matching tests are expanded, to make it easier to see the context of the matching tests.
+
+![UpdatedTestRunnerWindow.png](./images/updated-test-runner-window.png)]
+
+## Fixed
+This version includes many bug fixes and performance improvements. For a full list of changes and updates in this version, see the Unity Test Framework package [changelog](https://docs.unity3d.com/Packages/[email protected]/changelog/CHANGELOG.html).
diff --git a/Documentation~/workflow-create-playmode-test.md b/Documentation~/workflow-create-playmode-test.md
diff --git a/Documentation~/workflow-create-test-assembly.md b/Documentation~/workflow-create-test-assembly.md
@@ -1,35 +1,32 @@
-# Workflow: **How to create a new test assembly** 
+# Workflow: Creating test assemblies 
 
-Unity Test Framework looks for a test inside any assembly that references [NUnit](http://www.nunit.org/). We refer to such assemblies as **Test Assemblies**. **Play Mode** and **Edit Mode** tests need to be in separate assemblies.
+UTF looks for tests in a **Test Assembly**, which is any assembly that references Nunit.
 
-The [Test Runner](./getting-started.md) UI helps you set up **TestAssemblies**:
+The Test Runner UI helps you set up test assemblies:
 
-1. Select the **Assets** folder in your Project window.
-2. Open the **Test Runner** window (menu: **Window** > **General** > **Test Runner**).
-3. In the **Test Runner** window, the **EditMode** tab should be selected by default. Click **Create EditMode Test Assembly Folder**. 
+1. Select the Assets folder in your Project window.
+2. Open the Test Runner window (menu: **Window** > **General** > **Test Runner**).
+3. In the **Test Runner** window, select **Create a new Test Assembly Folder in the active path**.
 
-![Test Runner window EditMode tab](./images/editmode-tab.png)
+![Test Runner window EditMode tab](./images/create-test-assembly-folder.png)
 
-This creates a *Tests* folder in your project Assets with a corresponding `.asmdef` file with the required references. You can change the name of the new [Assembly Definition](https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html) and press Enter to accept it.
+Alternatively, you can go via the **Assets** menu:
 
-![New Test folder and assembly file](./images/tests-folder-assembly.png)
+1. Select the Assets folder in your Project window.
+2. Create a new Test Assembly Folder (menu: **Assets** > **Create** > **Testing** > **Test Assembly Folder**).
 
-Click on the assembly definition file to inspect it in the Inspector window. You'll see that it has references to **nunit.framework.dll** under Assembly References, and to **UnityEngine.TestRunner** and **UnityEditor.TestRunner** under Assembly Definition References. This tells Unity Test Framework that this is a test assembly.
+This creates a *Tests* folder in your project Assets with a corresponding `.asmdef` file with the required references. You can change the name of the new [Assembly Definition](https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html) and press Enter to accept it.
 
-The checkbox selections under **Platforms** determine which platforms the test assembly can run on. Assemblies created through the **Test Runner** target the **Editor** only by default. **Any Platform** or a specific platform other than **Editor** makes the tests you add run in Play Mode by default.
+![New Test folder and assembly file](./images/test-assembly-folder.png)
 
-> **Note**: The **UnityEditor.TestRunner** reference is only available for [Edit Mode tests](./edit-mode-vs-play-mode-tests.md#edit-mode-tests).
+Click on the assembly definition file to inspect it in the **Inspector** window. You'll see that it has references to **nunit.framework.dll***,* **UnityEngine.TestRunner,** and **UnityEditor.TestRunner** assemblies. This tells UTF that this is a test assembly.
 
-![Assembly definition import settings](./images/import-settings.png)
+The checkbox selections under **Platforms** determine which platforms the test assembly can run on. Assemblies created through the **Test Runner** target the **Editor** only by default. **Any Platform** or a specific platform other than **Editor** makes it possible to run any Play Mode tests in the assembly on standalone Players for the additional platforms.
 
-## Reference other assemblies in your Test Assemblies
-
-To reference other assemblies from your Test Assemblies:
+![Assembly definition import settings](./images/import-settings.png)
 
-1. Create your Test Assembly as described above.
-2. Create an Assembly Definition for the scripts you want to reference, refer to [Create an Assembly Definition asset](https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html#create-asmdef).
-3. Add a reference to the Assembly Definition created in Step 2 to your Test Assembly, refer to [Referencing another assembly](https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html#reference-another-assembly).
+> **Note**: The **UnityEditor.TestRunner** reference is only available for [Edit Mode tests](./edit-mode-vs-play-mode-tests.md#edit-mode-tests).
 
-## Additional resources
+You can repeat the steps above as many times as you like to create additional Test Assemblies. The first Test Assembly folder you create is named *Tests* by default and subsequent ones are named *Tests 1*, *Tests 2*, and so on. Remember that you can always rename the assemblies but each assembly name must be unique.
 
-Refer to the Unity Manual page on [Assembly definitions](https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html) for more detail on using Assembly Definition files to manage references.
+> **Note**: Changing the file name of the assembly definition file does not affect the value of the **Name** property in the file. Use the **Inspector** window or edit the .asmdef direclty in a text editor to make sure the name property is properly changed.
diff --git a/Documentation~/workflow-create-test.md b/Documentation~/workflow-create-test.md
@@ -1,53 +1,31 @@
-# Workflow: How to create a test
+# Workflow: Creating tests
 
-To create a test, do the following:
+To create a test through the **Test Runner** window:
 
-1. Create your *Test* [assembly folder](./workflow-create-test-assembly.md) and select it in the **Project** window.
-2. Click the button **Create Test Script in current folder** option in the **Test Runner** window.
-![EditMode create test script](./images/editmode-create-test-script.png)
-3. It creates a *NewTestScript.cs* file in the *Tests* folder. Change the name of the script, if necessary, and press Enter to accept it. 
+1. [Create your Test Assembly folder](./workflow-create-test-assembly.md) and select it in the **Project** window.
+2. Open the Test Runner window (menu: **Window** > **General** > **Test Runner**).
+3. Click the **Create a new Test Script in the active path** button in the **Test Runner** window.
 
-![NewTestScript.cs](./images/new-test-script.png)
-
-Now you’ll see two sample tests in the Test Runner window:
-
-![Test templates](./images/test-templates.png)
-
-Now you can open the tests in your favorite script editor.
-
-You can also create test scripts by navigating to **Assets** > **Create > Testing** > **C# Test Script**, unless adding a test script would result in a compilation error.
-
-> **Note**: Unity does not include `TestAssemblies` ([NUnit](http://www.nunit.org/), Unity Test Framework, and user script assemblies) when using the normal build pipeline, but does include them when using **Run on &lt;Platform&gt;** in the Test Runner window.
+Alternatively, use the **Asset** menu:
 
-## Create additional tests
+1. In the **Project** window, select the new folder.
+2. Create a new test script in the folder (menu: **Assets > Create > Testing > C# Test Script**).
 
-To create another set of tests:
+This creates a `NewTestScript.cs` file in the *Tests* folder with some sample tests to get you started. Change the name of the script, if necessary, and press Enter to accept it.
 
-1. In the **Project** window, select **Assets**.
-2. Create a new test assembly folder (menu: **Assets > Create> Testing > Tests Assembly Folder**).
-3. In the **Project** window, select the new folder.
-4. Create a new test script in the folder (menu: **Assets > Create > Testing > C# Test Script**).
-
-The assembly definition is assigned the same name as your new asset. To rename it, change the **Name** in the **Inspector** window. Assembly definition names must be unique.
-
-> **Note**: Changing the file name of the assembly definition file does not affect the value of the **Name** property in the file. Use the **Inspector** window to make sure the name is properly changed.
-
-By default **Any Platform** is preselected as the target platform for the new assembly, which means the test script appears as a PlayMode test in the TestRunner window. To change it to an EditMode test, in the **Inspector** window select **Editor** only under **Platforms**.
-
-New assemblies created through the **Assets** menu should automatically include references to `UnityEngine.TestRunner` and `UnityEditor.TestRunner`. If these references are missing, add them in the **Inspector** window under **Assembly Definition References**:
+![NewTestScript.cs](./images/new-test-script.png)
 
-![Assembly Definition References](./images/assembly-definition-references.png)
+Now you’ll see the sample tests in the Test Runner window:
 
-## Filters
+![Sample tests](./images/sample-test-tree.png)
 
-If you have a lot of tests, and you only want to view/run a sub-set of them, you can filter them in three ways (see image above): 
+Now you can open the tests in your favorite script editor.
 
-* Type in the search box in the top left
+> **Note**: Unity does not include test assemblies (NUnit, Unity Test Framework, and user script assemblies) when using the normal build pipeline, but does include them when using **Run on &lt;Platform&gt;** in the **Test Runner** window.
 
-* Click a test class or fixture (such as **NewTestScript** in the image above)
+## Creating Play Mode tests
 
-* Click one of the test result icon buttons in the top right
+The process for creating a Play Mode test is the same as for creating an Edit Mode test. The only relevant differences are:
 
-  
+* Play Mode tests that need to [run in a standalone platform Player](./workflow-run-playmode-test-standalone.md) should be in an assembly that references the required platform.
 
-For more information, see [Edit Mode vs. Play Mode tests](./edit-mode-vs-play-mode-tests.md).
diff --git a/Documentation~/workflow-run-playmode-test-standalone.md b/Documentation~/workflow-run-playmode-test-standalone.md
@@ -1,10 +1,10 @@
-# Workflow: How to run a Play Mode test in player
+# Workflow: Running Play Mode tests in a player
 
-If you run a **Play Mode** test in the same way as an [Editor test](./workflow-run-test.md), it runs inside the Unity Editor. You can also run Play Mode tests on specific platforms. Click **Run all in the player** to build and run your tests on the currently active target platform.
+If you run a **Play Mode** test in the same way as an [Editor test](./workflow-run-test.md), it runs inside the Unity Editor. You can also run Play Mode tests on specific platforms. Select **Player** tab to build and run your tests on the currently active target platform. 
 
-![Run PlayMode test in player](./images/playmode-run-standalone.png)
+![Run PlayMode test in player](./images/run-playmode-standalone.png)
 
-> **Note**: Your current platform displays in brackets on the button. For example, in the image above, the button reads **Run all in player (StandaloneWindows)**, because the current platform is Windows. The target platform is always the current Platform selected in [Build Settings](https://docs.unity3d.com/Manual/BuildSettings.html) (menu: **File** > **Build Settings**). 
+> **Note**: Your current platform shows at the top of the Test Runner window. For example, in the image above, the bar above the search bar reads **Running in StandaloneOSX**, because the current platform is MacOS. The target platform is always the current Platform selected in [Build Settings](https://docs.unity3d.com/Manual/BuildSettings.html) (menu: **File** > **Build Settings**). 
 
 The test result displays in the build once the test completes:
-Original file line number
+Diff line change
@@ Expand Up @@
     }
     ```
+    ## Ignore based on parameters
+    You can selectively ignore tests based on the parameters supplied to the test method by using the [ParameterizedIgnoreAttribute](xref:UnityEngine.TestTools.ParameterizedIgnoreAttribute).