Readme for the Testing - detailed instructions for functional testing usage and addition of new tests

[nanda-katikaneni]

🗣 Description

Create a high-level Readme for the testing folder with detailed instructions for:

• Functional Testing - Structure and High Level Details
• Functional Testing Usage with Examples
• Functional Testing - Adding New Test Cases with Examples

💭 Motivation and context

closes #735

🧪 Testing

• Run functional tests for a product using the instructions
• Add a new test using the instructions

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• PR targets the correct parent branch (e.g., main or release-name) for merge.
• Changes are limited to a single goal - eschew scope creep!
• Changes are sized such that they do not touch excessive number of files.
• All future TODOs are captured in issues, which are referenced in code comments.
• These code changes follow the ScubaGear content style guide.
• Related issues these changes resolve are linked preferably via closing keywords.
• All relevant type-of-change labels added.
• All relevant project fields are set.
• All relevant repo and/or project documentation updated to reflect these changes.
• All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

• PR passed smoke test check.

• Feature branch has been rebased against changes from parent branch, as needed

  Use Rebase branch button below or use this reference to rebase from the command line.

• Resolved all merge conflicts on branch

• Notified merge coordinator that PR is ready for merge via comment mention

✅ Post-merge checklist

• Feature branch deleted after merge to clean up repository.
• Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

[rgbrow1949]

The instructions look good, but this branch needs a rebase and there are unit tests that are currently failing

[tkol2022]

Pushed to halibut so that we have more time to make content revisions and complete a review.

[tkol2022]

In the instructions where Pester is installed and you have prerequisites, you are missing a step to install Selenium. That is before running the Update Selenium script.
Install-Module Selenium

[tkol2022]

I noticed in the section where we tell the reader to "kill any ChromeDriver.exe processes", I found that those processes are sometimes hidden under the Powershell app and you cannot locate them in the main process list. Although the ChromeDriver.exe is not shown in my screenshot below, that is where I saw an instance of it earlier. Add this information to the instructions.

[tkol2022]

Today when I tried running the functional tests, I was a getting a random error (forgot to take a screenshot) that was unrelated to any of our code. Selenium just wasn't working right for whatever reason and I cannot recall this ever happening before. I updated the ChromeDriver.exe but that didn't fix the problem so I ended up having to do what's below and now I can run the orchestrator again:

uninstall-module Selenium
install-module Selenium
./Testing/Functional/SmokeTest/UpdateSelenium.ps1

[tkol2022]

The README created for this PR is oriented to users (i.e. developers, testers) that want to execute the test orchestrator. I think you should add a couple of sentences at the top that makes this more obvious and change the title of the document to ScubaGear Functional Testing Automation for Developers and Testers.

On that note, I had developed a couple of ancillary guides for us admins which are related to setting up the infrastructure (service principals, user provisioning, etc.). Can we add a link to those GitHub issues in this document for easy reference?

• How to setup the permissions required to execute the automated functional test orchestrator How to setup the permissions required to execute the automated functional test orchestrator #589

• How to setup a tenant with the necessary AAD conditional access policies to run the Automated Functional Test Orchestrator How to setup a tenant with the necessary AAD conditional access policies to run the Automated Functional Test Orchestrator #591

[tkol2022]

In your Functional Testing Usage section make it clear that the addition of the Thumbprint field to the -Data parameter is what makes the orchestrator run in service principal mode and that the thumbprint must correspond to the end user's client certificate which should be installed on the machine running the test plans.

See #592 for example.

I would just copy the following sections verbatim because they contain a lot of details and some of that stuff is missing from your README.
#592 (comment)
#592 (comment)

[tkol2022]

Throughout the document I think it would be better to use the voice "you need to do this" instead of the third person "the user should do this". This makes the document easier to follow as a set of instructions directed at the reader.

[tkol2022]

Add a section with the Powershell code to generate a user client certificate.

$CertName = "ScubaServicePrincipal"
$CertParams = @{
    Subject = $CertName
    KeySpec = "KeyExchange"
    CertStoreLocation = "Cert:\CurrentUser\My"
}

$MyCert = New-SelfSignedCertificate @CertParams
Write-Output $MyCert


########## Exporting the cert public key (this is so you can upload the file to the Azure AD application as a credential)
Export-Certificate -Cert $MyCert -Type CERT -FilePath .\ScubaServicePrincipal.cer

[tkol2022]

Replace "AAD functional testing in service principal mode by filtering specific test cases" with "When developing or if you are testing pull requests you may need to execute just a single baseline policy in the respective YAML test plan file. To do that you will add Pester filter tags to the PesterConfig parameter. See an example to execute the policy MS.AAD.2.1v1 test cases below."

[tkol2022]

"Product functional tests are being run for multiple product-tenant combinations. Development team should check the successful completion of these runs for any regressions. "

It is not clear when the development team should check this? Is this more the job of the operations team?

Also, for performing regression tests, one option that a developer can exercise is to run the nightly product functional test via the GitHub actions against their branch although since that runs all of the products and all of the test cases, I think we should determine a more efficient way to do that if we ever want this option.

[tkol2022]

I don't see a reference to the sample functional test execution script that is reference in this comment. That might make it easier for developers.

#592 (comment)

[tkol2022]

Nice work. I have previously gone through the documentation and suggested tweaks. The overall team is doing a dry run of this document for release functional testing right now.

[james-garriss]

Good work!