Improving resiliency of test Infrastructure

[deviantintegral]

To frame this discussion, here's a goal I think there's likely broad agreement on:

DDEV tests in PRs should not be impacted if Randy's house has a power or internet outage, and other trusted ddev members should be able to manage or reboot all test runners connected to pull requests.

@rfay and I (along with a colleague of mine) had a good discussion at Drupalcon that touched on how ddev's testing is currently set up. This has also come up on some advisory group calls, from the lens of project sustainability.

One of the biggest challenges right now is automated tests for macOS and Windows hosts. Looking at an example PR at #4946 there are:

1. Many GitHub actions running in Actions runners. These are Linux and macOS, amd64 only.
2. Buildkite jobs. I believe the macOS Apple Silicon jobs are all at MacStadium. The wsl2 jobs are mini PC's in @rfay 's home? Are there any macOS runners there too?
3. A linux / arm CircleCI job.

What I haven't found is a job for non-WSL2 Windows, which I thought was in place? I know that's on the way to being deprecated but I thought it wasn't there yet.

There's great docs on setting up Buildkite runners at https://ddev.readthedocs.io/en/stable/developers/buildkite-testmachine-setup/. There's also https://ddev.readthedocs.io/en/stable/developers/github-selfhosted-setup/, but it's missing macOS setup docs?

My understanding is that with Windows tests, the biggest challenge is the combination of testing with a client version of Windows, and not Windows Server, as well as some providers not offering nested virtualization. For example, AWS only offers nested virtualization on bare-metal instances which are expensive.

Ideas for Next Steps

• Contact MacStadium, and see if they have a partner or sister company offering Windows hosting. If MacStadium has been working well, it seems like that would fit nicely for us.
• Put out a general call to the ddev community asking for Windows expertise in migrating Windows tests to some hosting provider, cloud or otherwise. For example, Azure has blog posts indicating they support nested virtualization. A Windows expert could probably tell us quickly if that's an option or not.
• Have a policy that if @rfay is unavailable to manage an issue with Windows runners, that him or another repository admin will disable any required checks on PRs for Windows tests.
• Re-evaluate to determine what Windows host OSs we need to run tests on. Do we need Windows 10 and 11? Are we able to use Windows Server, or do we really need client versions for accurate tests? Has this changed in Windows Server 2022?

[deviantintegral]

Re nested virtualization: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

Nested Virtualization is supported both Azure and on-premises. However, if using a non-Microsoft hypervisor such as KVM, Microsoft can not provide end-to-end support. Please ensure your vendor supports this scenario.

[rfay]

Thank you for opening this!

Some random answers:

• Traditional Windows (with mutagen) is now tested on pull into master only, but it's there. (Currently failing, because it wasn't tested on a recent PR :( ) https://buildkite.com/ddev/ddev-windows-mutagen/builds/4064
• A while back I was able to use traditional Windows on Azure, but it wasn't reliable.
• In my house are 3 amd64 mac minis and 1 arm64 mac mini
• Windows/WSL2 hasn't worked out for me because none of the services I've tried allows nested virtualization, or if they do, the performance is terrible. There might be a solution for this.
• We run Apple Silicon tests on one runner at MacStadium and one runner in my house (mac Mini)
• GitHub's mac amd64 is used for Colima tests, but it's so slow it's almost not worth doing.
• CircleCI has mac runners, and they may have Apple Silicon now. I think they actually use MacStadium, not sure.
• We test on Linux arm64, but on CircleCI. And I just this week changed that to only run on master branch commits as well, because it costs us a bit.
• Github self-hosted for macOS is not documented because we've never done it. Basically Buildkite is quite a lot better for self-hosted.
• Both @gilbertsoft and @mattstein have full remote access to the test runners in my house, so can reboot and do all the things... unless it's down or dead. I think only @gilbertsoft has actually done this, and it may have been a while. I see them most days. I use the remote access always.

My understanding is that with Windows tests, the biggest challenge is the combination of testing with a client version of Windows, and not Windows Server

Yes, agreed. There may be a way to reliably use Windows Server, but I haven't figured it out when I've tried. Also, I haven't tried it since we started using WSL-Docker-Inside, but still, requires performant nested virtualization.

[rfay]

I had forgotten about paperspace.com until their announcement of acquisition today. Last I tried them they couldn't do virtualization, but trying again. Update: Unfortunately unable to do WSL2, no virtualization.