Playwright results tab in Azure DevOps pipeline run

[chris-strudel]

Is your feature request related to a problem? Please describe.
I'm always frustrated when my Playwright tests finish running in Azure DevOps, and I'm left with virtually no results to review. What's the point of running tests without having access to their outcomes?

Describe the solution you'd like
Now that the Playwright Testing Service is operational, I'm looking for seamless integration with Azure DevOps (ADO). Whenever a pipeline runs and includes Playwright tests, a 'Playwright Tests' tab should be accessible in the run summary alongside other tabs. Each test case should have its associated video and trace available for immediate review within ADO. Additionally, a summary indicating pass versus fail statuses may also be beneficial.

Describe alternatives you've considered
Currently, I use BrowserStack for my Azure DevOps Playwright test runs because it offers video playback for each test case, facilitating easier diagnosis of issues. Without such functionality, the process feels ineffective.

Additional context

AB#1952583

[willbertSmillbert]

Hey man, not with Microsoft but i recently did the following to achieve the outcome you are wanting..

Adding a task to your pipeline will already allow you to do this

It might be a reasonable feature request to the azure devops guys to create a new task called publish playwright tests which would include more data like the videos and stuff your talking about. But you can already get to the second screenshot state using the publish test results task. See the below screenshot, taken from devops..

[puagarwa]

HI @chris-strudel There are few things which i want to decouple for clarity of issues.

Support for Test Results/Report in ADO

As @willbertSmillbert mentioned, this is possible today via ADO inbuilt Publish test Results Task via JUNIT file, you can enable JUNIT reporter in config and then add ADO Task in your pipeline to publish it.

Playwright Service Reporting and Playwright Test Tab

We have feature backlog to add reporting into MPT service and we are actively working on it and hopefully you will soon hear from us. Please add your suggestion around adding ADO integration via Playwright Test tab in your reporting feature ask #47

Known ADO Publish test Results Task bug around video and traces

ADO currently has known issue which doesnt allow video and traces to show up in thier report

[chris-strudel]

Hey man, not with Microsoft but i recently did the following to achieve the outcome you are wanting..

Adding a task to your pipeline will already allow you to do this

It might be a reasonable feature request to the azure devops guys to create a new task called publish playwright tests which would include more data like the videos and stuff your talking about. But you can already get to the second screenshot state using the publish test results task. See the below screenshot, taken from devops..

@willbertSmillbert - Yep! I currently do this, and it's a marginal reporting tool at best. I used to use the Allure reporter, but support for that stopped, and now all we have is the most minimal JUnit reporter. For Quality Engineering, those error messages often aren't enough. I think with the $20B that Microsoft is spending, it would be a good idea to up their game. If they really want the Playwright Testing Service to be a real player, then the reporting side needs a massive upgrade. The Playwright team worked really hard on all those great features with video and traces. Why not take advantage of all their work?

[willbertSmillbert]

@chris-strudel good point.

[puagarwa]

porting side needs a massive upgrade

HI @chris-strudel Your feedback is super valuable for us and it will help us develop reporting solution for playwright. I will keep this issue open.

[chris-strudel]

porting side needs a massive upgrade

HI @chris-strudel Your feedback is super valuable for us and it will help us develop reporting solution for playwright. I will keep this issue open.

I see so much potential in Playwright. It could finally be the testing tool that leaves everything else in the dust! However, it desperately needs the reporting piece. I just finished a POC for the Playwright Testing Service. It's great for spinning up 50 threads and running tests in a massive parallel run, but without reporting, trying to figure out where the issues are is just too painful. We'll be staying with our BrowserStack solution (and we're one of their biggest clients) until the Testing Service and ADO can tackle the reporting piece. We depend on those ADO pipeline runs. Our QEs and Devs need to be able to quickly find the problems so that our agile teams can move quickly.