[BUG]: Rerunning a Playwright shard on a Pull Request in Github Actions results in a report that gets stuck in In Progress

[angelo-loria]

When using the reporting dashboard with sharding in Github Actions, a re-run of a failed shard on a Pull Request results in a report that is stuck in In Progress.

To Reproduce
Steps to reproduce the behavior:

1. Execute tests in Github Actions with sharding enabled

 jobs:
  playwright_test:
    name: Playwright Test 
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4, 5]
        shardTotal: [5]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Yarn Install
        run: yarn install

      - name: Run Tests
        run: yarn test:ci --workers=2 --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          # Access token and regional endpoint for Microsoft Playwright Testing
          PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }}
          PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
          PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }} // note that this makes no difference

3. This is the result of our initial run:
   
4. The initial run results in a working report dashboard:
   
5. Rerun the failed jobs in Actions:
   
6. After rerunning jobs, we have different results:
   
7. The report gets stuck in In Progress and can't be accessed:
   

Setup information (please complete the following information):*

• Azure Subscription Id: af4ec0b8-998c-4cf2-900b-5352b121dbf9
• Workspace Name: Kaleidoscope
• Playwright Version: 1.48.1

[kj0171]

Sorry for the inconvenience caused.
For this particular run, for the 1st try can you please check whether the reporting was configured properly because we at the server end didn't receive any requests.
On the retry, we received requests for shards 2,4 and we were able to process them successfully.

[Test being stuck in InProgress]
Since, the run was for 5 shards, and we only received shards 2,4. That is why it took some to close the run from our end.
It should have been marked completed. Can you please verify if the run is marked as completed now and you are able to see results.

[Can we re-run shards and find the results to be updated]
For the service, every run is independent. If you do a re-run from GH Actions, the service will treat it as a new run and will show you the results in the workspace.

For any other queries or issues, you can post it out on GitHub issues.
Thanks

[angelo-loria]

I do see now that they are completing but it takes such a long time that it's not usable. What is an expected timeframe for the report to complete? For example, this one has been processing for over two hours:

[abranj19]

Hi @angelo-loria, Thanks for reporting this. We are thinking more on this about adding support for rerun failed jobs and will get back to you soon.