"Failed to load images" for Android Chrome results

[wangxianzhu]

Example

Permalink to an example reftest:
https://wpt.fyi/results/css/css-transforms/2d-rotate-001.html?run_id=5165372401778688&run_id=5189990978158592

Link to the reftest analyzer:
https://wpt.fyi/analyzer?screenshot=sha1%3Ae2a9949841092970ba6df81d17e309892ec5d383&screenshot=sha1%3A3b5e44231f8dd829ec6571bf679349c49307bc85

Description

Failed to load images.

[jcscottiii]

Notes while debugging this:

For the before picture, it expects a screenshot at https://storage.googleapis.com/wptd-screenshots/sha1:e2a9949841092970ba6df81d17e309892ec5d383.png

For the after picture, it expects a screenshot at https://storage.googleapis.com/wptd-screenshots/sha1:3b5e44231f8dd829ec6571bf679349c49307bc85.png

Both of those do not exist. Verified that they indeed do not exist in GCP too:

Next step: See where the upload happens for the bucket

[jcscottiii]

Uploading of the screenshots is handled by this handler at the /api/screenshots/upload route

wpt.fyi/api/screenshot/routes.go

Line 17 in cf42330

shared.AddRoute("/api/screenshots/upload", "api-screenshots-upload", uploadScreenshotHandler)

It's a private endpoint. And it used by the results-processor

wpt.fyi/results-processor/wptscreenshot.py

Line 97 in 0c1ef56

config.project_baseurl() + '/api/screenshots/upload')

Each screenshot is uploaded during the results processor:

wpt.fyi/results-processor/processor.py

Lines 349 to 357 in cf42330

	# ==== Beginning of tasks ====
	# Tasks are supposed to be independent; exceptions are ignored (but logged).
	# Each task is a function that takes a Processor.
	def _upload_screenshots(processor: Processor) -> None:
	for screenshot in processor.screenshots:
	with WPTScreenshot(screenshot, processor.report.run_info,
	auth=processor.auth) as s:
	s.process()

Which happens in process report:

wpt.fyi/results-processor/processor.py

Lines 377 to 388 in cf42330

	with Processor() as p:
	p.update_status(run_id, 'WPTFYI_PROCESSING', None, callback_url)
	if archives:
	_log.info("Downloading %d archives", len(archives))
	else:
	_log.info("Downloading %d results & %d screenshots",
	len(results), len(screenshots))
	p.download(results, screenshots, archives)
	if len(p.results) == 0:
	_log.error("No results successfully downloaded")
	p.update_status(run_id, 'EMPTY', None, callback_url)
	return ''

[jcscottiii]

Not sure if it's related. When I look at the logs for the failed 2d-rotate-001.html test, I see a normal failure message. But in the test below, I see a message around failing to take a screenshot. Not sure if the test runner hides that exception if the test fails.

Next steps: We need to look at the archives / uploaded reports

[jcscottiii]

Comparing the processor logs

For the Chrome Android run id=5068299723603968
(link to logs)

For Firefox Android run id=5100626969362432
(link to logs)

You'll see that the results received by the results processor are missing screenshots for Chrome Android

[jcscottiii]

I wonder if it's because Chromium CI tests are not run with --log-wptscreenshot. https://github.com/web-platform-tests/wpt/blob/38c248da8df5530cb270138878987e08cee8d725/docs/running-tests/from-local-system.md?plain=1#L200 ( at least I didn't find anything at first glance in chromium code search).

@WeizhongX Thoughts on this?

---

On another note, I did find out that a chromium developer can see the diffs locally. Example:

@wangxianzhu I will add a comment on the issue to unblock you so you can take a look.