Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CI to run a11y tests on more browser / OS combos #2095

Open
kaycebasques opened this issue Jan 13, 2025 · 3 comments
Open

Update CI to run a11y tests on more browser / OS combos #2095

kaycebasques opened this issue Jan 13, 2025 · 3 comments
Labels
tag: accessibility Issues related to accessibility issues or efforts tag: CI Pull requests that update GitHub Actions code

Comments

@kaycebasques
Copy link
Contributor

During review of #2093 we discovered an a11y test that should have failed in CI but did not: #2093 (comment)

The problem is that CI only runs the a11y tests on Ubuntu. If possible, they should run on more OS / browser combos.

@trallard
Copy link
Collaborator

We could expand the matrices for the a11y tests. Still, IIRC, when we first introduced them, we decided to keep them as minimal as possible to avoid significant disruptions to the contributor workflow, especially as we worked on accessibility remediation.

Separately, it seems that in the PR linked, there are indeed two tests failing.

tests/test_a11y.py ....................xx.............. [100%]
=========== 34 passed, 94 deselected, 2 xfailed in 69.29s (0:01:09) ============

see https://github.com/pydata/pydata-sphinx-theme/pull/2093/checks#step:4:1550
unless the expected behaviour was different?

@trallard
Copy link
Collaborator

🤔 ignore what I said about the failing tests (I will follow up on the corresponding PR).

In the meantime, since I already have a WIP PR for CI improvements, I expanded the a11y testing matrix. However, I separated these from the overall CI as we already have a lot of tests going on there. If you are interested, these are already working at https://github.com/pydata/pydata-sphinx-theme/actions/runs/12790504035

(though now some tests are failing in Firefox, so I need to look into those)

@kaycebasques
Copy link
Contributor Author

🤔 ignore what I said about the failing tests

Yes I was going to say "pretty sure I saw those tests fail on main also"… sounds like you reached similar conclusions!

Thank you expanding the a11y test matrix. It's a very compelling aspect of the value-add of pydata-sphinx-theme

@trallard trallard added tag: accessibility Issues related to accessibility issues or efforts tag: CI Pull requests that update GitHub Actions code labels Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tag: accessibility Issues related to accessibility issues or efforts tag: CI Pull requests that update GitHub Actions code
Projects
None yet
Development

No branches or pull requests

2 participants