Pytest's self tests are failing with pygments v2.19.0 #13112
Comments
|
Bisected to this Pygments change: Improve whitespace handling in the Python lexer. · pygments/pygments@d24e272 |
|
In the test, we match The Old: def # literal
\x1b[39;49;00m # reset fg/bg/formatting
␣ # literal space
\x1b[92m # bright green color for function name
test_this # literalNew: def # literal
\x1b[39;49;00m # reset fg/bg/formatting
! \x1b[90m # text color to "bright black"
␣ # literal space
! \x1b[39;49;00m # reset fg/bg/formatting
\x1b[92m # bright green color for function name
test_this # literalFrom what I understand, Pygments introduced a new Not sure why our style/formatter then decides to highlight that in "bright black" since it's space anyways, but from what I understand, Pygments isn't doing anything wrong here and we probably just need to adjust our expected output? |
With Pygments 2.19, the Python lexer now emits Text.Whitespace (rather than Text) tokens after "def", which get highlighted as "bright black". See pygments/pygments#1905 Fixes pytest-dev#13112
|
Fix: #13113 |
|
Sorry, I misread this. Thought it was a regression in Pygments, but it looks like a test on your side got tripped up by more precise whitespace handling? |
|
Yes, exactly. We have integration tests for pytest's output including the color codes, in order to ensure that pytest correctly shows Pygments-highlighted Python source code as part of its tracebacks. With Pygments now lexing the space in I don't think there is anything Pygments does wrong here. I'm somewhat surprised by spaces being colorized in the output as the foreground color would be invisible anyways, but I suppose there is some good reason for that. |
|
Thanks for getting back quickly. Yeah, we've been spending quite some time making sure that whitespace is correctly tagged. I'm also surprised there's a style which has a specific color for whitespace, I assume that "bright black space" is actually the background color and that's by mistake. Do you happen to know which style the tests use? |
|
Just the default style with dark terminal background, here is a reproducer: import pygments
from pygments.formatters.terminal import TerminalFormatter
from pygments.lexers import PythonLexer
source = "def func(): pass"
lexer = PythonLexer()
formatter = TerminalFormatter(bg="dark")
output = pygments.highlight(source, lexer, formatter)
print(output)
print(repr(output))Or with Pygments 2.18: $ echo "def func(): pass" | pygmentize -l python -f terminal -O bg=dark | cat -v
^[[94mdef^[[39;49;00m ^[[92mfunc^[[39;49;00m(): ^[[94mpass^[[39;49;00m^[[90m^[[39;49;00mbut 2.19: $ echo "def func(): pass" | pygmentize -l python -f terminal -O bg=dark | cat -v
^[[94mdef^[[39;49;00m^[[90m ^[[39;49;00m^[[92mfunc^[[39;49;00m(): ^[[94mpass^[[39;49;00m^[[90m^[[39;49;00m
|
pip listfrom the virtual environment you are usingDescription
When running tox, I am seeing 4 test failures in
test_terminal.py. I am not seeing these errors when I pinpygments>=2.7.2,<2.19.0. I suspect pygments is giving invalid control characters for resetting text formatting. I'm going to open a issue over there in a second.I'm using Centos, but have also confirmed the issue on ubuntu 20 and debian 12. I've also confirmed it on python 3.13, 3.12, and 3.8.
Specific tests failing:
How to recreate
tox -e py312Tox output
(Edited by @The-Compiler to fold away the lengthy output)
The text was updated successfully, but these errors were encountered: