[python/ci] Clean up a bit more after a test case (#3683)

[python/ci] Clean up a bit more after a test case (#3683) #3263