[red-knot] Property test improvements

[sharkdp]

Summary

• Add a workflow to run property tests on a daily basis (based on daily_fuzz.yaml)
• Mark assignable_to_is_reflexive as flaky (related to [red-knot] (Gradual) intersection types are not handled in assignability #14899)
• Add new (failing) intersection_assignable_to_both test (also related to [red-knot] (Gradual) intersection types are not handled in assignability #14899)

Test Plan

export QUICKCHECK_TESTS=100000
while cargo test --release -p red_knot_python_semantic -- \
  --ignored types::property_tests::stable; do :; done

[sharkdp]

Like the rest of this workflow, this is copied over from daily_fuzz.yaml. It seems like a reasonable idea (how would we notice otherwise?), but I haven't actually seen any "Daily parser fuzz failed on …" in the issue tracker. Did the daily_fuzz workflow never run, or is this issue-creation disabled somehow?

[carljm]

Looks good to me!

I have no idea how to verify that the cron thing actually works, other than to land this and wait and see? (I don't even know where to go look tomorrow to see if it ran, assuming it doesn't fail and notify us.)

[sharkdp]

I have no idea how to verify that the cron thing actually works, other than to land this and wait and see?

The fuzzer tests run daily at 00:00 (UTM?), so I decided to run the property test daily at 12:00, to distribute a bit among timezones, and so I will be awake when they run 😄

https://crontab.guru/#0_12___*

I don't even know where to go look tomorrow to see if it ran

Here: https://github.com/astral-sh/ruff/actions/workflows/daily_property_tests.yaml

Sample run is here: https://github.com/astral-sh/ruff/actions/runs/12678401447/job/35335911931

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

[AlexWaygood]

@sharkdp

Like the rest of this workflow, this is copied over from daily_fuzz.yaml. It seems like a reasonable idea (how would we notice otherwise?), but I haven't actually seen any "Daily parser fuzz failed on …" in the issue tracker. Did the daily_fuzz workflow never run, or is this issue-creation disabled somehow?

@carljm

I have no idea how to verify that the cron thing actually works, other than to land this and wait and see? (I don't even know where to go look tomorrow to see if it ran, assuming it doesn't fail and notify us.)

The py-fuzzer script found a number of bugs in our parser immediately after the big parser rewrite in early 2024, but to my knowledge, the daily cron job that we set up to fuzz the parser shortly after the parser rewrite has never failed. @dhruvmanila just wrote a really good parser! 😃 (And also, we haven't made any major updates since the big rewrite.)

However, I'm pretty confident that the issue-creation logic works, because I've used it to great success in multiple other repos. For examples, see:

• Typeshed:
  • Example issue: Daily tests failed on Sun Dec 29 2024 python/typeshed#13330
  • Workflow: https://github.com/python/typeshed/blob/main/.github/workflows/daily.yml
• typing_extensions:
  • Example issue: Third-party tests failed on Thu Dec 05 2024 python/typing_extensions#513
  • Workflow: https://github.com/python/typing_extensions/blob/main/.github/workflows/third_party.yml
• typeshed-stats:
  • Example issue: Daily test failed on Sat Sep 14 2024 AlexWaygood/typeshed-stats#255
  • Workflow: https://github.com/AlexWaygood/typeshed-stats/blob/main/.github/workflows/test.yml

[AlexWaygood]

this should be

Suggested change

	labels: ["bug", "red_knot", "testing"],
	labels: ["bug", "red-knot", "testing"],

(https://github.com/astral-sh/ruff/issues?q=is%3Aissue%20state%3Aopen%20label%3Ared-knot)

[sharkdp]

Thank you: #15361

[AlexWaygood]

A typeshed contributor recently made a great improvement to the typeshed version of this workflow so that the issue text links directly to the exact run that failed rather than a list of all the runs that have ever happened: https://github.com/python/typeshed/pull/13210/files

We could probably make the same improvement to this workflow and daily_fuzz.yaml!

[sharkdp]

#15361