Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up timeouts #165

Open
RagnarGrootKoerkamp opened this issue Nov 12, 2021 · 2 comments
Open

Clean up timeouts #165

RagnarGrootKoerkamp opened this issue Nov 12, 2021 · 2 comments

Comments

@RagnarGrootKoerkamp
Copy link
Owner

RagnarGrootKoerkamp commented Nov 12, 2021

  • There is the global program timeout for generators/visualizers/building (i.e.anything not submissions), which defaults to 30s and is overridden by --timeout for e.g. bt generate
  • There is a per problem submission timeout, which is derived from the timelimit and overridden using --timeout on bt run.

This makes for some inconsistency because now timeout is both a program and problem property.

We could 'fix' the code and always make timeout a program property, which is either derived from the global timeout or from the problem timeout, but I'm not sure that's actually worth fixing.

I think we should fix the duplicate --timeout flag name, since this makes bapctools.yaml configuration a bit annoying otherwise. I'm not exactly sure how though. One option is to give a different name to one of them, but have -T as shortcut for both anyway. (-t is currently the shortcut for timelimit already.)

I'm not sure whether --timelimit is more common/useful for the problem or generate case. For problems, I typically just change the timelimit, not the timeout, if I want to see how long things take. On the other hand, although rare, I could see generators taking more than 30 seconds.

CC @mpsijm

@mpsijm
Copy link
Collaborator

mpsijm commented Nov 12, 2021

For problems, I typically just change the timelimit, not the timeout, if I want to see how long things take.

I agree with that one, so in that sense I'd say we could rename/rework how the timeout works. As I have written in #163 (comment), we could change the submission timeout to work more like Kattis' safety_margin, so that the distinction with a Program timeout is clearer?

@RagnarGrootKoerkamp
Copy link
Owner Author

Implementing the limits: options as problemtools uses them is a long outstanding issue anyway, so safety_margin is definitely something we should do. Still, on the command line you would want to pass an absolute time, not a factor, I think.

Thinking about it, timeout: in a bapctools.yaml really should be for the generators, not the problem. So maybe the cleanest is to just drop the run --timeout flag (or make it do the same as generate --timeout), and force the user to do this using safety_margin in the problem config or by just doing -t/--timelimit. Alternatively, we could make --timeout override both generator and submission timeouts when it is specified on the command line, but that's probably confusing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants