add ~~memory~~ limits

[mzuenni]

implement some more limits

[mpsijm]

Haven't tested it yet, but here are some first comments based on the code 😄

[RagnarGrootKoerkamp]

No time to look at the code yet, but I could imagine the -m argument still being useful to override the yaml key for quick testing purposes?

[mzuenni]

but should the yaml key override it or be an additional limit (i.e. we take the min of both) and should that be for submissions or all subprocesses... ?

[mzuenni]

I think everything should work now, @mpsijm do you have time to give it a try? :)

[mpsijm]

I've used your branch today when preparing some problem ideas for the Freshmen Programming Contests, and it didn't crash 😄 I used at least bt run, bt generate, bt validate, bt constraints, and bt fuzz. Note that I haven't actually touched any of the new limits yet, nor did I explicitly try to push a submission/generator/validator past the time/memory/output limit (I only had some TLE submissions). But some testing is better than no testing, I guess 😛

[mpsijm]

How can the Program class know whether the timeout and memory keys in self.limits are the ones that it should use? It could be that we need to use validation_memory or some other key for the timeout/memory.

[mzuenni]

That is why we use self.limits not problem.limits the constructor of the subclass interferes the right values to use and passes it to the super class

[mpsijm]

Ah right, I got confused between the two kinds of limits, thanks 😄 Maybe we can give Program.limits a different name than Problem.limits to avoid confusion, but I'm not sure what 😅

A different suggestion could be to not store the five types of limits for a Program in a dict, but to have explicit fields (self.timeout, self.memory, self.compilation_time, etc.)?

[mzuenni]

but all of these are optional? (and maybe None needs to be handles differently at some point?)

[mpsijm]

They already are optional currently, right? The only difference is that you use self.limits.get(<key>, <default>) to fetch the values from the dict, falling back to <default> if no value exists for the <key> 😛

I think it would be nice if we set the defaults in the constructor already, instead of spread throughout the Program class. Then, the limits dict only needs to exist in the constructor, and the fields (self.timeout, self.memory, self.compilation_time, etc.) are no longer optional 😄 And that also makes it easy to change potential future differnt handling of None, because that all stays within the constructor 🙂

(even if we don't split up Program.limits to five separate fields, we still may want to set the defaults in the constructor)