Essence Problem Models Not Supported for generating Discriminating Instances

[vincepick]

After fixing the initial issue of key error for SRTimeLimit, it now says that solver ortools is not yet supported when trying to run it for discriminating instances.

+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
| |   Instance|      Alive|       Best|      Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
# 2024-10-23 18:12:58 UTC: /AutoIG/scripts/target-runner 1 1 719061008 dummy   -A 137
# 2024-10-23 18:13:01 UTC: DONE (1) Elapsed wall-clock seconds: 2.91
Inf
Error! Exception: Sorry, solver ortools is not yet supported.
Error: == irace == The output of targetRunner should not be more than two numbers!
== irace == The call to targetRunner was:
/AutoIG/scripts/target-runner 1 1 719061008 dummy   -A 137
== irace == The output was:
Inf
Error! Exception: Sorry, solver ortools is not yet supported.
== irace == This is not a bug in irace, but means that something failed when running the command(s) above or they were terminated before completion. Try to run the command(s) above from the execution directory '/AutoIG/experiments/simple_e_discriminating' to investigate the issue. See also Appendix B (targetRunner troubleshooting checklist) of the User Guide (https://cran.r-project.org/package=irace/vignettes/irace-package.pdf).

[vincepick]

It appears to be fully functional for the tiny.essence problem model, but for more complicated ones such as car-sequencing has another issue.

[ndangtt]

hi @vincepick, thank you for the updates. I guess SR doesn't support ortools yet, but to double-check, could you please post the detailed conjure/SR/solver command that caused the issue (produced when running the wrapper.py script), and the error message?

[ozgurakgun]

Try conjure solve --solver=or-tools ESSENCE PARAM. Should call or-tools. Passing -v as well will print the SR command conjure uses.

[vincepick]

Yes I can, I will take another look at it myself too I’ve just been trying to start documenting the issues more consistently using GitHub actions. This is an odd problem too because I know that ortools works for discriminating instances for the MiniZinc problem models, so I think that SR does support it, it may just be a part of wrapper causing an issue. I also made some changes to wrapper which fixed another bug but could have caused this one. Vincent Pickering

…

________________________________ From: Nguyen Dang ***@***.***> Sent: Thursday, October 24, 2024 10:08 AM To: stacs-cp/AutoIG ***@***.***> Cc: Vincent Pickering ***@***.***>; Mention ***@***.***> Subject: {Disarmed} Re: [stacs-cp/AutoIG] Essenc Problem Models Not Supported for generating Discriminating Instances (Issue #5) hi @vincepick<https://github.com/vincepick>, thank you for the updates. I guess SR doesn't support ortools yet, but to double-check, could you please post the detailed conjure/SR/solver command that caused the issue (produced when running the wrapper.py script), and the error message? — Reply to this email directly, view it on GitHub<#5 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A5XUL22IEXXEAX6OWCLPGYDZ5C2H3AVCNFSM6AAAAABQPM4XQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZUG4YTMMZQGE>. You are receiving this because you were mentioned.Message ID: ***@***.***> [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#5 (comment)", "url": "#5 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[vincepick]

Hi Dr. Dang and Dr. Oz,

Thank you both for your advice, I tried running with my current approach again and think we should discuss this issue further in our meeting tomorrow @ndangtt.

I'm afraid I am not positive which command you are referring to @ndangtt. I believe it was the following one, and this is the one I was examining, but please do let me know if you meant another:
python3 -u /AutoIG/scripts/wrapper.py 1 1 719061008 dummy -A 137

Its output:

2024-10-24 16:39:15: /AutoIG/scripts/wrapper.py 1 1 719061008 dummy -A 137
{'generalSettings': {'experimentType': 'discriminating', 'modelFile': '/AutoIG/data/models/simple_e/simple.essence', 'generatorFile': '/AutoIG/data/models/simple_e/simple_gen.essence', 'runDir': '/AutoIG/experiments/simple_e_discriminating'}, 'generatorSettings': {'genSRTimeLimit': 300, 'genSRFlags': '-S0 -no-bound-vars', 'genSolver': 'minion', 'genSolverTimeLimit': '5', 'genSolverFlags': '-varorder domoverwdeg -valorder random'}, 'evaluationSettings': {'nEvaluations': 1, 'gradedTypes': 'all', 'SRTimeLimit': 1, 'SRFlags': '', 'solverTimeLimit': 1, 'solverFlags': '-f', 'scoringMethod': 'complete', 'baseSolver': {'name': 'chuffed', 'solverMinTime': 1, 'totalTimeLimit': 3, 'solverFlags': '', 'SRTimeLimit': 1, 'SRFlags': '', 'solverTimeLimit': 1}, 'favouredSolver': {'name': 'ortools', 'totalTimeLimit': 3, 'solverFlags': '-f', 'SRTimeLimit': 1, 'SRFlags': '', 'solverTimeLimit': 1}}}


2024-10-24 16:39:15: Creating generator instance: ./detailed-output/gen-inst-1.param
2024-10-24 16:39:15: minion ./detailed-output/gen-inst-1.minion -solsout ./detailed-output/gen-inst-1.solution -randomseed 719061008 -timelimit 5 -varorder domoverwdeg -valorder random
2024-10-24 16:39:16: savilerow ./detailed-output/generator.eprime -mode ReadSolution -out-aux ./detailed-output/gen-inst-1.aux -out-solution ./detailed-output/gen-inst-1.solution.eprime-param -minion-sol-file ./detailed-output/gen-inst-1.solution
2024-10-24 16:39:16: conjure translate-solution --eprime=./detailed-output/generator.eprime --essence-param=./detailed-output/gen-inst-1.param --eprime-solution=./detailed-output/gen-inst-1.solution.eprime-param --essence-solution ./detailed-output/gen-inst-1.solution.param


2024-10-24 16:39:16: 
Generator results: genInstance=gen-inst-1, genStatus=sat, genSRTime=0, genSolverTime=0.11559700965881348


2024-10-24 16:39:16: Solving ./detailed-output/inst-1-719061008.param...


---- With random seed 0th (719061008) and solver ortools (favouredSolver)
Traceback (most recent call last):
  File "/AutoIG/scripts/wrapper.py", line 1110, in <module>
    main()
  File "/AutoIG/scripts/wrapper.py", line 1044, in main
    score, instanceResults = globals()[evaluationFunctionName](
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/AutoIG/scripts/wrapper.py", line 228, in evaluate_essence_instance_discriminating
    runStatus, SRTime, solverTime = call_conjure_solve(
                                    ^^^^^^^^^^^^^^^^^^^
  File "/AutoIG/scripts/essence_pipeline_utils.py", line 384, in call_conjure_solve
    conjureCmd, tempFiles = make_conjure_solve_command(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/AutoIG/scripts/essence_pipeline_utils.py", line 318, in make_conjure_solve_command
    raise Exception("Sorry, solver " + solver + " is not yet supported.")
Exception: Sorry, solver ortools is not yet supported.

I found the dictionary where the solver infos are defined in essence_pipeline_utils.py, and ortools indeed isn't listed. I can try adding it and then going from there, but am not sure that is the best approach. I believe we should discuss options and Dr. Oz's recommendation in our meeting tomorrow then go from there, I have quite a few questions and also want to confirm that what I already changed doesn't seem to introduce any more bugs before working off of it.

solverInfo = {}
solverInfo["cplex"] = {
    "timelimitUnit": "ms",
    "timelimitPrefix": "--time-limit ",
    "randomSeedPrefix": "via text file",
}
... rest of solvers (ortools indeed not listed among them)

Thank you,
Vincent

[vincepick]

Hi Dr. Dang,

Thank you for meeting with us today.

Here is the command and attached files we talked about. Please let me know if there is anything else I should add and I will get to it right away. I had to attach the files as a zip because GitHub doesn't recognize any of the file formats. In the meantime, I will continue to experiment by comparing the MZN implementation for generating discriminating instances with the Essence implementation.

Thank you,
Vincent

Command:
conjure solve -v ./problem.essence ./detailed-output/inst-1-719061008.param -o ./detailed-output --use-existing-models=problem.eprime --savilerow-options "-timelimit 1 " --solver-options "--time-limit 1000 -r 719061008 -f" --solver=ortools

Command Output (With additional verbose flag):

conjure solve -v ./problem.essence ./detailed-output/inst-1-719061008.param -o ./detailed-output --use-existing-models=problem.eprime --savilerow-options "-timelimit 1 " --solver-options "--time-limit 1000 -r 719061008 -f" --solver=ortools Command line options: Solve {essence = "./problem.essence", essenceParams = ["./detailed-output inst-1-719061008.param"], validateSolutionsOpt = False, outputDirectory = "./detailed-output", numberingStart = 1, smartFilenames = False, responses = "", responsesRepresentation = "", solutionsInOneFile = False, runsolverCPUTimeLimit = Nothing, runsolverWallTimeLimit = Nothing, runsolverMemoryLimit = Nothing, logLevel = LogDebug, verboseTrail = False, rewritesTrail = False, logRuleFails = False, logRuleSuccesses = False, logRuleAttempts = False, logChoices = False, portfolio = Nothing, strategyQ = "f", strategyA = "c", representations = Nothing, representationsFinds = Nothing, representationsGivens = Nothing, representationsAuxiliaries = Nothing, representationsQuantifieds = Nothing, representationsCuts = Nothing, channelling = True, representationLevels = True, followModel = "", seed = Nothing, limitModels = Nothing, limitTime = Nothing, useExistingModels = ["problem.eprime"], savilerowOptions = ["-timelimit 1 "], solverOptions = ["--time-limit 1000 -r 719061008 -f"], solver = "ortools", graphSolver = False, cgroups = False, nbSolutions = "1", copySolutions = True, outputFormat = Plain, lineWidth = 120, generateStreamliners = ""} Error: Unsupported solver: ortools

AutoIG Files.zip

[ndangtt]

thanks, @vincepick!

@ozgurakgun: can we ask for your help on this please? We're using conjure's latest release and cannot make or-tools works via conjure solve. The command is:
conjure solve -v ./problem.essence inst-1-719061008.param -o ./ --use-existing-models=problem.eprime --savilerow-options "-timelimit 10 " --solver-options "-t 10000 -r 719061008 -f" --solver=or-tools

The files are attached in previous message by Vincent. When I switch to --solver=chuffed, the command works. I have fzn-ortools in my $PATH.

[ozgurakgun]

the release may be too old, you could try the latest docker file?

https://github.com/conjure-cp/conjure/pkgs/container/conjure/285546501?tag=main

specific versions can also be selected here: https://github.com/conjure-cp/conjure/pkgs/container/conjure/versions

example command to run this via podman: https://github.com/conjure-cp/conjure-aas/blob/c0c94ad615b1540dde4169a32d1a5790cccddcd6/conjure-aas.js#L94

a major advantage is that you won't have to install any of the tools.

alternatively you can build the latest version on main

alternative 3: I can make a new release, I probably should!

[ozgurakgun]

also, the binary was renamed to fzn-cp-sat - is this the issue I wonder?

[vincepick]

My container is built off of the most recent Conjure base image, so it should be fully up to date for Conjure unless it has had any updates in the last month or so. (It's using v2.5.1, 2024-09-25 to be specific)

If there have been updates, I would be happy to rebuild my container and see if that makes any difference.

Also, I have been having some odd new issues. Perhaps they are related to the renamed binary, thank you for pointing it out!

Vincent

[ozgurakgun]

if you are building on top of that docker image everything should work. what error message do you get?

[vincepick]

I tried remaking my current container earlier today but seemed to be running into issues with the path. I think they're probably setup errors on my end that will be quickly fixed, but I'll keep an eye out for if it may be related to the renamed binary when I start poking around at it again tomorrow.

[ndangtt]

thanks very much @ozgurakgun!

@vincepick: I think there are a couple of issues here:

• conjure solve should call or-tools, not ortools
• as mentioned by Oz, ortools's executable is now named fzn-cp-sat, not fzn-ortools. We may want to update the current docker file and make use of conjure's chuffed/ortools/minizinc binaries there instead (not running the install scripts of those solvers). I'll open a new issue regarding docker file for detailed discussion.
• Savile Row's time limit is too small in the current example command, which may result in SR timeout most of the times in the tuning experiment.

[ndangtt]

@vincepick: another thing that I forgot to mention is the flag for time limit. We should use -t <timelimit> for or-tools, chuffed, yuck and picat instead of --time-limit <timelimit>.

[vincepick]

Sounds good, thank you for all the suggestions @ndangtt . I will try experimenting with all of them and update you tomorrow afternoon.

Could you also please give us permissions to open a GitHub "project" if possible, so we can set some internal deadlines and tie them to the issues to document our progress more clearly? I saw you gave us "maintainer", but unfortunately it still doesn't let us open one. I am also happy to use another issue tracker if preferred!

[ndangtt]

@vincepick: I'm not sure how to give you permission to create a project. If you find a guidelines on how to do it, please let me know and I'll try. Otherwise, if you can tell me what you want me to create, I can create one and add you and Tudor into it.