how to run benchmark script with a .aider.model.settings.yml configuration?

[rawwerks]

Issue

@paul-gauthier

i'm having trouble running https://github.com/Aider-AI/aider/blob/a44ebfe99fdb5bb578d22fd292c3b94dcd4d05a9/benchmark/benchmark.py with a .aider.model.settings.yml file.

inside the docker container, i can successfully run aider --model my-custom-model --verbose --no-show-model-warnings and get everything to run, but if i run ./benchmark/benchmark.py test --model my-custom-model --exercises-dir polyglot-benchmark --num-tests 1 --verbose --new, i get:

litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed 
model=test-config
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` 
Learn more: https://docs.litellm.ai/docs/providers

and

- dirname: 2025-01-03-17-19-14--test
  test_cases: 1
  model: my-custom-model
  edit_format: whole
  commit_hash: 94f83eb-dirty
  pass_rate_1: 0.0
  pass_rate_2: 0.0
  pass_num_1: 0
  pass_num_2: 0
  percent_cases_well_formed: 100.0
  error_outputs: 2
  num_malformed_responses: 0
  num_with_malformed_responses: 0
  user_asks: 2
  lazy_comments: 0
  syntax_errors: 0
  indentation_errors: 0
  exhausted_context_windows: 0
  test_timeouts: 0
  total_tests: 225
  command: aider --model my-custom-model
  date: 2025-01-03
  versions: 0.70.1.dev
  seconds_per_case: 0.7
  total_cost: 0.0000

perhaps there are some additional flags that need to be added to benchmark.py (that would get passed to aider as command line flags?)

or perhaps it's just an issue of the config not being in the right folder? my suspicion is that it has to do with launching aider in all the different subfolders, and that the benchmark script might need to be updated to point to the .aider.model.settings.yml in the parent directory.

Version and model info

versions: 0.70.1.dev

models: https://github.com/mixtcha/mixtcha/tree/main/integrations/aider

current workaround

https://github.com/mixtcha/mixtcha/tree/main/integrations/aider#benchmarking-a-mixtcha

[liudan193]

How do you add your my-custom-model?