some questions

[dpcdsg]

I want to ask if only https://bigcode-bigcodebench-evaluator.hf.space/ can be used to generate scores after the results are generated.

[terryyz]

Yes, unless you just want to check the ground-truth pass rate. To evaluate any models, you need to do the generation first.

[dpcdsg]

I already have results generated by other models, and now I want to score them. Can I only use the https://bigcode-bigcodebench-evaluator.hf.space/ you provided to get the scores?

[terryyz]

Regarding the gradio (HF space) endpoint, please refer to the following note:

The gradio backend is hosted on the Hugging Face space by default. The default space can be sometimes slow, so we recommend you to use the gradio backend with a cloned bigcodebench-evaluator endpoint for faster evaluation. Otherwise, you can also use the e2b sandbox for evaluation, which is also pretty slow on the default machine.

For any other execution methods, please refer to ADVANCED_USAGE., where you can choose the execution from e2b, gradio, local. e2b is typically slower than the others. local requires you to either use the provided docker image or manually configure the local environment.

[dpcdsg]

OK, thanks