Skip to content

Commit

Permalink
fix save generation path arg
Browse files Browse the repository at this point in the history
  • Loading branch information
loubnabnl committed Apr 22, 2023
1 parent 26c38c0 commit f2ac68e
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 7 deletions.
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,21 +131,24 @@ Suppose you generated text with the `bigcode/santacoder` model and saved it in `
```bash
accelerate launch main.py \
--model bigcode/santacoder \
--tasks multiple-java \
--tasks multiple-py \
--max_length_generation 650 \
--temperature 0.8 \
--do_sample True \
--n_samples 200 \
--batch_size 120 \
--generation_only
--batch_size 200 \
--trsut_remote_code \
--generation_only \
--save_generations \
--save_generations_path generations_py.json
```

To run the container (here from image `evaluation-harness`) to evaluate on `generations.json`, or another file mount it with `-v`, specify `n_samples` and allow code execution with `--allow_code_execution` (and add the number of problems `--limit` if it was used during generation):
```bash
$ sudo docker run -v $(pwd)/generations_py.json:/app/generations_py.json:ro -it evaluation-harness-multiple python3 main.py \
--model bigcode/santacoder \
--tasks multiple-py \
--generations_path /app/generations_py.json \
--load_generations_path /app/generations_py.json \
--allow_code_execution \
--temperature 0.8 \
--n_samples 200
Expand Down
4 changes: 2 additions & 2 deletions main.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,9 +195,9 @@ def main():
print("generation mode only")
generations, references = evaluator.generate_text(task)
if accelerator.is_main_process:
with open("generations.json", "w") as fp:
with open(args.save_generations_path, "w") as fp:
json.dump(generations, fp)
print("generations were saved")
print(f"generations were saved at {args.save_generations_path}")
if args.save_references:
with open("references.json", "w") as fp:
json.dump(references, fp)
Expand Down
2 changes: 1 addition & 1 deletion makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ build:
test:
docker run -v $(CURDIR)/tests/docker_test/test_generations.json:/app/test_generations.json:ro \
-it $(IMAGE_NAME) python3 main.py --model dummy_model --tasks humaneval --limit 4 \
--generations_path /app/test_generations.json --allow_code_execution
--load_generations_path /app/test_generations.json --allow_code_execution

@echo "If pass@1 is 0.25 then your configuration for standard benchmarks is correct"

Expand Down

0 comments on commit f2ac68e

Please sign in to comment.