Merge remote-tracking branch 'refs/remotes/origin/main'

ChiSym · Jun 17, 2024 · 11ad036 · 11ad036
2 parents 1ce8acf + cde94b2
commit 11ad036
Show file tree

Hide file tree

Showing 6 changed files with 1,484 additions and 5 deletions.
diff --git a/bench/README.md b/bench/README.md
@@ -13,18 +13,20 @@ pip install gdown
 assuming you're currently in the `bench` directory, do
 
 ```bash
+mkdir spider/data
 cd spider/data
 gdown 'https://drive.google.com/u/0/uc?id=1iRDVHLr4mX2wQKSgA9J8Pire73Jahh0m&export=download'
 unzip spider.zip
 ```
 
-also, download the evaluation codebase:
+also, go back to the `genparse/bench` directory and download the evaluation codebase:
 
 ```bash
+cd ../..
 git clone https://github.com/taoyds/spider.git spider-eval
 ```
 
-before running any evaluation, `spider-eval` depends on `punkt` package of `nltk, so download that first:
+before running any evaluation, `spider-eval` depends on `punkt` package of `nltk`, so download that first:
 
 ```
   >>> import nltk

diff --git a/bench/run_spider_llama2_chat.py b/bench/run_spider_llama2_chat.py
@@ -155,11 +155,11 @@ def main():
         predicted.append(output[0]['generated_text'][len(prompt) :])
 
     gold = spider_dev_data[:n_query]
-    with open(f'spider-eval/gold-{args.exp_name}.txt', 'w+') as f:
+    with open(f'bench/spider-eval/gold-{args.exp_name}.txt', 'w+') as f:
         for datum in gold:
             print(f'{datum.query}\t{datum.schema_name}', file=f)
 
-    with open(f'spider-eval/predicted-{args.exp_name}.txt', 'w+') as f:
+    with open(f'bench/spider-eval/predicted-{args.exp_name}.txt', 'w+') as f:
         for datum in predicted:
             datum = datum.replace('\n', ' ')
             assert '\t' not in datum