[Feature] When using unitxt with lm-evaluation-harness, need to provide a mechanism to specify the local metrics location

[yhwang]

When using unitxt with lm-evaluation-harness, users can only put the local metrics modules in the current working directory. Need a better mechanism to allow users to specify the location of the local metrics modules.

[yhwang]

cc @yoavkatz

[yoavkatz]

One proposal is this:

To have a HF_UNITXT_CACHE_DIR, such that when your run in online mode, it will store the metrics and models there. To create offline mode, you would run all the relevant metrics (downloaded to this dedicated cache dir). You can then package this cache within the image, and when run in offline mode, it will search in that cache.

Example flow:

1. When online :

python -c "import evaluate; print(evaluate.load('exact_match',cache_dir='unitxt_cache').compute(references=['hello'], predictions=['hello']))"
{'exact_match': 1.0}

2. Then disconnect from internet

3. Then we can run again:
   python -c "import evaluate; print(evaluate.load('exact_match',cache_dir='unitxt_cache').compute(references=['hello'], predictions=['hello']))"

(The code works , and only returns this warning):

Using the latest cached version of the module from /Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/9d3b67e0c429cd7460b2b05aab53419b48eea369b73e1d9f185a56ca90c373d4 (last modified on Mon Sep 30 08:35:23 2024) since it couldn't be found locally at evaluate-metric--exact_match, or remotely on the Hugging Face Hub.
{'exact_match': 1.0}

[yhwang]

I believe this approach's key factor is not the HF_UNITXT_CACHE_DIR. Actually, it directly relies on the cached metrics on HF_HOME. Once you populate the metrics into the HF_HOME/modules/evaluate_modules/metrics using the evaluate.load() API, the following evaluate.load() should be able to use the cached version.

The reason I raised this question is:

1. The caching uses the SHA for the file name and is not human-readable. When preparing the pre-load modules, having a human-readable file structure would be better.
2. The preload modules need to be put exactly on the HF_HOME/modules/evaluate_modules/metrics path. I am seeking an alternative way to have a custom location. I saw a relevant env: HF_METRICS_CACHE, but it seems equivalent to the cache_dir arg of the evaluate.load() API. This means it uses the SHA for the files and later on these cached files would be copied to HF_HOME/modules/evaluate_modules/metrics

I am looking for a more intuitive way to prepare the metrics module and Unitxt can load/use them.

[yoavkatz]

You are right. The metric code seems to be always downloaded to HF_HOME . The cache_dir parameter is only used to store temporary reference and predictions.

So I think we have no option but to use the same approach we used in lm eval. Add a prefix to the metric name and store the code under a subdirectory with the same name.

@elronbandel . Please give your input if you see any other option.