Fix/unit tests

[mreso]

What does this PR do?

This PR removes our usage of decapoda-research/llama-7b-hf in the unit tests as a surrogate for the actual llama tokenizer. As an alternative we skip the tests if we can not access the gated model folder.

Fixes # (issue)

Feature/Issue validation/testing

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

• Test A pytest tests

============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/mreso/llama-recipes
configfile: pyproject.toml
plugins: mock-3.12.0
collected 23 items

tests/test_batching.py ..                                                                                                                                                                                                              [  8%]
tests/test_finetuning.py .....                                                                                                                                                                                                         [ 30%]
tests/test_sampler.py ..........                                                                                                                                                                                                       [ 73%]
tests/test_train_utils.py .                                                                                                                                                                                                            [ 78%]
tests/datasets/test_alpaca_dataset.py s                                                                                                                                                                                                [ 82%]
tests/datasets/test_custom_dataset.py ..                                                                                                                                                                                               [ 91%]
tests/datasets/test_grammar_datasets.py .                                                                                                                                                                                              [ 95%]
tests/datasets/test_samsum_datasets.py .                                                                                                                                                                                               [100%]

============================================================================================================== warnings summary ==============================================================================================================
../.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:441
  /home/mreso/.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    _torch_pytree._register_pytree_node(

src/llama_recipes/finetuning.py:5
  /home/mreso/llama-recipes/src/llama_recipes/finetuning.py:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging

../.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:309
../.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:309
  /home/mreso/.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    _torch_pytree._register_pytree_node(

../.conda/envs/llama/lib/python3.10/site-packages/torch/distributed/_shard/checkpoint/__init__.py:8
  /home/mreso/.conda/envs/llama/lib/python3.10/site-packages/torch/distributed/_shard/checkpoint/__init__.py:8: DeprecationWarning: torch.distributed._shard.checkpoint will be deprecated, use torch.distributed.checkpoint instead
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================================= 22 passed, 1 skipped, 5 warnings in 48.58s =================================================================================================

• Test B $huggingface-cli logout Successfully logged out. $rm -rf ~/.cache/huggingface/ $pytest tests

============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/mreso/llama-recipes
configfile: pyproject.toml
plugins: mock-3.12.0
collected 23 items

tests/test_batching.py ss                                                                                                                                                                                                              [  8%]
tests/test_finetuning.py .....                                                                                                                                                                                                         [ 30%]
tests/test_sampler.py ..........                                                                                                                                                                                                       [ 73%]
tests/test_train_utils.py .                                                                                                                                                                                                            [ 78%]
tests/datasets/test_alpaca_dataset.py s                                                                                                                                                                                                [ 82%]
tests/datasets/test_custom_dataset.py s.                                                                                                                                                                                               [ 91%]
tests/datasets/test_grammar_datasets.py s                                                                                                                                                                                              [ 95%]
tests/datasets/test_samsum_datasets.py s                                                                                                                                                                                               [100%]

============================================================================================================== warnings summary ==============================================================================================================
../.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:441
  /home/mreso/.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    _torch_pytree._register_pytree_node(

src/llama_recipes/finetuning.py:5
  /home/mreso/llama-recipes/src/llama_recipes/finetuning.py:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging

../.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:309
../.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:309
  /home/mreso/.conda/envs/llama/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
    _torch_pytree._register_pytree_node(

../.conda/envs/llama/lib/python3.10/site-packages/torch/distributed/_shard/checkpoint/__init__.py:8
  /home/mreso/.conda/envs/llama/lib/python3.10/site-packages/torch/distributed/_shard/checkpoint/__init__.py:8: DeprecationWarning: torch.distributed._shard.checkpoint will be deprecated, use torch.distributed.checkpoint instead
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================================= 17 passed, 6 skipped, 5 warnings in 8.32s ==================================================================================================

• Test C $pytest tests --unskip-missing-tokenizer

========================================================================================================== short test summary info ===========================================================================================================
ERROR tests/test_batching.py::test_packing - OSError: You are trying to access a gated repo.
ERROR tests/test_batching.py::test_distributed_packing - OSError: You are trying to access a gated repo.
ERROR tests/test_finetuning.py::test_finetuning_no_validation - OSError: You are trying to access a gated repo.
ERROR tests/test_finetuning.py::test_finetuning_with_validation - OSError: You are trying to access a gated repo.
ERROR tests/test_finetuning.py::test_finetuning_peft - OSError: You are trying to access a gated repo.
ERROR tests/test_finetuning.py::test_finetuning_weight_decay - OSError: You are trying to access a gated repo.
ERROR tests/test_finetuning.py::test_batching_strategy - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_array[2-False] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_array[8-False] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_array[2-True] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_array[8-True] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_dict[2-False] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_dict[8-False] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_dict[2-True] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_batch_sampler_dict[8-True] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_dist_batch_sampling[2] - OSError: You are trying to access a gated repo.
ERROR tests/test_sampler.py::test_dist_batch_sampling[8] - OSError: You are trying to access a gated repo.
ERROR tests/test_train_utils.py::test_gradient_accumulation - OSError: You are trying to access a gated repo.
ERROR tests/datasets/test_custom_dataset.py::test_custom_dataset - OSError: You are trying to access a gated repo.
ERROR tests/datasets/test_custom_dataset.py::test_unknown_dataset_error - OSError: You are trying to access a gated repo.
ERROR tests/datasets/test_grammar_datasets.py::test_grammar_dataset - OSError: You are trying to access a gated repo.
ERROR tests/datasets/test_samsum_datasets.py::test_samsum_dataset - OSError: You are trying to access a gated repo.
================================================================================================= 1 skipped, 6 warnings, 22 errors in 9.18s ==================================================================================================

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Did you write any new necessary tests?

Thanks for contributing 🎉!