Skip to content

Commit

Permalink
Update for rolling leaderboard
Browse files Browse the repository at this point in the history
  • Loading branch information
fsschneider committed Feb 11, 2025
1 parent 1ee83e2 commit 45a9b9a
Showing 1 changed file with 5 additions and 48 deletions.
53 changes: 5 additions & 48 deletions docs/GETTING_STARTED.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@
- [Docker Tips](#docker-tips)
- [Score your Submission](#score-your-submission)
- [Running workloads](#running-workloads)
- [Package your Submission code](#package-your-submission-code)
- [Package Logs for Self-Reporting Submissions](#package-logs-for-self-reporting-submissions)
- [Submit your Submission](#submit-your-submission)

## Set Up and Installation

Expand All @@ -31,7 +30,7 @@ The specs on the benchmarking machines are:
- 8xV100 16GB GPUs
- 240 GB in RAM
- 2 TB in storage (for datasets).
3. Install the algorithmic package and dependencies either in a [Python virtual environment](#python-virtual-environment) or use a [Docker](#docker) (recommended) or [Singularity/Apptainer container](#using-singularityapptainer-instead-of-docker).
3. Install the `algoperf` package and dependencies either in a [Python virtual environment](#python-virtual-environment) or use a [Docker](#docker) (recommended) or [Singularity/Apptainer container](#using-singularityapptainer-instead-of-docker).

### Python Virtual Environment

Expand Down Expand Up @@ -252,7 +251,7 @@ python3 submission_runner.py \
--tuning_search_space=<path_to_tuning_search_space>
```
**Pytorch**: to score your submission on a workload, from the algorithmic-efficency directory run:
**PyTorch**: to score your submission on a workload, from the algorithmic-efficency directory run:
```bash
python3 submission_runner.py \
Expand Down Expand Up @@ -389,50 +388,8 @@ python score_submissions.py --submission_directory <directory_with_submissions>

We provide the scores and performance profiles for the [paper baseline algorithms](/reference_algorithms/paper_baselines/) in the "Baseline Results" section in [Benchmarking Neural Network Training Algorithms](https://arxiv.org/abs/2306.07179).

## Package your Submission code
## Submit your Submission

If you have registered for the AlgoPerf competition you will receive
an email on 3/27/2024 with a link to a UI to upload a compressed submission folder.

To package your submission modules please make sure your submission folder is structured as follows:

```bash
submission_folder/
├── external_tuning
│ ├── algorithm_name
│ │ ├── helper_module.py
│ │ ├── requirements.txt
│ │ ├── submission.py
│ │ └── tuning_search_space.json
│ └── other_algorithm_name
│ ├── requirements.txt
│ ├── submission.py
│ └── tuning_search_space.json
└── self_tuning
└── algorithm_name
├── requirements.txt
└── submission.py
```

Specifically we require that:

1. There exist subdirectories in the the submission folder named after the ruleset: `external_tuning` or `self_tuning`.
2. The ruleset subdirectories contain directories named according to
some identifier of the algorithm.
3. Each algorithm subdirectory contains a `submission.py` module. Additional helper modules are allowed if prefer to you organize your code into multiple files. If there are additional python packages that have to be installed for the algorithm also include a `requirements.txt` with package names and versions in the algorithm subdirectory.
4. For `external_tuning` algorithms the algorithm subdirectory
should contain a `tuning_search_space.json`.

To check that your submission folder meets the above requirements you can run the `submissions/repo_checker.py` script.

## Package Logs for Self-Reporting Submissions

To prepare your submission for self reporting run:

```bash
python3 package_logs.py --experiment_dir <experiment_dir> --destination_dir <destination_dir>
```

The destination directiory will contain the logs packed in studies and trials required for self-reporting.
To submit your submission, please create a PR on the [submission repository](https://github.com/mlcommons/submissions_algorithms). You can find more details in the submission repositories [How to Submit](https://github.com/mlcommons/submissions_algorithms?tab=readme-ov-file#how-to-submit) section. The working group will review your PR and select the most promising submissions for scoring.

**Good Luck!**

0 comments on commit 45a9b9a

Please sign in to comment.