From 44a7a48196ddd4a2976c41f0966c2eea2989f20a Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Wed, 31 Aug 2022 09:46:51 +0100 Subject: [PATCH 1/2] Reproducibility requirement of results Clarification regarding reproducibility requirement of inference submission results --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index a5ab3cc..d895e29 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -528,7 +528,7 @@ For Inference, the results logs must have been produced by the [standard load ge The following section applies to all submissions in all divisions. -The source code must be sufficient to reproduce the results of the submission, given all source components specified in Section 5.11 (for Inference) or Section 5.12 (for Training) are provided in the submission repo for all Categories, including Available, Preview, and RDI. In addition, any software component that would be required to substantially reproduce the submission must be uniquely identified using one of the following methods: +The source code must be sufficient to reproduce the results of the submission, given all source components specified in Section 5.11 (for Inference) or Section 5.12 (for Training) are provided in the submission repo for all Categories, including Available, Preview, and RDI. For inference submissions, the reproducibility must, at minimum, produce all the mandatory loadgen log files used for the submission. In addition, any software component that would be required to substantially reproduce the submission must be uniquely identified using one of the following methods: |=== From 66c673c3c1404587041c7c8583cb777f11d0443a Mon Sep 17 00:00:00 2001 From: Arjun Suresh Date: Wed, 14 Sep 2022 14:33:16 +0100 Subject: [PATCH 2/2] Required files mentioned in the reproduce section Required files of submission clearly mentioned in the reproduce section --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index d895e29..b7f9f38 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -528,7 +528,7 @@ For Inference, the results logs must have been produced by the [standard load ge The following section applies to all submissions in all divisions. -The source code must be sufficient to reproduce the results of the submission, given all source components specified in Section 5.11 (for Inference) or Section 5.12 (for Training) are provided in the submission repo for all Categories, including Available, Preview, and RDI. For inference submissions, the reproducibility must, at minimum, produce all the mandatory loadgen log files used for the submission. In addition, any software component that would be required to substantially reproduce the submission must be uniquely identified using one of the following methods: +The source code must be sufficient to reproduce all the required files of the submission, given all source components specified in Section 5.11 (for Inference) or Section 5.12 (for Training) are provided in the submission repo for all Categories, including Available, Preview, and RDI. In addition, any software component that would be required to substantially reproduce the submission must be uniquely identified using one of the following methods: |===