-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathre.html
355 lines (346 loc) · 20.8 KB
/
re.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
<!DOCTYPE html>
<html>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>HSCC 2020 - Repeatability Evaluation</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<link rel="stylesheet" href="styling.css" type="text/css" media="screen" />
</html>
<body>
<main>
<h1>HSCC 2020 - Repeatability Evaluation</h1>
<h3> Submission Guidelines</h3>
<p>The Repeatability Evaluation Package (REP) consists of three components:</p>
<ul>
<li>A <span style="font-weight: bold;">copy</span> (in pdf
format) of the submitted paper. This copy will
be used by the REC to evaluate how well the elements of the REP
match the paper. </li>
<li>A <span style="font-weight: bold;">document</span> (either
a webpage, a pdf, or a plain text file) explaining at a
minimum:</li>
<ul>
<li>What elements of the paper are included in the REP (e.g.:
specific figures, tables, etc.).</li>
<li>The system requirements for running the REP (e.g.: OS,
compilers, environments, etc.).</li>
<li>Instructions for installing and running the software and
extracting the corresponding results. </li>
</ul>
</ul>
<ul>
<li>The <span style="font-weight: bold;">software</span> and <span style="font-weight: bold;">any
accompanying data</span>. We will accept at least the
following formats:</li>
<ul>
<li>A link to a public online repository, such as
bitbucket.org, github.com, or
sourceforge.net. </li>
<li>An archive in a standard file format (e.g.: zip, gz, tgz)
containing all the necessary components. </li>
<li>A link to a virtual machine image (using either VirtualBox
or VMware) which can be downloaded.</li>
</ul>
</ul>
If you would like to submit software and/or data in another
format, please contact the RE committee chair in advance to
discuss options. <br>
<br>
The RP submission website: <a href="https://easychair.org/conferences/?conf=hscc20re
">RE submission link</a>.
<br>
<br>
When preparing your RP, keep in mind
that other conferences have reported that the most common reason
for reproducibility failure is installation problems. We
recommend that you have an independent member of your lab test
your installation instructions and REP on a clean machine before
final submission.<br>
<br>
The repeatability evaluation process uses anonymous reviews so as
to solicit honest feedback. Authors of REPs should make a
genuine effort to avoid learning the identity of the
reviewers. This effort may require turning off analytics or
only using systems with high enough traffic that REC accesses will
not be apparent. In all cases where tracing is unavoidable
the authors should provide warnings in the documentation so that
reviewers can take necessary precautions to maintain anonymity. <br>
<br>
REPs are considered confidential material in the same sense as initial
paper submissions: committee members agree not to share REP contents
and to delete them after evaluation. REPs remain the property of
the authors, and there is no requirement to post them publicly
(although we encourage you to do so).
<br>
<br>
<h5 style= "display: inline";>Note for tool/case study paper authors on double-blind process: </h5>
Tool and case study paper authors are strongly encouraged to submit a
repeatability package due by October 28th. While we expect the tool/case study papers submitted
to the main conference to follow the double-blind instructions, the RE packages do not need
to be anonymized. Therefore please remove any links to a repository that could reveal your
identity from the paper but include these in your RE package submission. RE packages and papers
will be evaluated by different committees. RE committee's findings will be communicated to
the regular reviewers of the paper before the final decisions are made.
<br>
<br>
<h5 style= "display: inline";>Second round of repeatability evaluation: </h5>
Authors of the accepted papers will be encouraged to submit a
repeatability package due early January. Among the RE packages submitted, the papers, results of
which are deemed repeatable will receive a repeatability badge that will be included
on the first page of the published version. These papers will also be highlighted on
the conference website. The timeline for the first round is as follows:
<ol>
<li style="display: flex; flex-direction: row; justify-content: space-between; flex-wrap: wrap;">
RE package submission for tool/case study papers:
<span style="color: #1583E8;" class="blue">October 28th, 2019, AOE</span>
</li>
<li style="display: flex; flex-direction: row; justify-content: space-between; flex-wrap: wrap;">
RE notification for tool/case study papers:
<span style="color: #1583E8;" class="blue">December 23rd, 2019, AOE (tentative)</span>
</li>
</ol>
<h3><span style="font-weight: bold;">Background and Goals <br></span></h3>
HSCC has a rich history of publishing strong papers emphasizing
computational contributions; however, subsequent re-creation of
these computational elements is often challenging because details
of the implementation are unavoidably absent in the paper. Some
authors post their code and data to their websites, but there is
little formal incentive to do so and no easy way to determine
whether others can actually use the result. As a
consequence, computational results often become non reproducible
-- even by the research group which originally produced them --
after just a few years. <br>
<br>
The goal of the HSCC repeatability evaluation process is to
improve the reproducibility of computational results in the papers
selected for the conference. <br>
<h3><span style="font-weight: bold;">Benefits for Authors</span></h3>
We hope that this process will provide the following benefits to
authors: <br>
<ul>
<li>Raise the profile of papers containing repeatable
computational results by highlighting them at the conference
and online.</li>
<li>Raise the profile of HSCC as a whole, by making it easier to
build upon the published results.</li>
<li>Provide authors with an incentive to adopt best-practices
for code and data management that are known to improve the
quality and extendability of computational results.</li>
<li>Provide authors an opportunity to receive feedback from
independent reviewers about whether their computational
results can be repeated.</li>
<li>Obtain a special mention in the conference proceedings, and
take part in the competition for the best RE award.<strong style="font-weight: bold;"><br>
</strong></li>
</ul>
<br>
While creating a repeatability package will require some work from
the authors, we believe the cost of that extra work is outweighed
by a direct benefit to members of the authors' research lab: if an
independent reviewer can replicate the results with a minimum of
effort, it is much more likely that future members of the lab will
also be able to do so, even if the primary author has departed.<br>
<br>
The repeatability evaluation process for HSCC draws upon several
similar efforts at other conferences (SIGMOD, SAS, CAV, ECOOP,
OOPSLA), and <a target="_blank" href="https://sites.google.com/site/hscc2014repeatability/home">a
first experimental run was held at HSCC14</a>. <br>
<h3><span style="font-weight: bold;">Repeatability Evaluation
Criteria</span></h3>
Each member of the Repeatability Evaluation Committee assigned to
review a Repeatability Package (RP) will judge it based on three
criteria -- <span style="font-weight: bold;">coverage,
instructions, and quality</span> -- where each criteria is
assessed on the following scale:<br>
<ul>
<li>significantly exceeds expectations (5),</li>
<li>exceeds expectations (4),</li>
<li>meets expectations (3),</li>
<li>falls below expectations (2),</li>
<li>missing or significantly falls below expectations (1).</li>
</ul>
In order to be judged "repeatable" an RP must "meet expectations"
(average score of 3), and must not have any missing elements (no
scores of 1). Each RP is evaluated independently according
to the objective criteria. The higher scores ("exceeds" or
"significantly exceeds expectations") in the criteria should be
considered aspirational goals, not requirements for acceptance. <br>
<h4>Coverage</h4>
What fraction of the appropriate figures and tables are reproduced
by the RP? Note that some figures and tables should not be
included in this calculation; for example, figures generated in a
drawing program, or tables listing only parameter values.
The focus is on those figures or tables in the paper containing
computationally generated or processed experimental evidence to
support the claims of the paper.<br>
<br>
Note that satisfying this criterion does <span style="font-weight: bold;">not</span>
require that the corresponding figures or tables be recreated in
exactly the same format as appears in the paper, merely that the
data underlying those figures or tables be generated in a
recognizable format.<br>
<br>
A <span style="font-weight: bold;">repeatable</span> element is
one for which the computation can be rerun by following the
instructions in the RP in a suitably equipped environment.
An <span style="font-weight: bold;">extensible</span> element is
one for which variations of the original computation can be run by
modifying elements of the code and/or data. Consequently,
necessary conditions for extensibility include that the modifiable
elements be identified in the instructions or documentation, and
that all source code must be available and/or involve calls to
commonly available and trusted software (eg: Windows, Linux, C or
Python standard libraries, Matlab, etc.).<br>
<br>
The categories for this criterion are:<br>
<ul>
<li><span style="font-weight: bold;">None (missing / 1)</span>:
There are no repeatable elements. This case
automatically applies to papers which do not submit a RP or
papers which contain no computational elements.</li>
<li><span style="font-weight: bold;">Some (falls below
expectations / 2)</span>: There is at least one repeatable
element.</li>
<li><span style="font-weight: bold;">Most (meets expectations /
3)</span>: The majority (at least half) of the elements are
repeatable.</li>
<li><span style="font-weight: bold;">All repeatable or most
extensible (exceeds expectations / 4)</span>: All elements
are repeatable or most are repeatable and easily
modified. Note that if there is only one computational
element and it is repeatable, then this score should be
awarded.</li>
<li><span style="font-weight: bold;">All extensible
(significantly exceeds expectations / 5)</span>: All
elements are repeatable and easily modified.</li>
</ul>
<h4>Instructions</h4>
<p>This criterion is focused on the instructions which will allow
another user to recreate the computational results from the
paper. </p>
<ul>
<li><span style="font-weight: bold;">None (missing / 1)</span>:
No instructions were included in the RP.</li>
<li><span style="font-weight: bold;">Rudimentary (falls below
expectations / 2)</span>: The instructions specify a script
or command to run, but little else.</li>
<li><span style="font-weight: bold;">Complete (meets
expectations / 3)</span>: For every computational element
that is repeatable, there is a specific instruction which
explains how to repeat it. The environment under which
the software was originally run is described.</li>
<li><span style="font-weight: bold;">Comprehensive (exceeds
expectations / 4)</span>: For every computational element
that is repeatable there is a single command which recreates
that element almost exactly as it appears in the published
paper (eg: file format, fonts, line styles, etc. might not be
the same, but the content of the element is the same).
In addition to identifying the specific environment under
which the software was originally run, a broader class of
environments is identified under which it could run.</li>
<li><span style="font-weight: bold;">Outstanding (significantly
exceeds expectations / 5)</span>: In addition to the
criteria for a comprehensive set of instructions, explanations
are provided of: </li>
<ul>
<li>all the major components / modules in the software,</li>
<li>important design decisions made during implementation,</li>
<li>how to modify / extend the software, and/or</li>
<li>what environments / modifications would break the
software.</li>
</ul>
</ul>
<h4>Quality</h4>
<p>This criterion explores the documentation and trustworthiness
of the software and its results. While a set of scripts
which exactly recreate, for example, the figures from the paper
certainly aid in repeatability, without well-documented code it
is hard to understand how the data in that figure were
processed, without well-documented data it is hard to determine
whether the input is correct, and without testing it is hard to
determine whether the results can be trusted.<br>
<br>
If there are tests in the RP which are not included in the
paper, they should at least be mentioned in the instructions
document. Documentation of test details can be put into
the instructions document or into a separate document in the RP.<br>
<br>
The categories for this criterion are:
</p>
<ul>
<li><span style="font-weight: bold;">None (missing / 1)</span>:
There is no evidence of documentation or testing.</li>
<li><span style="font-weight: bold;">Rudimentary documentation
(falls below expectations / 2)</span>: The purpose of almost
all files is documented (preferably within the file, but
otherwise in the instructions or a separate readme file).</li>
<li><span style="font-weight: bold;">Comprehensive documentation
(meets expectations / 3)</span>: The purpose of almost all
files is documented. Within source code files, almost
all classes, methods, attributes and variables are given
lengthy clear names and/or documentation of their
purpose. Within data files, the format and structure of
the data is documented; for example, in comma separated value
(csv) files there is a header row and/or comments explaining
the contents of each column.</li>
<li><span style="font-weight: bold;">Comprehensive documentation
and rudimentary testing (exceeds expectations / 4)</span>:
In addition to the criteria for comprehensive documentation,
there are identified test cases with known solutions which can
be run to validate at least some components of the code.</li>
<li><span style="font-weight: bold;">Comprehensive documentation
and testing (significantly exceeds expectations / 5)</span></li>
<li>In addition to the criteria for comprehensive documentation,
there are clearly identified unit tests (preferably run with a
unit test framework) which exercise a significant fraction of
the smaller components of the code (individual functions and
classes) and system level tests which exercise a significant
fraction of the full package. Unit tests are typically
self-documenting, but the system level tests will require
documentation of at least the source of the known solution.</li>
</ul>
Note that tests are a form of documentation, so it is not really
possible to have testing without documentation.<br>
<h3><br>
<span style="font-weight: bold;"></span></h3>
<h3><span style="font-weight: bold;">Sample Repeatability Package</span></h3>
<ul>
<li>Citation: Ian M. Mitchell, "Scalable calculation of reach
sets and tubes for nonlinear systems with terminal
integrators: a mixed implicit explicit formulation" in Hybrid
Systems Computation and Control, pp. 103-112 (2011).</li>
<li>Official version: <a target="_blank" href="http://dx.doi.org/10.1145/1967701.1967718">http://dx.doi.org/10.1145/1967701.1967718</a></li>
<li>Author postprint: <a target="_blank" href=" https://www.cs.ubc.ca/~mitchell/Papers/myHSCC11.pdf"> https://www.cs.ubc.ca/~mitchell/Papers/myHSCC11.pdf</a></li>
<li>Repeatability package: <a target="_blank" href="https://www.cs.ubc.ca/~mitchell/ToolboxLS/PublicationCode/hscc2011.zip">https://www.cs.ubc.ca/~mitchell/ToolboxLS/PublicationCode/hscc2011.zip</a></li>
<li>Repeatability Evaluation: meets or exceeds expectations
(average 3⅓).</li>
<ul>
<li><span style="font-weight: bold;">Coverage</span>: all
repeatable (exceeds expectations / 4). Code to
recreate figures 1-5 and 7-8 is provided. Figure 6 is
a hand-drawn coordinate system. There are no tables.</li>
<li><span style="font-weight: bold;">Instructions</span>:
complete (meets expectations / 3). The included
readme.txt file lists which m-files are used to recreate
which figures. The environment is described (Matlab
R2010b or later, link to the Toolbox of Level Set
Methods). However, some effort is required to extract
certain figures (eg: figures 7 & 8).</li>
<li><span style="font-weight: bold;">Quality</span>:
comprehensive documentation (meets expectations / 3).
All source files include Matlab help entries for every
function as well as numerous comments. There are no
data files. However, there is no sign of testing.</li>
</ul>
</ul>
<div style="text-align: right;"><br>
<br>
[Thanks to Ian M. Mitchell for the content of this page]<br>
<span style="font-weight: bold;"></span>
</div>
<span style="font-weight: bold;"> </span>
</br>
<footer>
<p>© Copyright 2014-20 HSCC</p>
</footer>
</main>
</body>