-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathtodo.txt
116 lines (104 loc) · 5.33 KB
/
todo.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
----------------------------------------------------------------------
Copyright 2007-2009 Alexey Radul.
----------------------------------------------------------------------
This file is part of Test Manager.
Test Manager is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Test Manager is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with Test Manager. If not, see <http://www.gnu.org/licenses/>.
----------------------------------------------------------------------
Add a general-purpose pattern matcher, or normal form converter, or
something (incl good handling of records) for use with check.
- A fail function that always fails (but emits its message, yes)
The quickcheck experience suggests the following feature: allow test
definitions to take arguments, thereby becoming properties; and allow
separate specification of examples to try them on (hand-crafted
examples, generators that make examples randomly, and maybe various
model checking or theorem proving machinery).
Api features
- Is it ok to write (run-test foo bar) inside of a surround definition
if the tests are all defined to take arguments? What use can I put
this to?
- I could use this for manually simulating something like quickcheck
- Implement by making the actual run-test take any set of args, and
applying the internal procedure.
- (in-test-group '(chain of subgroups) ...) could descend down the
whole chain in one swoop
- (define-deep-set-up ...) could define a setup to be run before each
test, no really, test not subgroup.
- Do the assertions want to be macros that include the actual code
of their input expressions in their messages?
- Should probably provide the procedural versions anyway, for, say, map
- Do I want to offer some find of (let-fixtures (binding ...)
test-definition ...) facility (that would allow mutation of the fixtures
internally, but restore them at every test)?
- A way to mark that certain tests or assertions are expected to fail?
Perhaps with keyword arguments to the appropriate functions? See
comment below.
- with-test-info macro (and function) adds extra information
to the failure display
Example failure display in SchemeUnit
--------------------
A test
... has a FAILURE
name: check-pred
location: (#<path:/Users/noel/programming/schematics/schemeunit/branches/v3/doc/file-test.scm> 14 6 252 22)
expression: (check-pred even? elt)
params: (#<procedure:even?> 9)
--------------------
On expected failures, Gremio says:
Sometimes we find that something is broken, and we don't know just
how to fix it at the time we find the bug, and would like to commit
a broken test case for it, but, well, it's a broken test case, and
that sucks. So we'd like the test case to show how it should look
when it's not broken, but also not to complain just because it is
broken. To wit, the following suggested optional arguments to
assert-foo:
#:fails
says that the test is supposed to fail
#:fails-with value
says that the test should fail, and in particular, instead of expecting
what it should expect, check that the wrong thing returned is the given
value (or matches it, etc., as per the assertion this is supplied to.
This is more interesting in the assert-throws world, and generally for
exceptions thrown from things that shouldn't -- should there be a
separate [#:fails-throwing exception]?)
#:fails-because "explanation"
like #:fails, just says that the test should fail, but also specifies a
human-readable explanation. If the test at some point no longer fails,
for example, or fails a different way than specified with #:fails-with,
then this might get displayed somehow.
Emacs interface:
- run-test-at-point
- with the group's setup and teardown
- run-test-group-at-point
- run-tests-in-file
- syntax-highlight define-test
- rerun-last-test-{suite/group/single test}
- run-last-defined-{test/suite/group} ?
Bring the Guile port up to feature parity?
Problems:
- The test-runner system is in *desperate* need of refactoring.
- Use SOS? Portability issues?
- Perhaps a function that returns a result object, from which
the results can then be printed?
- How much do I care about this extensibility thing?
I am beginning to realise what languages I am trying to create here.
The current main focus is a language for defining test suites. It
consists of assertions, tests, and test groups. There is an
interesting collection of thoughts to be thought about implicitness
and explicitness. For example, in-test-group arranges an implicit
change in the "current test group", which affects things like
define-test and possibly define-set-up, define-group-set-up, etc. In
contrast, there is also the explict set-tg:group-set-up! and their
ilk. The function (current-test-group) functions as a bridge between
the two worlds.
- Language for defining test suites
- Language for defining individual tests
- Language for running tests and getting results