|
sl@0
|
1 |
README -- Tcl test suite design document.
|
|
sl@0
|
2 |
|
|
sl@0
|
3 |
RCS: @(#) $Id: README,v 1.11.2.1 2003/04/01 21:13:07 dgp Exp $
|
|
sl@0
|
4 |
|
|
sl@0
|
5 |
Contents:
|
|
sl@0
|
6 |
---------
|
|
sl@0
|
7 |
|
|
sl@0
|
8 |
1. Introduction
|
|
sl@0
|
9 |
2. Running tests
|
|
sl@0
|
10 |
3. Adding tests
|
|
sl@0
|
11 |
4. Incompatibilities with prior Tcl versions
|
|
sl@0
|
12 |
|
|
sl@0
|
13 |
1. Introduction:
|
|
sl@0
|
14 |
----------------
|
|
sl@0
|
15 |
|
|
sl@0
|
16 |
This directory contains a set of validation tests for the Tcl commands
|
|
sl@0
|
17 |
and C Library procedures for Tcl. Each of the files whose name ends
|
|
sl@0
|
18 |
in ".test" is intended to fully exercise the functions in the C source
|
|
sl@0
|
19 |
file that corresponds to the file prefix. The C functions and/or Tcl
|
|
sl@0
|
20 |
commands tested by a given file are listed in the first line of the
|
|
sl@0
|
21 |
file.
|
|
sl@0
|
22 |
|
|
sl@0
|
23 |
2. Running tests:
|
|
sl@0
|
24 |
-----------------
|
|
sl@0
|
25 |
|
|
sl@0
|
26 |
We recommend that you use the "test" target of Tcl's Makefile to run
|
|
sl@0
|
27 |
the test suite. From the directory in which you build Tcl, simply
|
|
sl@0
|
28 |
type "make test". This will create a special executable named
|
|
sl@0
|
29 |
tcltest in which the testing scripts will be evaluated. To create
|
|
sl@0
|
30 |
the tcltest executable without running the test suite, simple type
|
|
sl@0
|
31 |
"make tcltest".
|
|
sl@0
|
32 |
|
|
sl@0
|
33 |
All the configuration options of the tcltest package are available
|
|
sl@0
|
34 |
during a "make test" by defining the TESTFLAGS environment variable.
|
|
sl@0
|
35 |
For example,if you wish to run only those tests in the file append.test,
|
|
sl@0
|
36 |
you can type:
|
|
sl@0
|
37 |
|
|
sl@0
|
38 |
make test TESTFLAGS="-file append.test"
|
|
sl@0
|
39 |
|
|
sl@0
|
40 |
For interactive testing, the Tcl Makefile provides the "runtest" target.
|
|
sl@0
|
41 |
Type "make runtest" in your build directory, and the tcltest executable
|
|
sl@0
|
42 |
will be created, if necessary, then it will run interactively. At the
|
|
sl@0
|
43 |
command prompt, you may type any Tcl commands. If you type
|
|
sl@0
|
44 |
"source ../tests/all.tcl", the test suite will run. You may use the
|
|
sl@0
|
45 |
tcltest::configure command to configure the test suite run as an
|
|
sl@0
|
46 |
alternative to command line options via TESTFLAGS. You might also
|
|
sl@0
|
47 |
wish to use the tcltest::testConstraint command to select the constraints
|
|
sl@0
|
48 |
that govern which tests are run. See the documentation for the tcltest
|
|
sl@0
|
49 |
package for details.
|
|
sl@0
|
50 |
|
|
sl@0
|
51 |
3. Adding tests:
|
|
sl@0
|
52 |
----------------
|
|
sl@0
|
53 |
|
|
sl@0
|
54 |
Please see the tcltest man page for more information regarding how to
|
|
sl@0
|
55 |
write and run tests.
|
|
sl@0
|
56 |
|
|
sl@0
|
57 |
Please note that the all.tcl file will source your new test file if
|
|
sl@0
|
58 |
the filename matches the tests/*.test pattern (as it should). The
|
|
sl@0
|
59 |
names of test files that contain regression (or glass-box) tests
|
|
sl@0
|
60 |
should correspond to the Tcl or C code file that they are testing.
|
|
sl@0
|
61 |
For example, the test file for the C file "tclCmdAH.c" is
|
|
sl@0
|
62 |
"cmdAH.test". Test files that contain black-box tests may not
|
|
sl@0
|
63 |
correspond to any Tcl or C code file so they should match the pattern
|
|
sl@0
|
64 |
"*_bb.test".
|
|
sl@0
|
65 |
|
|
sl@0
|
66 |
Be sure your new test file can be run from any working directory.
|
|
sl@0
|
67 |
|
|
sl@0
|
68 |
Be sure no temporary files are left behind by your test file.
|
|
sl@0
|
69 |
Use [tcltest::makeFile], [tcltest::removeFile], and [tcltest::cleanupTests]
|
|
sl@0
|
70 |
properly to be sure of this.
|
|
sl@0
|
71 |
|
|
sl@0
|
72 |
Be sure your tests can run cross-platform in both a build environment
|
|
sl@0
|
73 |
as well as an installation environment. If your test file contains
|
|
sl@0
|
74 |
tests that should not be run in one or more of those cases, please use
|
|
sl@0
|
75 |
the constraints mechanism to skip those tests.
|
|
sl@0
|
76 |
|
|
sl@0
|
77 |
4. Incompatibilities of package tcltest 2.1 with
|
|
sl@0
|
78 |
testing machinery of very old versions of Tcl:
|
|
sl@0
|
79 |
------------------------------------------------
|
|
sl@0
|
80 |
|
|
sl@0
|
81 |
1) Global variables such as VERBOSE, TESTS, and testConfig of the
|
|
sl@0
|
82 |
old machinery correspond to the [configure -verbose],
|
|
sl@0
|
83 |
[configure -match], and [testConstraint] commands of tcltest 2.1,
|
|
sl@0
|
84 |
respectively.
|
|
sl@0
|
85 |
|
|
sl@0
|
86 |
2) VERBOSE values were longer numeric. [configure -verbose] values
|
|
sl@0
|
87 |
are lists of keywords.
|
|
sl@0
|
88 |
|
|
sl@0
|
89 |
3) When you run "make test", the working dir for the test suite is now
|
|
sl@0
|
90 |
the one from which you called "make test", rather than the "tests"
|
|
sl@0
|
91 |
directory. This change allows for both unix and windows test
|
|
sl@0
|
92 |
suites to be run simultaneously without interference with each
|
|
sl@0
|
93 |
other or with existing files. All tests must now run independently
|
|
sl@0
|
94 |
of their working directory.
|
|
sl@0
|
95 |
|
|
sl@0
|
96 |
4) The "all" file is now called "all.tcl"
|
|
sl@0
|
97 |
|
|
sl@0
|
98 |
5) The "defs" and "defs.tcl" files no longer exist.
|
|
sl@0
|
99 |
|
|
sl@0
|
100 |
6) Instead of creating a doAllTests file in the tests directory, to
|
|
sl@0
|
101 |
run all nonPortable tests, just use the "-constraints nonPortable"
|
|
sl@0
|
102 |
command line flag. If you are running interactively, you can run
|
|
sl@0
|
103 |
[tcltest::testConstraint nonPortable 1] (after loading the tcltest
|
|
sl@0
|
104 |
package).
|
|
sl@0
|
105 |
|
|
sl@0
|
106 |
7) Direct evaluation of the *.test files by the "source" command is no
|
|
sl@0
|
107 |
longer recommended. Instead, "source all.tcl" and use the "-file" and
|
|
sl@0
|
108 |
"-notfile" options of tcltest::configure to control which *.test files
|
|
sl@0
|
109 |
are evaluated.
|