1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
|
Running regression tests:
-------------------------
Prerequisites: cmake, python, ImageMagick 6.5.9.3 or newer
First, get a working qmake GUI build. It is used by the tests.
Next, get MCAD installed by using 'git submodule update --init'
A) Building test environment
Linux, Mac:
$ cd tests
$ cmake .
$ make
Windows + MSVC:
From the QT command prompt:
> cd tests
> cmake . -DCMAKE_BUILD_TYPE=Release
> sed -i s/\/MD/\/MT/ CMakeCache.txt
> cmake .
> nmake -f Makefile
Cross compiling Linux->Win32:
Please see openscad/tests/CMingw-cross-env.cmake for instructions.
B) Running tests
$ ctest Runs tests enabled by default
$ ctest -R <regex> Runs only matching tests, e.g. ctest -R dxf
$ ctest -C <configs> Adds extended tests belonging to configs.
Valid configs:
Default - Run default tests
Heavy - Run more time consuming tests (> ~10 seconds)
Examples - test all examples
All - test everything
Adding a new regression test:
------------------------------
1) create a test file at an appropriate location under testdata/
2) if the test is non-obvious, create a human readable description as comments in the test (or in another file in the same directory in case the file isn't human readable)
3) if a new test app was written, this must be added to tests/CMakeLists.txt
4) Add the tests to the test apps for which you want them to run (in tests/CMakeLists.txt)
5) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -R mytest
(this will generate a mytest-expected.txt file which is used for regression testing)
6) manually verify that the output is correct (tests/regression/<testapp>/mytest-expected.<suffix>)
7) run the test normally and verify that it passes:
$ ctest -R mytest
Adding a new example:
---------------------
This is almost the same as adding a new regression test:
1) Create the example under examples/
2) run the test with the environment variable TEST_GENERATE=1, e.g.:
$ TEST_GENERATE=1 ctest -C Examples -R exampleNNN
(this will generate a exampleNNN-expected.txt file which is used for regression testing)
3) manually verify that the output is correct (tests/regression/<testapp>/exampleNNN.<suffix>)
4) run the test normally and verify that it passes:
$ ctest -C Examples -R exampleNNN
Migration away from dedicated regression tests:
-----------------------------------------------
As the main OpenSCAD binary gained the capability to output PNG renderings,
the necessity for dedicated test binaries went away; for many test, OpenSCAD
is now directly called, resulting in better test coverage and less code to
maintain.
So far, the following tests have been fully converted to use the main binary:
* moduledumptest
* csgtermtest
* cgalpngtest
* opencsgtest
Two tests still need an intermediate script that mangles away timestamps and
near-zero floating point numbers:
* dumptest
* echotest
Some tests are yet to be converted:
* csgtexttest -- verify whether this is not redundant with dumptest
* throwntogethertest -- should be a .png output mode
* cgalstlsanitytest -- could probably be replaced by a shell script grepping
for nan and possibly counting faces
These look like tests, but are not actually in use:
* modulecachetest
* cgalcachetest
In the course of migration, the possibilities of using the OPENSCAD_TESTING
compile time flag go away; that flag used to strip timestamps out of files,
to unify float output, and to tweak the file inclusion paths. With that flag
unused, we get better coverage of the primary code paths, at the cost of
having to do some normalization in the unit testing process (thus the dumptest
and echo test scripts). Especially, having a nonstandard MCAD library in the
user include path can now break things -- but so can having misbehaving
programs in your PATH.
Troubleshooting:
------------------------------
0. Headless unix servers
If you are attempting to run the tests on a unix-like system but only
have shell-console access, you may be able to run the tests by using a
virtual framebuffer program like Xvnc or Xvfb. For example:
$ Xvfb :5 -screen 0 800x600x24 &
$ DISPLAY=:5 ctest
or
$ xvfb-run ctest
Some versions of Xvfb may fail, however.
1. Trouble finding libraries on unix
To help CMAKE find eigen, OpenCSG, CGAL, Boost, and GLEW, you can use
environment variables, just like for the main qmake & openscad.pro. Examples:
OPENSCAD_LIBRARIES=$HOME cmake .
CGALDIR=$HOME/CGAL-3.9 BOOSTDIR=$HOME/boost-1.47.0 cmake .
Valid variables are as follows:
BOOSTDIR, CGALDIR, EIGENDIR, GLEWDIR, OPENCSGDIR, OPENSCAD_LIBRARIES
When running, this might help find your locally built libraries (assuming
you installed into $HOME)
Linux: export LD_LIBRARY_PATH=$HOME/lib:$HOME/lib64
Mac: export DYLD_LIBRARY_PATH=$HOME/lib
2. Location of logs
Logs of test runs are found in tests/build/Testing/Temporary
A pretty-printed index.html is in a subdir of tests/build/Testing/Temporary
Expected results are found in tests/regression/*
Actual results are found in tests/build/testname-output/*
3. Image-based tests takes a long time, they fail, and the log says 'return -11'
Imagemagick may have crashed while comparing the expected images to the
test-run generated (actual) images. You can try using the alternate
ImageMagick comparison method by by erasing CMakeCache, and re-running
cmake with -DCOMPARATOR=ncc. This will enable the Normalized Cross
Comparison method.
4. Testing images fails with 'morphology not found" for ImageMagick in the log
Your version of imagemagick is old. Upgrade, or pass -DCOMPARATOR=old to
cmake. The comparison will be of lowered reliability.
5. Locale errors
"terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid"
Is a boost/libstdc++ bug. Fix like so:
$ export LC_MESSAGES=
6. Other issues
The OpenSCAD User Manual has a section on buildling. Please check there
for updates:
http://en.wikibooks.org/wiki/OpenSCAD_User_Manual
|
