author | Mathieu Lacage <mathieu.lacage@sophia.inria.fr> |
Mon, 23 Mar 2009 09:56:27 +0100 | |
changeset 4359 | 8d689bf4405f |
parent 4202 | 272cabf60878 |
child 4308 | b8528d30dfb3 |
permissions | -rw-r--r-- |
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
1 |
# python lib modules |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
2 |
import os |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
3 |
import sys |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
4 |
import shutil |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
5 |
import pproc as subprocess |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
6 |
import urllib |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
7 |
|
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
8 |
# WAF modules |
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
9 |
import Build |
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
10 |
import Options |
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
11 |
import Utils |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
12 |
import Task |
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
13 |
|
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
14 |
# local modules |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
15 |
import wutils |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
16 |
|
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
17 |
|
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
18 |
def dev_null(): |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
19 |
if sys.platform == 'win32': |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
20 |
return open("NUL:", "w") |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
21 |
else: |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
22 |
return open("/dev/null", "w") |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
23 |
|
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
24 |
|
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
25 |
def _find_tests(testdir): |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
26 |
"""Return a list of test modules in the test directory |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
27 |
|
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
28 |
Arguments: |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
29 |
testdir -- the directory to look in for tests |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
30 |
""" |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
31 |
names = os.listdir(testdir) |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
32 |
tests = [] |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
33 |
for name in names: |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
34 |
if name[:5] == "test-" and name[-3:] == ".py": |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
35 |
testname = name[:-3] |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
36 |
tests.append(testname) |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
37 |
tests.sort() |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
38 |
return tests |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
39 |
|
4202
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
40 |
def diff(dir1, dir2, verbose): |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
41 |
import filecmp |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
42 |
comp = filecmp.dircmp(dir1, dir2) |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
43 |
differ = (comp.left_only or comp.right_only or comp.diff_files) |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
44 |
if differ: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
45 |
if verbose: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
46 |
comp.report() |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
47 |
import difflib |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
48 |
for diff_fname in comp.diff_files: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
49 |
if not (diff_fname.endswith(".tr") or diff_fname.endswith(".mob")): |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
50 |
print "The different file %r does not sound like a text file, not compared." % (diff_fname,) |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
51 |
diff_file1 = open(os.path.join(dir1, diff_fname), "rt").readlines() |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
52 |
diff_file2 = open(os.path.join(dir2, diff_fname), "rt").readlines() |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
53 |
diff = difflib.unified_diff(diff_file1, diff_file2) |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
54 |
count = 0 |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
55 |
print "Differences in file %r" % (diff_fname,) |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
56 |
for line in diff: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
57 |
print line |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
58 |
count += 1 |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
59 |
if count > 100: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
60 |
break |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
61 |
return 1 |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
62 |
else: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
63 |
return 0 |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
64 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
65 |
class regression_test_task(Task.TaskBase): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
66 |
after = 'cc cxx cc_link cxx_link' |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
67 |
color = 'BLUE' |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
68 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
69 |
def __init__(self, env, test_name, test_scripts_dir, build_traces_dir, reference_traces): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
70 |
super(regression_test_task, self).__init__() |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
71 |
self.env = env |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
72 |
self.test_name = test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
73 |
self.test_scripts_dir = test_scripts_dir |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
74 |
self.build_traces_dir = build_traces_dir |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
75 |
self.reference_traces_dir = reference_traces |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
76 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
77 |
def __str__(self): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
78 |
return 'regression-test (%s)\n' % self.test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
79 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
80 |
def run(self): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
81 |
"""Run a single test""" |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
82 |
sys.path.insert(0, self.test_scripts_dir) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
83 |
try: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
84 |
mod = __import__(self.test_name, globals(), locals(), []) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
85 |
finally: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
86 |
sys.path.remove(self.test_scripts_dir) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
87 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
88 |
assert self.test_name.startswith('test-') |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
89 |
short_name = self.test_name[len('test-'):] |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
90 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
91 |
trace_dir_name = getattr(mod, "trace_dir_name", None) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
92 |
if trace_dir_name is None: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
93 |
trace_dir_name = "%s.ref" % short_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
94 |
trace_output_path = os.path.join(self.build_traces_dir, trace_dir_name) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
95 |
reference_traces_path = os.path.join(self.reference_traces_dir, trace_dir_name) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
96 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
97 |
if hasattr(mod, 'get_arguments'): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
98 |
arguments = mod.get_arguments(self.env, '..') |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
99 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
100 |
arguments = getattr(mod, "arguments", []) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
101 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
102 |
pyscript = getattr(mod, "pyscript", None) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
103 |
if pyscript: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
104 |
is_pyscript = True |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
105 |
program = pyscript |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
106 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
107 |
is_pyscript = False |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
108 |
program = getattr(mod, "program", short_name) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
109 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
110 |
if hasattr(mod, 'may_run'): |
4130
9bddd87225a9
Skip the NSC test when valgrind is enabled.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4129
diff
changeset
|
111 |
reason_cannot_run = mod.may_run(self.env, Options.options) |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
112 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
113 |
reason_cannot_run = None |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
114 |
if reason_cannot_run: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
115 |
print "SKIP %s (%s)" % (self.test_name, reason_cannot_run) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
116 |
self.result = None |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
117 |
return 0 |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
118 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
119 |
if Options.options.regression_generate: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
120 |
# clean the target dir |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
121 |
shutil.rmtree(reference_traces_path, ignore_errors=True) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
122 |
os.makedirs(reference_traces_path) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
123 |
result = self.run_reference_generate(reference_traces_path, program, arguments, is_pyscript) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
124 |
if result == 0: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
125 |
print "GENERATE " + self.test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
126 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
127 |
print "GENERATE FAIL " + self.test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
128 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
129 |
# clean the target dir |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
130 |
shutil.rmtree(trace_output_path, ignore_errors=True) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
131 |
os.makedirs(trace_output_path) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
132 |
# run it |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
133 |
result = self.run_reference_test(reference_traces_path, trace_output_path, program, arguments, is_pyscript) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
134 |
if result == 0: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
135 |
print "PASS " + self.test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
136 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
137 |
print "FAIL " + self.test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
138 |
self.result = result |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
139 |
return 0 |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
140 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
141 |
def run_reference_test(self, reference_traces_path, trace_output_path, program, arguments, is_pyscript): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
142 |
if not os.path.exists(reference_traces_path): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
143 |
print "Cannot locate reference traces in " + reference_traces_path |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
144 |
return 1 |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
145 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
146 |
if is_pyscript: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
147 |
script = os.path.abspath(os.path.join('..', *os.path.split(program))) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
148 |
argv = [self.env['PYTHON'], script] + arguments |
4129
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
149 |
try: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
150 |
wutils.run_argv(argv, cwd=trace_output_path) |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
151 |
except Utils.WafError, ex: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
152 |
print >> sys.stderr, ex |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
153 |
return 1 |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
154 |
else: |
4129
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
155 |
try: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
156 |
wutils.run_program(program, |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
157 |
command_template=wutils.get_command_template(self.env, arguments), |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
158 |
cwd=trace_output_path) |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
159 |
except Utils.WafError, ex: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
160 |
print >> sys.stderr, ex |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
161 |
return 1 |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
162 |
|
4202
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
163 |
rc = diff(trace_output_path, reference_traces_path, Options.options.verbose) |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
164 |
if rc: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
165 |
print "----------" |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
166 |
print "Traces differ in test: ", self.test_name |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
167 |
print "Reference traces in directory: " + reference_traces_path |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
168 |
print "Traces in directory: " + trace_output_path |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
169 |
print "Run the following command for details:" |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
170 |
print "\tdiff -u %s %s" % (reference_traces_path, trace_output_path) |
4202
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
171 |
if not Options.options.verbose: |
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset
|
172 |
print "Or re-run regression testing with option -v" |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
173 |
print "----------" |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
174 |
return rc |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
175 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
176 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
177 |
def run_reference_generate(self, trace_output_path, program, arguments, is_pyscript): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
178 |
if is_pyscript: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
179 |
script = os.path.abspath(os.path.join('..', *os.path.split(program))) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
180 |
argv = [self.env['PYTHON'], script] + arguments |
4129
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
181 |
try: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
182 |
retval = wutils.run_argv(argv, cwd=trace_output_path) |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
183 |
except Utils.WafError, ex: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
184 |
print >> sys.stderr, ex |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
185 |
return 1 |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
186 |
else: |
4129
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
187 |
try: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
188 |
retval = wutils.run_program(program, |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
189 |
command_template=wutils.get_command_template(self.env, arguments), |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
190 |
cwd=trace_output_path) |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
191 |
except Utils.WafError, ex: |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
192 |
print >> sys.stderr, ex |
9b1f5d58dfb6
Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset
|
193 |
return 1 |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
194 |
return retval |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
195 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
196 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
197 |
class regression_test_collector_task(Task.TaskBase): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
198 |
after = 'regression_test_task' |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
199 |
color = 'BLUE' |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
200 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
201 |
def __init__(self, test_tasks): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
202 |
super(regression_test_collector_task, self).__init__() |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
203 |
self.test_tasks = test_tasks |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
204 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
205 |
def __str__(self): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
206 |
return 'regression-test-collector\n' |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
207 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
208 |
def run(self): |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
209 |
failed_tests = [test for test in self.test_tasks if test.result is not None and test.result != 0] |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
210 |
skipped_tests = [test for test in self.test_tasks if test.result is None] |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
211 |
print "Regression testing summary:" |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
212 |
if skipped_tests: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
213 |
print "SKIP: %i of %i tests have been skipped (%s)" % ( |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
214 |
len(skipped_tests), len(self.test_tasks), |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
215 |
', '.join([test.test_name for test in skipped_tests])) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
216 |
if failed_tests: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
217 |
print "FAIL: %i of %i tests have failed (%s)" % ( |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
218 |
len(failed_tests), len(self.test_tasks), |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
219 |
', '.join([test.test_name for test in failed_tests])) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
220 |
return 1 |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
221 |
else: |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
222 |
print "PASS: %i of %i tests passed" % (len(self.test_tasks) - len(skipped_tests), |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
223 |
len(self.test_tasks)) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
224 |
return 0 |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
225 |
|
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
226 |
def run_regression(bld, reference_traces): |
3872
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
227 |
"""Execute regression tests. Called with cwd set to the 'regression' subdir of ns-3. |
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
228 |
|
4109
1a251c8ad317
Cleanup: remove regression reference traces downloading code (moved to allinone)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4064
diff
changeset
|
229 |
@param reference_traces: reference traces directory. |
3872
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
230 |
|
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
231 |
""" |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
232 |
|
4110
04170734fa8b
Don't abuse os.chdir so much (not thread safe)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4109
diff
changeset
|
233 |
testdir = os.path.join("regression", "tests") |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
234 |
if not os.path.exists(testdir): |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
235 |
print "Tests directory does not exist" |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
236 |
sys.exit(3) |
3872
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
237 |
|
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
238 |
if Options.options.regression_tests: |
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset
|
239 |
tests = Options.options.regression_tests.split(',') |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
240 |
else: |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
241 |
tests = _find_tests(testdir) |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
242 |
|
3872
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
243 |
if not os.path.exists(reference_traces): |
8e757a83fb36
Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset
|
244 |
print "Reference traces directory (%s) does not exist" % reference_traces |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
245 |
return 3 |
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
246 |
|
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
247 |
test_scripts_dir = bld.path.find_dir('regression/tests').abspath() |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
248 |
build_traces_dir = bld.path.find_or_declare('regression/traces').abspath(bld.env) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
249 |
tasks = [] |
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset
|
250 |
for test in tests: |
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
251 |
tasks.append(regression_test_task(bld.env, test, test_scripts_dir, build_traces_dir, reference_traces)) |
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset
|
252 |
regression_test_collector_task(tasks) |