author  Tom Henderson <tomh@tomh.org> 
Wed, 08 Apr 2009 16:07:34 0700  
changeset 4373  e493e80274bd 
parent 4308  b8528d30dfb3 
child 4326  179f86838e62 
permissions  rwrr 
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

1 
# python lib modules 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

2 
import os 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

3 
import sys 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

4 
import shutil 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

5 
import pproc as subprocess 
4308
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

6 
import errno 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

7 

4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

8 
# WAF modules 
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

9 
import Build 
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

10 
import Options 
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

11 
import Utils 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

12 
import Task 
4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

13 

10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

14 
# local modules 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

15 
import wutils 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

16 

4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

17 

3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

18 
def dev_null(): 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

19 
if sys.platform == 'win32': 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

20 
return open("NUL:", "w") 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

21 
else: 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

22 
return open("/dev/null", "w") 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

23 

9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

24 

9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

25 
def _find_tests(testdir): 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

26 
"""Return a list of test modules in the test directory 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

27 

9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

28 
Arguments: 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

29 
testdir  the directory to look in for tests 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

30 
""" 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

31 
names = os.listdir(testdir) 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

32 
tests = [] 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

33 
for name in names: 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

34 
if name[:5] == "test" and name[3:] == ".py": 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

35 
testname = name[:3] 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

36 
tests.append(testname) 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

37 
tests.sort() 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

38 
return tests 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

39 

4202
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

40 
def diff(dir1, dir2, verbose): 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

41 
import filecmp 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

42 
comp = filecmp.dircmp(dir1, dir2) 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

43 
differ = (comp.left_only or comp.right_only or comp.diff_files) 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

44 
if differ: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

45 
if verbose: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

46 
comp.report() 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

47 
import difflib 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

48 
for diff_fname in comp.diff_files: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

49 
if not (diff_fname.endswith(".tr") or diff_fname.endswith(".mob")): 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

50 
print "The different file %r does not sound like a text file, not compared." % (diff_fname,) 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

51 
diff_file1 = open(os.path.join(dir1, diff_fname), "rt").readlines() 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

52 
diff_file2 = open(os.path.join(dir2, diff_fname), "rt").readlines() 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

53 
diff = difflib.unified_diff(diff_file1, diff_file2) 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

54 
count = 0 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

55 
print "Differences in file %r" % (diff_fname,) 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

56 
for line in diff: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

57 
print line 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

58 
count += 1 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

59 
if count > 100: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

60 
break 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

61 
return 1 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

62 
else: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

63 
return 0 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

64 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

65 
class regression_test_task(Task.TaskBase): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

66 
after = 'cc cxx cc_link cxx_link' 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

67 
color = 'BLUE' 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

68 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

69 
def __init__(self, env, test_name, test_scripts_dir, build_traces_dir, reference_traces): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

70 
super(regression_test_task, self).__init__() 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

71 
self.env = env 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

72 
self.test_name = test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

73 
self.test_scripts_dir = test_scripts_dir 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

74 
self.build_traces_dir = build_traces_dir 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

75 
self.reference_traces_dir = reference_traces 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

76 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

77 
def __str__(self): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

78 
return 'regressiontest (%s)\n' % self.test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

79 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

80 
def run(self): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

81 
"""Run a single test""" 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

82 
sys.path.insert(0, self.test_scripts_dir) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

83 
try: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

84 
mod = __import__(self.test_name, globals(), locals(), []) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

85 
finally: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

86 
sys.path.remove(self.test_scripts_dir) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

87 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

88 
assert self.test_name.startswith('test') 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

89 
short_name = self.test_name[len('test'):] 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

90 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

91 
trace_dir_name = getattr(mod, "trace_dir_name", None) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

92 
if trace_dir_name is None: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

93 
trace_dir_name = "%s.ref" % short_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

94 
trace_output_path = os.path.join(self.build_traces_dir, trace_dir_name) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

95 
reference_traces_path = os.path.join(self.reference_traces_dir, trace_dir_name) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

96 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

97 
if hasattr(mod, 'get_arguments'): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

98 
arguments = mod.get_arguments(self.env, '..') 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

99 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

100 
arguments = getattr(mod, "arguments", []) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

101 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

102 
pyscript = getattr(mod, "pyscript", None) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

103 
if pyscript: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

104 
is_pyscript = True 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

105 
program = pyscript 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

106 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

107 
is_pyscript = False 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

108 
program = getattr(mod, "program", short_name) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

109 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

110 
if hasattr(mod, 'may_run'): 
4130
9bddd87225a9
Skip the NSC test when valgrind is enabled.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4129
diff
changeset

111 
reason_cannot_run = mod.may_run(self.env, Options.options) 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

112 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

113 
reason_cannot_run = None 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

114 
if reason_cannot_run: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

115 
print "SKIP %s (%s)" % (self.test_name, reason_cannot_run) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

116 
self.result = None 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

117 
return 0 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

118 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

119 
if Options.options.regression_generate: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

120 
# clean the target dir 
4308
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

121 
try: 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

122 
shutil.rmtree(trace_output_path) 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

123 
except OSError, ex: 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

124 
if ex.errno not in [errno.ENOENT]: 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

125 
raise 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

126 
os.makedirs(reference_traces_path) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

127 
result = self.run_reference_generate(reference_traces_path, program, arguments, is_pyscript) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

128 
if result == 0: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

129 
print "GENERATE " + self.test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

130 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

131 
print "GENERATE FAIL " + self.test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

132 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

133 
# clean the target dir 
4308
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

134 
try: 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

135 
shutil.rmtree(trace_output_path) 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

136 
except OSError, ex: 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

137 
if ex.errno not in [errno.ENOENT]: 
b8528d30dfb3
Less error hiding antipattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4202
diff
changeset

138 
raise 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

139 
os.makedirs(trace_output_path) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

140 
# run it 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

141 
result = self.run_reference_test(reference_traces_path, trace_output_path, program, arguments, is_pyscript) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

142 
if result == 0: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

143 
print "PASS " + self.test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

144 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

145 
print "FAIL " + self.test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

146 
self.result = result 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

147 
return 0 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

148 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

149 
def run_reference_test(self, reference_traces_path, trace_output_path, program, arguments, is_pyscript): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

150 
if not os.path.exists(reference_traces_path): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

151 
print "Cannot locate reference traces in " + reference_traces_path 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

152 
return 1 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

153 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

154 
if is_pyscript: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

155 
script = os.path.abspath(os.path.join('..', *os.path.split(program))) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

156 
argv = [self.env['PYTHON'], script] + arguments 
4129
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

157 
try: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

158 
wutils.run_argv(argv, cwd=trace_output_path) 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

159 
except Utils.WafError, ex: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

160 
print >> sys.stderr, ex 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

161 
return 1 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

162 
else: 
4129
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

163 
try: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

164 
wutils.run_program(program, 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

165 
command_template=wutils.get_command_template(self.env, arguments), 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

166 
cwd=trace_output_path) 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

167 
except Utils.WafError, ex: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

168 
print >> sys.stderr, ex 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

169 
return 1 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

170 

4202
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

171 
rc = diff(trace_output_path, reference_traces_path, Options.options.verbose) 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

172 
if rc: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

173 
print "" 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

174 
print "Traces differ in test: ", self.test_name 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

175 
print "Reference traces in directory: " + reference_traces_path 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

176 
print "Traces in directory: " + trace_output_path 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

177 
print "Run the following command for details:" 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

178 
print "\tdiff u %s %s" % (reference_traces_path, trace_output_path) 
4202
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

179 
if not Options.options.verbose: 
272cabf60878
For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4130
diff
changeset

180 
print "Or rerun regression testing with option v" 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

181 
print "" 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

182 
return rc 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

183 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

184 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

185 
def run_reference_generate(self, trace_output_path, program, arguments, is_pyscript): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

186 
if is_pyscript: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

187 
script = os.path.abspath(os.path.join('..', *os.path.split(program))) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

188 
argv = [self.env['PYTHON'], script] + arguments 
4129
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

189 
try: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

190 
retval = wutils.run_argv(argv, cwd=trace_output_path) 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

191 
except Utils.WafError, ex: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

192 
print >> sys.stderr, ex 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

193 
return 1 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

194 
else: 
4129
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

195 
try: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

196 
retval = wutils.run_program(program, 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

197 
command_template=wutils.get_command_template(self.env, arguments), 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

198 
cwd=trace_output_path) 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

199 
except Utils.WafError, ex: 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

200 
print >> sys.stderr, ex 
9b1f5d58dfb6
Interpret regression program nonzero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4119
diff
changeset

201 
return 1 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

202 
return retval 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

203 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

204 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

205 
class regression_test_collector_task(Task.TaskBase): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

206 
after = 'regression_test_task' 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

207 
color = 'BLUE' 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

208 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

209 
def __init__(self, test_tasks): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

210 
super(regression_test_collector_task, self).__init__() 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

211 
self.test_tasks = test_tasks 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

212 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

213 
def __str__(self): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

214 
return 'regressiontestcollector\n' 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

215 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

216 
def run(self): 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

217 
failed_tests = [test for test in self.test_tasks if test.result is not None and test.result != 0] 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

218 
skipped_tests = [test for test in self.test_tasks if test.result is None] 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

219 
print "Regression testing summary:" 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

220 
if skipped_tests: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

221 
print "SKIP: %i of %i tests have been skipped (%s)" % ( 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

222 
len(skipped_tests), len(self.test_tasks), 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

223 
', '.join([test.test_name for test in skipped_tests])) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

224 
if failed_tests: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

225 
print "FAIL: %i of %i tests have failed (%s)" % ( 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

226 
len(failed_tests), len(self.test_tasks), 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

227 
', '.join([test.test_name for test in failed_tests])) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

228 
return 1 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

229 
else: 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

230 
print "PASS: %i of %i tests passed" % (len(self.test_tasks)  len(skipped_tests), 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

231 
len(self.test_tasks)) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

232 
return 0 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

233 

a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

234 
def run_regression(bld, reference_traces): 
3872
8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

235 
"""Execute regression tests. Called with cwd set to the 'regression' subdir of ns3. 
8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

236 

4109
1a251c8ad317
Cleanup: remove regression reference traces downloading code (moved to allinone)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4064
diff
changeset

237 
@param reference_traces: reference traces directory. 
3872
8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

238 

8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

239 
""" 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

240 

4110
04170734fa8b
Don't abuse os.chdir so much (not thread safe)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4109
diff
changeset

241 
testdir = os.path.join("regression", "tests") 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

242 
if not os.path.exists(testdir): 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

243 
print "Tests directory does not exist" 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

244 
sys.exit(3) 
3872
8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

245 

4064
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

246 
if Options.options.regression_tests: 
10222f483860
Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3872
diff
changeset

247 
tests = Options.options.regression_tests.split(',') 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

248 
else: 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

249 
tests = _find_tests(testdir) 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

250 

3872
8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

251 
if not os.path.exists(reference_traces): 
8e757a83fb36
Add a withregressiontraces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
3866
diff
changeset

252 
print "Reference traces directory (%s) does not exist" % reference_traces 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

253 
return 3 
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

254 

4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

255 
test_scripts_dir = bld.path.find_dir('regression/tests').abspath() 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

256 
build_traces_dir = bld.path.find_or_declare('regression/traces').abspath(bld.env) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

257 
tasks = [] 
3866
9e946fee902c
Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff
changeset

258 
for test in tests: 
4114
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

259 
tasks.append(regression_test_task(bld.env, test, test_scripts_dir, build_traces_dir, reference_traces)) 
a6c5ccfa3451
Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
4110
diff
changeset

260 
regression_test_collector_task(tasks) 