regression.py
author Tom Henderson <tomh@tomh.org>
Wed, 08 Apr 2009 16:07:34 -0700
changeset 4373 e493e80274bd
parent 4308 b8528d30dfb3
child 4326 179f86838e62
permissions -rw-r--r--
implementation and plumbing of Ipv4InterfaceAddress class
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
4064
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
     1
# python lib modules
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     2
import os
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     3
import sys
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     4
import shutil
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     5
import pproc as subprocess
4308
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
     6
import errno
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     7
4064
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
     8
# WAF modules
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
     9
import Build
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
    10
import Options
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
    11
import Utils
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    12
import Task
4064
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
    13
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
    14
# local modules
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    15
import wutils
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    16
4064
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
    17
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    18
def dev_null():
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    19
    if sys.platform == 'win32':
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    20
        return open("NUL:", "w")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    21
    else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    22
        return open("/dev/null", "w")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    23
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    24
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    25
def _find_tests(testdir):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    26
    """Return a list of test modules in the test directory
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    27
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    28
    Arguments:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    29
    testdir -- the directory to look in for tests
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    30
    """
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    31
    names = os.listdir(testdir)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    32
    tests = []
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    33
    for name in names:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    34
        if name[:5] == "test-" and name[-3:] == ".py":
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    35
            testname = name[:-3]
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    36
            tests.append(testname)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    37
    tests.sort()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    38
    return tests
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    39
4202
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    40
def diff(dir1, dir2, verbose):
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    41
    import filecmp
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    42
    comp = filecmp.dircmp(dir1, dir2)
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    43
    differ = (comp.left_only or comp.right_only or comp.diff_files)
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    44
    if differ:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    45
        if verbose:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    46
            comp.report()
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    47
            import difflib
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    48
            for diff_fname in comp.diff_files:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    49
                if not (diff_fname.endswith(".tr") or diff_fname.endswith(".mob")):
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    50
                    print "The different file %r does not sound like a text file, not compared." % (diff_fname,)
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    51
                diff_file1 = open(os.path.join(dir1, diff_fname), "rt").readlines()
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    52
                diff_file2 = open(os.path.join(dir2, diff_fname), "rt").readlines()
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    53
                diff = difflib.unified_diff(diff_file1, diff_file2)
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    54
                count = 0
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    55
                print "Differences in file %r" % (diff_fname,)
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    56
                for line in diff:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    57
                    print line
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    58
                    count += 1
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    59
                    if count > 100:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    60
                        break
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    61
        return 1
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    62
    else:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
    63
        return 0
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    64
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    65
class regression_test_task(Task.TaskBase):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    66
    after = 'cc cxx cc_link cxx_link'
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    67
    color = 'BLUE'
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    68
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    69
    def __init__(self, env, test_name, test_scripts_dir, build_traces_dir, reference_traces):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    70
        super(regression_test_task, self).__init__()
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    71
        self.env = env
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    72
        self.test_name = test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    73
        self.test_scripts_dir = test_scripts_dir
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    74
        self.build_traces_dir = build_traces_dir
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    75
        self.reference_traces_dir = reference_traces
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    76
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    77
    def __str__(self):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    78
        return 'regression-test (%s)\n' % self.test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    79
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    80
    def run(self):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    81
        """Run a single test"""
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    82
        sys.path.insert(0, self.test_scripts_dir)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    83
        try:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    84
            mod = __import__(self.test_name, globals(), locals(), [])
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    85
        finally:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    86
            sys.path.remove(self.test_scripts_dir)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    87
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    88
        assert self.test_name.startswith('test-')
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    89
        short_name = self.test_name[len('test-'):]
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    90
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    91
        trace_dir_name = getattr(mod, "trace_dir_name", None)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    92
        if trace_dir_name is None:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    93
            trace_dir_name = "%s.ref" % short_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    94
        trace_output_path = os.path.join(self.build_traces_dir, trace_dir_name)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    95
        reference_traces_path = os.path.join(self.reference_traces_dir, trace_dir_name)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    96
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    97
        if hasattr(mod, 'get_arguments'):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    98
            arguments = mod.get_arguments(self.env, '..')
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
    99
        else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   100
            arguments = getattr(mod, "arguments", [])
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   101
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   102
        pyscript = getattr(mod, "pyscript", None)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   103
        if pyscript:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   104
            is_pyscript = True
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   105
            program = pyscript
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   106
        else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   107
            is_pyscript = False
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   108
            program = getattr(mod, "program", short_name)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   109
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   110
        if hasattr(mod, 'may_run'):
4130
9bddd87225a9 Skip the NSC test when valgrind is enabled.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4129
diff changeset
   111
            reason_cannot_run = mod.may_run(self.env, Options.options)
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   112
        else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   113
            reason_cannot_run = None
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   114
        if reason_cannot_run:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   115
            print "SKIP %s (%s)" % (self.test_name, reason_cannot_run)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   116
            self.result = None
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   117
            return 0
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   118
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   119
        if Options.options.regression_generate:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   120
            # clean the target dir
4308
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   121
            try:
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   122
                shutil.rmtree(trace_output_path)
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   123
            except OSError, ex:
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   124
                if ex.errno not in [errno.ENOENT]:
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   125
                    raise
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   126
            os.makedirs(reference_traces_path)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   127
            result = self.run_reference_generate(reference_traces_path, program, arguments, is_pyscript)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   128
            if result == 0:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   129
                print "GENERATE " + self.test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   130
            else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   131
                print "GENERATE FAIL " + self.test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   132
        else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   133
            # clean the target dir
4308
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   134
            try:
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   135
                shutil.rmtree(trace_output_path)
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   136
            except OSError, ex:
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   137
                if ex.errno not in [errno.ENOENT]:
b8528d30dfb3 Less error hiding anti-pattern when running regression tests.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4202
diff changeset
   138
                    raise
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   139
            os.makedirs(trace_output_path)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   140
            # run it
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   141
            result = self.run_reference_test(reference_traces_path, trace_output_path, program, arguments, is_pyscript)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   142
            if result == 0:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   143
                print "PASS " + self.test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   144
            else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   145
                print "FAIL " + self.test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   146
        self.result = result
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   147
        return 0
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   148
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   149
    def run_reference_test(self, reference_traces_path, trace_output_path, program, arguments, is_pyscript):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   150
        if not os.path.exists(reference_traces_path):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   151
            print "Cannot locate reference traces in " + reference_traces_path
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   152
            return 1
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   153
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   154
        if is_pyscript:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   155
            script = os.path.abspath(os.path.join('..', *os.path.split(program)))
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   156
            argv = [self.env['PYTHON'], script] + arguments
4129
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   157
            try:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   158
                wutils.run_argv(argv, cwd=trace_output_path)
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   159
            except Utils.WafError, ex:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   160
                print >> sys.stderr, ex
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   161
                return 1
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   162
        else:
4129
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   163
            try:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   164
                wutils.run_program(program,
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   165
                                   command_template=wutils.get_command_template(self.env, arguments),
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   166
                                   cwd=trace_output_path)
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   167
            except Utils.WafError, ex:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   168
                print >> sys.stderr, ex
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   169
                return 1
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   170
4202
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
   171
        rc = diff(trace_output_path, reference_traces_path, Options.options.verbose)
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   172
        if rc:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   173
            print "----------"
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   174
            print "Traces differ in test: ", self.test_name
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   175
            print "Reference traces in directory: " + reference_traces_path
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   176
            print "Traces in directory: " + trace_output_path
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   177
            print "Run the following command for details:"
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   178
            print "\tdiff -u %s %s" % (reference_traces_path, trace_output_path)
4202
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
   179
            if not Options.options.verbose:
272cabf60878 For regression testing, use python modules filecmp and difflib instead of an external diff command, to improve portability (e.g. mingw). Closes #330.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4130
diff changeset
   180
                print "Or re-run regression testing with option -v"
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   181
            print "----------"
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   182
        return rc
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   183
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   184
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   185
    def run_reference_generate(self, trace_output_path, program, arguments, is_pyscript):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   186
        if is_pyscript:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   187
            script = os.path.abspath(os.path.join('..', *os.path.split(program)))
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   188
            argv = [self.env['PYTHON'], script] + arguments
4129
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   189
            try:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   190
                retval = wutils.run_argv(argv, cwd=trace_output_path)
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   191
            except Utils.WafError, ex:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   192
                print >> sys.stderr, ex
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   193
                return 1
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   194
        else:
4129
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   195
            try:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   196
                retval = wutils.run_program(program,
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   197
                                            command_template=wutils.get_command_template(self.env, arguments),
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   198
                                            cwd=trace_output_path)
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   199
            except Utils.WafError, ex:
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   200
                print >> sys.stderr, ex
9b1f5d58dfb6 Interpret regression program non-zero exit as a test failure, instead of showing a traceback.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4119
diff changeset
   201
                return 1
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   202
        return retval
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   203
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   204
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   205
class regression_test_collector_task(Task.TaskBase):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   206
    after = 'regression_test_task'
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   207
    color = 'BLUE'
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   208
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   209
    def __init__(self, test_tasks):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   210
        super(regression_test_collector_task, self).__init__()
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   211
        self.test_tasks = test_tasks
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   212
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   213
    def __str__(self):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   214
        return 'regression-test-collector\n'
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   215
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   216
    def run(self):
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   217
        failed_tests = [test for test in self.test_tasks if test.result is not None and test.result != 0]
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   218
        skipped_tests = [test for test in self.test_tasks if test.result is None]
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   219
        print "Regression testing summary:"
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   220
        if skipped_tests:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   221
            print "SKIP: %i of %i tests have been skipped (%s)" % (
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   222
                len(skipped_tests), len(self.test_tasks),
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   223
                ', '.join([test.test_name for test in skipped_tests]))
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   224
        if failed_tests:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   225
            print "FAIL: %i of %i tests have failed (%s)" % (
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   226
                len(failed_tests), len(self.test_tasks),
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   227
                ', '.join([test.test_name for test in failed_tests]))
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   228
            return 1
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   229
        else:
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   230
            print "PASS: %i of %i tests passed" % (len(self.test_tasks) - len(skipped_tests),
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   231
                                                   len(self.test_tasks))
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   232
            return 0
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   233
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   234
def run_regression(bld, reference_traces):
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   235
    """Execute regression tests.  Called with cwd set to the 'regression' subdir of ns-3.
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   236
4109
1a251c8ad317 Cleanup: remove regression reference traces downloading code (moved to allinone)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4064
diff changeset
   237
    @param reference_traces: reference traces directory.
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   238
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   239
    """
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   240
4110
04170734fa8b Don't abuse os.chdir so much (not thread safe)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4109
diff changeset
   241
    testdir = os.path.join("regression", "tests")
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   242
    if not os.path.exists(testdir):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   243
        print "Tests directory does not exist"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   244
        sys.exit(3)
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   245
4064
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
   246
    if Options.options.regression_tests:
10222f483860 Upgrade to new WAF, work in progress
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3872
diff changeset
   247
        tests = Options.options.regression_tests.split(',')
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   248
    else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   249
        tests = _find_tests(testdir)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   250
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   251
    if not os.path.exists(reference_traces):
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   252
        print "Reference traces directory (%s) does not exist" % reference_traces
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   253
        return 3
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   254
    
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   255
    test_scripts_dir = bld.path.find_dir('regression/tests').abspath()
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   256
    build_traces_dir = bld.path.find_or_declare('regression/traces').abspath(bld.env)
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   257
    tasks = []
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   258
    for test in tests:
4114
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   259
        tasks.append(regression_test_task(bld.env, test, test_scripts_dir, build_traces_dir, reference_traces))
a6c5ccfa3451 Run regression tests as WAF tasks, plus simplify how unit tests are declared. Closes #480.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 4110
diff changeset
   260
    regression_test_collector_task(tasks)