regression.py
author Tom Henderson <tomh@tomh.org>
Mon, 15 Dec 2008 07:25:27 -0800
changeset 4024 dfd0bc16dc99
parent 3872 8e757a83fb36
child 4064 10222f483860
permissions -rw-r--r--
add device chapters to manual
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     1
import os
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     2
import sys
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     3
import Params
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     4
import shutil
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     5
import pproc as subprocess
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     6
import urllib
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     7
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     8
import wutils
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
     9
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    10
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    11
# The directory in which the tarball of the reference traces lives.  This is
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    12
# used if Mercurial is not on the system.
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    13
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    14
REGRESSION_TRACES_URL = "http://www.nsnam.org/releases/"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    15
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    16
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    17
# The path to the Mercurial repository used to find the reference traces if
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    18
# we find "hg" on the system.  We expect that the repository will be named
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    19
# identically to refDirName below
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    20
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    21
REGRESSION_TRACES_REPO = "http://code.nsnam.org/"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    22
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    23
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    24
# Name of the local directory where the regression code lives.
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    25
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    26
REGRESSION_DIR = "regression"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    27
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    28
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    29
# The last part of the path name to use to find the regression traces.  The
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    30
# path will be APPNAME + '-' + VERSION + REGRESSION_SUFFIX, e.g.,
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    31
# ns-3-dev-ref-traces
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    32
#
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    33
REGRESSION_SUFFIX = "-ref-traces"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    34
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    35
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    36
def dev_null():
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    37
    if sys.platform == 'win32':
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    38
        return open("NUL:", "w")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    39
    else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    40
        return open("/dev/null", "w")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    41
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    42
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    43
### Regression testing
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    44
class Regression(object):
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    45
    def __init__(self, testdir, reference_traces):
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    46
        self.testdir = testdir
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    47
        self.reference_traces = reference_traces
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    48
        self.env = Params.g_build.env_of_name('default')
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    49
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    50
    def run_test(self, verbose, generate, testName, arguments=[], pyscript=None, refTestName=None):
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    51
        """
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    52
        @param verbose: enable verbose execution
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    53
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    54
        @param generate: generate new traces instead of comparing with the reference
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    55
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    56
        @param testName: name of the test
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    57
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    58
        @arguments: list of extra parameters to pass to the program to be tested
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    59
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    60
        @pyscript: if not None, the test is written in Python and this
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    61
        parameter contains the path to the python script, relative to
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    62
        the project root dir
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    63
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    64
        @param refTestName: if not None, this is the name of the directory under refDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    65
        that contains the reference traces. Otherwise "refDirname/testName + .ref" is used.
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    66
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    67
        """
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    68
        if not isinstance(arguments, list):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    69
            raise TypeError
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    70
        
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    71
        if refTestName is None:
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    72
            refTestDirName = os.path.join(self.reference_traces, (testName + ".ref"))
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    73
        else:
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    74
            refTestDirName = os.path.join(self.reference_traces, refTestName)
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    75
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    76
        if not os.path.exists(self.reference_traces):
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    77
            print"No reference trace repository"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    78
            return 1
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    79
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    80
        if generate:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    81
            if not os.path.exists(refTestDirName):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    82
                print "creating new " + refTestDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    83
                os.mkdir(refTestDirName)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    84
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    85
            if pyscript is None:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    86
                Params.g_options.cwd_launch = refTestDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    87
                tmpl = "%s"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    88
                for arg in arguments:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    89
                    tmpl = tmpl + " " + arg
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    90
                wutils.run_program(testName, tmpl)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    91
            else:
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
    92
                argv = [self.env['PYTHON'], os.path.join(Params.g_cwd_launch, *os.path.split(pyscript))] + arguments
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    93
                before = os.getcwd()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    94
                os.chdir(refTestDirName)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    95
                try:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    96
                    wutils.run_argv(argv)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    97
                finally:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    98
                    os.chdir(before)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
    99
            print "Remember to commit " + refTestDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   100
            return 0
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   101
        else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   102
            if not os.path.exists(refTestDirName):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   103
                print "Cannot locate reference traces in " + refTestDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   104
                return 1
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   105
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   106
            
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   107
            if refTestName is None:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   108
                traceDirName = testName + ".ref"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   109
            else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   110
                traceDirName = refTestName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   111
            traceDirName = os.path.join ('traces', traceDirName)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   112
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   113
            try:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   114
                shutil.rmtree(traceDirName)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   115
            except OSError:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   116
                pass
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   117
            os.mkdir(traceDirName)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   118
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   119
            #os.system("./waf --cwd regression/traces --run " +
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   120
            #  testName + " > /dev/null 2>&1")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   121
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   122
            if pyscript is None:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   123
                Params.g_options.cwd_launch = traceDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   124
                wutils.run_program(testName, command_template=wutils.get_command_template(*arguments))
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   125
            else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   126
                argv = [self.env['PYTHON'], os.path.join('..', '..', '..', *os.path.split(pyscript))] + arguments
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   127
                before = os.getcwd()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   128
                os.chdir(traceDirName)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   129
                try:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   130
                    wutils.run_argv(argv)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   131
                finally:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   132
                    os.chdir(before)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   133
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   134
            if verbose:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   135
                #diffCmd = "diff traces " + refTestDirName + " | head"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   136
                diffCmd = subprocess.Popen([self.env['DIFF'], traceDirName, refTestDirName],
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   137
                                           stdout=subprocess.PIPE)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   138
                headCmd = subprocess.Popen("head", stdin=diffCmd.stdout)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   139
                rc2 = headCmd.wait()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   140
                diffCmd.stdout.close()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   141
                rc1 = diffCmd.wait()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   142
                rc = rc1 or rc2
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   143
            else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   144
                rc = subprocess.Popen([self.env['DIFF'], traceDirName, refTestDirName], stdout=dev_null()).wait()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   145
            if rc:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   146
                print "----------"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   147
                print "Traces differ in test: test-" + testName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   148
                print "Reference traces in directory: regression/" + refTestDirName
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   149
                print "Traces in directory: traces"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   150
                print "Rerun regression test as: " + \
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   151
                    "\"./waf --regression --regression-tests=test-" + testName + "\""
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   152
                print "Then do \"diff -u regression/" + refTestDirName + " regression/" + traceDirName +\
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   153
                    "\" for details"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   154
                print "----------"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   155
            return rc
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   156
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   157
def _find_tests(testdir):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   158
    """Return a list of test modules in the test directory
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   159
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   160
    Arguments:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   161
    testdir -- the directory to look in for tests
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   162
    """
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   163
    names = os.listdir(testdir)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   164
    tests = []
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   165
    for name in names:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   166
        if name[:5] == "test-" and name[-3:] == ".py":
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   167
            testname = name[:-3]
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   168
            tests.append(testname)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   169
    tests.sort()
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   170
    return tests
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   171
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   172
def run_regression(reference_traces):
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   173
    """Execute regression tests.  Called with cwd set to the 'regression' subdir of ns-3.
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   174
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   175
    @param reference_traces: reference traces directory, or None for default.
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   176
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   177
    """
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   178
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   179
    testdir = "tests"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   180
    if not os.path.exists(testdir):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   181
        print "Tests directory does not exist"
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   182
        sys.exit(3)
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   183
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   184
    dir_name = (wutils.APPNAME + '-' + wutils.VERSION + REGRESSION_SUFFIX)
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   185
    if reference_traces is None:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   186
        reference_traces = dir_name
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   187
        no_net = False
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   188
    else:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   189
        no_net = True
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   190
    
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   191
    sys.path.append(testdir)
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   192
    sys.modules['tracediff'] = Regression(testdir, reference_traces)
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   193
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   194
    if Params.g_options.regression_tests:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   195
        tests = Params.g_options.regression_tests.split(',')
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   196
    else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   197
        tests = _find_tests(testdir)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   198
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   199
    print "========== Running Regression Tests =========="
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   200
    env = Params.g_build.env_of_name('default')
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   201
    if not no_net:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   202
        if env['MERCURIAL']:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   203
            print "Synchronizing reference traces using Mercurial."
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   204
            if not os.path.exists(reference_traces):
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   205
                print "Cloning " + REGRESSION_TRACES_REPO + dir_name + " from repo."
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   206
                argv = ["hg", "clone", REGRESSION_TRACES_REPO + dir_name, reference_traces]
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   207
                rv = subprocess.Popen(argv).wait()
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   208
            else:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   209
                _dir = os.getcwd()
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   210
                os.chdir(reference_traces)
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   211
                try:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   212
                    print "Pulling " + REGRESSION_TRACES_REPO + dir_name + " from repo."
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   213
                    result = subprocess.Popen(["hg", "-q", "pull", REGRESSION_TRACES_REPO + dir_name]).wait()
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   214
                    if not result:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   215
                        result = subprocess.Popen(["hg", "-q", "update"]).wait()
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   216
                finally:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   217
                    os.chdir("..")
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   218
                if result:
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   219
                    Params.fatal("Synchronizing reference traces using Mercurial failed.")
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   220
        else:
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   221
            if not os.path.exists(reference_traces):
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   222
                traceball = dir_name + wutils.TRACEBALL_SUFFIX
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   223
                print "Retrieving " + traceball + " from web."
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   224
                urllib.urlretrieve(REGRESSION_TRACES_URL + traceball, traceball)
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   225
                os.system("tar -xjf %s -C .." % (traceball))
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   226
                print "Done."
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   227
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   228
    if not os.path.exists(reference_traces):
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   229
        print "Reference traces directory (%s) does not exist" % reference_traces
3866
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   230
        return 3
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   231
    
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   232
    bad = []
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   233
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   234
    for test in tests:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   235
        try:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   236
            result = _run_regression_test(test)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   237
            if result == 0:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   238
                if Params.g_options.regression_generate:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   239
                    print "GENERATE " + test
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   240
                else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   241
                    print "PASS " + test
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   242
            else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   243
                bad.append(test)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   244
                print "FAIL " + test
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   245
        except NotImplementedError:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   246
                print "SKIP " + test            
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   247
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   248
    return len(bad) > 0
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   249
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   250
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   251
def _run_regression_test(test):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   252
    """Run a single test.
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   253
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   254
    Arguments:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   255
    test -- the name of the test
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   256
    """
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   257
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   258
    if os.path.exists("traces"):
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   259
        files = os.listdir("traces")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   260
        for file in files:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   261
            if file == '.' or file == '..':
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   262
                continue
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   263
            shutil.rmtree(os.path.join("traces", file), ignore_errors=True)
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   264
    else:
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   265
        os.mkdir("traces")
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   266
    
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   267
    mod = __import__(test, globals(), locals(), [])
9e946fee902c Refactor wscript code to move regression testing code to a separate python module (bug 325)
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents:
diff changeset
   268
    return mod.run(verbose=(Params.g_options.verbose > 0),
3872
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   269
                   generate=Params.g_options.regression_generate)
8e757a83fb36 Add a --with-regression-traces configure option, to allow using externally supplied regression traces instead of fetching them from the network.
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 3866
diff changeset
   270