test.py
author Sébastien Deronne <sebastien.deronne@gmail.com>
Sun, 09 Dec 2018 12:14:03 +0100
changeset 13895 5df69eea72d6
parent 13841 55a1a4bbf970
permissions -rwxr-xr-x
wifi: Do not allow S-MPDU transmissions as long as ADDBA handshake is not established
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     1
#! /usr/bin/env python
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     2
## -*- Mode: python; py-indent-offset: 4; indent-tabs-mode: nil; coding: utf-8; -*-
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     3
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     4
# Copyright (c) 2009 University of Washington
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     5
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     6
# This program is free software; you can redistribute it and/or modify
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     7
# it under the terms of the GNU General Public License version 2 as
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     8
# published by the Free Software Foundation;
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     9
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    10
# This program is distributed in the hope that it will be useful,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    11
# but WITHOUT ANY WARRANTY; without even the implied warranty of
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    12
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    13
# GNU General Public License for more details.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    14
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    15
# You should have received a copy of the GNU General Public License
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    16
# along with this program; if not, write to the Free Software
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    17
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    18
#
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
    19
from __future__ import print_function
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    20
import os
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    21
import sys
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
    22
import time
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    23
import optparse
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    24
import subprocess
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    25
import threading
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    26
import signal
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    27
import xml.dom.minidom
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
    28
import shutil
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
    29
import re
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    30
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
    31
from utils import get_list_from_file
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
    32
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
    33
try:
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
    34
    import queue
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
    35
except ImportError:
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
    36
    import Queue as queue
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    37
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    38
# XXX This should really be part of a waf command to list the configuration
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    39
# items relative to optional ns-3 pieces.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    40
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    41
# A list of interesting configuration items in the waf configuration 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    42
# cache which we may be interested in when deciding on which examples
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    43
# to run and how to run them.  These are set by waf during the 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    44
# configuration phase and the corresponding assignments are usually
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    45
# found in the associated subdirectory wscript files.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    46
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    47
interesting_config_items = [
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
    48
    "NS3_ENABLED_MODULES",
12808
f2dba79bb5d1 build: (fixes #2630) Add contrib directory in parallel to src
Lauri Sormunen <lauri.sormunen@magister.fi>
parents: 12235
diff changeset
    49
    "NS3_ENABLED_CONTRIBUTED_MODULES",
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    50
    "NS3_MODULE_PATH",
6619
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
    51
    "NSC_ENABLED",
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    52
    "ENABLE_REAL_TIME",
7028
3963d3678649 Make examples that were in samples directory run
Mitch Watrous <watrous@u.washington.edu>
parents: 6890
diff changeset
    53
    "ENABLE_THREADING",
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    54
    "ENABLE_EXAMPLES",
7245
4b245d55faa2 Bug 1146 - help string for test.py with disabled tests
Mitch Watrous <watrous@u.washington.edu>
parents: 7199
diff changeset
    55
    "ENABLE_TESTS",
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
    56
    "EXAMPLE_DIRECTORIES",
6243
a597d6d2da85 Bug 882 - ./test.py reports 6 FAILs with --disable-python
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 6227
diff changeset
    57
    "ENABLE_PYTHON_BINDINGS",
11868
94ac3e991329 click: Fix detection of Waf configuration variable
Tom Henderson <tomh@tomh.org>
parents: 11741
diff changeset
    58
    "NSCLICK",
9164
347452534b2e Brite Integration
Brian Swenson <bswenson3@gatech.edu>
parents: 9076
diff changeset
    59
    "ENABLE_BRITE",
6890
e5da7045526e Merge OpenFlow
Josh Pelkey <jpelkey@gatech.edu>
parents: 6886
diff changeset
    60
    "ENABLE_OPENFLOW",
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    61
    "APPNAME",
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    62
    "BUILD_PROFILE",
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    63
    "VERSION",
7687
9918e57b9f8d bug 1317: allow test.py to learn which python version was used to build ns-3
Tom Henderson <tomh@tomh.org>
parents: 7682
diff changeset
    64
    "PYTHON",
9832
02642e139f19 Bug 1695 - improve test.py output when -g selected without valgrind
Mitch Watrous <watrous@u.washington.edu>
parents: 9827
diff changeset
    65
    "VALGRIND_FOUND",
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    66
]
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    67
6619
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
    68
NSC_ENABLED = False
5295
c1bd4ffb5e47 fix test.py -v
Craig Dowell <craigdo@ee.washington.edu>
parents: 5279
diff changeset
    69
ENABLE_REAL_TIME = False
7028
3963d3678649 Make examples that were in samples directory run
Mitch Watrous <watrous@u.washington.edu>
parents: 6890
diff changeset
    70
ENABLE_THREADING = False
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    71
ENABLE_EXAMPLES = True
7245
4b245d55faa2 Bug 1146 - help string for test.py with disabled tests
Mitch Watrous <watrous@u.washington.edu>
parents: 7199
diff changeset
    72
ENABLE_TESTS = True
11868
94ac3e991329 click: Fix detection of Waf configuration variable
Tom Henderson <tomh@tomh.org>
parents: 11741
diff changeset
    73
NSCLICK = False
9164
347452534b2e Brite Integration
Brian Swenson <bswenson3@gatech.edu>
parents: 9076
diff changeset
    74
ENABLE_BRITE = False
6890
e5da7045526e Merge OpenFlow
Josh Pelkey <jpelkey@gatech.edu>
parents: 6886
diff changeset
    75
ENABLE_OPENFLOW = False
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
    76
EXAMPLE_DIRECTORIES = []
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    77
APPNAME = ""
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    78
BUILD_PROFILE = ""
9850
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
    79
BUILD_PROFILE_SUFFIX = ""
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    80
VERSION = ""
7689
8d3e8dbc5305 postfix bug 1317: support -v option in test.py when python not enabled
Tom Henderson <tomh@tomh.org>
parents: 7687
diff changeset
    81
PYTHON = ""
9832
02642e139f19 Bug 1695 - improve test.py output when -g selected without valgrind
Mitch Watrous <watrous@u.washington.edu>
parents: 9827
diff changeset
    82
VALGRIND_FOUND = True
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    83
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    84
#
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    85
# This will be given a prefix and a suffix when the waf config file is
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    86
# read.
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    87
#
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
    88
test_runner_name = "test-runner"
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    89
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    90
#
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    91
# If the user has constrained us to run certain kinds of tests, we can tell waf
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    92
# to only build
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    93
#
9072
7bfaded450be Bug 1284 - ./test.py performance tests do not exist
Mitch Watrous <watrous@u.washington.edu>
parents: 8919
diff changeset
    94
core_kinds = ["bvt", "core", "performance", "system", "unit"]
5295
c1bd4ffb5e47 fix test.py -v
Craig Dowell <craigdo@ee.washington.edu>
parents: 5279
diff changeset
    95
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    96
#
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    97
# There are some special cases for test suites that kill valgrind.  This is
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    98
# because NSC causes illegal instruction crashes when run under valgrind.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    99
#
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   100
core_valgrind_skip_tests = [
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   101
    "ns3-tcp-cwnd",
6198
cd1b101cae62 suppress valgrind testing of new nsc test suite
Tom Henderson <tomh@tomh.org>
parents: 6127
diff changeset
   102
    "nsc-tcp-loss",
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   103
    "ns3-tcp-interoperability",
7259
5c63124c3955 remove click from valgrind tests
Josh Pelkey <jpelkey@gatech.edu>
parents: 7245
diff changeset
   104
    "routing-click",
9219
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   105
    "lte-rr-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   106
    "lte-tdmt-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   107
    "lte-fdmt-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   108
    "lte-pf-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   109
    "lte-tta-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   110
    "lte-fdbet-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   111
    "lte-ttbet-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   112
    "lte-fdtbfq-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   113
    "lte-tdtbfq-ff-mac-scheduler",
f09a0822157e bug 1563: reduce valgrind test scope
Tom Henderson <tomh@tomh.org>
parents: 9164
diff changeset
   114
    "lte-pss-ff-mac-scheduler",
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   115
]
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   116
6619
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   117
# 
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   118
# There are some special cases for test suites that fail when NSC is
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   119
# missing.
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   120
#
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   121
core_nsc_missing_skip_tests = [
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   122
    "ns3-tcp-cwnd",
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   123
    "nsc-tcp-loss",
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   124
    "ns3-tcp-interoperability",
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   125
]
c6e03f378655 Fixed problems due to missing NSC.
Mitch Watrous <watrous@u.washington.edu>
parents: 6374
diff changeset
   126
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   127
#
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   128
# Parse the examples-to-run file if it exists.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   129
#
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   130
# This function adds any C++ examples or Python examples that are to be run
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   131
# to the lists in example_tests and python_tests, respectively.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   132
#
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   133
def parse_examples_to_run_file(
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   134
    examples_to_run_path,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   135
    cpp_executable_dir,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   136
    python_script_dir,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   137
    example_tests,
9220
fd38b9505eb9 Add shell script that runs all examples with full logging turned on
Mitch Watrous <watrous@u.washington.edu>
parents: 9219
diff changeset
   138
    example_names_original,
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   139
    python_tests):
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   140
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   141
    # Look for the examples-to-run file exists.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   142
    if os.path.exists(examples_to_run_path):
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   143
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   144
        # Each tuple in the C++ list of examples to run contains
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   145
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   146
        #     (example_name, do_run, do_valgrind_run)
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   147
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   148
        # where example_name is the executable to be run, do_run is a
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   149
        # condition under which to run the example, and do_valgrind_run is
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   150
        # a condition under which to run the example under valgrind.  This
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   151
        # is needed because NSC causes illegal instruction crashes with
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   152
        # some tests when they are run under valgrind.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   153
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   154
        # Note that the two conditions are Python statements that
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   155
        # can depend on waf configuration variables.  For example,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   156
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   157
        #     ("tcp-nsc-lfn", "NSC_ENABLED == True", "NSC_ENABLED == False"),
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   158
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   159
        cpp_examples = get_list_from_file(examples_to_run_path, "cpp_examples")
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   160
        for example_name, do_run, do_valgrind_run in cpp_examples:
9220
fd38b9505eb9 Add shell script that runs all examples with full logging turned on
Mitch Watrous <watrous@u.washington.edu>
parents: 9219
diff changeset
   161
13665
b2861f3dc03d Spellcheck
Alexander Krotov <krotov@iitp.ru>
parents: 13237
diff changeset
   162
            # Separate the example name from its arguments.
9220
fd38b9505eb9 Add shell script that runs all examples with full logging turned on
Mitch Watrous <watrous@u.washington.edu>
parents: 9219
diff changeset
   163
            example_name_original = example_name
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   164
            example_name_parts = example_name.split(' ', 1)
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   165
            if len(example_name_parts) == 1:
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   166
                example_name      = example_name_parts[0]
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   167
                example_arguments = ""
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   168
            else:
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   169
                example_name      = example_name_parts[0]
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   170
                example_arguments = example_name_parts[1]
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   171
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
   172
            # Add the proper prefix and suffix to the example name to
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
   173
            # match what is done in the wscript file.
10554
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   174
            example_path = "%s%s-%s%s" % (APPNAME, VERSION, example_name, BUILD_PROFILE_SUFFIX)
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   175
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   176
            # Set the full path for the example.
10554
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   177
            example_path = os.path.join(cpp_executable_dir, example_path)
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   178
            example_name = os.path.join(
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   179
                os.path.relpath(cpp_executable_dir, NS3_BUILDDIR),
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   180
                example_name)
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   181
            # Add all of the C++ examples that were built, i.e. found
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   182
            # in the directory, to the list of C++ examples to run.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   183
            if os.path.exists(example_path):
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   184
                # Add any arguments to the path.
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   185
                if len(example_name_parts) != 1:
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   186
                    example_path = "%s %s" % (example_path, example_arguments)
10554
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   187
                    example_name = "%s %s" % (example_name, example_arguments)
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   188
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   189
                # Add this example.
10554
ee9c5221e7bd Strip NS3_BUILDDIR from Example path.
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 9852
diff changeset
   190
                example_tests.append((example_name, example_path, do_run, do_valgrind_run))
9220
fd38b9505eb9 Add shell script that runs all examples with full logging turned on
Mitch Watrous <watrous@u.washington.edu>
parents: 9219
diff changeset
   191
                example_names_original.append(example_name_original)
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   192
    
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   193
        # Each tuple in the Python list of examples to run contains
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   194
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   195
        #     (example_name, do_run)
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   196
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   197
        # where example_name is the Python script to be run and
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   198
        # do_run is a condition under which to run the example.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   199
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   200
        # Note that the condition is a Python statement that can
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   201
        # depend on waf configuration variables.  For example,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   202
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   203
        #     ("realtime-udp-echo.py", "ENABLE_REAL_TIME == True"),
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   204
        #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   205
        python_examples = get_list_from_file(examples_to_run_path, "python_examples")
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   206
        for example_name, do_run in python_examples:
13665
b2861f3dc03d Spellcheck
Alexander Krotov <krotov@iitp.ru>
parents: 13237
diff changeset
   207
            # Separate the example name from its arguments.
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   208
            example_name_parts = example_name.split(' ', 1)
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   209
            if len(example_name_parts) == 1:
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   210
                example_name      = example_name_parts[0]
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   211
                example_arguments = ""
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   212
            else:
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   213
                example_name      = example_name_parts[0]
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   214
                example_arguments = example_name_parts[1]
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   215
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   216
            # Set the full path for the example.
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   217
            example_path = os.path.join(python_script_dir, example_name)
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   218
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   219
            # Add all of the Python examples that were found to the
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   220
            # list of Python examples to run.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   221
            if os.path.exists(example_path):
7864
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   222
                # Add any arguments to the path.
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   223
                if len(example_name_parts) != 1:
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   224
                    example_path = "%s %s" % (example_path, example_arguments)
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   225
d0df2d56ae9c Bug 1415 - examples-to-run.py doesn't work with command line arguments
Mitch Watrous <watrous@u.washington.edu>
parents: 7819
diff changeset
   226
                # Add this example.
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
   227
                python_tests.append((example_path, do_run))
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   228
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   229
#
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   230
# The test suites are going to want to output status.  They are running
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   231
# concurrently.  This means that unless we are careful, the output of
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   232
# the test suites will be interleaved.  Rather than introducing a lock
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   233
# file that could unintentionally start serializing execution, we ask
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   234
# the tests to write their output to a temporary directory and then 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   235
# put together the final output file when we "join" the test tasks back
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   236
# to the main thread.  In addition to this issue, the example programs
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   237
# often write lots and lots of trace files which we will just ignore.
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   238
# We put all of them into the temp directory as well, so they can be
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   239
# easily deleted.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   240
#
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   241
TMP_OUTPUT_DIR = "testpy-output"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   242
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   243
def read_test(test):
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   244
    result = test.find('Result').text
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   245
    name = test.find('Name').text
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   246
    if not test.find('Reason') is None:
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   247
        reason = test.find('Reason').text
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   248
    else:
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   249
        reason = ''
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   250
    if not test.find('Time') is None:
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   251
        time_real = test.find('Time').get('real')
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   252
    else:
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   253
        time_real = ''
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   254
    return (result, name, reason, time_real)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   255
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   256
#
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   257
# A simple example of writing a text file with a test result summary.  It is 
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   258
# expected that this output will be fine for developers looking for problems.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   259
#
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   260
def node_to_text (test, f):
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   261
    (result, name, reason, time_real) = read_test(test)
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   262
    if reason:
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   263
        reason = " (%s)" % reason
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   264
        
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   265
    output = "%s: Test Suite \"%s\" (%s)%s\n" % (result, name, time_real, reason)
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   266
    f.write(output)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   267
    for details in test.findall('FailureDetails'):
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   268
        f.write("    Details:\n")
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   269
        f.write("      Message:   %s\n" % details.find('Message').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   270
        f.write("      Condition: %s\n" % details.find('Condition').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   271
        f.write("      Actual:    %s\n" % details.find('Actual').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   272
        f.write("      Limit:     %s\n" % details.find('Limit').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   273
        f.write("      File:      %s\n" % details.find('File').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   274
        f.write("      Line:      %s\n" % details.find('Line').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   275
    for child in test.findall('Test'):
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   276
        node_to_text(child, f)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   277
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   278
def translate_to_text(results_file, text_file):
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   279
    text_file += '.txt'
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   280
    print('Writing results to text file \"%s\"...' % text_file, end='')
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   281
    f = open(text_file, 'w')
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   282
    import xml.etree.ElementTree as ET
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   283
    et = ET.parse (results_file)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   284
    for test in et.findall('Test'):
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   285
        node_to_text (test, f)
7199
eae56a101979 Bug 957 - Issue with test.py
Mitch Watrous <watrous@u.washington.edu>
parents: 7096
diff changeset
   286
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   287
    for example in et.findall('Example'):
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   288
        result = example.find('Result').text
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   289
        name = example.find('Name').text
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   290
        if not example.find('Time') is None:
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   291
            time_real = example.find('Time').get('real')
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   292
        else:
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   293
            time_real = ''
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   294
        output = "%s: Example \"%s\" (%s)\n" % (result, name, time_real)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   295
        f.write(output)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   296
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   297
    f.close()
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   298
    print('done.')
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   299
    
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   300
#
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   301
# A simple example of writing an HTML file with a test result summary.  It is 
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   302
# expected that this will eventually be made prettier as time progresses and
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   303
# we have time to tweak it.  This may end up being moved to a separate module
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   304
# since it will probably grow over time.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   305
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   306
def translate_to_html(results_file, html_file):
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   307
    html_file += '.html'
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   308
    print('Writing results to html file %s...' % html_file, end='')
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   309
    f = open(html_file, 'w')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   310
    f.write("<html>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   311
    f.write("<body>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   312
    f.write("<center><h1>ns-3 Test Results</h1></center>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   313
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   314
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   315
    # Read and parse the whole results file.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   316
    #
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   317
    import xml.etree.ElementTree as ET
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   318
    et = ET.parse(results_file)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   319
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   320
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   321
    # Iterate through the test suites
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   322
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   323
    f.write("<h2>Test Suites</h2>\n")
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   324
    for suite in et.findall('Test'):     
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   325
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   326
        # For each test suite, get its name, result and execution time info
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   327
        #
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   328
        (result, name, reason, time) = read_test (suite)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   329
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   330
        # 
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   331
        # Print a level three header with the result, name and time.  If the 
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   332
        # test suite passed, the header is printed in green. If the suite was
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   333
        # skipped, print it in orange, otherwise assume something bad happened
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   334
        # and print in red.
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   335
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   336
        if result == "PASS":
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   337
            f.write("<h3 style=\"color:green\">%s: %s (%s)</h3>\n" % (result, name, time))
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   338
        elif result == "SKIP":
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   339
            f.write("<h3 style=\"color:#ff6600\">%s: %s (%s) (%s)</h3>\n" % (result, name, time, reason))
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   340
        else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   341
            f.write("<h3 style=\"color:red\">%s: %s (%s)</h3>\n" % (result, name, time))
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   342
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   343
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   344
        # The test case information goes in a table.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   345
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   346
        f.write("<table border=\"1\">\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   347
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   348
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   349
        # The first column of the table has the heading Result
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   350
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   351
        f.write("<th> Result </th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   352
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   353
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   354
        # If the suite crashed or is skipped, there is no further information, so just
13665
b2861f3dc03d Spellcheck
Alexander Krotov <krotov@iitp.ru>
parents: 13237
diff changeset
   355
        # declare a new table row with the result (CRASH or SKIP) in it.  Looks like:
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   356
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   357
        #   +--------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   358
        #   | Result |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   359
        #   +--------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   360
        #   | CRASH  |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   361
        #   +--------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   362
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   363
        # Then go on to the next test suite.  Valgrind and skipped errors look the same.
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   364
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   365
        if result in ["CRASH", "SKIP", "VALGR"]:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   366
            f.write("<tr>\n")
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   367
            if result == "SKIP":
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   368
                f.write("<td style=\"color:#ff6600\">%s</td>\n" % result)
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   369
            else:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   370
                f.write("<td style=\"color:red\">%s</td>\n" % result)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   371
            f.write("</tr>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   372
            f.write("</table>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   373
            continue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   374
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   375
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   376
        # If the suite didn't crash, we expect more information, so fill out
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   377
        # the table heading row.  Like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   378
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   379
        #   +--------+----------------+------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   380
        #   | Result | Test Case Name | Time |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   381
        #   +--------+----------------+------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   382
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   383
        f.write("<th>Test Case Name</th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   384
        f.write("<th> Time </th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   385
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   386
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   387
        # If the test case failed, we need to print out some failure details
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   388
        # so extend the heading row again.  Like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   389
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   390
        #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   391
        #   | Result | Test Case Name | Time | Failure Details |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   392
        #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   393
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   394
        if result == "FAIL":
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   395
            f.write("<th>Failure Details</th>\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   396
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   397
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   398
        # Now iterate through all of the test cases.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   399
        #
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   400
        for case in suite.findall('Test'):
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   401
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   402
            #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   403
            # Get the name, result and timing information from xml to use in
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   404
            # printing table below.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   405
            #
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   406
            (result, name, reason, time) = read_test(case)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   407
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   408
            #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   409
            # If the test case failed, we iterate through possibly multiple
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   410
            # failure details
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   411
            #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   412
            if result == "FAIL":
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   413
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   414
                # There can be multiple failures for each test case.  The first
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   415
                # row always gets the result, name and timing information along
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   416
                # with the failure details.  Remaining failures don't duplicate
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   417
                # this information but just get blanks for readability.  Like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   418
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   419
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   420
                #   | Result | Test Case Name | Time | Failure Details |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   421
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   422
                #   |  FAIL  | The name       | time | It's busted     |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   423
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   424
                #   |        |                |      | Really broken   |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   425
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   426
                #   |        |                |      | Busted bad      |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   427
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   428
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   429
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   430
                first_row = True
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   431
                for details in case.findall('FailureDetails'):
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   432
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   433
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   434
                    # Start a new row in the table for each possible Failure Detail
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   435
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   436
                    f.write("<tr>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   437
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   438
                    if first_row:
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   439
                        first_row = False
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   440
                        f.write("<td style=\"color:red\">%s</td>\n" % result)
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   441
                        f.write("<td>%s</td>\n" % name)
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   442
                        f.write("<td>%s</td>\n" % time)
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   443
                    else:
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   444
                        f.write("<td></td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   445
                        f.write("<td></td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   446
                        f.write("<td></td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   447
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   448
                    f.write("<td>")
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   449
                    f.write("<b>Message: </b>%s, " % details.find('Message').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   450
                    f.write("<b>Condition: </b>%s, " % details.find('Condition').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   451
                    f.write("<b>Actual: </b>%s, " % details.find('Actual').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   452
                    f.write("<b>Limit: </b>%s, " % details.find('Limit').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   453
                    f.write("<b>File: </b>%s, " % details.find('File').text)
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   454
                    f.write("<b>Line: </b>%s" % details.find('Line').text)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   455
                    f.write("</td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   456
                    
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   457
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   458
                    # End the table row
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   459
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   460
                    f.write("</td>\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   461
            else:
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   462
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   463
                # If this particular test case passed, then we just print the PASS
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   464
                # result in green, followed by the test case name and its execution
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   465
                # time information.  These go off in <td> ... </td> table data.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   466
                # The details table entry is left blank.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   467
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   468
                #   +--------+----------------+------+---------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   469
                #   | Result | Test Case Name | Time | Details |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   470
                #   +--------+----------------+------+---------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   471
                #   |  PASS  | The name       | time |         |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   472
                #   +--------+----------------+------+---------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   473
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   474
                f.write("<tr>\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   475
                f.write("<td style=\"color:green\">%s</td>\n" % result)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   476
                f.write("<td>%s</td>\n" % name)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   477
                f.write("<td>%s</td>\n" % time)
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   478
                f.write("<td>%s</td>\n" % reason)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   479
                f.write("</tr>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   480
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   481
        # All of the rows are written, so we need to end the table.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   482
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   483
        f.write("</table>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   484
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   485
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   486
    # That's it for all of the test suites.  Now we have to do something about 
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   487
    # our examples.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   488
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   489
    f.write("<h2>Examples</h2>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   490
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   491
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   492
    # Example status is rendered in a table just like the suites.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   493
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   494
    f.write("<table border=\"1\">\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   495
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   496
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   497
    # The table headings look like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   498
    #
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   499
    #   +--------+--------------+--------------+---------+
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   500
    #   | Result | Example Name | Elapsed Time | Details |
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   501
    #   +--------+--------------+--------------+---------+
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   502
    #                                           
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   503
    f.write("<th> Result </th>\n")              
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   504
    f.write("<th>Example Name</th>\n")
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   505
    f.write("<th>Elapsed Time</th>\n")
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   506
    f.write("<th>Details</th>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   507
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   508
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   509
    # Now iterate through all of the examples
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   510
    #
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
   511
    for example in et.findall("Example"):
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   512
        
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   513
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   514
        # Start a new row for each example
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   515
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   516
        f.write("<tr>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   517
        
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   518
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   519
        # Get the result and name of the example in question
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   520
        #
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   521
        (result, name, reason, time) = read_test(example)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   522
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   523
        #
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   524
        # If the example either failed or crashed, print its result status
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   525
        # in red; otherwise green.  This goes in a <td> ... </td> table data
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   526
        #
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   527
        if result == "PASS":
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   528
            f.write("<td style=\"color:green\">%s</td>\n" % result)
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   529
        elif result == "SKIP":
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   530
            f.write("<td style=\"color:#ff6600\">%s</fd>\n" % result)
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   531
        else:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   532
            f.write("<td style=\"color:red\">%s</td>\n" % result)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   533
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   534
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   535
        # Write the example name as a new tag data.
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   536
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   537
        f.write("<td>%s</td>\n" % name)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   538
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   539
        #
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   540
        # Write the elapsed time as a new tag data.
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   541
        #
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   542
        f.write("<td>%s</td>\n" % time)
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   543
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   544
        #
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   545
        # Write the reason, if it exist
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   546
        #
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   547
        f.write("<td>%s</td>\n" % reason)
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   548
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   549
        #
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   550
        # That's it for the current example, so terminate the row.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   551
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   552
        f.write("</tr>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   553
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   554
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   555
    # That's it for the table of examples, so terminate the table.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   556
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   557
    f.write("</table>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   558
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   559
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   560
    # And that's it for the report, so finish up.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   561
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   562
    f.write("</body>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   563
    f.write("</html>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   564
    f.close()
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   565
    print('done.')
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   566
    
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   567
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   568
# Python Control-C handling is broken in the presence of multiple threads.  
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   569
# Signals get delivered to the runnable/running thread by default and if 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   570
# it is blocked, the signal is simply ignored.  So we hook sigint and set 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   571
# a global variable telling the system to shut down gracefully.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   572
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   573
thread_exit = False
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   574
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   575
def sigint_hook(signal, frame):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   576
    global thread_exit
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   577
    thread_exit = True
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   578
    return 0
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   579
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   580
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   581
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   582
# In general, the build process itself naturally takes care of figuring out
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   583
# which tests are built into the test runner.  For example, if waf configure
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   584
# determines that ENABLE_EMU is false due to some missing dependency,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   585
# the tests for the emu net device simply will not be built and will 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   586
# therefore not be included in the built test runner.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   587
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   588
# Examples, however, are a different story.  In that case, we are just given
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   589
# a list of examples that could be run.  Instead of just failing, for example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   590
# nsc-tcp-zoo if NSC is not present, we look into the waf saved configuration
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   591
# for relevant configuration items.  
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   592
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   593
# XXX This function pokes around in the waf internal state file.  To be a
13665
b2861f3dc03d Spellcheck
Alexander Krotov <krotov@iitp.ru>
parents: 13237
diff changeset
   594
# little less hacky, we should add a command to waf to return this info
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   595
# and use that result.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   596
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   597
def read_waf_config():
7669
ab2af491785a align test.py with waf-1.6.10 changes
Tom Henderson <tomh@tomh.org>
parents: 7488
diff changeset
   598
    for line in open(".lock-waf_" + sys.platform + "_build", "rt"):
7713
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   599
        if line.startswith("top_dir ="):
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   600
            key, val = line.split('=')
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   601
            top_dir = eval(val.strip())
7488
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   602
        if line.startswith("out_dir ="):
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   603
            key, val = line.split('=')
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   604
            out_dir = eval(val.strip())
7713
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   605
    global NS3_BASEDIR
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   606
    NS3_BASEDIR = top_dir
7488
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   607
    global NS3_BUILDDIR
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   608
    NS3_BUILDDIR = out_dir
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   609
    for line in open("%s/c4che/_cache.py" % out_dir).readlines():
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   610
        for item in interesting_config_items:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   611
            if line.startswith(item):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   612
                exec(line, globals())
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   613
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   614
    if options.verbose:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   615
        for item in interesting_config_items:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   616
            print("%s ==" % item, eval(item))
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   617
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   618
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   619
# It seems pointless to fork a process to run waf to fork a process to run
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   620
# the test runner, so we just run the test runner directly.  The main thing 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   621
# that waf would do for us would be to sort out the shared library path but
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   622
# we can deal with that easily and do here.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   623
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   624
# There can be many different ns-3 repositories on a system, and each has 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   625
# its own shared libraries, so ns-3 doesn't hardcode a shared library search
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   626
# path -- it is cooked up dynamically, so we do that too.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   627
#
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   628
def make_paths():
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   629
    have_DYLD_LIBRARY_PATH = False
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   630
    have_LD_LIBRARY_PATH = False
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   631
    have_PATH = False
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   632
    have_PYTHONPATH = False
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   633
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   634
    keys = list(os.environ.keys())
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   635
    for key in keys:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   636
        if key == "DYLD_LIBRARY_PATH":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   637
            have_DYLD_LIBRARY_PATH = True
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   638
        if key == "LD_LIBRARY_PATH":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   639
            have_LD_LIBRARY_PATH = True
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   640
        if key == "PATH":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   641
            have_PATH = True
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   642
        if key == "PYTHONPATH":
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   643
            have_PYTHONPATH = True
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   644
7488
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   645
    pypath = os.environ["PYTHONPATH"] = os.path.join (NS3_BUILDDIR, "bindings", "python")
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   646
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   647
    if not have_PYTHONPATH:
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   648
        os.environ["PYTHONPATH"] = pypath
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   649
    else:
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   650
        os.environ["PYTHONPATH"] += ":" + pypath
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   651
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   652
    if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   653
        print("os.environ[\"PYTHONPATH\"] == %s" % os.environ["PYTHONPATH"])
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   654
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   655
    if sys.platform == "darwin":
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   656
        if not have_DYLD_LIBRARY_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   657
            os.environ["DYLD_LIBRARY_PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   658
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   659
            os.environ["DYLD_LIBRARY_PATH"] += ":" + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   660
        if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   661
            print("os.environ[\"DYLD_LIBRARY_PATH\"] == %s" % os.environ["DYLD_LIBRARY_PATH"])
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   662
    elif sys.platform == "win32":
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   663
        if not have_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   664
            os.environ["PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   665
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   666
            os.environ["PATH"] += ';' + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   667
        if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   668
            print("os.environ[\"PATH\"] == %s" % os.environ["PATH"])
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   669
    elif sys.platform == "cygwin":
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   670
        if not have_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   671
            os.environ["PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   672
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   673
            os.environ["PATH"] += ":" + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   674
        if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   675
            print("os.environ[\"PATH\"] == %s" % os.environ["PATH"])
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   676
    else:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   677
        if not have_LD_LIBRARY_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   678
            os.environ["LD_LIBRARY_PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   679
        for path in NS3_MODULE_PATH:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   680
            os.environ["LD_LIBRARY_PATH"] += ":" + str(path)
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   681
        if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   682
            print("os.environ[\"LD_LIBRARY_PATH\"] == %s" % os.environ["LD_LIBRARY_PATH"])
5275
d4008f2981ba Windows needs entire path
Craig Dowell <craigdo@ee.washington.edu>
parents: 5274
diff changeset
   683
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   684
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   685
# Short note on generating suppressions:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   686
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   687
# See the valgrind documentation for a description of suppressions.  The easiest
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   688
# way to generate a suppression expression is by using the valgrind 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   689
# --gen-suppressions option.  To do that you have to figure out how to run the 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   690
# test in question.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   691
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   692
# If you do "test.py -v -g -s <suitename> then test.py will output most of what
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   693
# you need.  For example, if you are getting a valgrind error in the
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   694
# devices-mesh-dot11s-regression test suite, you can run:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   695
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   696
#   ./test.py -v -g -s devices-mesh-dot11s-regression 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   697
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   698
# You should see in the verbose output something that looks like:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   699
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   700
#   Synchronously execute valgrind --suppressions=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/testpy.supp
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
   701
#   --leak-check=full --error-exitcode=2 /home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/build/debug/utils/ns3-dev-test-runner-debug 
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   702
#   --suite=devices-mesh-dot11s-regression --basedir=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   703
#   --tempdir=testpy-output/2010-01-12-22-47-50-CUT 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   704
#   --out=testpy-output/2010-01-12-22-47-50-CUT/devices-mesh-dot11s-regression.xml
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   705
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   706
# You need to pull out the useful pieces, and so could run the following to 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   707
# reproduce your error:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   708
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   709
#   valgrind --suppressions=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/testpy.supp
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
   710
#   --leak-check=full --error-exitcode=2 /home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/build/debug/utils/ns3-dev-test-runner-debug 
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   711
#   --suite=devices-mesh-dot11s-regression --basedir=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   712
#   --tempdir=testpy-output 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   713
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   714
# Hint: Use the first part of the command as is, and point the "tempdir" to 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   715
# somewhere real.  You don't need to specify an "out" file.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   716
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   717
# When you run the above command you should see your valgrind error.  The 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   718
# suppression expression(s) can be generated by adding the --gen-suppressions=yes
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   719
# option to valgrind.  Use something like:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   720
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   721
#   valgrind --gen-suppressions=yes --suppressions=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/testpy.supp
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
   722
#   --leak-check=full --error-exitcode=2 /home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/build/debug/utils/ns3-dev-test-runner-debug 
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   723
#   --suite=devices-mesh-dot11s-regression --basedir=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   724
#   --tempdir=testpy-output 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   725
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   726
# Now when valgrind detects an error it will ask:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   727
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   728
#   ==27235== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ----
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   729
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   730
# to which you just enter 'y'<ret>.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   731
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   732
# You will be provided with a suppression expression that looks something like
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   733
# the following:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   734
#   {
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   735
#     <insert_a_suppression_name_here>
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   736
#     Memcheck:Addr8
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   737
#     fun:_ZN3ns36dot11s15HwmpProtocolMac8SendPreqESt6vectorINS0_6IePreqESaIS3_EE
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   738
#     fun:_ZN3ns36dot11s15HwmpProtocolMac10SendMyPreqEv
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   739
#     fun:_ZN3ns36dot11s15HwmpProtocolMac18RequestDestinationENS_12Mac48AddressEjj
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   740
#     ...
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   741
#     the rest of the stack frame
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   742
#     ...
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   743
#   }
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   744
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   745
# You need to add a supression name which will only be printed out by valgrind in 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   746
# verbose mode (but it needs to be there in any case).  The entire stack frame is
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   747
# shown to completely characterize the error, but in most cases you won't need 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   748
# all of that info.  For example, if you want to turn off all errors that happen
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   749
# when the function (fun:) is called, you can just delete the rest of the stack
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   750
# frame.  You can also use wildcards to make the mangled signatures more readable.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   751
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   752
# I added the following to the testpy.supp file for this particular error:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   753
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   754
#   {
13665
b2861f3dc03d Spellcheck
Alexander Krotov <krotov@iitp.ru>
parents: 13237
diff changeset
   755
#     Suppress invalid read size errors in SendPreq() when using HwmpProtocolMac
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   756
#     Memcheck:Addr8
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   757
#     fun:*HwmpProtocolMac*SendPreq*
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   758
#   }
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   759
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   760
# Now, when you run valgrind the error will be suppressed.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   761
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   762
VALGRIND_SUPPRESSIONS_FILE = "testpy.supp"
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   763
7087
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   764
def run_job_synchronously(shell_command, directory, valgrind, is_python, build_path=""):
7713
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   765
    suppressions_path = os.path.join (NS3_BASEDIR, VALGRIND_SUPPRESSIONS_FILE)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   766
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   767
    if is_python:
7713
6b3d1a5495f7 Bug 1325
Mitch Watrous <watrous@u.washington.edu>
parents: 7689
diff changeset
   768
        path_cmd = PYTHON[0] + " " + os.path.join (NS3_BASEDIR, shell_command)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   769
    else:
7087
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   770
        if len(build_path):
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   771
            path_cmd = os.path.join (build_path, shell_command)
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   772
        else:
7488
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
   773
            path_cmd = os.path.join (NS3_BUILDDIR, shell_command)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   774
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   775
    if valgrind:
12888
6dcf3064c1ff test.py: (fixes #2743) Treat memory leaks as errors without parsing valgrind output
Alexander Krotov <krotov@iitp.ru>
parents: 12808
diff changeset
   776
        cmd = "valgrind --suppressions=%s --leak-check=full --show-reachable=yes --error-exitcode=2 --errors-for-leak-kinds=all %s" % (suppressions_path,
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   777
            path_cmd)
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   778
    else:
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   779
        cmd = path_cmd
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   780
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   781
    if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   782
        print("Synchronously execute %s" % cmd)
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   783
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   784
    start_time = time.time()
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   785
    proc = subprocess.Popen(cmd, shell = True, cwd = directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   786
    stdout_results, stderr_results = proc.communicate()
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   787
    elapsed_time = time.time() - start_time
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   788
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   789
    retval = proc.returncode
11741
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   790
    try:
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   791
        stdout_results = stdout_results.decode()
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   792
    except UnicodeDecodeError:
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   793
        print("Non-decodable character in stdout output of %s" % cmd)
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   794
        print(stdout_results)
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   795
        retval = 1
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   796
    try:
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   797
        stderr_results = stderr_results.decode()
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   798
    except UnicodeDecodeError:
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   799
        print("Non-decodable character in stderr output of %s" % cmd)
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   800
        print(stderr_results)
823fe9fed974 bug 2199 postfix: make unicode decode() more robust
Tom Henderson <tomh@tomh.org>
parents: 11731
diff changeset
   801
        retval = 1
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   802
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   803
    if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   804
        print("Return code = ", retval)
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   805
        print("stderr = ", stderr_results)
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   806
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   807
    return (retval, stdout_results, stderr_results, elapsed_time)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   808
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   809
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   810
# This class defines a unit of testing work.  It will typically refer to
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   811
# a test suite to run using the test-runner, or an example to run directly.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   812
#
5415
5c2f96b507a9 Stray () in test.py (bug 718)
Craig Dowell <craigdo@ee.washington.edu>
parents: 5414
diff changeset
   813
class Job:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   814
    def __init__(self):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   815
        self.is_break = False
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   816
        self.is_skip = False
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   817
        self.skip_reason = ""
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   818
        self.is_example = False
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   819
        self.is_pyexample = False
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   820
        self.shell_command = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   821
        self.display_name = ""
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   822
        self.basedir = ""
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   823
        self.tempdir = ""
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   824
        self.cwd = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   825
        self.tmp_file_name = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   826
        self.returncode = False
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   827
        self.elapsed_time = 0
7087
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   828
        self.build_path = ""
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   829
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   830
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   831
    # A job is either a standard job or a special job indicating that a worker
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   832
    # thread should exist.  This special job is indicated by setting is_break 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   833
    # to true.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   834
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   835
    def set_is_break(self, is_break):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   836
        self.is_break = is_break
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   837
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   838
    #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   839
    # If a job is to be skipped, we actually run it through the worker threads
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   840
    # to keep the PASS, FAIL, CRASH and SKIP processing all in one place.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   841
    #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   842
    def set_is_skip(self, is_skip):
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   843
        self.is_skip = is_skip
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   844
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   845
    #
13841
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   846
    # If a job is to be skipped, log the reason.
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   847
    #
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   848
    def set_skip_reason(self, skip_reason):
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   849
        self.skip_reason = skip_reason
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   850
55a1a4bbf970 test: show reason a test/example was skipped
Peter D. Barnes, Jr. <barnes26@llnl.gov>
parents: 13745
diff changeset
   851
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   852
    # Examples are treated differently than standard test suites.  This is
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   853
    # mostly because they are completely unaware that they are being run as 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   854
    # tests.  So we have to do some special case processing to make them look
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   855
    # like tests.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   856
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   857
    def set_is_example(self, is_example):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   858
        self.is_example = is_example
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   859
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   860
    #
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   861
    # Examples are treated differently than standard test suites.  This is
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   862
    # mostly because they are completely unaware that they are being run as 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   863
    # tests.  So we have to do some special case processing to make them look
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   864
    # like tests.
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   865
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   866
    def set_is_pyexample(self, is_pyexample):
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   867
        self.is_pyexample = is_pyexample
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   868
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   869
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   870
    # This is the shell command that will be executed in the job.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   871
    #
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
   872
    #  "utils/ns3-dev-test-runner-debug --test-name=some-test-suite"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   873
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   874
    def set_shell_command(self, shell_command):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   875
        self.shell_command = shell_command
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   876
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   877
    #
7087
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   878
    # This is the build path where ns-3 was built.  For example,
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   879
    #
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   880
    #  "/home/craigdo/repos/ns-3-allinone-test/ns-3-dev/build/debug"
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   881
    #
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   882
    def set_build_path(self, build_path):
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   883
        self.build_path = build_path
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   884
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   885
    #
13665
b2861f3dc03d Spellcheck
Alexander Krotov <krotov@iitp.ru>
parents: 13237
diff changeset
   886
    # This is the display name of the job, typically the test suite or example 
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   887
    # name.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   888
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   889
    #  "some-test-suite" or "udp-echo"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   890
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   891
    def set_display_name(self, display_name):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   892
        self.display_name = display_name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   893
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   894
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   895
    # This is the base directory of the repository out of which the tests are
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   896
    # being run.  It will be used deep down in the testing framework to determine
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   897
    # where the source directory of the test was, and therefore where to find 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   898
    # provided test vectors.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   899
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   900
    #  "/home/user/repos/ns-3-dev"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   901
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   902
    def set_basedir(self, basedir):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   903
        self.basedir = basedir
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   904
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   905
    #
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   906
    # This is the directory to which a running test suite should write any 
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   907
    # temporary files.
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   908
    #
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   909
    def set_tempdir(self, tempdir):
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   910
        self.tempdir = tempdir
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   911
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   912
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   913
    # This is the current working directory that will be given to an executing
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   914
    # test as it is being run.  It will be used for examples to tell them where
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   915
    # to write all of the pcap files that we will be carefully ignoring.  For
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   916
    # example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   917
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   918
    #  "/tmp/unchecked-traces"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   919
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   920
    def set_cwd(self, cwd):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   921
        self.cwd = cwd
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   922
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   923
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   924
    # This is the temporary results file name that will be given to an executing 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   925
    # test as it is being run.  We will be running all of our tests in parallel
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   926
    # so there must be multiple temporary output files.  These will be collected
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   927
    # into a single XML file at the end and then be deleted.  
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   928
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   929
    def set_tmp_file_name(self, tmp_file_name):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   930
        self.tmp_file_name = tmp_file_name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   931
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   932
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   933
    # The return code received when the job process is executed.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   934
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   935
    def set_returncode(self, returncode):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   936
        self.returncode = returncode
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   937
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   938
    #
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   939
    # The elapsed real time for the job execution.
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   940
    #
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   941
    def set_elapsed_time(self, elapsed_time):
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   942
        self.elapsed_time = elapsed_time
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   943
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   944
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   945
# The worker thread class that handles the actual running of a given test.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   946
# Once spawned, it receives requests for work through its input_queue and
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   947
# ships the results back through the output_queue.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   948
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   949
class worker_thread(threading.Thread):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   950
    def __init__(self, input_queue, output_queue):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   951
        threading.Thread.__init__(self)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   952
        self.input_queue = input_queue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   953
        self.output_queue = output_queue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   954
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   955
    def run(self):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   956
        while True:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   957
            job = self.input_queue.get()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   958
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   959
            # Worker threads continue running until explicitly told to stop with
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   960
            # a special job.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   961
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   962
            if job.is_break:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   963
                return
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   964
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   965
            # If the global interrupt handler sets the thread_exit variable,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   966
            # we stop doing real work and just report back a "break" in the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   967
            # normal command processing has happened.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   968
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   969
            if thread_exit == True:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   970
                job.set_is_break(True)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   971
                self.output_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   972
                continue
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   973
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   974
            #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   975
            # If we are actually supposed to skip this job, do so.  Note that
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   976
            # if is_skip is true, returncode is undefined.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   977
            #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   978
            if job.is_skip:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   979
                if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   980
                    print("Skip %s" % job.shell_command)
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   981
                self.output_queue.put(job)
6253
3aa30fa758f5 bug 886
Craig Dowell <craigdo@ee.washington.edu>
parents: 6247
diff changeset
   982
                continue
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   983
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   984
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   985
            # Otherwise go about the business of running tests as normal.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   986
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   987
            else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   988
                if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
   989
                    print("Launch %s" % job.shell_command)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   990
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   991
                if job.is_example or job.is_pyexample:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   992
                    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   993
                    # If we have an example, the shell command is all we need to
7819
30638cb5fd62 fix test.py comments
Tom Henderson <tomh@tomh.org>
parents: 7713
diff changeset
   994
                    # know.  It will be something like "examples/udp/udp-echo" or 
30638cb5fd62 fix test.py comments
Tom Henderson <tomh@tomh.org>
parents: 7713
diff changeset
   995
                    # "examples/wireless/mixed-wireless.py"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   996
                    #
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   997
                    (job.returncode, standard_out, standard_err, et) = run_job_synchronously(job.shell_command, 
7087
18ae03e01c6f Bug 1108 - test.py -e option needs repair
Mitch Watrous <watrous@u.washington.edu>
parents: 7028
diff changeset
   998
                        job.cwd, options.valgrind, job.is_pyexample, job.build_path)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   999
                else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1000
                    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1001
                    # If we're a test suite, we need to provide a little more info
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1002
                    # to the test runner, specifically the base directory and temp
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1003
                    # file name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1004
                    #
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
  1005
                    if options.update_data:
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
  1006
                        update_data = '--update-data'
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
  1007
                    else:
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
  1008
                        update_data = ''
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1009
                    (job.returncode, standard_out, standard_err, et) = run_job_synchronously(job.shell_command + 
7379
ff0d074f885a add support for --update-data to test.py, move test runner code to library for modular build, add support for hierarchical tests beyond the previous two-level hierarchy.
Mathieu Lacage <mathieu.lacage@gmail.com>
parents: 7259
diff changeset
  1010
                        " --xml --tempdir=%s --out=%s %s" % (job.tempdir, job.tmp_file_name, update_data), 
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1011
                        job.cwd, options.valgrind, False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1012
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1013
                job.set_elapsed_time(et)
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1014
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1015
                if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1016
                    print("returncode = %d" % job.returncode)
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1017
                    print("---------- begin standard out ----------")
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1018
                    print(standard_out)
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1019
                    print("---------- begin standard err ----------")
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1020
                    print(standard_err)
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1021
                    print("---------- end standard err ----------")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1022
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1023
                self.output_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1024
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1025
#
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1026
# This is the main function that does the work of interacting with the
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1027
# test-runner itself.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1028
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1029
def run_tests():
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1030
    #
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1031
    # Pull some interesting configuration information out of waf, primarily
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1032
    # so we can know where executables can be found, but also to tell us what
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1033
    # pieces of the system have been built.  This will tell us what examples 
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1034
    # are runnable.
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1035
    #
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1036
    read_waf_config()
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1037
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1038
    #
9850
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1039
    # Set the proper suffix.
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1040
    #
9851
33cfe647ba55 Make examples again work for nonrelease builds
Mitch Watrous <watrous@u.washington.edu>
parents: 9850
diff changeset
  1041
    global BUILD_PROFILE_SUFFIX
9850
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1042
    if BUILD_PROFILE == 'release': 
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1043
        BUILD_PROFILE_SUFFIX = ""
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1044
    else:
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1045
        BUILD_PROFILE_SUFFIX = "-" + BUILD_PROFILE
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1046
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1047
    #
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1048
    # Add the proper prefix and suffix to the test-runner name to
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1049
    # match what is done in the wscript file.
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1050
    #
9850
aff1dc7687f1 Bug 1707 - test.py doesn't work with the release profile
Mitch Watrous <watrous@u.washington.edu>
parents: 9832
diff changeset
  1051
    test_runner_name = "%s%s-%s%s" % (APPNAME, VERSION, "test-runner", BUILD_PROFILE_SUFFIX)
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1052
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1053
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1054
    # Run waf to make sure that everything is built, configured and ready to go
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1055
    # unless we are explicitly told not to.  We want to be careful about causing
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1056
    # our users pain while waiting for extraneous stuff to compile and link, so
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1057
    # we allow users that know what they''re doing to not invoke waf at all.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1058
    #
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1059
    if not options.nowaf:
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1060
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1061
        #
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1062
        # If the user is running the "kinds" or "list" options, there is an 
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1063
        # implied dependency on the test-runner since we call that program
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1064
        # if those options are selected.  We will exit after processing those
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1065
        # options, so if we see them, we can safely only build the test-runner.
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1066
        #
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1067
        # If the user has constrained us to running only a particular type of
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1068
        # file, we can only ask waf to build what we know will be necessary.
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1069
        # For example, if the user only wants to run BVT tests, we only have
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1070
        # to build the test-runner and can ignore all of the examples.
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1071
        #
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1072
        # If the user only wants to run a single example, then we can just build
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1073
        # that example.
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1074
        #
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1075
        # If there is no constraint, then we have to build everything since the
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1076
        # user wants to run everything.
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1077
        #
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1078
        if options.kinds or options.list or (len(options.constrain) and options.constrain in core_kinds):
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1079
            if sys.platform == "win32":
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1080
                waf_cmd = sys.executable + " waf --target=test-runner"
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1081
            else:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1082
                waf_cmd = sys.executable + " waf --target=test-runner"
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1083
        elif len(options.example):
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1084
            if sys.platform == "win32": #Modify for windows
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1085
                waf_cmd = sys.executable + " waf --target=%s" % os.path.basename(options.example)
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1086
            else:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1087
                waf_cmd = sys.executable + " waf --target=%s" % os.path.basename(options.example)
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1088
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1089
        else:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1090
            if sys.platform == "win32": #Modify for windows
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1091
                waf_cmd = sys.executable + " waf"
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1092
            else:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1093
                waf_cmd = sys.executable + " waf"
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1094
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1095
        if options.verbose:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1096
            print("Building: %s" % waf_cmd)
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1097
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1098
        proc = subprocess.Popen(waf_cmd, shell = True)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1099
        proc.communicate()
6315
1f53e122254b fix bug 908
Andrey Mazo <mazo@iitp.ru>
parents: 6312
diff changeset
  1100
        if proc.returncode:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1101
            print("Waf died. Not running tests", file=sys.stderr)
6315
1f53e122254b fix bug 908
Andrey Mazo <mazo@iitp.ru>
parents: 6312
diff changeset
  1102
            return proc.returncode
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1103
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1104
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1105
    #
7682
8813ef4cec0f Bug 1339 - test.py uses the old name for the test runner
Mitch Watrous <watrous@u.washington.edu>
parents: 7669
diff changeset
  1106
    # Dynamically set up paths.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1107
    #
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
  1108
    make_paths()
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1109
9076
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1110
    #
7096
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1111
    # Get the information from the build status file.
9076
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1112
    #
7488
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
  1113
    build_status_file = os.path.join (NS3_BUILDDIR, 'build-status.py')
7096
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1114
    if os.path.exists(build_status_file):
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1115
        ns3_runnable_programs = get_list_from_file(build_status_file, "ns3_runnable_programs")
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1116
        ns3_runnable_scripts = get_list_from_file(build_status_file, "ns3_runnable_scripts")
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1117
    else:
11634
99173c0ad09b port wscripts, test.py, and waf-tools to Python3
Siddharth Santurkar <siddharth.santurkar@ieee.org>
parents: 11412
diff changeset
  1118
        print('The build status file was not found.  You must do waf build before running test.py.', file=sys.stderr)
7096
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1119
        sys.exit(2)
7357a40d2dde Make test.py not run examples that are not runnable
Mitch Watrous <watrous@u.washington.edu>
parents: 7087
diff changeset
  1120
9076
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1121
    #
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1122
    # Make a dictionary that maps the name of a program to its path.
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1123
    #
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1124
    ns3_runnable_programs_dictionary = {}
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1125
    for program in ns3_runnable_programs:
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1126
        # Remove any directory names from path.
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1127
        program_name = os.path.basename(program)
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1128
        ns3_runnable_programs_dictionary[program_name] = program
7c9f34fa2dd6 Bug 1463 - ./test.py --example should automatically detect the path
Mitch Watrous <watrous@u.washington.edu>
parents: 9075
diff changeset
  1129
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1130
    # Generate the lists of examples to run as smoke tests in order to
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1131
    # ensure that they remain buildable and runnable over time.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1132
    #
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1133
    example_tests = []
9220
fd38b9505eb9 Add shell script that runs all examples with full logging turned on
Mitch Watrous <watrous@u.washington.edu>
parents: 9219
diff changeset
  1134
    example_names_original = []
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1135
    python_tests = []
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1136
    for directory in EXAMPLE_DIRECTORIES:
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1137
        # Set the directories and paths for this example. 
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1138
        example_directory   = os.path.join("examples", directory)
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1139
        examples_to_run_path = os.path.join(example_directory, "examples-to-run.py")
7488
72d0c878f3c7 More waf 1.6.7 build fixes
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 7411
diff changeset
  1140
        cpp_executable_dir   = os.path.join(NS3_BUILDDIR, example_directory)
6886
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1141
        python_script_dir    = os.path.join(example_directory)
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1142
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1143
        # Parse this example directory's file.
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1144
        parse_examples_to_run_file(
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1145
            examples_to_run_path,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1146
            cpp_executable_dir,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1147
            python_script_dir,
690b7a78bbd0 Make test.py handle examples-to-run.py files
Mitch Watrous <watrous@u.washington.edu>
parents: 6863
diff changeset
  1148
            example_tests,
9220
fd38b9505eb9 Add shell script that runs all examples with full logging turned on
Mitch Watrous <watrous@u.washington.edu>
parents: 9219
diff changeset
  1149
            example_names_original,
6886