test.py
author Dean Armstrong <deanarm@gmail.com>
Thu, 03 Jun 2010 17:23:55 +0100
changeset 6331 eee2eab36748
parent 6327 246934d7b912
child 6337 92c95748a915
permissions -rwxr-xr-x
Bug 910: Replace AccessClass with 802.11-style AcIndex
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     1
#! /usr/bin/env python
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     2
## -*- Mode: python; py-indent-offset: 4; indent-tabs-mode: nil; coding: utf-8; -*-
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     3
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     4
# Copyright (c) 2009 University of Washington
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     5
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     6
# This program is free software; you can redistribute it and/or modify
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     7
# it under the terms of the GNU General Public License version 2 as
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     8
# published by the Free Software Foundation;
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
     9
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    10
# This program is distributed in the hope that it will be useful,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    11
# but WITHOUT ANY WARRANTY; without even the implied warranty of
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    12
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    13
# GNU General Public License for more details.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    14
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    15
# You should have received a copy of the GNU General Public License
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    16
# along with this program; if not, write to the Free Software
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    17
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    18
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    19
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    20
import os
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    21
import sys
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
    22
import time
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    23
import optparse
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    24
import subprocess
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    25
import threading
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    26
import Queue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    27
import signal
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    28
import xml.dom.minidom
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
    29
import shutil
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
    30
import re
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    31
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    32
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    33
# XXX This should really be part of a waf command to list the configuration
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    34
# items relative to optional ns-3 pieces.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    35
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    36
# A list of interesting configuration items in the waf configuration 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    37
# cache which we may be interested in when deciding on which examples
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    38
# to run and how to run them.  These are set by waf during the 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    39
# configuration phase and the corresponding assignments are usually
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    40
# found in the associated subdirectory wscript files.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    41
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    42
interesting_config_items = [
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    43
    "NS3_BUILDDIR",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    44
    "NS3_MODULE_PATH",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    45
    "ENABLE_NSC",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    46
    "ENABLE_REAL_TIME",
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    47
    "ENABLE_EXAMPLES",
6243
a597d6d2da85 Bug 882 - ./test.py reports 6 FAILs with --disable-python
Gustavo J. A. M. Carneiro <gjc@inescporto.pt>
parents: 6227
diff changeset
    48
    "ENABLE_PYTHON_BINDINGS",
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    49
]
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    50
5295
c1bd4ffb5e47 fix test.py -v
Craig Dowell <craigdo@ee.washington.edu>
parents: 5279
diff changeset
    51
ENABLE_NSC = False
c1bd4ffb5e47 fix test.py -v
Craig Dowell <craigdo@ee.washington.edu>
parents: 5279
diff changeset
    52
ENABLE_REAL_TIME = False
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    53
ENABLE_EXAMPLES = True
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    54
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    55
#
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    56
# If the user has constrained us to run certain kinds of tests, we can tell waf
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    57
# to only build
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    58
#
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    59
core_kinds = ["bvt", "core", "system", "unit"]
5295
c1bd4ffb5e47 fix test.py -v
Craig Dowell <craigdo@ee.washington.edu>
parents: 5279
diff changeset
    60
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    61
#
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    62
# There are some special cases for test suites that kill valgrind.  This is
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    63
# because NSC causes illegal instruction crashes when run under valgrind.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    64
#
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    65
core_valgrind_skip_tests = [
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    66
    "ns3-tcp-cwnd",
6198
cd1b101cae62 suppress valgrind testing of new nsc test suite
Tom Henderson <tomh@tomh.org>
parents: 6127
diff changeset
    67
    "nsc-tcp-loss",
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    68
    "ns3-tcp-interoperability",
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    69
]
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    70
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    71
#
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    72
# A list of examples to run as smoke tests just to ensure that they remain 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    73
# buildable and runnable over time.  Also a condition under which to run
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    74
# the example (from the waf configuration), and a condition under which to
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    75
# run the example under valgrind.  This is because NSC causes illegal 
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    76
# instruction crashes when run under valgrind.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    77
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    78
# XXX Should this not be read from a configuration file somewhere and not
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    79
# hardcoded.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    80
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
    81
example_tests = [
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    82
    ("csma/csma-bridge", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    83
    ("csma/csma-bridge-one-hop", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    84
    ("csma/csma-broadcast", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    85
    ("csma/csma-multicast", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    86
    ("csma/csma-one-subnet", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    87
    ("csma/csma-packet-socket", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    88
    ("csma/csma-ping", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    89
    ("csma/csma-raw-ip-socket", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    90
    ("csma/csma-star", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    91
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    92
    ("emulation/emu-ping", "False", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    93
    ("emulation/emu-udp-echo", "False", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    94
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    95
    ("error-model/simple-error-model", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
    96
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    97
    ("ipv6/icmpv6-redirect", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    98
    ("ipv6/ping6", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
    99
    ("ipv6/radvd", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   100
    ("ipv6/radvd-two-prefix", "True", "True"),    
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   101
    ("ipv6/test-ipv6", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   102
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   103
    ("mesh/mesh", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   104
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   105
    ("naming/object-names", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   106
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   107
    ("realtime/realtime-udp-echo", "ENABLE_REAL_TIME == True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   108
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   109
    ("routing/dynamic-global-routing", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   110
    ("routing/global-injection-slash32", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   111
    ("routing/global-routing-slash32", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   112
    ("routing/mixed-global-routing", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   113
    ("routing/nix-simple", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   114
    ("routing/nms-p2p-nix", "False", "True"), # Takes too long to run
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   115
    ("routing/simple-alternate-routing", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   116
    ("routing/simple-global-routing", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   117
    ("routing/simple-point-to-point-olsr", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   118
    ("routing/simple-routing-ping6", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   119
    ("routing/static-routing-slash32", "True", "True"),
5741
83919caa6208 [aodv] examples/routing/aodv added to tested examples
Pavel Boyko <boyko@iitp.ru>
parents: 5483
diff changeset
   120
    ("routing/aodv", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   121
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   122
    ("stats/wifi-example-sim", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   123
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   124
    ("tap/tap-wifi-dumbbell", "False", "True"), # Requires manual configuration
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   125
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   126
    ("tcp/star", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   127
    ("tcp/tcp-large-transfer", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   128
    ("tcp/tcp-nsc-lfn", "ENABLE_NSC == True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   129
    ("tcp/tcp-nsc-zoo", "ENABLE_NSC == True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   130
    ("tcp/tcp-star-server", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   131
6127
b5bc10de166d merge topology read system
Tommaso Pecorella <tpecorella@mac.com>
parents: 6111
diff changeset
   132
    ("topology-read/topology-read --input=../../examples/topology-read/Inet_small_toposample.txt", "True", "True"),
b5bc10de166d merge topology read system
Tommaso Pecorella <tpecorella@mac.com>
parents: 6111
diff changeset
   133
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   134
    ("tunneling/virtual-net-device", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   135
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   136
    ("tutorial/first", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   137
    ("tutorial/hello-simulator", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   138
    ("tutorial/second", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   139
    ("tutorial/third", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   140
    ("tutorial/fourth", "True", "True"),
5483
ad83d869c08c Make tests pass on MinGW, add fifth tutorial example
Craig Dowell <craigdo@ee.washington.edu>
parents: 5481
diff changeset
   141
    ("tutorial/fifth", "True", "True"),
6046
46665d75667d output stream object inherits from simple ref count
Craig Dowell <craigdo@ee.washington.edu>
parents: 5917
diff changeset
   142
    ("tutorial/sixth", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   143
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   144
    ("udp/udp-echo", "True", "True"),
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   145
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   146
    ("wireless/mixed-wireless", "True", "True"),
6311
f8ca463163fd update multirate example for test.py
Duy Nguyen <duy@soe.ucsc.edu>
parents: 6310
diff changeset
   147
    ("wireless/multirate --totalTime=0.3s --rateManager=ns3::AarfcdWifiManager", "True", "True"), 
f8ca463163fd update multirate example for test.py
Duy Nguyen <duy@soe.ucsc.edu>
parents: 6310
diff changeset
   148
    ("wireless/multirate --totalTime=0.3s --rateManager=ns3::AmrrWifiManager", "True", "True"), 
f8ca463163fd update multirate example for test.py
Duy Nguyen <duy@soe.ucsc.edu>
parents: 6310
diff changeset
   149
    ("wireless/multirate --totalTime=0.3s --rateManager=ns3::CaraWifiManager", "True", "True"), 
f8ca463163fd update multirate example for test.py
Duy Nguyen <duy@soe.ucsc.edu>
parents: 6310
diff changeset
   150
    ("wireless/multirate --totalTime=0.3s --rateManager=ns3::IdealWifiManager", "True", "True"), 
f8ca463163fd update multirate example for test.py
Duy Nguyen <duy@soe.ucsc.edu>
parents: 6310
diff changeset
   151
    ("wireless/multirate --totalTime=0.3s --rateManager=ns3::OnoeWifiManager", "True", "True"), 
f8ca463163fd update multirate example for test.py
Duy Nguyen <duy@soe.ucsc.edu>
parents: 6310
diff changeset
   152
    ("wireless/multirate --totalTime=0.3s --rateManager=ns3::RraaWifiManager", "True", "True"), 
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   153
    ("wireless/simple-wifi-frame-aggregation", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   154
    ("wireless/wifi-adhoc", "False", "True"), # Takes too long to run
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   155
    ("wireless/wifi-ap --verbose=0", "True", "True"), # Don't let it spew to stdout
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   156
    ("wireless/wifi-clear-channel-cmu", "False", "True"), # Requires specific hardware
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   157
    ("wireless/wifi-simple-adhoc", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   158
    ("wireless/wifi-simple-adhoc-grid", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   159
    ("wireless/wifi-simple-infra", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   160
    ("wireless/wifi-simple-interference", "True", "True"),
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   161
    ("wireless/wifi-wired-bridging", "True", "True"),
6111
0b22ae082153 Merge WiMAX module
Mohamed Amine Ismail <amine.ismail@sophia.inria.fr>
parents: 6083
diff changeset
   162
0b22ae082153 Merge WiMAX module
Mohamed Amine Ismail <amine.ismail@sophia.inria.fr>
parents: 6083
diff changeset
   163
    ("wimax/wimax-simple", "True", "True"),
0b22ae082153 Merge WiMAX module
Mohamed Amine Ismail <amine.ismail@sophia.inria.fr>
parents: 6083
diff changeset
   164
    ("wimax/wimax-ipv4", "True", "True"),
0b22ae082153 Merge WiMAX module
Mohamed Amine Ismail <amine.ismail@sophia.inria.fr>
parents: 6083
diff changeset
   165
    ("wimax/wimax-multicast", "True", "True"),
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   166
]
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   167
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   168
#
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   169
# A list of python examples to run as smoke tests just to ensure that they 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   170
# runnable over time.  Also a condition under which to run the example (from
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   171
# the waf configuration)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   172
#
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   173
# XXX Should this not be read from a configuration file somewhere and not
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   174
# hardcoded.
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   175
#
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   176
python_tests = [
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   177
    ("csma/csma-bridge.py", "True"),
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   178
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   179
    ("flowmon/wifi-olsr-flowmon.py", "True"),
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   180
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   181
    ("routing/simple-routing-ping6.py", "True"),
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   182
6202
c6b85405f6fe some python examples require enable-sudo, skip
Craig Dowell <craigdo@ee.washington.edu>
parents: 6201
diff changeset
   183
    ("tap/tap-csma-virtual-machine.py", "False"), # requires enable-sudo
c6b85405f6fe some python examples require enable-sudo, skip
Craig Dowell <craigdo@ee.washington.edu>
parents: 6201
diff changeset
   184
    ("tap/tap-wifi-virtual-machine.py", "False"), # requires enable-sudo
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   185
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   186
    ("tutorial/first.py", "True"),
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   187
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   188
    ("wireless/wifi-ap.py", "True"),
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   189
    ("wireless/mixed-wireless.py", "True"),
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   190
]
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   191
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   192
#
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   193
# The test suites are going to want to output status.  They are running
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   194
# concurrently.  This means that unless we are careful, the output of
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   195
# the test suites will be interleaved.  Rather than introducing a lock
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   196
# file that could unintentionally start serializing execution, we ask
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   197
# the tests to write their output to a temporary directory and then 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   198
# put together the final output file when we "join" the test tasks back
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   199
# to the main thread.  In addition to this issue, the example programs
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   200
# often write lots and lots of trace files which we will just ignore.
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   201
# We put all of them into the temp directory as well, so they can be
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   202
# easily deleted.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   203
#
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   204
TMP_OUTPUT_DIR = "testpy-output"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   205
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   206
def get_node_text(node):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   207
    for child in node.childNodes:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   208
        if child.nodeType == child.TEXT_NODE:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   209
            return child.nodeValue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   210
    return "None"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   211
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   212
#
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   213
# A simple example of writing a text file with a test result summary.  It is 
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   214
# expected that this output will be fine for developers looking for problems.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   215
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   216
def translate_to_text(results_file, text_file):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   217
    f = open(text_file, 'w')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   218
    dom = xml.dom.minidom.parse(results_file)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   219
    for suite in dom.getElementsByTagName("TestSuite"):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   220
        result = get_node_text(suite.getElementsByTagName("SuiteResult")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   221
        name = get_node_text(suite.getElementsByTagName("SuiteName")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   222
        time = get_node_text(suite.getElementsByTagName("SuiteTime")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   223
        output = "%s: Test Suite \"%s\" (%s)\n" % (result, name, time)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   224
        f.write(output)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   225
        if result != "CRASH":
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   226
            for case in suite.getElementsByTagName("TestCase"):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   227
                result = get_node_text(case.getElementsByTagName("CaseResult")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   228
                name = get_node_text(case.getElementsByTagName("CaseName")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   229
                time = get_node_text(case.getElementsByTagName("CaseTime")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   230
                output =   "  %s: Test Case \"%s\" (%s)\n" % (result, name, time)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   231
                f.write(output)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   232
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   233
                if result == "FAIL":
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   234
                    for details in case.getElementsByTagName("FailureDetails"):
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   235
                        f.write("    Details:\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   236
                        f.write("      Message:   %s\n" % get_node_text(details.getElementsByTagName("Message")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   237
                        f.write("      Condition: %s\n" % get_node_text(details.getElementsByTagName("Condition")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   238
                        f.write("      Actual:    %s\n" % get_node_text(details.getElementsByTagName("Actual")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   239
                        f.write("      Limit:     %s\n" % get_node_text(details.getElementsByTagName("Limit")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   240
                        f.write("      File:      %s\n" % get_node_text(details.getElementsByTagName("File")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   241
                        f.write("      Line:      %s\n" % get_node_text(details.getElementsByTagName("Line")[0]))
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   242
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   243
    for example in dom.getElementsByTagName("Example"):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   244
        result = get_node_text(example.getElementsByTagName("Result")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   245
        name = get_node_text(example.getElementsByTagName("Name")[0])
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   246
        time = get_node_text(example.getElementsByTagName("ElapsedTime")[0])
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   247
        output = "%s: Example \"%s\" (%s)\n" % (result, name, time)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   248
        f.write(output)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   249
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   250
    f.close()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   251
    
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   252
#
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   253
# A simple example of writing an HTML file with a test result summary.  It is 
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   254
# expected that this will eventually be made prettier as time progresses and
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   255
# we have time to tweak it.  This may end up being moved to a separate module
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   256
# since it will probably grow over time.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   257
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   258
def translate_to_html(results_file, html_file):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   259
    f = open(html_file, 'w')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   260
    f.write("<html>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   261
    f.write("<body>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   262
    f.write("<center><h1>ns-3 Test Results</h1></center>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   263
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   264
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   265
    # Read and parse the whole results file.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   266
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   267
    dom = xml.dom.minidom.parse(results_file)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   268
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   269
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   270
    # Iterate through the test suites
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   271
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   272
    f.write("<h2>Test Suites</h2>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   273
    for suite in dom.getElementsByTagName("TestSuite"):
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   274
     
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   275
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   276
        # For each test suite, get its name, result and execution time info
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   277
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   278
        name = get_node_text(suite.getElementsByTagName("SuiteName")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   279
        result = get_node_text(suite.getElementsByTagName("SuiteResult")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   280
        time = get_node_text(suite.getElementsByTagName("SuiteTime")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   281
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   282
        # 
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   283
        # Print a level three header with the result, name and time.  If the 
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   284
        # test suite passed, the header is printed in green. If the suite was
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   285
        # skipped, print it in orange, otherwise assume something bad happened
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   286
        # and print in red.
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   287
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   288
        if result == "PASS":
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   289
            f.write("<h3 style=\"color:green\">%s: %s (%s)</h3>\n" % (result, name, time))
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   290
        elif result == "SKIP":
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   291
            f.write("<h3 style=\"color:#ff6600\">%s: %s (%s)</h3>\n" % (result, name, time))
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   292
        else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   293
            f.write("<h3 style=\"color:red\">%s: %s (%s)</h3>\n" % (result, name, time))
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   294
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   295
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   296
        # The test case information goes in a table.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   297
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   298
        f.write("<table border=\"1\">\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   299
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   300
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   301
        # The first column of the table has the heading Result
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   302
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   303
        f.write("<th> Result </th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   304
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   305
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   306
        # If the suite crashed or is skipped, there is no further information, so just
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   307
        # delare a new table row with the result (CRASH or SKIP) in it.  Looks like:
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   308
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   309
        #   +--------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   310
        #   | Result |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   311
        #   +--------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   312
        #   | CRASH  |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   313
        #   +--------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   314
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   315
        # Then go on to the next test suite.  Valgrind and skipped errors look the same.
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   316
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   317
        if result in ["CRASH", "SKIP", "VALGR"]:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   318
            f.write("<tr>\n")
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   319
            if result == "SKIP":
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   320
                f.write("<td style=\"color:#ff6600\">%s</td>\n" % result)
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   321
            else:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   322
                f.write("<td style=\"color:red\">%s</td>\n" % result)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   323
            f.write("</tr>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   324
            f.write("</table>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   325
            continue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   326
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   327
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   328
        # If the suite didn't crash, we expect more information, so fill out
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   329
        # the table heading row.  Like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   330
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   331
        #   +--------+----------------+------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   332
        #   | Result | Test Case Name | Time |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   333
        #   +--------+----------------+------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   334
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   335
        f.write("<th>Test Case Name</th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   336
        f.write("<th> Time </th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   337
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   338
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   339
        # If the test case failed, we need to print out some failure details
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   340
        # so extend the heading row again.  Like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   341
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   342
        #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   343
        #   | Result | Test Case Name | Time | Failure Details |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   344
        #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   345
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   346
        if result == "FAIL":
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   347
            f.write("<th>Failure Details</th>\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   348
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   349
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   350
        # Now iterate through all of the test cases.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   351
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   352
        for case in suite.getElementsByTagName("TestCase"):
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   353
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   354
            #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   355
            # Get the name, result and timing information from xml to use in
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   356
            # printing table below.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   357
            #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   358
            name = get_node_text(case.getElementsByTagName("CaseName")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   359
            result = get_node_text(case.getElementsByTagName("CaseResult")[0])
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   360
            time = get_node_text(case.getElementsByTagName("CaseTime")[0])
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   361
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   362
            #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   363
            # If the test case failed, we iterate through possibly multiple
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   364
            # failure details
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   365
            #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   366
            if result == "FAIL":
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   367
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   368
                # There can be multiple failures for each test case.  The first
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   369
                # row always gets the result, name and timing information along
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   370
                # with the failure details.  Remaining failures don't duplicate
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   371
                # this information but just get blanks for readability.  Like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   372
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   373
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   374
                #   | Result | Test Case Name | Time | Failure Details |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   375
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   376
                #   |  FAIL  | The name       | time | It's busted     |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   377
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   378
                #   |        |                |      | Really broken   |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   379
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   380
                #   |        |                |      | Busted bad      |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   381
                #   +--------+----------------+------+-----------------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   382
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   383
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   384
                first_row = True
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   385
                for details in case.getElementsByTagName("FailureDetails"):
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   386
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   387
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   388
                    # Start a new row in the table for each possible Failure Detail
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   389
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   390
                    f.write("<tr>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   391
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   392
                    if first_row:
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   393
                        first_row = False
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   394
                        f.write("<td style=\"color:red\">%s</td>\n" % result)
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   395
                        f.write("<td>%s</td>\n" % name)
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   396
                        f.write("<td>%s</td>\n" % time)
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   397
                    else:
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   398
                        f.write("<td></td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   399
                        f.write("<td></td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   400
                        f.write("<td></td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   401
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   402
                    f.write("<td>")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   403
                    f.write("<b>Message: </b>%s, " % get_node_text(details.getElementsByTagName("Message")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   404
                    f.write("<b>Condition: </b>%s, " % get_node_text(details.getElementsByTagName("Condition")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   405
                    f.write("<b>Actual: </b>%s, " % get_node_text(details.getElementsByTagName("Actual")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   406
                    f.write("<b>Limit: </b>%s, " % get_node_text(details.getElementsByTagName("Limit")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   407
                    f.write("<b>File: </b>%s, " % get_node_text(details.getElementsByTagName("File")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   408
                    f.write("<b>Line: </b>%s" % get_node_text(details.getElementsByTagName("Line")[0]))
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   409
                    f.write("</td>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   410
                    
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   411
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   412
                    # End the table row
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   413
                    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   414
                    f.write("</td>\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   415
            else:
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   416
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   417
                # If this particular test case passed, then we just print the PASS
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   418
                # result in green, followed by the test case name and its execution
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   419
                # time information.  These go off in <td> ... </td> table data.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   420
                # The details table entry is left blank.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   421
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   422
                #   +--------+----------------+------+---------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   423
                #   | Result | Test Case Name | Time | Details |
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   424
                #   +--------+----------------+------+---------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   425
                #   |  PASS  | The name       | time |         |   
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   426
                #   +--------+----------------+------+---------+
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   427
                #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   428
                f.write("<tr>\n")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   429
                f.write("<td style=\"color:green\">%s</td>\n" % result)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   430
                f.write("<td>%s</td>\n" % name)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   431
                f.write("<td>%s</td>\n" % time)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   432
                f.write("<td></td>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   433
                f.write("</tr>\n")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   434
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   435
        # All of the rows are written, so we need to end the table.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   436
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   437
        f.write("</table>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   438
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   439
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   440
    # That's it for all of the test suites.  Now we have to do something about 
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   441
    # our examples.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   442
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   443
    f.write("<h2>Examples</h2>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   444
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   445
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   446
    # Example status is rendered in a table just like the suites.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   447
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   448
    f.write("<table border=\"1\">\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   449
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   450
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   451
    # The table headings look like,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   452
    #
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   453
    #   +--------+--------------+--------------+
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   454
    #   | Result | Example Name | Elapsed Time |
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   455
    #   +--------+--------------+--------------+
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   456
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   457
    f.write("<th> Result </th>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   458
    f.write("<th>Example Name</th>\n")
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   459
    f.write("<th>Elapsed Time</th>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   460
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   461
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   462
    # Now iterate through all of the examples
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   463
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   464
    for example in dom.getElementsByTagName("Example"):
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   465
        
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   466
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   467
        # Start a new row for each example
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   468
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   469
        f.write("<tr>\n")
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   470
        
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   471
        #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   472
        # Get the result and name of the example in question
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   473
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   474
        result = get_node_text(example.getElementsByTagName("Result")[0])
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   475
        name =   get_node_text(example.getElementsByTagName("Name")[0])
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   476
        time =   get_node_text(example.getElementsByTagName("ElapsedTime")[0])
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   477
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   478
        #
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   479
        # If the example either failed or crashed, print its result status
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   480
        # in red; otherwise green.  This goes in a <td> ... </td> table data
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   481
        #
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   482
        if result == "PASS":
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   483
            f.write("<td style=\"color:green\">%s</td>\n" % result)
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   484
        elif result == "SKIP":
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   485
            f.write("<td style=\"color:#ff6600\">%s</fd>\n" % result)
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   486
        else:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   487
            f.write("<td style=\"color:red\">%s</td>\n" % result)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   488
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   489
        #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   490
        # Write the example name as a new tag data.
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   491
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   492
        f.write("<td>%s</td>\n" % name)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   493
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   494
        #
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   495
        # Write the elapsed time as a new tag data.
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   496
        #
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   497
        f.write("<td>%s</td>\n" % time)
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   498
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   499
        #
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   500
        # That's it for the current example, so terminate the row.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   501
        #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   502
        f.write("</tr>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   503
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   504
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   505
    # That's it for the table of examples, so terminate the table.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   506
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   507
    f.write("</table>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   508
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   509
    #
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   510
    # And that's it for the report, so finish up.
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
   511
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   512
    f.write("</body>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   513
    f.write("</html>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   514
    f.close()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   515
    
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   516
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   517
# Python Control-C handling is broken in the presence of multiple threads.  
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   518
# Signals get delivered to the runnable/running thread by default and if 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   519
# it is blocked, the signal is simply ignored.  So we hook sigint and set 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   520
# a global variable telling the system to shut down gracefully.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   521
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   522
thread_exit = False
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   523
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   524
def sigint_hook(signal, frame):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   525
    global thread_exit
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   526
    thread_exit = True
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   527
    return 0
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   528
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   529
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   530
# Waf can be configured to compile in debug or optimized modes.  In each
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   531
# case, the resulting built goes into a different directory.  If we want
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   532
# test tests to run from the correct code-base, we have to figure out which
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   533
# mode waf is running in.  This is called its active variant.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   534
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   535
# XXX This function pokes around in the waf internal state file.  To be a
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   536
# little less hacky, we should add a commmand to waf to return this info
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   537
# and use that result.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   538
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   539
def read_waf_active_variant():
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   540
    for line in open("build/c4che/default.cache.py").readlines():
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   541
        if line.startswith("NS3_ACTIVE_VARIANT"):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   542
            exec(line, globals())
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   543
            break
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   544
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   545
    if options.verbose:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   546
        print "NS3_ACTIVE_VARIANT == %s" % NS3_ACTIVE_VARIANT
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   547
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   548
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   549
# In general, the build process itself naturally takes care of figuring out
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   550
# which tests are built into the test runner.  For example, if waf configure
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   551
# determines that ENABLE_EMU is false due to some missing dependency,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   552
# the tests for the emu net device simply will not be built and will 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   553
# therefore not be included in the built test runner.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   554
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   555
# Examples, however, are a different story.  In that case, we are just given
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   556
# a list of examples that could be run.  Instead of just failing, for example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   557
# nsc-tcp-zoo if NSC is not present, we look into the waf saved configuration
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   558
# for relevant configuration items.  
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   559
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   560
# XXX This function pokes around in the waf internal state file.  To be a
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   561
# little less hacky, we should add a commmand to waf to return this info
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   562
# and use that result.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   563
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   564
def read_waf_config():
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   565
    for line in open("build/c4che/%s.cache.py" % NS3_ACTIVE_VARIANT).readlines():
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   566
        for item in interesting_config_items:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   567
            if line.startswith(item):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   568
                exec(line, globals())
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   569
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   570
    if options.verbose:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   571
        for item in interesting_config_items:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   572
            print "%s ==" % item, eval(item)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   573
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   574
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   575
# It seems pointless to fork a process to run waf to fork a process to run
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   576
# the test runner, so we just run the test runner directly.  The main thing 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   577
# that waf would do for us would be to sort out the shared library path but
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   578
# we can deal with that easily and do here.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   579
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   580
# There can be many different ns-3 repositories on a system, and each has 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   581
# its own shared libraries, so ns-3 doesn't hardcode a shared library search
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   582
# path -- it is cooked up dynamically, so we do that too.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   583
#
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   584
def make_paths():
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   585
    have_DYLD_LIBRARY_PATH = False
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   586
    have_LD_LIBRARY_PATH = False
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   587
    have_PATH = False
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   588
    have_PYTHONPATH = False
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   589
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   590
    keys = os.environ.keys()
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   591
    for key in keys:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   592
        if key == "DYLD_LIBRARY_PATH":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   593
            have_DYLD_LIBRARY_PATH = True
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   594
        if key == "LD_LIBRARY_PATH":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   595
            have_LD_LIBRARY_PATH = True
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   596
        if key == "PATH":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   597
            have_PATH = True
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   598
        if key == "PYTHONPATH":
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   599
            have_PYTHONPATH = True
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   600
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   601
    pypath = os.environ["PYTHONPATH"] = os.path.join (NS3_BUILDDIR, NS3_ACTIVE_VARIANT, "bindings", "python")
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   602
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   603
    if not have_PYTHONPATH:
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   604
        os.environ["PYTHONPATH"] = pypath
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   605
    else:
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   606
        os.environ["PYTHONPATH"] += ":" + pypath
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   607
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   608
    if options.verbose:
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
   609
        print "os.environ[\"PYTHONPATH\"] == %s" % os.environ["PYTHONPATH"]
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   610
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   611
    if sys.platform == "darwin":
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   612
        if not have_DYLD_LIBRARY_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   613
            os.environ["DYLD_LIBRARY_PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   614
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   615
            os.environ["DYLD_LIBRARY_PATH"] += ":" + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   616
        if options.verbose:
5466
c206a8ce5644 Fix testing under Mac OS X
fmoatamr
parents: 5461
diff changeset
   617
            print "os.environ[\"DYLD_LIBRARY_PATH\"] == %s" % os.environ["DYLD_LIBRARY_PATH"]
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   618
    elif sys.platform == "win32":
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   619
        if not have_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   620
            os.environ["PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   621
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   622
            os.environ["PATH"] += ';' + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   623
        if options.verbose:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   624
            print "os.environ[\"PATH\"] == %s" % os.environ["PATH"]
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   625
    elif sys.platform == "cygwin":
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   626
        if not have_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   627
            os.environ["PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   628
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   629
            os.environ["PATH"] += ":" + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   630
        if options.verbose:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   631
            print "os.environ[\"PATH\"] == %s" % os.environ["PATH"]
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   632
    else:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   633
        if not have_LD_LIBRARY_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   634
            os.environ["LD_LIBRARY_PATH"] = ""
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   635
        for path in NS3_MODULE_PATH:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   636
            os.environ["LD_LIBRARY_PATH"] += ":" + path
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   637
        if options.verbose:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   638
            print "os.environ[\"LD_LIBRARY_PATH\"] == %s" % os.environ["LD_LIBRARY_PATH"]
5275
d4008f2981ba Windows needs entire path
Craig Dowell <craigdo@ee.washington.edu>
parents: 5274
diff changeset
   639
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   640
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   641
# Short note on generating suppressions:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   642
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   643
# See the valgrind documentation for a description of suppressions.  The easiest
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   644
# way to generate a suppression expression is by using the valgrind 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   645
# --gen-suppressions option.  To do that you have to figure out how to run the 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   646
# test in question.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   647
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   648
# If you do "test.py -v -g -s <suitename> then test.py will output most of what
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   649
# you need.  For example, if you are getting a valgrind error in the
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   650
# devices-mesh-dot11s-regression test suite, you can run:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   651
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   652
#   ./test.py -v -g -s devices-mesh-dot11s-regression 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   653
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   654
# You should see in the verbose output something that looks like:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   655
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   656
#   Synchronously execute valgrind --suppressions=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/testpy.supp
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   657
#   --leak-check=full --error-exitcode=2 /home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/build/debug/utils/test-runner 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   658
#   --suite=devices-mesh-dot11s-regression --basedir=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   659
#   --tempdir=testpy-output/2010-01-12-22-47-50-CUT 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   660
#   --out=testpy-output/2010-01-12-22-47-50-CUT/devices-mesh-dot11s-regression.xml
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   661
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   662
# You need to pull out the useful pieces, and so could run the following to 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   663
# reproduce your error:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   664
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   665
#   valgrind --suppressions=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/testpy.supp
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   666
#   --leak-check=full --error-exitcode=2 /home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/build/debug/utils/test-runner 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   667
#   --suite=devices-mesh-dot11s-regression --basedir=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   668
#   --tempdir=testpy-output 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   669
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   670
# Hint: Use the first part of the command as is, and point the "tempdir" to 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   671
# somewhere real.  You don't need to specify an "out" file.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   672
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   673
# When you run the above command you should see your valgrind error.  The 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   674
# suppression expression(s) can be generated by adding the --gen-suppressions=yes
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   675
# option to valgrind.  Use something like:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   676
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   677
#   valgrind --gen-suppressions=yes --suppressions=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/testpy.supp
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   678
#   --leak-check=full --error-exitcode=2 /home/craigdo/repos/ns-3-allinone-dev/ns-3-dev/build/debug/utils/test-runner 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   679
#   --suite=devices-mesh-dot11s-regression --basedir=/home/craigdo/repos/ns-3-allinone-dev/ns-3-dev 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   680
#   --tempdir=testpy-output 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   681
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   682
# Now when valgrind detects an error it will ask:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   683
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   684
#   ==27235== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ----
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   685
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   686
# to which you just enter 'y'<ret>.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   687
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   688
# You will be provided with a suppression expression that looks something like
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   689
# the following:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   690
#   {
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   691
#     <insert_a_suppression_name_here>
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   692
#     Memcheck:Addr8
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   693
#     fun:_ZN3ns36dot11s15HwmpProtocolMac8SendPreqESt6vectorINS0_6IePreqESaIS3_EE
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   694
#     fun:_ZN3ns36dot11s15HwmpProtocolMac10SendMyPreqEv
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   695
#     fun:_ZN3ns36dot11s15HwmpProtocolMac18RequestDestinationENS_12Mac48AddressEjj
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   696
#     ...
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   697
#     the rest of the stack frame
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   698
#     ...
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   699
#   }
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   700
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   701
# You need to add a supression name which will only be printed out by valgrind in 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   702
# verbose mode (but it needs to be there in any case).  The entire stack frame is
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   703
# shown to completely characterize the error, but in most cases you won't need 
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   704
# all of that info.  For example, if you want to turn off all errors that happen
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   705
# when the function (fun:) is called, you can just delete the rest of the stack
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   706
# frame.  You can also use wildcards to make the mangled signatures more readable.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   707
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   708
# I added the following to the testpy.supp file for this particular error:
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   709
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   710
#   {
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   711
#     Supress invalid read size errors in SendPreq() when using HwmpProtocolMac
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   712
#     Memcheck:Addr8
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   713
#     fun:*HwmpProtocolMac*SendPreq*
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   714
#   }
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   715
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   716
# Now, when you run valgrind the error will be suppressed.
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   717
#
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   718
VALGRIND_SUPPRESSIONS_FILE = "testpy.supp"
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   719
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   720
def run_job_synchronously(shell_command, directory, valgrind, is_python):
5909
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   721
    (base, build) = os.path.split (NS3_BUILDDIR)
766442c7f240 Bug 781: Suppress the valgrind error: Invalid read size of 8 in TestSuite devices-mesh-dot11s-regression
Faker Moatamri <faker.moatamri@sophia.inria.fr>
parents: 5741
diff changeset
   722
    suppressions_path = os.path.join (base, VALGRIND_SUPPRESSIONS_FILE)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   723
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   724
    if is_python:
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   725
        path_cmd = "python " + os.path.join (base, shell_command)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   726
    else:
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   727
        path_cmd = os.path.join (NS3_BUILDDIR, NS3_ACTIVE_VARIANT, shell_command)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   728
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   729
    if valgrind:
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   730
        cmd = "valgrind --suppressions=%s --leak-check=full --show-reachable=yes --error-exitcode=2 %s" % (suppressions_path, 
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   731
            path_cmd)
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   732
    else:
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   733
        cmd = path_cmd
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   734
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   735
    if options.verbose:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   736
        print "Synchronously execute %s" % cmd
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   737
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
   738
    start_time = time.time()
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   739
    proc = subprocess.Popen(cmd, shell = True, cwd = directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   740
    stdout_results, stderr_results = proc.communicate()
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   741
    elapsed_time = time.time() - start_time
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   742
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   743
    retval = proc.returncode
5917
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   744
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   745
    #
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   746
    # valgrind sometimes has its own idea about what kind of memory management
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   747
    # errors are important.  We want to detect *any* leaks, so the way to do 
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   748
    # that is to look for the presence of a valgrind leak summary section.
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   749
    #
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   750
    # If another error has occurred (like a test suite has failed), we don't 
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   751
    # want to trump that error, so only do the valgrind output scan if the 
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   752
    # test has otherwise passed (return code was zero).
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   753
    #
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   754
    if valgrind and retval == 0 and "== LEAK SUMMARY:" in stderr_results:
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   755
        retval = 2
aa8364846523 Bug 792: Neither test.py nor waf --valgrind catch all kinds of memory leaks
Craig Dowell <craigdo@ee.washington.edu>
parents: 5912
diff changeset
   756
    
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   757
    if options.verbose:
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   758
        print "Return code = ", retval
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   759
        print "stderr = ", stderr_results
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   760
5912
679c04aa43e7 Update test.py to detect still reachable memory blocks as errors, it will fail if there is some reachable memory blocks reported by valgrind
fmoatamr
parents: 5909
diff changeset
   761
    return (retval, stdout_results, stderr_results, elapsed_time)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   762
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   763
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   764
# This class defines a unit of testing work.  It will typically refer to
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   765
# a test suite to run using the test-runner, or an example to run directly.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   766
#
5415
5c2f96b507a9 Stray () in test.py (bug 718)
Craig Dowell <craigdo@ee.washington.edu>
parents: 5414
diff changeset
   767
class Job:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   768
    def __init__(self):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   769
        self.is_break = False
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   770
        self.is_skip = False
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   771
        self.is_example = False
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   772
        self.is_pyexample = False
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   773
        self.shell_command = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   774
        self.display_name = ""
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   775
        self.basedir = ""
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   776
        self.tempdir = ""
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   777
        self.cwd = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   778
        self.tmp_file_name = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   779
        self.returncode = False
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   780
        self.elapsed_time = 0
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   781
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   782
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   783
    # A job is either a standard job or a special job indicating that a worker
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   784
    # thread should exist.  This special job is indicated by setting is_break 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   785
    # to true.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   786
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   787
    def set_is_break(self, is_break):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   788
        self.is_break = is_break
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   789
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   790
    #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   791
    # If a job is to be skipped, we actually run it through the worker threads
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   792
    # to keep the PASS, FAIL, CRASH and SKIP processing all in one place.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   793
    #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   794
    def set_is_skip(self, is_skip):
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   795
        self.is_skip = is_skip
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   796
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   797
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   798
    # Examples are treated differently than standard test suites.  This is
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   799
    # mostly because they are completely unaware that they are being run as 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   800
    # tests.  So we have to do some special case processing to make them look
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   801
    # like tests.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   802
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   803
    def set_is_example(self, is_example):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   804
        self.is_example = is_example
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   805
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   806
    #
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   807
    # Examples are treated differently than standard test suites.  This is
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   808
    # mostly because they are completely unaware that they are being run as 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   809
    # tests.  So we have to do some special case processing to make them look
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   810
    # like tests.
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   811
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   812
    def set_is_pyexample(self, is_pyexample):
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   813
        self.is_pyexample = is_pyexample
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   814
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   815
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   816
    # This is the shell command that will be executed in the job.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   817
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   818
    #  "utils/test-runner --suite=some-test-suite"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   819
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   820
    def set_shell_command(self, shell_command):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   821
        self.shell_command = shell_command
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   822
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   823
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   824
    # This is the dispaly name of the job, typically the test suite or example 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   825
    # name.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   826
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   827
    #  "some-test-suite" or "udp-echo"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   828
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   829
    def set_display_name(self, display_name):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   830
        self.display_name = display_name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   831
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   832
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   833
    # This is the base directory of the repository out of which the tests are
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   834
    # being run.  It will be used deep down in the testing framework to determine
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   835
    # where the source directory of the test was, and therefore where to find 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   836
    # provided test vectors.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   837
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   838
    #  "/home/user/repos/ns-3-dev"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   839
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   840
    def set_basedir(self, basedir):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   841
        self.basedir = basedir
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   842
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   843
    #
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   844
    # This is the directory to which a running test suite should write any 
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   845
    # temporary files.
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   846
    #
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   847
    def set_tempdir(self, tempdir):
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   848
        self.tempdir = tempdir
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   849
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   850
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   851
    # This is the current working directory that will be given to an executing
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   852
    # test as it is being run.  It will be used for examples to tell them where
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   853
    # to write all of the pcap files that we will be carefully ignoring.  For
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   854
    # example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   855
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   856
    #  "/tmp/unchecked-traces"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   857
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   858
    def set_cwd(self, cwd):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   859
        self.cwd = cwd
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   860
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   861
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   862
    # This is the temporary results file name that will be given to an executing 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   863
    # test as it is being run.  We will be running all of our tests in parallel
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   864
    # so there must be multiple temporary output files.  These will be collected
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
   865
    # into a single XML file at the end and then be deleted.  
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   866
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   867
    def set_tmp_file_name(self, tmp_file_name):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   868
        self.tmp_file_name = tmp_file_name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   869
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   870
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   871
    # The return code received when the job process is executed.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   872
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   873
    def set_returncode(self, returncode):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   874
        self.returncode = returncode
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   875
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   876
    #
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   877
    # The elapsed real time for the job execution.
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   878
    #
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   879
    def set_elapsed_time(self, elapsed_time):
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   880
        self.elapsed_time = elapsed_time
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   881
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   882
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   883
# The worker thread class that handles the actual running of a given test.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   884
# Once spawned, it receives requests for work through its input_queue and
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   885
# ships the results back through the output_queue.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   886
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   887
class worker_thread(threading.Thread):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   888
    def __init__(self, input_queue, output_queue):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   889
        threading.Thread.__init__(self)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   890
        self.input_queue = input_queue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   891
        self.output_queue = output_queue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   892
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   893
    def run(self):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   894
        while True:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   895
            job = self.input_queue.get()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   896
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   897
            # Worker threads continue running until explicitly told to stop with
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   898
            # a special job.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   899
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   900
            if job.is_break:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   901
                return
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   902
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   903
            # If the global interrupt handler sets the thread_exit variable,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   904
            # we stop doing real work and just report back a "break" in the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   905
            # normal command processing has happened.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   906
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   907
            if thread_exit == True:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   908
                job.set_is_break(True)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   909
                self.output_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   910
                continue
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   911
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   912
            #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   913
            # If we are actually supposed to skip this job, do so.  Note that
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   914
            # if is_skip is true, returncode is undefined.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   915
            #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   916
            if job.is_skip:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   917
                if options.verbose:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   918
                    print "Skip %s" % job.shell_command
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   919
                self.output_queue.put(job)
6253
3aa30fa758f5 bug 886
Craig Dowell <craigdo@ee.washington.edu>
parents: 6247
diff changeset
   920
                continue
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
   921
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   922
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   923
            # Otherwise go about the business of running tests as normal.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   924
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   925
            else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   926
                if options.verbose:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   927
                    print "Launch %s" % job.shell_command
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   928
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   929
                if job.is_example or job.is_pyexample:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   930
                    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   931
                    # If we have an example, the shell command is all we need to
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   932
                    # know.  It will be something like "examples/udp-echo" or 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   933
                    # "examples/mixed-wireless.py"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   934
                    #
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   935
                    (job.returncode, standard_out, standard_err, et) = run_job_synchronously(job.shell_command, 
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   936
                        job.cwd, options.valgrind, job.is_pyexample)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   937
                else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   938
                    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   939
                    # If we're a test suite, we need to provide a little more info
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   940
                    # to the test runner, specifically the base directory and temp
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   941
                    # file name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   942
                    #
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   943
                    (job.returncode, standard_out, standard_err, et) = run_job_synchronously(job.shell_command + 
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
   944
                        " --basedir=%s --tempdir=%s --out=%s" % (job.basedir, job.tempdir, job.tmp_file_name), 
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
   945
                        job.cwd, options.valgrind, False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   946
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   947
                job.set_elapsed_time(et)
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
   948
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   949
                if options.verbose:
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
   950
                    print "returncode = %d" % job.returncode
6327
246934d7b912 Fixed typo in test.py
Lalith Suresh
parents: 6315
diff changeset
   951
                    print "---------- begin standard out ----------"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   952
                    print standard_out
5351
6abced63cd7f keep wifi-ap from spewing all over std_out during tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5350
diff changeset
   953
                    print "---------- begin standard err ----------"
6abced63cd7f keep wifi-ap from spewing all over std_out during tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5350
diff changeset
   954
                    print standard_err
6abced63cd7f keep wifi-ap from spewing all over std_out during tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5350
diff changeset
   955
                    print "---------- end standard err ----------"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   956
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   957
                self.output_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   958
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   959
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   960
# This is the main function that does the work of interacting with the test-runner
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   961
# itself.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   962
#
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   963
def run_tests():
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   964
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   965
    # Run waf to make sure that everything is built, configured and ready to go
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   966
    # unless we are explicitly told not to.  We want to be careful about causing
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   967
    # our users pain while waiting for extraneous stuff to compile and link, so
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   968
    # we allow users that know what they''re doing to not invoke waf at all.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
   969
    #
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   970
    if not options.nowaf:
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   971
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   972
        #
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   973
        # If the user is running the "kinds" or "list" options, there is an 
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   974
        # implied dependency on the test-runner since we call that program
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   975
        # if those options are selected.  We will exit after processing those
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   976
        # options, so if we see them, we can safely only build the test-runner.
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   977
        #
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   978
        # If the user has constrained us to running only a particular type of
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   979
        # file, we can only ask waf to build what we know will be necessary.
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   980
        # For example, if the user only wants to run BVT tests, we only have
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   981
        # to build the test-runner and can ignore all of the examples.
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   982
        #
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   983
        # If the user only wants to run a single example, then we can just build
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   984
        # that example.
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   985
        #
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   986
        # If there is no constraint, then we have to build everything since the
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   987
        # user wants to run everything.
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   988
        #
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
   989
        if options.kinds or options.list or (len(options.constrain) and options.constrain in core_kinds):
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   990
            if sys.platform == "win32":
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   991
                waf_cmd = "waf --target=test-runner"
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   992
            else:
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   993
                waf_cmd = "./waf --target=test-runner"
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   994
        elif len(options.example):
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   995
            if sys.platform == "win32":
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   996
                waf_cmd = "waf --target=%s" % os.path.basename(options.example)
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   997
            else:
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   998
                waf_cmd = "./waf --target=%s" % os.path.basename(options.example)
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
   999
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1000
        else:
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1001
            if sys.platform == "win32":
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1002
                waf_cmd = "waf"
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1003
            else:
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1004
                waf_cmd = "./waf"
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1005
5470
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1006
        if options.verbose:
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1007
            print "Building: %s" % waf_cmd
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1008
d3aabb63dd12 remind test.py how to build on unix and limit build scope when posible
Craig Dowell <craigdo@ee.washington.edu>
parents: 5466
diff changeset
  1009
        proc = subprocess.Popen(waf_cmd, shell = True)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1010
        proc.communicate()
6315
1f53e122254b fix bug 908
Andrey Mazo <mazo@iitp.ru>
parents: 6312
diff changeset
  1011
        if proc.returncode:
1f53e122254b fix bug 908
Andrey Mazo <mazo@iitp.ru>
parents: 6312
diff changeset
  1012
            print >> sys.stderr, "Waf died. Not running tests"
1f53e122254b fix bug 908
Andrey Mazo <mazo@iitp.ru>
parents: 6312
diff changeset
  1013
            return proc.returncode
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1014
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1015
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1016
    # Pull some interesting configuration information out of waf, primarily
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1017
    # so we can know where executables can be found, but also to tell us what
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1018
    # pieces of the system have been built.  This will tell us what examples 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1019
    # are runnable.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1020
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1021
    read_waf_active_variant()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1022
    read_waf_config()
6227
38a3d54b5742 python path helps when running python programs
Craig Dowell <craigdo@ee.washington.edu>
parents: 6203
diff changeset
  1023
    make_paths()
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1024
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1025
    #
6083
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1026
    # If lots of logging is enabled, we can crash Python when it tries to 
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1027
    # save all of the text.  We just don't allow logging to be turned on when
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1028
    # test.py runs.  If you want to see logging output from your tests, you
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1029
    # have to run them using the test-runner directly.
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1030
    #
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1031
    os.environ["NS_LOG"] = ""
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1032
a4a7ded174dc test.py turns off NS_LOG locally
Craig Dowell <craigdo@ee.washington.edu>
parents: 6046
diff changeset
  1033
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1034
    # There are a couple of options that imply we can to exit before starting
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1035
    # up a bunch of threads and running tests.  Let's detect these cases and 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1036
    # handle them without doing all of the hard work.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1037
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1038
    if options.kinds:
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1039
        path_cmd = os.path.join("utils", "test-runner --kinds")
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1040
        (rc, standard_out, standard_err, et) = run_job_synchronously(path_cmd, os.getcwd(), False, False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1041
        print standard_out
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1042
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1043
    if options.list:
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1044
        path_cmd = os.path.join("utils", "test-runner --list")
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1045
        (rc, standard_out, standard_err, et) = run_job_synchronously(path_cmd, os.getcwd(), False, False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1046
        print standard_out
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1047
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1048
    if options.kinds or options.list:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1049
        return
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1050
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1051
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1052
    # We communicate results in two ways.  First, a simple message relating 
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1053
    # PASS, FAIL, CRASH or SKIP is always written to the standard output.  It 
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1054
    # is expected that this will be one of the main use cases.  A developer can
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1055
    # just run test.py with no options and see that all of the tests still 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1056
    # pass.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1057
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1058
    # The second main use case is when detailed status is requested (with the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1059
    # --text or --html options).  Typicall this will be text if a developer
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1060
    # finds a problem, or HTML for nightly builds.  In these cases, an
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1061
    # XML file is written containing the status messages from the test suites.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1062
    # This file is then read and translated into text or HTML.  It is expected
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1063
    # that nobody will really be interested in the XML, so we write it somewhere
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1064
    # with a unique name (time) to avoid collisions.  In case an error happens, we
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1065
    # provide a runtime option to retain the temporary files.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1066
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1067
    # When we run examples as smoke tests, they are going to want to create
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1068
    # lots and lots of trace files.  We aren't really interested in the contents
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1069
    # of the trace files, so we also just stash them off in the temporary dir.
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1070
    # The retain option also causes these unchecked trace files to be kept.
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1071
    #
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1072
    date_and_time = time.strftime("%Y-%m-%d-%H-%M-%S-CUT", time.gmtime())
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1073
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1074
    if not os.path.exists(TMP_OUTPUT_DIR):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1075
        os.makedirs(TMP_OUTPUT_DIR)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1076
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1077
    testpy_output_dir = os.path.join(TMP_OUTPUT_DIR, date_and_time);
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1078
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1079
    if not os.path.exists(testpy_output_dir):
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1080
        os.makedirs(testpy_output_dir)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1081
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1082
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1083
    # Create the main output file and start filling it with XML.  We need to 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1084
    # do this since the tests will just append individual results to this file.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1085
    #
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1086
    xml_results_file = os.path.join(testpy_output_dir, "results.xml")
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1087
    f = open(xml_results_file, 'w')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1088
    f.write('<?xml version="1.0"?>\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1089
    f.write('<TestResults>\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1090
    f.close()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1091
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1092
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1093
    # We need to figure out what test suites to execute.  We are either given one 
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1094
    # suite or example explicitly via the --suite or --example/--pyexample option,
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1095
    # or we need to call into the test runner and ask it to list all of the available
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1096
    # test suites.  Further, we need to provide the constraint information if it
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1097
    # has been given to us.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1098
    # 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1099
    # This translates into allowing the following options with respect to the 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1100
    # suites
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1101
    #
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1102
    #  ./test,py:                                           run all of the suites and examples
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1103
    #  ./test.py --constrain=core:                          run all of the suites of all kinds
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1104
    #  ./test.py --constrain=unit:                          run all unit suites
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1105
    #  ./test,py --suite=some-test-suite:                   run a single suite
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1106
    #  ./test,py --example=udp/udp-echo:                    run no test suites
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1107
    #  ./test,py --pyexample=wireless/mixed-wireless.py:    run no test suites
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1108
    #  ./test,py --suite=some-suite --example=some-example: run the single suite
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1109
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1110
    # We can also use the --constrain option to provide an ordering of test 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1111
    # execution quite easily.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1112
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1113
    if len(options.suite):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1114
        suites = options.suite + "\n"
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1115
    elif len(options.example) == 0 and len(options.pyexample) == 0:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1116
        if len(options.constrain):
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1117
            path_cmd = os.path.join("utils", "test-runner --list --constrain=%s" % options.constrain)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1118
            (rc, suites, standard_err, et) = run_job_synchronously(path_cmd, os.getcwd(), False, False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1119
        else:
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1120
            path_cmd = os.path.join("utils", "test-runner --list")
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1121
            (rc, suites, standard_err, et) = run_job_synchronously(path_cmd, os.getcwd(), False, False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1122
    else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1123
        suites = ""
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1124
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1125
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1126
    # suite_list will either a single test suite name that the user has 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1127
    # indicated she wants to run or a list of test suites provided by
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1128
    # the test-runner possibly according to user provided constraints.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1129
    # We go through the trouble of setting up the parallel execution 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1130
    # even in the case of a single suite to avoid having two process the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1131
    # results in two different places.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1132
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1133
    suite_list = suites.split('\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1134
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1135
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1136
    # We now have a possibly large number of test suites to run, so we want to
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1137
    # run them in parallel.  We're going to spin up a number of worker threads
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1138
    # that will run our test jobs for us.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1139
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1140
    input_queue = Queue.Queue(0)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1141
    output_queue = Queue.Queue(0)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1142
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1143
    jobs = 0
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1144
    threads=[]
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1145
5273
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1146
    #
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1147
    # In Python 2.6 you can just use multiprocessing module, but we don't want
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1148
    # to introduce that dependency yet; so we jump through a few hoops.
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1149
    #
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1150
    processors = 1
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1151
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1152
    if sys.platform != "win32":
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1153
        if 'SC_NPROCESSORS_ONLN'in os.sysconf_names:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1154
            processors = os.sysconf('SC_NPROCESSORS_ONLN')
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1155
        else:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1156
            proc = subprocess.Popen("sysctl -n hw.ncpu", shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1157
            stdout_results, stderr_results = proc.communicate()
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1158
            if len(stderr_results) == 0:
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1159
                processors = int(stdout_results)
5273
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1160
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1161
    #
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1162
    # Now, spin up one thread per processor which will eventually mean one test
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1163
    # per processor running concurrently.
c95bd39cc980 Remove multiprocessing dependency in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5257
diff changeset
  1164
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1165
    for i in range(processors):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1166
        thread = worker_thread(input_queue, output_queue)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1167
        threads.append(thread)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1168
        thread.start()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1169
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1170
    #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1171
    # Keep track of some summary statistics
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1172
    #
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1173
    total_tests = 0
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1174
    skipped_tests = 0
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1175
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1176
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1177
    # We now have worker threads spun up, and a list of work to do.  So, run 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1178
    # through the list of test suites and dispatch a job to run each one.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1179
    # 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1180
    # Dispatching will run with unlimited speed and the worker threads will 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1181
    # execute as fast as possible from the queue.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1182
    #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1183
    # Note that we actually dispatch tests to be skipped, so all of the 
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1184
    # PASS, FAIL, CRASH and SKIP processing is done in the same place.
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1185
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1186
    for test in suite_list:
5461
ac47584eca3e Make test.py work on MinGW
Craig Dowell <craigdo@ee.washington.edu>
parents: 5460
diff changeset
  1187
        test = test.strip()
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1188
        if len(test):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1189
            job = Job()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1190
            job.set_is_example(False)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1191
            job.set_is_pyexample(False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1192
            job.set_display_name(test)
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1193
            job.set_tmp_file_name(os.path.join(testpy_output_dir, "%s.xml" % test))
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1194
            job.set_cwd(os.getcwd())
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1195
            job.set_basedir(os.getcwd())
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
  1196
            job.set_tempdir(testpy_output_dir)
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1197
            if (options.multiple):
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1198
                multiple = " --multiple"
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1199
            else:
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1200
                multiple = ""
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1201
5461
ac47584eca3e Make test.py work on MinGW
Craig Dowell <craigdo@ee.washington.edu>
parents: 5460
diff changeset
  1202
            path_cmd = os.path.join("utils", "test-runner --suite=%s%s" % (test, multiple))
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1203
            job.set_shell_command(path_cmd)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1204
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1205
            if options.valgrind and test in core_valgrind_skip_tests:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1206
                job.set_is_skip(True)
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1207
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1208
            if options.verbose:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1209
                print "Queue %s" % test
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1210
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1211
            input_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1212
            jobs = jobs + 1
5279
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1213
            total_tests = total_tests + 1
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1214
    
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1215
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1216
    # We've taken care of the discovered or specified test suites.  Now we
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1217
    # have to deal with examples run as smoke tests.  We have a list of all of
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1218
    # the example programs it makes sense to try and run.  Each example will
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1219
    # have a condition associated with it that must evaluate to true for us
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1220
    # to try and execute it.  This is used to determine if the example has
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1221
    # a dependency that is not satisfied.  For example, if an example depends
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1222
    # on NSC being configured by waf, that example should have a condition
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1223
    # that evaluates to true if NSC is enabled.  For example,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1224
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1225
    #      ("tcp-nsc-zoo", "ENABLE_NSC == True"),
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1226
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1227
    # In this case, the example "tcp-nsc-zoo" will only be run if we find the
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1228
    # waf configuration variable "ENABLE_NSC" to be True.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1229
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1230
    # We don't care at all how the trace files come out, so we just write them 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1231
    # to a single temporary directory.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1232
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1233
    # XXX As it stands, all of the trace files have unique names, and so file
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1234
    # collisions can only happen if two instances of an example are running in
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1235
    # two versions of the test.py process concurrently.  We may want to create
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1236
    # uniquely named temporary traces directories to avoid this problem.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1237
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1238
    # We need to figure out what examples to execute.  We are either given one 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1239
    # suite or example explicitly via the --suite or --example option, or we
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1240
    # need to walk the list of examples looking for available example 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1241
    # conditions.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1242
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1243
    # This translates into allowing the following options with respect to the 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1244
    # suites
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1245
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1246
    #  ./test,py:                                           run all of the examples
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1247
    #  ./test.py --constrain=unit                           run no examples
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1248
    #  ./test.py --constrain=example                        run all of the examples
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1249
    #  ./test.py --suite=some-test-suite:                   run no examples
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1250
    #  ./test.py --example=some-example:                    run the single example
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1251
    #  ./test.py --suite=some-suite --example=some-example: run the single example
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1252
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1253
    # XXX could use constrain to separate out examples used for performance 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1254
    # testing
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1255
    #
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1256
    if len(options.suite) == 0 and len(options.example) == 0 and len(options.pyexample) == 0:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1257
        if len(options.constrain) == 0 or options.constrain == "example":
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1258
            if ENABLE_EXAMPLES:
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1259
                for test, do_run, do_valgrind_run in example_tests:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1260
                    if eval(do_run):
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1261
                        job = Job()
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1262
                        job.set_is_example(True)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1263
                        job.set_is_pyexample(False)
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1264
                        job.set_display_name(test)
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1265
                        job.set_tmp_file_name("")
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1266
                        job.set_cwd(testpy_output_dir)
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1267
                        job.set_basedir(os.getcwd())
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
  1268
                        job.set_tempdir(testpy_output_dir)
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1269
                        job.set_shell_command("examples/%s" % test)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1270
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1271
                        if options.valgrind and not eval(do_valgrind_run):
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1272
                            job.set_is_skip (True)
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1273
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1274
                        if options.verbose:
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1275
                            print "Queue %s" % test
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1276
5369
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1277
                        input_queue.put(job)
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1278
                        jobs = jobs + 1
86beb5869f67 split examples, add examples, tweak test.py to minimize builds
Craig Dowell <craigdo@ee.washington.edu>
parents: 5351
diff changeset
  1279
                        total_tests = total_tests + 1
5279
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1280
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1281
    elif len(options.example):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1282
        #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1283
        # If you tell me to run an example, I will try and run the example
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1284
        # irrespective of any condition.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1285
        #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1286
        job = Job()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1287
        job.set_is_example(True)
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1288
        job.set_is_pyexample(False)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1289
        job.set_display_name(options.example)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1290
        job.set_tmp_file_name("")
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1291
        job.set_cwd(testpy_output_dir)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1292
        job.set_basedir(os.getcwd())
5481
de01c67af975 pass explicit temp directory to test suites
Craig Dowell <craigdo@ee.washington.edu>
parents: 5470
diff changeset
  1293
        job.set_tempdir(testpy_output_dir)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1294
        job.set_shell_command("examples/%s" % options.example)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1295
        
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1296
        if options.verbose:
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1297
            print "Queue %s" % options.example
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1298
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1299
        input_queue.put(job)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1300
        jobs = jobs + 1
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1301
        total_tests = total_tests + 1
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1302
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1303
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1304
    # Run some Python examples as smoke tests.  We have a list of all of
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1305
    # the example programs it makes sense to try and run.  Each example will
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1306
    # have a condition associated with it that must evaluate to true for us
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1307
    # to try and execute it.  This is used to determine if the example has
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1308
    # a dependency that is not satisfied.
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1309
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1310
    # We don't care at all how the trace files come out, so we just write them 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1311
    # to a single temporary directory.
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1312
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1313
    # We need to figure out what python examples to execute.  We are either 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1314
    # given one pyexample explicitly via the --pyexample option, or we
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1315
    # need to walk the list of python examples
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1316
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1317
    # This translates into allowing the following options with respect to the 
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1318
    # suites
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1319
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1320
    #  ./test.py --constrain=pyexample           run all of the python examples
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1321
    #  ./test.py --pyexample=some-example.py:    run the single python example
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1322
    #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1323
    if len(options.suite) == 0 and len(options.example) == 0 and len(options.pyexample) == 0:
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1324
        if len(options.constrain) == 0 or options.constrain == "pyexample":
6247
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1325
            if ENABLE_EXAMPLES:
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1326
                for test, do_run in python_tests:
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1327
                    if eval(do_run):
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1328
                        job = Job()
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1329
                        job.set_is_example(False)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1330
                        job.set_is_pyexample(True)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1331
                        job.set_display_name(test)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1332
                        job.set_tmp_file_name("")
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1333
                        job.set_cwd(testpy_output_dir)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1334
                        job.set_basedir(os.getcwd())
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1335
                        job.set_tempdir(testpy_output_dir)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1336
                        job.set_shell_command("examples/%s" % test)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1337
6247
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1338
                        #
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1339
                        # Python programs and valgrind do not work and play
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1340
                        # well together, so we skip them under valgrind.
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1341
                        # We go through the trouble of doing all of this
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1342
                        # work to report the skipped tests in a consistent
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1343
                        # way throught the output formatter.
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1344
                        #
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1345
                        if options.valgrind:
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1346
                            job.set_is_skip (True)
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1347
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1348
                        #
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1349
                        # The user can disable python bindings, so we need
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1350
                        # to pay attention to that and give some feedback
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1351
                        # that we're not testing them
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1352
                        #
ab7f2498373a report skipping python tests, don't valgrind them
Craig Dowell <craigdo@ee.washington.edu>
parents: 6243
diff changeset
  1353
                        if not ENABLE_PYTHON_BINDINGS:
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1354
                            job.set_is_skip (True)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1355
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1356
                        if options.verbose:
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1357
                            print "Queue %s" % test
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1358
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1359
                        input_queue.put(job)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1360
                        jobs = jobs + 1
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1361
                        total_tests = total_tests + 1
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1362
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1363
    elif len(options.pyexample):
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1364
        #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1365
        # If you tell me to run a python example, I will try and run the example
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1366
        # irrespective of any condition.
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1367
        #
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1368
        job = Job()
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1369
        job.set_is_pyexample(True)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1370
        job.set_display_name(options.pyexample)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1371
        job.set_tmp_file_name("")
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1372
        job.set_cwd(testpy_output_dir)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1373
        job.set_basedir(os.getcwd())
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1374
        job.set_tempdir(testpy_output_dir)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1375
        job.set_shell_command("examples/%s" % options.pyexample)
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1376
        
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1377
        if options.verbose:
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1378
            print "Queue %s" % options.pyexample
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1379
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1380
        input_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1381
        jobs = jobs + 1
5279
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1382
        total_tests = total_tests + 1
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1383
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1384
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1385
    # Tell the worker threads to pack up and go home for the day.  Each one
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1386
    # will exit when they see their is_break task.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1387
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1388
    for i in range(processors):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1389
        job = Job()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1390
        job.set_is_break(True)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1391
        input_queue.put(job)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1392
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1393
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1394
    # Now all of the tests have been dispatched, so all we have to do here
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1395
    # in the main thread is to wait for them to complete.  Keyboard interrupt
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1396
    # handling is broken as mentioned above.  We use a signal handler to catch
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1397
    # sigint and set a global variable.  When the worker threads sense this
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1398
    # they stop doing real work and will just start throwing jobs back at us
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1399
    # with is_break set to True.  In this case, there are no real results so we 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1400
    # ignore them.  If there are real results, we always print PASS or FAIL to
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1401
    # standard out as a quick indication of what happened.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1402
    #
5279
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1403
    passed_tests = 0
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1404
    failed_tests = 0
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1405
    crashed_tests = 0
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1406
    valgrind_errors = 0
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1407
    for i in range(jobs):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1408
        job = output_queue.get()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1409
        if job.is_break:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1410
            continue
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1411
6203
1204777b0bcf bug 856, initialize vbl
Craig Dowell <craigdo@ee.washington.edu>
parents: 6202
diff changeset
  1412
        if job.is_example or job.is_pyexample:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1413
            kind = "Example"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1414
        else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1415
            kind = "TestSuite"
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1416
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1417
        if job.is_skip:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1418
            status = "SKIP"
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1419
            skipped_tests = skipped_tests + 1
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1420
        else:
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1421
            if job.returncode == 0:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1422
                status = "PASS"
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1423
                passed_tests = passed_tests + 1
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1424
            elif job.returncode == 1:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1425
                failed_tests = failed_tests + 1
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1426
                status = "FAIL"
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1427
            elif job.returncode == 2:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1428
                valgrind_errors = valgrind_errors + 1
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1429
                status = "VALGR"
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1430
            else:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1431
                crashed_tests = crashed_tests + 1
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1432
                status = "CRASH"
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1433
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1434
        print "%s: %s %s" % (status, kind, job.display_name)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1435
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1436
        if job.is_example or job.is_pyexample:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1437
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1438
            # Examples are the odd man out here.  They are written without any
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1439
            # knowledge that they are going to be run as a test, so we need to 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1440
            # cook up some kind of output for them.  We're writing an xml file,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1441
            # so we do some simple XML that says we ran the example.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1442
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1443
            # XXX We could add some timing information to the examples, i.e. run
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1444
            # them through time and print the results here.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1445
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1446
            f = open(xml_results_file, 'a')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1447
            f.write('<Example>\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1448
            example_name = "  <Name>%s</Name>\n" % job.display_name
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1449
            f.write(example_name)
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1450
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1451
            if status == "PASS":
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1452
                f.write('  <Result>PASS</Result>\n')
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1453
            elif status == "FAIL":
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1454
                f.write('  <Result>FAIL</Result>\n')
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1455
            elif status == "VALGR":
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1456
                f.write('  <Result>VALGR</Result>\n')
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1457
            elif status == "SKIP":
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1458
                f.write('  <Result>SKIP</Result>\n')
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1459
            else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1460
                f.write('  <Result>CRASH</Result>\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1461
5460
7c6f8ebf4a34 test.py prints example elapsed times
craigdo@ee.washington.edu
parents: 5459
diff changeset
  1462
            f.write('  <ElapsedTime>%.3f</ElapsedTime>\n' % job.elapsed_time)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1463
            f.write('</Example>\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1464
            f.close()
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1465
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1466
        else:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1467
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1468
            # If we're not running an example, we're running a test suite.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1469
            # These puppies are running concurrently and generating output
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1470
            # that was written to a temporary file to avoid collisions.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1471
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1472
            # Now that we are executing sequentially in the main thread, we can
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1473
            # concatenate the contents of the associated temp file to the main 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1474
            # results file and remove that temp file.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1475
            #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1476
            # One thing to consider is that a test suite can crash just as
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1477
            # well as any other program, so we need to deal with that 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1478
            # possibility as well.  If it ran correctly it will return 0
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1479
            # if it passed, or 1 if it failed.  In this case, we can count
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1480
            # on the results file it saved being complete.  If it crashed, it 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1481
            # will return some other code, and the file should be considered 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1482
            # corrupt and useless.  If the suite didn't create any XML, then
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1483
            # we're going to have to do it ourselves.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1484
            #
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1485
            # Another issue is how to deal with a valgrind error.  If we run
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1486
            # a test suite under valgrind and it passes, we will get a return
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1487
            # code of 0 and there will be a valid xml results file since the code
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1488
            # ran to completion.  If we get a return code of 1 under valgrind,
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1489
            # the test case failed, but valgrind did not find any problems so the
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1490
            # test case return code was passed through.  We will have a valid xml
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1491
            # results file here as well since the test suite ran.  If we see a 
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1492
            # return code of 2, this means that valgrind found an error (we asked
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1493
            # it to return 2 if it found a problem in run_job_synchronously) but
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1494
            # the suite ran to completion so there is a valid xml results file.
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1495
            # If the suite crashes under valgrind we will see some other error 
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1496
            # return code (like 139).  If valgrind finds an illegal instruction or
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1497
            # some other strange problem, it will die with its own strange return
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1498
            # code (like 132).  However, if the test crashes by itself, not under
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1499
            # valgrind we will also see some other return code.
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1500
            #
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1501
            # If the return code is 0, 1, or 2, we have a valid xml file.  If we 
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1502
            # get another return code, we have no xml and we can't really say what
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1503
            # happened -- maybe the TestSuite crashed, maybe valgrind crashed due
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1504
            # to an illegal instruction.  If we get something beside 0-2, we assume
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1505
            # a crash and fake up an xml entry.  After this is all done, we still
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1506
            # need to indicate a valgrind error somehow, so we fake up an xml entry
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1507
            # with a VALGR result.  Thus, in the case of a working TestSuite that
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1508
            # fails valgrind, we'll see the PASS entry for the working TestSuite
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1509
            # followed by a VALGR failing test suite of the same name.
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1510
            #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1511
            if job.is_skip:
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1512
                f = open(xml_results_file, 'a')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1513
                f.write("<TestSuite>\n")
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1514
                f.write("  <SuiteName>%s</SuiteName>\n" % job.display_name)
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1515
                f.write('  <SuiteResult>SKIP</SuiteResult>\n')
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1516
                f.write('  <SuiteTime>Execution times not available</SuiteTime>\n')
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1517
                f.write("</TestSuite>\n")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1518
                f.close()
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1519
            else:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1520
                if job.returncode == 0 or job.returncode == 1 or job.returncode == 2:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1521
                    f_to = open(xml_results_file, 'a')
5459
3c240ed38d99 try and make test.py a bit more portable
Craig Dowell <craigdo@ee.washington.edu>
parents: 5415
diff changeset
  1522
                    f_from = open(job.tmp_file_name)
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1523
                    f_to.write(f_from.read())
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1524
                    f_to.close()
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1525
                    f_from.close()
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1526
                else:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1527
                    f = open(xml_results_file, 'a')
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1528
                    f.write("<TestSuite>\n")
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1529
                    f.write("  <SuiteName>%s</SuiteName>\n" % job.display_name)
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1530
                    f.write('  <SuiteResult>CRASH</SuiteResult>\n')
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1531
                    f.write('  <SuiteTime>Execution times not available</SuiteTime>\n')
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1532
                    f.write("</TestSuite>\n")
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1533
                    f.close()
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1534
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1535
                    if job.returncode == 2:
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1536
                        f = open(xml_results_file, 'a')
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1537
                        f.write("<TestSuite>\n")
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1538
                        f.write("  <SuiteName>%s</SuiteName>\n" % job.display_name)
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1539
                        f.write('  <SuiteResult>VALGR</SuiteResult>\n')
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1540
                        f.write('  <SuiteTime>Execution times not available</SuiteTime>\n')
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1541
                        f.write("</TestSuite>\n")
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1542
                        f.close()
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1543
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1544
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1545
    # We have all of the tests run and the results written out.  One final 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1546
    # bit of housekeeping is to wait for all of the threads to close down
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1547
    # so we can exit gracefully.
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1548
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1549
    for thread in threads:
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1550
        thread.join()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1551
    
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1552
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1553
    # Back at the beginning of time, we started the body of an XML document
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1554
    # since the test suites and examples were going to just write their 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1555
    # individual pieces.  So, we need to finish off and close out the XML 
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1556
    # document
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1557
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1558
    f = open(xml_results_file, 'a')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1559
    f.write('</TestResults>\n')
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1560
    f.close()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1561
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1562
    #
5279
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1563
    # Print a quick summary of events
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1564
    #
5402
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1565
    print "%d of %d tests passed (%d passed, %d skipped, %d failed, %d crashed, %d valgrind errors)" % (passed_tests, 
b73b6da8ead6 teach test.py to skip selected tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5390
diff changeset
  1566
        total_tests, passed_tests, skipped_tests, failed_tests, crashed_tests, valgrind_errors)
5279
1aec8ea805d2 update tcp interop response vectors
Craig Dowell <craigdo@ee.washington.edu>
parents: 5275
diff changeset
  1567
    #
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1568
    # The last things to do are to translate the XML results file to "human
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1569
    # readable form" if the user asked for it (or make an XML file somewhere)
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1570
    #
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1571
    if len(options.html):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1572
        translate_to_html(xml_results_file, options.html)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1573
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1574
    if len(options.text):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1575
        translate_to_text(xml_results_file, options.text)
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1576
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1577
    if len(options.xml):
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1578
        shutil.copyfile(xml_results_file, options.xml)
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1579
5412
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1580
    #
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1581
    # If we have been asked to retain all of the little temporary files, we
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1582
    # don't delete tm.  If we do delete the temporary files, delete only the
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1583
    # directory we just created.  We don't want to happily delete any retained
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1584
    # directories, which will probably surprise the user.
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1585
    #
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1586
    if not options.retain:
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1587
        shutil.rmtree(testpy_output_dir)
00d7fe69d024 refalgamize temporary files in test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5403
diff changeset
  1588
5403
530155ecdbcb fix ns3-tcp-cwnd unit test
Craig Dowell <craigdo@ee.washington.edu>
parents: 5402
diff changeset
  1589
    if passed_tests + skipped_tests == total_tests:
5351
6abced63cd7f keep wifi-ap from spewing all over std_out during tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5350
diff changeset
  1590
        return 0 # success
5350
9f457bebbcf4 have test.py return status
Craig Dowell <craigdo@ee.washington.edu>
parents: 5324
diff changeset
  1591
    else:
5351
6abced63cd7f keep wifi-ap from spewing all over std_out during tests
Craig Dowell <craigdo@ee.washington.edu>
parents: 5350
diff changeset
  1592
        return 1 # catchall for general errors
5350
9f457bebbcf4 have test.py return status
Craig Dowell <craigdo@ee.washington.edu>
parents: 5324
diff changeset
  1593
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1594
def main(argv):
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1595
    parser = optparse.OptionParser()
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1596
    parser.add_option("-c", "--constrain", action="store", type="string", dest="constrain", default="",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1597
                      metavar="KIND",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1598
                      help="constrain the test-runner by kind of test")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1599
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1600
    parser.add_option("-e", "--example", action="store", type="string", dest="example", default="",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1601
                      metavar="EXAMPLE",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1602
                      help="specify a single example to run")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1603
5370
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1604
    parser.add_option("-g", "--grind", action="store_true", dest="valgrind", default=False,
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1605
                      help="run the test suites and examples using valgrind")
a27163124701 add valgrind powers back to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 5369
diff changeset
  1606
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1607
    parser.add_option("-k", "--kinds", action="store_true", dest="kinds", default=False,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1608
                      help="print the kinds of tests available")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1609
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1610
    parser.add_option("-l", "--list", action="store_true", dest="list", default=False,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1611
                      help="print the list of known tests")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1612
5324
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1613
    parser.add_option("-m", "--multiple", action="store_true", dest="multiple", default=False,
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1614
                      help="report multiple failures from test suites and test cases")
0ba73cdd2a43 Refalgamize test framework to allow multiple test failures
Craig Dowell <craigdo@ee.washington.edu>
parents: 5295
diff changeset
  1615
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1616
    parser.add_option("-n", "--nowaf", action="store_true", dest="nowaf", default=False,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1617
                      help="do not run waf before starting testing")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1618
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1619
    parser.add_option("-p", "--pyexample", action="store", type="string", dest="pyexample", default="",
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1620
                      metavar="PYEXAMPLE",
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1621
                      help="specify a single python example to run")
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1622
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1623
    parser.add_option("-r", "--retain", action="store_true", dest="retain", default=False,
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1624
                      help="retain all temporary files (which are normally deleted)")
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1625
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1626
    parser.add_option("-s", "--suite", action="store", type="string", dest="suite", default="",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1627
                      metavar="TEST-SUITE",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1628
                      help="specify a single test suite to run")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1629
6200
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1630
    parser.add_option("-t", "--text", action="store", type="string", dest="text", default="",
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1631
                      metavar="TEXT-FILE",
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1632
                      help="write detailed test results into TEXT-FILE.txt")
2bf2700b6e67 Add some python examples to test.py
Craig Dowell <craigdo@ee.washington.edu>
parents: 6127
diff changeset
  1633
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1634
    parser.add_option("-v", "--verbose", action="store_true", dest="verbose", default=False,
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1635
                      help="print progress and informational messages")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1636
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1637
    parser.add_option("-w", "--web", "--html", action="store", type="string", dest="html", default="",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1638
                      metavar="HTML-FILE",
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1639
                      help="write detailed test results into HTML-FILE.html")
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1640
5239
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1641
    parser.add_option("-x", "--xml", action="store", type="string", dest="xml", default="",
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1642
                      metavar="XML-FILE",
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1643
                      help="write detailed test results into XML-FILE.xml")
2e753de86174 Remove and replace attribute unit tests (bug 675)
Craig Dowell <craigdo@ee.washington.edu>
parents: 4772
diff changeset
  1644
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1645
    global options
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1646
    options = parser.parse_args()[0]
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1647
    signal.signal(signal.SIGINT, sigint_hook)
5350
9f457bebbcf4 have test.py return status
Craig Dowell <craigdo@ee.washington.edu>
parents: 5324
diff changeset
  1648
    
9f457bebbcf4 have test.py return status
Craig Dowell <craigdo@ee.washington.edu>
parents: 5324
diff changeset
  1649
    return run_tests()
4772
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1650
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1651
if __name__ == '__main__':
7b6ae6bf0055 add test and validation framework
Craig Dowell <craigdo@ee.washington.edu>
parents:
diff changeset
  1652
    sys.exit(main(sys.argv))