doc/manual/source/distributed.rst
author Mitch Watrous <watrous@u.washington.edu>
Fri, 06 May 2011 13:21:20 -0700
changeset 7140 d203296efb63
parent 7024 4392d52b3536
permissions -rw-r--r--
Fix an example dependency problem and fix some paths in the documentation
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
6742
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     1
.. include:: replace.txt
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     2
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     3
Distributed
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     4
-----------
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     5
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     6
Parallel and distributed discrete event simulation allows the execution of a
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     7
single simulation program on multiple processors. By splitting up the simulation
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     8
into logical processes, LPs, each LP can be executed by a different processor.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
     9
This simulation methodology enables very large-scale simulations by leveraging
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    10
increased processing power and memory availability. In order to ensure proper
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    11
execution of a distributed simulation, message passing between LPs is required.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    12
To support distributed simulation in |ns3|, the standard Message Passing
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    13
Interface (MPI) is used, along with a new distributed simulator class.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    14
Currently, dividing a simulation for distributed purposes in |ns3| can only occur
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    15
across point-to-point links.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    16
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    17
Current Implementation Details
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    18
******************************
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    19
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    20
During the course of a distributed simulation, many packets must cross simulator
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    21
boundaries. In other words, a packet that originated on one LP is destined for a
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    22
different LP, and in order to make this transition, a message containing the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    23
packet contents must be sent to the remote LP. Upon receiving this message, the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    24
remote LP can rebuild the packet and proceed as normal. The process of sending
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    25
an receiving messages between LPs is handled easily by the new MPI interface in
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    26
|ns3|.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    27
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    28
Along with simple message passing between LPs, a distributed simulator is used
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    29
on each LP to determine which events to process. It is important to process
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    30
events in time-stamped order to ensure proper simulation execution. If a LP
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    31
receives a message containing an event from the past, clearly this is an issue,
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    32
since this event could change other events which have already been executed. To
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    33
address this problem, a conservative synchronization algorithm with lookahead is
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    34
used in |ns3|. For more information on different synchronization approaches and
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    35
parallel and distributed simulation in general, please refer to "Parallel and
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    36
Distributed Simulation Systems" by Richard Fujimoto.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    37
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    38
Remote point-to-point links
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    39
+++++++++++++++++++++++++++
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    40
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    41
As described in the introduction, dividing a simulation for distributed purposes
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    42
in |ns3| currently can only occur across point-to-point links; therefore, the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    43
idea of remote point-to-point links is very important for distributed simulation
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    44
in |ns3|. When a point-to-point link is installed, connecting two nodes, the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    45
point-to-point helper checks the system id, or rank, of both nodes. The rank
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    46
should be assigned during node creation for distributed simulation and is
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    47
intended to signify on which LP a node belongs. If the two nodes are on the same
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    48
rank, a regular point-to-point link is created. If, however, the two nodes are
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    49
on different ranks, then these nodes are intended for different LPs, and a
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    50
remote point-to-point link is used. If a packet is to be sent across a remote
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    51
point-to-point link, MPI is used to send the message to the remote LP.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    52
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    53
Distributing the topology
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    54
+++++++++++++++++++++++++
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    55
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    56
Currently, the full topology is created on each rank, regardless of the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    57
individual node system ids. Only the applications are specific to a rank. For
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    58
example, consider node 1 on LP 1 and node 2 on LP 2, with a traffic generator on
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    59
node 1. Both node 1 and node 2 will be created on both LP1 and LP2; however, the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    60
traffic generator will only be installed on LP1. While this is not optimal for
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    61
memory efficiency, it does simplify routing, since all current routing
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    62
implementations in |ns3| will work with distributed simulation.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    63
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    64
Running Distributed Simulations
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    65
*******************************
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    66
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    67
Prerequisites
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    68
+++++++++++++
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    69
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    70
Ensure that MPI is installed, as well as mpic++. In Ubuntu repositories, 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    71
these are openmpi-bin, openmpi-common, openmpi-doc, libopenmpi-dev. In 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    72
Fedora, these are openmpi and openmpi-devel.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    73
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    74
Note: 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    75
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    76
There is a conflict on some Fedora systems between libotf and openmpi. A 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    77
possible "quick-fix" is to yum remove libotf before installing openmpi. 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    78
This will remove conflict, but it will also remove emacs. Alternatively, 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    79
these steps could be followed to resolve the conflict:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    80
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    81
    1) Rename the tiny otfdump which emacs says it needs:
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    82
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    83
         mv /usr/bin/otfdump /usr/bin/otfdump.emacs-version
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    84
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    85
    2) Manually resolve openmpi dependencies:
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    86
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    87
         sudo yum install libgfortran libtorque numactl
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    88
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    89
    3) Download rpm packages: 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    90
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    91
         openmpi-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    92
         openmpi-devel-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    93
         openmpi-libs-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    94
         openmpi-vt-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    95
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    96
         from
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    97
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    98
         http://mirrors.kernel.org/fedora/releases/11/Everything/i386/os/Packages/
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
    99
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   100
    4) Force the packages in:
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   101
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   102
         sudo rpm -ivh --force openmpi-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   103
         openmpi-libs-1.3.1-1.fc11.i586.rpm openmpi-devel-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   104
         openmpi-vt-1.3.1-1.fc11.i586.rpm
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   105
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   106
Also, it may be necessary to add the openmpi bin directory to PATH in order to
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   107
execute mpic++ and mpirun from the command line. Alternatively, the full path to
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   108
these executables can be used. Finally, if openmpi complains about the inability
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   109
to open shared libraries, such as libmpi_cxx.so.0, it may be necessary to add
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   110
the openmpi lib directory to LD_LIBRARY_PATH.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   111
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   112
Building and Running Examples
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   113
+++++++++++++++++++++++++++++
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   114
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   115
If you already built |ns3| without MPI enabled, you must re-build:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   116
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   117
    ./waf distclean
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   118
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   119
Configure |ns3| with the --enable-mpi option:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   120
7024
4392d52b3536 Make examples not be built by default
Mitch Watrous <watrous@u.washington.edu>
parents: 6998
diff changeset
   121
    ./waf -d debug configure --enable-examples --enable-tests --enable-mpi
6742
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   122
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   123
Ensure that MPI is enabled by checking the optional features shown from the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   124
output of configure.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   125
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   126
Next, build |ns3|:
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   127
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   128
    ./waf
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   129
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   130
After building |ns3| with mpi enabled, the example programs are now ready to run
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   131
with mpirun. Here are a few examples (from the root |ns3| directory):::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   132
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   133
    mpirun -np 2 ./waf --run simple-distributed
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   134
    mpirun -np 4 -machinefile mpihosts ./waf --run 'nms-udp-nix --LAN=2 --CN=4 --nix=1'
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   135
            
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   136
The np switch is the number of logical processors to use. The machinefile switch
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   137
is which machines to use. In order to use machinefile, the target file must
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   138
exist (in this case mpihosts). This can simply contain something like:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   139
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   140
    localhost
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   141
    localhost
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   142
    localhost
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   143
    ...
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   144
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   145
Or if you have a cluster of machines, you can name them.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   146
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   147
NOTE: Some users have experienced issues using mpirun and waf together. An
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   148
alternative way to run distributed examples is shown below:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   149
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   150
    ./waf shell
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   151
    cd build/debug
7140
d203296efb63 Fix an example dependency problem and fix some paths in the documentation
Mitch Watrous <watrous@u.washington.edu>
parents: 7024
diff changeset
   152
    mpirun -np 2 src/mpi/examples/simple-distributed
6742
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   153
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   154
Creating custom topologies
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   155
++++++++++++++++++++++++++
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   156
7140
d203296efb63 Fix an example dependency problem and fix some paths in the documentation
Mitch Watrous <watrous@u.washington.edu>
parents: 7024
diff changeset
   157
The example programs in src/mpi/examples give a good idea of how to create different
6742
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   158
topologies for distributed simulation. The main points are assigning system ids
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   159
to individual nodes, creating point-to-point links where the simulation should
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   160
be divided, and installing applications only on the LP associated with the
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   161
target node.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   162
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   163
Assigning system ids to nodes is simple and can be handled two different ways.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   164
First, a NodeContainer can be used to create the nodes and assign system ids:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   165
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   166
    NodeContainer nodes;
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   167
    nodes.Create (5, 1); // Creates 5 nodes with system id 1.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   168
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   169
Alternatively, nodes can be created individually, assigned system ids, and added
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   170
to a NodeContainer. This is useful if a NodeContainer holds nodes with different
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   171
system ids:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   172
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   173
    NodeContainer nodes;
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   174
    Ptr<Node> node1 = CreateObject<Node> (0); // Create node1 with system id 0
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   175
    Ptr<Node> node2 = CreateObject<Node> (1); // Create node2 with system id 1
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   176
    nodes.Add (node1);
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   177
    nodes.Add (node2);
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   178
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   179
Next, where the simulation is divided is determined by the placement of 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   180
point-to-point links. If a point-to-point link is created between two 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   181
nodes with different system ids, a remote point-to-point link is created, 
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   182
as described in :ref:`Current Implementation Details`.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   183
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   184
Finally, installing applications only on the LP associated with the target node
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   185
is very important. For example, if a traffic generator is to be placed on node
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   186
0, which is on LP0, only LP0 should install this application.  This is easily
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   187
accomplished by first checking the simulator system id, and ensuring that it
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   188
matches the system id of the target node before installing the application.
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   189
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   190
Tracing During Distributed Simulations
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   191
**************************************
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   192
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   193
Depending on the system id (rank) of the simulator, the information traced will
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   194
be different, since traffic originating on one simulator is not seen by another
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   195
simulator until it reaches nodes specific to that simulator. The easiest way to
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   196
keep track of different traces is to just name the trace files or pcaps
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   197
differently, based on the system id of the simulator. For example, something
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   198
like this should work well, assuming all of these local variables were
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   199
previously defined:::
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   200
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   201
    if (MpiInterface::GetSystemId () == 0)
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   202
      {
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   203
        pointToPoint.EnablePcapAll ("distributed-rank0");
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   204
        phy.EnablePcap ("distributed-rank0", apDevices.Get (0));
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   205
        csma.EnablePcap ("distributed-rank0", csmaDevices.Get (0), true);
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   206
      }
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   207
    else if (MpiInterface::GetSystemId () == 1)
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   208
      {
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   209
        pointToPoint.EnablePcapAll ("distributed-rank1");
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   210
        phy.EnablePcap ("distributed-rank1", apDevices.Get (0));
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   211
        csma.EnablePcap ("distributed-rank1", csmaDevices.Get (0), true);
a1759a95842c convert manual to sphinx format
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
parents:
diff changeset
   212
      }