--- a/doc/manual/emulation.texi Fri Dec 05 13:19:40 2008 -0800
+++ b/doc/manual/emulation.texi Fri Dec 05 16:49:05 2008 -0800
@@ -3,75 +3,374 @@
@anchor{chap:Emulation}
ns-3 has been designed for integration into testbed and virtual machine
-environments. These two environments require distinct approaches. An
-architecture for the first approach, used in testbeds, is shown in the
-following figure.
+environments. We have addressed this need by providing two kinds of
+net devices. The first kind, which we call an @code{Emu} @code {NetDevice}
+allows ns-3 simulations to send data on a ``real'' network. The second kind,
+called a @code{Tap} @code{NetDevice} allows a ``real'' host to participate
+in an ns-3 simulation as if it were one of the simulated nodes. An ns-3
+simulation may be constructed with any combination of simulated, @code{Emu},
+or @code{Tap} devices.
+
+One of the use-cases we want to support is that of a testbed. A concrete
+example of an environment of this kind is the ORBIT testbed. ORBIT is a
+laboratory emulator/field trial network arranged as a two dimensional grid of
+400 802.11 radio nodes. We integrate with ORBIT by using their ``imaging''
+process to load and run ns-3 simulations on the ORBIT array. We use our
+@code{Emu} @code{NetDevice}s to drive the hardware in the testbed and we can
+accumulate results either using the ns-3 tracing and logging functions, or the
+native ORBIT data gathering techniques. See @uref{http://www.orbit-lab.org/}
+for details on the ORBIT testbed.
+
+A simulation of this kind is shown in the following figure:
@float Figure,fig:testbed
-@caption{Implementation overview of testbed.}
-@image{figures/testbed}
+@center @caption{Example Implementation of Testbed Emulation.}
+@center @image{figures/testbed, 5in}
@end float
-You can see that there are separate hosts, each running a subset of a
+You can see that there are separate hosts, each running a subset of a ``global''
simulation. Instead of an ns-3 channel connecting the hosts, we use real
hardware provided by the testbed. This allows ns-3 applications and protocol
stacks attached to a simulation node to communicate over real hardware.
-We expect the primary use for this configuration will be to analyze experimental
-applications and protocols in a real network environment that includes all of
+We expect the primary use for this configuration will be to generate repeatable
+experimental results in a real-world network environment that includes all of
the ns-3 tracing, loging, visualization and statistics gathering tools.
-A concrete example of a testbed environment is the ORBIT testbed which provides
-a rectangular array of hosts with various types of WiFi cards and other radios.
-We integrate with ORBIT in that we can load and run ns-3 simulations on the ORBIT
-array and use our simulator to drive the hardware in the testbed and accumulate
-results either using the ns-3 tracing and logging functions, or the native ORBIT
-data gathering techniques.
-
-In what is essentially an inverse configuration, we allow virtual machines
-running native applications and protocol stacks to communicate over an ns-3
-simulated channel. This type of scenario is shown in the following figure.
+In what can be viewed as essentially an inverse configuration, we allow ``real''
+machines running native applications and protocol stacks to integrate with
+an ns-3 simulation. This allows for the simulation of large networks connected
+to a real mahince, and also enables virtualization. A simulation of this kind
+is shown in the following figure:
@float Figure,fig:emulated-channel
@caption{Implementation overview of emulated channel.}
-@image{figures/emulated-channel}
+@image{figures/emulated-channel, 5in}
@end float
-In this figure, you will see that there is a single host with a number of
-virtual machines running on it. An ns-3 simulation is shown running in the
-center virtual machine. This simulation has a number of nodes with associated
-ns-3 applications and protocol stacks that are talking to an ns-3 channel
-through ns-3 net devices.
+Here, you will see that there is a single host with a number of virtual machines
+running on it. An ns-3 simulation is shown running in the virtual machine shown
+in the center of the figure. This simulation has a number of nodes with associated
+ns-3 applications and protocol stacks that are talking to an ns-3 channel through
+native simulated ns-3 net devices.
There are also two virtual machines shown at the far left and far right of the
-figure. These VMs are running native applications and native protocol stacks,
-and are connected into the simulation by a Tap net device. The user-mode handler
-for the Tap device is instantiated in the simulation and attached to a proxy node
-that represents the native VM in the simulation.
-
-The nodes at either end of the simulation VM have only a Tap handler and net
-device associated with them. These handlers allow the Tap devices on the native
-VMs to behave as an ns-3 net device in the simulation VM. This, in turn, allows
-the native software and protocol suites in the native VMs to believe that they
-are connected to a real network represented by the simulation.
+figure. These VMs are running native (Linux) applications and protocol stacks.
+The VM is connected into the simulation by a Linux Tap net device. The user-mode
+handler for the Tap device is instantiated in the simulation and attached to a
+proxy node that represents the native VM in the simulation. These handlers allow
+the Tap devices on the native VMs to behave as if they were ns-3 net devices in
+the simulation VM. This, in turn, allows the native software and protocol suites
+in the native VMs to believe that they are connected to the simulated ns-3
+channel.
We expect the typical use case for this environment will be to analyze the
behavior of native applications and protocol suites in the presence of large
-simulated ns-3 networks. For example, the channel in the simulation VM could
-be an ns-3 Wifi channel and could also be connected to any ns-3 topology. This
-would allow native applications or protocol suites to be analyzed in the presence
-of very complex simulated networks.
+simulated ns-3 networks.
@section Behavior
-TBD
+@subsection Emu Net Device
+
+The @code{Emu} net device allows a simulation node to send and receive packets
+over a real network. The emulated net device relies on a specified interface
+being in promiscuous mode. It opens a raw socket and binds to that interface.
+We perform MAC spoofing to separate simulation network traffic from other
+network traffic that may be flowing to and from the host.
+
+Normally, the use case for emulated net devices is in collections of small
+simulations that connect to the outside world through specific interfaces.
+For example, one could construct a number of virtual machines and connect them
+via a host-only network. To use the emulated net device, you would need to
+set all of the host-only interfaces in promiscuous mode and provide an
+appropriate device name, "eth1" for example.
+
+One could also use the @code{Emu} net device in a testbed situation where the
+host on which the simulation is running has a specific interface of interest
+which drives the testbed hardware. You would also need to set this specific
+interface into promiscuous mode and provide an appropriate device name to the
+ns-3 emulated net device. An example of this environment is the ORBIT testbed
+as described above.
+
+The @code{Emu} net device only works if the underlying interface is up and in
+promiscuous mode. Packets will be sent out over the device, but we use MAC
+spoofing. The MAC addresses will be generated (by default) using the
+Organizationally Unique Identifier (OUI) 00:00:00 as a base. This vendor code
+is not assigned to any organization and so should not conflict with any real
+hardware.
+
+It is always up to the user to determine that using these MAC addresses is
+okay on your network and won't conflict with anything else (including another
+simulation using @code{Emu} devices) on your network. If you are using the
+emulated net device in separate simulations you must consider global MAC
+address assignment issues and ensure that MAC addresses are unique across
+all simulations. The emulated net device respects the MAC address provided
+in the @code{SetAddress} method so you can do this manually. For larger
+simulations, you may want to set the OUI in the MAC address allocation function.
+
+IP addresses corresponding to the emulated net devices are the addresses
+generated in the simulation, which are generated in the usual way via helper
+functions. Since we are using MAC spoofing, there will not be a conflict
+between ns-3 network stacks and any native network stacks.
+
+The emulated net device comes with a helper function as all ns-3 devices do.
+One unique aspect is that there is no channel associated with the underlying
+medium. We really have no idea what this external medium is, and so have not
+made an effort to model it abstractly. The primary thing to be aware of is the
+implication this has for static global routing. The global router module
+attempts to walk the channels looking for adjacent networks. Since there
+is no channel, the global router will be unable to do this and you must then
+use a dynamic routing protocol such as OLSR to include routing in
+@code{Emu}-based networks.
+
+@subsection Tap Net Device
+
+The @code{Tap} Net Device is scheduled for inclusion in ns-3.4 at the writing
+of this section. We will include details as soon as the @code{Tap} device is
+merged.
@section Usage
-TBD
+@subsection Emu Net Device
+
+The usage of the @code{Emu} net device is straightforward once the network of
+simulations has been configured. Since most of the work involved in working
+with this device is in network configuration before even starting a simulation,
+you may want to take a moment to review a couple of HOWTO pages on the ns-3 wiki
+that describe how to set up a virtual test network using VMware and how to run
+a set of example (client server) simulations that use @code{Emu} net devices.
+
+@uref{http://www.nsnam.org/wiki/index.php/HOWTO_use_VMware_to_set_up_virtual_networks_(Windows)}
+@uref{http://www.nsnam.org/wiki/index.php/HOWTO_use_ns-3_scripts_to_drive_real_hardware_(experimental)}
+
+Once you are over the configuration hurdle, the script changes required to use
+an @code{Emu} device are trivial. The main structural difference is that you
+will need to create an ns-3 simulation script for each node. In the case of
+the HOWTOs above, there is one client script and one server script. The only
+``challenge'' is to get the addresses set correctly.
+
+Just as with all other ns-3 net devices, we provide a helper class for the
+@code{Emu} net device. The following code snippet illustrates how one would
+declare an EmuHelper and use it to set the ``DeviceName'' attribute to ``eth1''
+and install @code{Emu} devices on a group of nodes. You would do this on both
+the client and server side in the case of the HOWTO seen above.
+
+@verbatim
+ EmuHelper emu;
+ emu.SetAttribute ("DeviceName", StringValue ("eth1"));
+ NetDeviceContainer d = emu.Install (n);
+@end verbatim
+
+The only other change that may be required is to make sure that the address
+spaces (MAC and IP) on the client and server simulations are compatible. First
+the MAC address is set to a unique well-known value in both places (illustrated
+here for one side).
+
+@verbatim
+ //
+ // We've got the devices in place. Since we're using MAC address
+ // spoofing under the sheets, we need to make sure that the MAC addresses
+ // we have assigned to our devices are unique. Ns-3 will happily
+ // automatically assign the same MAC addresses to the devices in both halves
+ // of our two-script pair, so let's go ahead and just manually change them
+ // to something we ensure is unique.
+ //
+ Ptr<NetDevice> nd = d.Get (0);
+ Ptr<EmuNetDevice> ed = nd->GetObject<EmuNetDevice> ();
+ ed->SetAddress ("00:00:00:00:00:02");
+@end verbatim
+
+And then the IP address of the client or serveris set in the usual way using
+helpers.
+
+@verbatim
+ //
+ // We've got the "hardware" in place. Now we need to add IP addresses.
+ // This is the server half of a two-script pair. We need to make sure
+ // that the addressing in both of these applications is consistent, so
+ // we use provide an initial address in both cases. Here, the client
+ // will reside on one machine running ns-3 with one node having ns-3
+ // with IP address "10.1.1.2" and talk to a server script running in
+ // another ns-3 on another computer that has an ns-3 node with IP
+ // address "10.1.1.3"
+ //
+ Ipv4AddressHelper ipv4;
+ ipv4.SetBase ("10.1.1.0", "255.255.255.0", "0.0.0.2");
+ Ipv4InterfaceContainer i = ipv4.Assign (d);
+@end verbatim
+
+You will use application helpers to generate traffic exactly as you do in any
+ns-3 simulation script. Note that the server address shown below in a snippet
+from the client, must correspond to the IP address assigned to the server node
+similarly to the snippet above.
+
+@verbatim
+ uint32_t packetSize = 1024;
+ uint32_t maxPacketCount = 2000;
+ Time interPacketInterval = Seconds (0.001);
+ UdpEchoClientHelper client ("10.1.1.3", 9);
+ client.SetAttribute ("MaxPackets", UintegerValue (maxPacketCount));
+ client.SetAttribute ("Interval", TimeValue (interPacketInterval));
+ client.SetAttribute ("PacketSize", UintegerValue (packetSize));
+ ApplicationContainer apps = client.Install (n.Get (0));
+ apps.Start (Seconds (1.0));
+ apps.Stop (Seconds (2.0));
+@end verbatim
+
+The @code{Emu} net device and helper provide access to ASCII and pcap tracing
+functionality just as other ns-3 net devices to. You enable tracing similarly
+to these other net devices:
+
+@verbatim
+ EmuHelper::EnablePcapAll ("emu-udp-echo-client");
+@end verbatim
+
+To see an example of a client script using the @code{Emu} net device, see
+@code{examples/emu-udp-echo-client.cc} and @code{examples/emu-udp-echo-server.cc}
+in the repository @uref{http://code.nsnam.org/craigdo/ns-3-emu/}.
+
+@subsection Tap Net Device
+
+The @code{Tap} Net Device is scheduled for inclusion in ns-3.4 at the writing
+of this section. We will include details as soon as the @code{Tap} device is
+merged.
@section Implementation
-TBD
+Perhaps the most unusual part of the @code{Emu} and @code{Tap} device
+implementation relates to the requirement for executing some of the code
+with super-user permissions. Rather than force the user to execute the entire
+simulation as root, we provide a small ``creator'' program that runs as root
+and does any required high-permission sockets work.
+
+We do a similar thing for both the @code{Emu} and the @code{Tap} devices.
+The high-level view is that the @code{CreateSocket} method creates a local
+interprocess (Unix) socket, forks, and executes the small creation program.
+The small program, which runs as suid root, creates a raw socket and sends
+back the raw socket file descriptor over the Unix socket that is passed to
+it as a parameter. The raw socket is passed as a control message (sometimes
+called ancillary data) of type SCM_RIGHTS.
+
+@subsection Emu Net Device
+
+The @code{Emu} net device uses the ns-3 threading and multithreaded real-time
+scheduler extensions. The interesting work in the @code{Emu} device is done
+when the net device is started (@code{EmuNetDevice::StartDevice ()}). An
+attribute (``Start'') provides a simulation time at which to spin up the
+net device. At this specified time (which defaults to t=0), the socket
+creation function is called and executes as described above. You may also
+specify a time at which to stop the device using the ``Stop'' attribute.
+
+Once the (promiscuous mode) socket is created, we bind it to an interface name
+also provided as an attribute (``DeviceName'') that is stored internally as
+@code{m_deviceName}:
+
+@verbatim
+ struct ifreq ifr;
+ bzero (&ifr, sizeof(ifr));
+ strncpy ((char *)ifr.ifr_name, m_deviceName.c_str (), IFNAMSIZ);
+
+ int32_t rc = ioctl (m_sock, SIOCGIFINDEX, &ifr);
+
+ struct sockaddr_ll ll;
+ bzero (&ll, sizeof(ll));
+
+ ll.sll_family = AF_PACKET;
+ ll.sll_ifindex = m_sll_ifindex;
+ ll.sll_protocol = htons(ETH_P_ALL);
+
+ rc = bind (m_sock, (struct sockaddr *)&ll, sizeof (ll));
+@end verbatim
+
+After the promiscuous raw socket is set up, a separate thread is spawned to do
+reads from that socket and the link state is set to @code{Up}.
+
+@verbatim
+ m_readThread = Create<SystemThread> (
+ MakeCallback (&EmuNetDevice::ReadThread, this));
+ m_readThread->Start ();
+
+ NotifyLinkUp ();
+@end verbatim
+
+The @code{EmuNetDevice::ReadThread} function basically just sits in an infinite
+loop reading from the promiscuous mode raw socket and scheduling packet
+receptions using the real-time simulator extensions.
+
+@verbatim
+ for (;;)
+ {
+ ...
+
+ len = recvfrom (m_sock, buf, bufferSize, 0, (struct sockaddr *)&addr,
+ &addrSize);
+
+ ...
+
+ DynamicCast<RealtimeSimulatorImpl> (Simulator::GetImplementation ())->
+ ScheduleRealtimeNow (
+ MakeEvent (&EmuNetDevice::ForwardUp, this, buf, len));
+
+ ...
+ }
+@end verbatim
+
+The line starting with our templated DynamicCast function probably deserves a
+comment. It gains access to the simulator implementation object using
+the @code{Simulator::GetImplementation} method and then casts to the real-time
+simulator implementation to use the real-time schedule method
+@code{ScheduleRealtimeNow}. This function will cause a handler for the newly
+received packet to be scheduled for execution at the current real time clock
+value. This will, in turn cause the simulation clock to be advanced to that
+real time value when the scheduled event (@code{EmuNetDevice::ForwardUp}) is
+fired.
+
+The @code{ForwardUp} function operates as most other similar ns-3 net device
+methods do. The packet is first filtered based on the destination address. In
+the case of the @code{Emu} device, the MAC destination address will be the
+address of the @code{Emu} device and not the hardware address of the real
+device. Headers are then stripped off and the trace hooks are hit. Finally,
+the packet is passed up the ns-3 protocol stack using the receive callback
+function of the net device.
+
+Sending a packet is equally straightforward as shown below. The first thing
+we do is to add the ethernet header and trailer to the ns-3 @code{Packet} we
+are sending. The source address corresponds to the address of the @code{Emu}
+device and not the underlying native device MAC address. This is where the
+MAC address spoofing is done. The trailer is added and we enqueue and dequeue
+the packet from the net device queue to hit the trace hooks.
+
+@verbatim
+ header.SetSource (source);
+ header.SetDestination (destination);
+ header.SetLengthType (packet->GetSize ());
+ packet->AddHeader (header);
+
+ EthernetTrailer trailer;
+ trailer.CalcFcs (packet);
+ packet->AddTrailer (trailer);
+
+ m_queue->Enqueue (packet);
+ packet = m_queue->Dequeue ();
+
+ struct sockaddr_ll ll;
+ bzero (&ll, sizeof (ll));
+
+ ll.sll_family = AF_PACKET;
+ ll.sll_ifindex = m_sll_ifindex;
+ ll.sll_protocol = htons(ETH_P_ALL);
+
+ rc = sendto (m_sock, packet->PeekData (), packet->GetSize (), 0,
+ reinterpret_cast<struct sockaddr *> (&ll), sizeof (ll));
+@end verbatim
+Finally, we simply send the packet to the raw socket which puts it out on the
+real network.
+
+@subsection Tap Net Device
+
+The @code{Tap} Net Device is scheduled for inclusion in ns-3.4 at the writing
+of this section. We will include details as soon as the @code{Tap} device is
+merged.
+