--- a/doc/manual/emu.texi Thu Oct 15 23:55:06 2009 -0700
+++ b/doc/manual/emu.texi Fri Oct 16 07:13:05 2009 -0700
@@ -1,183 +1,293 @@
@node Emu NetDevice
@chapter Emu NetDevice
-This is the introduction to Emu NetDevice chapter, to complement the
-Emu model doxygen.
-
-@menu
-* Overview of the model::
-* Using the EmuNetDevice::
-* Emu Tracing::
-@end menu
+@section Behavior
-@node Overview of the model
-@section Overview of the model
-
-The emulated net device allows a simulation node to send and receive packets
-a real network.
-
-The Emu net device is not a complete net device and channel combination as is
-typical in ns-3. The Emu device can be thought of as a proxy for a real
-device that resides in an ns-3 simulation. The Emu net device talks to that
-real device using raw sockets and binds to the device via the Linux interface.
-There is no related Emu channel since other devices will most likely reside on
-different computers running entirely separate simulations.
+The @code{Emu} net device allows a simulation node to send and receive packets
+over a real network. The emulated net device relies on a specified interface
+being in promiscuous mode. It opens a raw socket and binds to that interface.
+We perform MAC spoofing to separate simulation network traffic from other
+network traffic that may be flowing to and from the host.
-The Emu net device relies on a specified interface (``eth1, for example) being
-in promiscuous mode. It opens a raw socket and binds to that interface. We
-perform MAC spoofing to separate simulation network traffic from other network
-traffic that may be flowing to and from the host.
-
-Normally, the use case for emulated net devices is in collections of
-small simulations that connect to the outside world through specific
-interfaces. For example, one could construct a number of virtual
-machines and connect them via a host-only network. To use the emulated
-net device, you would need to set all of the host-only interfaces in
-promiscuous mode and provide an appropriate device name, "eth1" for example.
+Normally, the use case for emulated net devices is in collections of small
+simulations that connect to the outside world through specific interfaces.
+For example, one could construct a number of virtual machines and connect them
+via a host-only network. To use the emulated net device, you would need to
+set all of the host-only interfaces in promiscuous mode and provide an
+appropriate device name, "eth1" for example.
-One could also use the emulated net device in a testbed situation
-where the host on which the simulation is running has a specific interface
-of interest which drives the testbed hardware. You would also need to set
-this specific interface into promiscuous mode and provide an appropriate
-device name to the ns-3 emulated net device.
+One could also use the @code{Emu} net device in a testbed situation where the
+host on which the simulation is running has a specific interface of interest
+which drives the testbed hardware. You would also need to set this specific
+interface into promiscuous mode and provide an appropriate device name to the
+ns-3 emulated net device. An example of this environment is the ORBIT testbed
+as described above.
-The emulated net device only works if the underlying interface is up in
-promiscuous mode. We could just turn it on, but the situation is that we
-expect the other considerations listed above to have been dealt with.
-To verify that these issues are dealt with, we just make sure that the end
-result of that process has taken place and that the specified interface is
-in promiscuous mode.
-
-@subsection Address Concerns
-
-Packets will be sent out over the device, but as mentioned, we use MAC spoofing.
-By default in the simulation, the MAC addresses will be generated using the
+The @code{Emu} net device only works if the underlying interface is up and in
+promiscuous mode. Packets will be sent out over the device, but we use MAC
+spoofing. The MAC addresses will be generated (by default) using the
Organizationally Unique Identifier (OUI) 00:00:00 as a base. This vendor code
is not assigned to any organization and so should not conflict with any real
hardware.
-It is always up to you to determine that using these MAC addresses is
+It is always up to the user to determine that using these MAC addresses is
okay on your network and won't conflict with anything else (including another
-simulation using emu devices) on your network. If you are using the
+simulation using @code{Emu} devices) on your network. If you are using the
emulated net device in separate simulations you must consider global MAC
address assignment issues and ensure that MAC addresses are unique across
all simulations. The emulated net device respects the MAC address provided
-in the SetAddress method so you can do this manually. For larger simulations,
-you may want to set the OUI in the MAC address allocation function.
+in the @code{SetAddress} method so you can do this manually. For larger
+simulations, you may want to set the OUI in the MAC address allocation function.
IP addresses corresponding to the emulated net devices are the addresses
generated in the simulation, which are generated in the usual way via helper
-functions.
-
-@subsection Attributes
-
-The Emu network device appears to the ns-3 system just as any other device and
-can be controlled through the attribute system, and traced through conventional
-trace hooks. The EmuNetDevice provides following Attributes:
-
-@itemize @bullet
-@item Address: The Mac48Address of the device;
-@item DeviceName: The name of the underlying real device (e.g., ``eth1'');
-@item Start: The simulation time at which to enable the underlying socket;
-@item Stop: The simulation time at which to stop receiving from the underlying socket;
-@item TxQueue: The transmit queue used by the device;
-@item InterframeGap: The optional time to wait between "frames";
-@item Rx: A trace source for received packets;
-@end itemize
-
-Packets sent over the EmuNetDevice are always routed through the
-transmit queue to provide a trace hook for packets sent out over the
-network. This transmit queue can be set (via attribute) to model different
-queuing strategies.
-
-@node Using the EmuNetDevice
-@section Using the EmuNetDevice
+functions. Since we are using MAC spoofing, there will not be a conflict
+between ns-3 network stacks and any native network stacks.
The emulated net device comes with a helper function as all ns-3 devices do.
One unique aspect is that there is no channel associated with the underlying
-medium. We really have no idea what this medium is, and so have not made an
-effort to model it abstractly. The primary thing to be aware of is the
+medium. We really have no idea what this external medium is, and so have not
+made an effort to model it abstractly. The primary thing to be aware of is the
implication this has for static global routing. The global router module
attempts to walk the channels looking for adjacent networks. Since there
-is no channel, the global router will be unable to do this.
+is no channel, the global router will be unable to do this and you must then
+use a dynamic routing protocol such as OLSR to include routing in
+@code{Emu}-based networks.
-The Emu net devices are typically created and configured using the associated
-@code{EmuHelper} object. The various ns3 device helpers generally work in a
-similar way, and their use is seen in many of our example programs.
+@section Usage
-The conceptual model of interest is that of a bare computer ``husk'' into which
-you plug net devices. The bare computers are created using a @code{NodeContainer}
-helper. You just ask this helper to create as many computers (we call them
-@code{Nodes}) as you need on your network:
-
+Any mixing of ns-3 objects with real objects will typically require that
+ns-3 compute checksums in its protocols. By default, checksums are not
+computed by ns-3. To enable checksums (e.g. UDP, TCP, IP), users must set
+the attribute @code{ChecksumEnabled} to true, such as follows:
@verbatim
- NodeContainer nodes;
- nodes.Create (nEmuNodes);
+GlobalValue::Bind ("ChecksumEnabled", BooleanValue (true));
@end verbatim
-Once you have your nodes, you need to instantiate a @code{EmuHelper} and set
-any attributes you may want to change.
+The usage of the @code{Emu} net device is straightforward once the network of
+simulations has been configured. Since most of the work involved in working
+with this device is in network configuration before even starting a simulation,
+you may want to take a moment to review a couple of HOWTO pages on the ns-3 wiki
+that describe how to set up a virtual test network using VMware and how to run
+a set of example (client server) simulations that use @code{Emu} net devices.
+
+@uref{http://www.nsnam.org/wiki/index.php/HOWTO_use_VMware_to_set_up_virtual_networks_(Windows)}
+@uref{http://www.nsnam.org/wiki/index.php/HOWTO_use_ns-3_scripts_to_drive_real_hardware_(experimental)}
+
+Once you are over the configuration hurdle, the script changes required to use
+an @code{Emu} device are trivial. The main structural difference is that you
+will need to create an ns-3 simulation script for each node. In the case of
+the HOWTOs above, there is one client script and one server script. The only
+``challenge'' is to get the addresses set correctly.
+
+Just as with all other ns-3 net devices, we provide a helper class for the
+@code{Emu} net device. The following code snippet illustrates how one would
+declare an EmuHelper and use it to set the ``DeviceName'' attribute to ``eth1''
+and install @code{Emu} devices on a group of nodes. You would do this on both
+the client and server side in the case of the HOWTO seen above.
@verbatim
EmuHelper emu;
- csma.SetAttribute ("DeviceName", StringValue ("eth1"));
+ emu.SetAttribute ("DeviceName", StringValue ("eth1"));
+ NetDeviceContainer d = emu.Install (n);
@end verbatim
-
-Once the attributes are set, all that remains is to create the devices
-and install them on the required nodes. When we create the net devices,
-we add them to a container to allow you to use them in the future. This
-all takes just one line of code.
+
+The only other change that may be required is to make sure that the address
+spaces (MAC and IP) on the client and server simulations are compatible. First
+the MAC address is set to a unique well-known value in both places (illustrated
+here for one side).
+
+@verbatim
+ //
+ // We've got the devices in place. Since we're using MAC address
+ // spoofing under the sheets, we need to make sure that the MAC addresses
+ // we have assigned to our devices are unique. Ns-3 will happily
+ // automatically assign the same MAC addresses to the devices in both halves
+ // of our two-script pair, so let's go ahead and just manually change them
+ // to something we ensure is unique.
+ //
+ Ptr<NetDevice> nd = d.Get (0);
+ Ptr<EmuNetDevice> ed = nd->GetObject<EmuNetDevice> ();
+ ed->SetAddress ("00:00:00:00:00:02");
+@end verbatim
+
+And then the IP address of the client or server is set in the usual way using
+helpers.
@verbatim
- NetDeviceContainer emuDevices = emu.Install (nodes);
+ //
+ // We've got the "hardware" in place. Now we need to add IP addresses.
+ // This is the server half of a two-script pair. We need to make sure
+ // that the addressing in both of these applications is consistent, so
+ // we use provide an initial address in both cases. Here, the client
+ // will reside on one machine running ns-3 with one node having ns-3
+ // with IP address "10.1.1.2" and talk to a server script running in
+ // another ns-3 on another computer that has an ns-3 node with IP
+ // address "10.1.1.3"
+ //
+ Ipv4AddressHelper ipv4;
+ ipv4.SetBase ("10.1.1.0", "255.255.255.0", "0.0.0.2");
+ Ipv4InterfaceContainer i = ipv4.Assign (d);
+@end verbatim
+
+You will use application helpers to generate traffic exactly as you do in any
+ns-3 simulation script. Note that the server address shown below in a snippet
+from the client, must correspond to the IP address assigned to the server node
+similarly to the snippet above.
+
+@verbatim
+ uint32_t packetSize = 1024;
+ uint32_t maxPacketCount = 2000;
+ Time interPacketInterval = Seconds (0.001);
+ UdpEchoClientHelper client ("10.1.1.3", 9);
+ client.SetAttribute ("MaxPackets", UintegerValue (maxPacketCount));
+ client.SetAttribute ("Interval", TimeValue (interPacketInterval));
+ client.SetAttribute ("PacketSize", UintegerValue (packetSize));
+ ApplicationContainer apps = client.Install (n.Get (0));
+ apps.Start (Seconds (1.0));
+ apps.Stop (Seconds (2.0));
+@end verbatim
+
+The @code{Emu} net device and helper provide access to ASCII and pcap tracing
+functionality just as other ns-3 net devices to. You enable tracing similarly
+to these other net devices:
+
+@verbatim
+ EmuHelper::EnablePcapAll ("emu-udp-echo-client");
@end verbatim
-@node Emu Tracing
-@section Emu Tracing
+To see an example of a client script using the @code{Emu} net device, see
+@code{examples/emu-udp-echo-client.cc} and @code{examples/emu-udp-echo-server.cc}
+in the repository @uref{http://code.nsnam.org/craigdo/ns-3-emu/}.
+
+@section Implementation
+
+Perhaps the most unusual part of the @code{Emu} and @code{Tap} device
+implementation relates to the requirement for executing some of the code
+with super-user permissions. Rather than force the user to execute the entire
+simulation as root, we provide a small ``creator'' program that runs as root
+and does any required high-permission sockets work.
-Like all ns-3 devices, the Emu Model provides a number of trace sources.
-These trace sources can be hooked using your own custom trace code, or you
-can use our helper functions to arrange for tracing to be enabled on devices
-you specify.
+We do a similar thing for both the @code{Emu} and the @code{Tap} devices.
+The high-level view is that the @code{CreateSocket} method creates a local
+interprocess (Unix) socket, forks, and executes the small creation program.
+The small program, which runs as suid root, creates a raw socket and sends
+back the raw socket file descriptor over the Unix socket that is passed to
+it as a parameter. The raw socket is passed as a control message (sometimes
+called ancillary data) of type SCM_RIGHTS.
-@subsection Upper-Level (MAC) Hooks
+The @code{Emu} net device uses the ns-3 threading and multithreaded real-time
+scheduler extensions. The interesting work in the @code{Emu} device is done
+when the net device is started (@code{EmuNetDevice::StartDevice ()}). An
+attribute (``Start'') provides a simulation time at which to spin up the
+net device. At this specified time (which defaults to t=0), the socket
+creation function is called and executes as described above. You may also
+specify a time at which to stop the device using the ``Stop'' attribute.
+
+Once the (promiscuous mode) socket is created, we bind it to an interface name
+also provided as an attribute (``DeviceName'') that is stored internally as
+@code{m_deviceName}:
-From the point of view of tracing in the net device, there are several
-interesting points to insert trace hooks. A convention inherited from other
-simulators is that packets destined for transmission onto attached networks
-pass through a single "transmit queue" in the net device. We provide trace
-hooks at this point in packet flow, which corresponds (abstractly) only to a
-transition from the network to data link layer, and call them collectively
-the device MAC hooks.
+@verbatim
+ struct ifreq ifr;
+ bzero (&ifr, sizeof(ifr));
+ strncpy ((char *)ifr.ifr_name, m_deviceName.c_str (), IFNAMSIZ);
+
+ int32_t rc = ioctl (m_sock, SIOCGIFINDEX, &ifr);
+
+ struct sockaddr_ll ll;
+ bzero (&ll, sizeof(ll));
+
+ ll.sll_family = AF_PACKET;
+ ll.sll_ifindex = m_sll_ifindex;
+ ll.sll_protocol = htons(ETH_P_ALL);
-When a packet is sent to the Emu net device for transmission it always
-passes through the transmit queue. The transmit queue in the
-EmuNetDevice inherits from Queue, and therefore inherits three
-trace sources:
+ rc = bind (m_sock, (struct sockaddr *)&ll, sizeof (ll));
+@end verbatim
+
+After the promiscuous raw socket is set up, a separate thread is spawned to do
+reads from that socket and the link state is set to @code{Up}.
+
+@verbatim
+ m_readThread = Create<SystemThread> (
+ MakeCallback (&EmuNetDevice::ReadThread, this));
+ m_readThread->Start ();
+
+ NotifyLinkUp ();
+@end verbatim
+
+The @code{EmuNetDevice::ReadThread} function basically just sits in an infinite
+loop reading from the promiscuous mode raw socket and scheduling packet
+receptions using the real-time simulator extensions.
-@itemize @bullet
-@item An Enqueue operation source (see Queue::m_traceEnqueue);
-@item A Dequeue operation source (see Queue::m_traceDequeue);
-@item A Drop operation source (see Queue::m_traceDrop).
-@end itemize
+@verbatim
+ for (;;)
+ {
+ ...
+
+ len = recvfrom (m_sock, buf, bufferSize, 0, (struct sockaddr *)&addr,
+ &addrSize);
+
+ ...
+
+ DynamicCast<RealtimeSimulatorImpl> (Simulator::GetImplementation ())->
+ ScheduleRealtimeNow (
+ MakeEvent (&EmuNetDevice::ForwardUp, this, buf, len));
+
+ ...
+ }
+@end verbatim
-The upper-level (MAC) trace hooks for the EmuNetDevice are, in fact,
-exactly these three trace sources on the single transmit queue of the device.
+The line starting with our templated DynamicCast function probably deserves a
+comment. It gains access to the simulator implementation object using
+the @code{Simulator::GetImplementation} method and then casts to the real-time
+simulator implementation to use the real-time schedule method
+@code{ScheduleRealtimeNow}. This function will cause a handler for the newly
+received packet to be scheduled for execution at the current real time clock
+value. This will, in turn cause the simulation clock to be advanced to that
+real time value when the scheduled event (@code{EmuNetDevice::ForwardUp}) is
+fired.
+
+The @code{ForwardUp} function operates as most other similar ns-3 net device
+methods do. The packet is first filtered based on the destination address. In
+the case of the @code{Emu} device, the MAC destination address will be the
+address of the @code{Emu} device and not the hardware address of the real
+device. Headers are then stripped off and the trace hooks are hit. Finally,
+the packet is passed up the ns-3 protocol stack using the receive callback
+function of the net device.
-The m_traceEnqueue event is triggered when a packet is placed on the transmit
-queue. This happens at the time that EmuNetDevice::Send or
-EmuNetDevice::SendFrom is called by a higher layer to queue a packet for
-transmission.
+Sending a packet is equally straightforward as shown below. The first thing
+we do is to add the ethernet header and trailer to the ns-3 @code{Packet} we
+are sending. The source address corresponds to the address of the @code{Emu}
+device and not the underlying native device MAC address. This is where the
+MAC address spoofing is done. The trailer is added and we enqueue and dequeue
+the packet from the net device queue to hit the trace hooks.
+
+@verbatim
+ header.SetSource (source);
+ header.SetDestination (destination);
+ header.SetLengthType (packet->GetSize ());
+ packet->AddHeader (header);
+
+ EthernetTrailer trailer;
+ trailer.CalcFcs (packet);
+ packet->AddTrailer (trailer);
-The m_traceDequeue event is triggered when a packet is removed from the
-transmit queue. Dequeues from the transmit queue happen immediately after
-the Enqueue event and just prior to the packet being sent to the underlying
-socket. This means that the transmit queue really only exists to fire on
-enqueue and dequeue operations so the Emu device behaves like other ns-3
-devices in this respect.
+ m_queue->Enqueue (packet);
+ packet = m_queue->Dequeue ();
+
+ struct sockaddr_ll ll;
+ bzero (&ll, sizeof (ll));
-@subsection Lower-Level (PHY) Hooks
+ ll.sll_family = AF_PACKET;
+ ll.sll_ifindex = m_sll_ifindex;
+ ll.sll_protocol = htons(ETH_P_ALL);
-There are no lower level trace hooks implemented in the Emu net device since
-we rely on the underlying OS implementation of the raw socket to perform
-the low level operations required to send and receive packets.
+ rc = sendto (m_sock, packet->PeekData (), packet->GetSize (), 0,
+ reinterpret_cast<struct sockaddr *> (&ll), sizeof (ll));
+@end verbatim
+
+
+Finally, we simply send the packet to the raw socket which puts it out on the
+real network.
+