.. include:: replace.txt

+++++++++++++++++++++

Testing Documentation

+++++++++++++++++++++

Overview

********

To test and validate the ns-3 LTE module, several test suites are provided which are integrated with the ns-3 test framework.

To run them, you need to have configured the build of the simulator in this way::

    ./waf configure --enable-tests --enable-modules=lte --enable-examples

    ./test.py

The above will run not only the test suites belonging to the LTE module, but also those belonging to all the other ns-3 modules on which the LTE module depends. See the ns-3 manual for generic information on the testing framework.

You can get a more detailed report in HTML format in this way::

    ./test.py -w results.html

After the above command has run, you can view the detailed result for each test by opening the file ``results.html`` with a web browser. 

You can run each test suite separately using this command::

    ./test.py -s test-suite-name

For more details about ``test.py`` and the ns-3 testing framework, please refer to the ns-3 manual.

Description of the test suites

******************************

Unit Tests

~~~~~~~~~~

SINR calculation in the Downlink

--------------------------------

The test suite ``lte-downlink-sinr`` 

checks that the SINR calculation in

downlink is performed correctly. The SINR in the downlink is calculated for each

RB assigned to data transmissions by dividing the power of the

intended signal from the considered eNB by the sum of the noise power plus all

the transmissions on the same RB coming from other eNBs (the interference

signals):

.. math::

  \gamma = \frac{ P_\mathrm{signal} }{ P_\mathrm{noise} + \sum P_\mathrm{interference} }

In general, different signals can be active during different periods

of time. We define a *chunk* as the time interval between any two

events of type either start or end of a waveform. In other words, a

chunk identifies a time interval during which the set of active

waveforms does not change. Let :math:`i` be the generic chunk,

:math:`T_i` its duration and :math:`\mathrm{SINR_i}` its SINR,

calculated with the above equation. The calculation of the average

SINR :math:`\overline{\gamma}` to be used for CQI feedback reporting

uses the following formula:  

.. math::

  \overline{\gamma} = \frac{ \sum_i {\gamma}_i  T_i }{ \sum_i T_{i} }

The test suite checks that the above calculation is performed

correctly in the simulator. The test vectors are obtained offline by

an Octave script that implements the above equation, and that

recreates a number of random transmitted signals and interference

signals that mimic a scenario where an UE is trying to decode a signal

from an eNB while facing interference from other eNBs. The test passes

if the calculated values are equal to the test vector within a

tolerance of :math:`10^{-7}`. The tolerance is meant to account for

the approximation errors typical of floating point arithmetic. 

SINR calculation in the Uplink

------------------------------

The test suite ``lte-uplink-sinr`` checks that the SINR calculation in

uplink is performed correctly. This test suite is identical to

``lte-downlink-sinr`` described in the previous section, with the

difference than both the signal and the interference now refer to

transmissions by the UEs, and reception is performed by the eNB. 

This test suite recreates a number of random transmitted signals and

interference signals to mimic a scenario where an eNB is trying to

decode the signal from several UEs simultaneously (the ones in the

cell of the eNB) while facing interference from other UEs (the ones

belonging to other cells).  

The test vectors are obtained by a dedicated Octave script. The test

passes if the calculated values are equal to the test vector within a

tolerance of :math:`10^{-7}` which, as for the downlink SINR test,

deals with floating point arithmetic approximation issues. 

E-UTRA Absolute Radio Frequency Channel Number (EARFCN)

-------------------------------------------------------

The test suite ``lte-earfcn`` checks that the carrier frequency used

by the LteSpectrumValueHelper class (which implements the LTE spectrum

model) is done in compliance with [TS36101]_, where the E-UTRA

Absolute Radio Frequency Channel Number (EARFCN) is defined. The test

vector for this test suite comprises a set of EARFCN values and the

corresponding carrier frequency calculated by hand following the

specification of [TS36101]_. The test passes if the carrier frequency

returned by LteSpectrumValueHelper is the same as the known value for

each element in the test vector.

System Tests

~~~~~~~~~~~~

.. _sec-lte-amc-tests:

Adaptive Modulation and Coding Tests

------------------------------------

The test suite ``lte-link-adaptation`` provides system tests recreating a

scenario with a single eNB and a single UE. Different test cases are created

corresponding to different SNR values perceived by the UE. The aim of the test

is to check that in each test case the chosen MCS corresponds to some known

reference values. These reference values are obtained by

re-implementing in Octave (see `src/lte/test/reference/lte_amc.m`) the

model described in Section :ref:`sec-lte-amc` for the calculation of the

spectral efficiency, and determining the corresponding MCS index

by manually looking up the tables in [R1-081483]_. The resulting test vector is

represented in Figure :ref:`fig-lte-mcs-index`.

The MCS which is used by the simulator is measured by

obtaining the tracing output produced by the scheduler after 4ms (this

is needed to account for the initial delay in CQI reporting). The SINR

which is calcualted by the simulator is also obtained using the

``LteSinrChunkProcessor`` interface. The test

passes if both the following conditions are satisfied:

 #. the SINR calculated by the simulator correspond to the SNR

    of the test vector within an absolute tolerance of :math:`10^{-7}`;

 #. the MCS index used by the simulator exactly corresponds to

    the one in the test vector.

.. _fig-lte-mcs-index:

.. figure:: figures/lte-mcs-index.*

   :align: center

   Test vector for Adaptive Modulation and Coding

Inter-cell Interference Tests

-----------------------------

The test suite `lte-interference` provides system tests recreating an

inter-cell interference scenario with two eNBs, each having a single

UE attached to it and employing Adaptive Modulation and Coding both in

the downlink and in the uplink. The topology of the scenario

is depicted in Figure :ref:`fig-lte-interference-test-scenario`. The

:math:`d_1` parameter represents the distance of each UE to the eNB it

is attached to, whereas the :math:`d_2` parameter represent the

interferer distance. We note that the scenario topology is such that

the interferer distance is the same for uplink and downlink; still,

the actual interference power perceived will be different, because of

the different propagation loss in the uplink and downlink

bands. Different test cases are obtained by varying the :math:`d_1`

and :math:`d_2` parameters.

.. _fig-lte-interference-test-scenario:

.. figure:: figures/lte-interference-test-scenario.*

   :align: center

   Topology for the inter-cell interference test

The test vectors are obtained by use of a dedicated octave script

(available in

`src/lte/test/reference/lte_link_budget_interference.m`), which does

the link budget calculations (including interference) corresponding to the topology of each

test case, and outputs the resulting SINR and spectral efficiency. The

latter is then used to determine (using the same procedure adopted for 

:ref:`sec-lte-amc-tests`. We note that the test vector

contains separate values for uplink and downlink.

UE Measurements Tests

---------------------

The test suite `lte-ue-measurements` provides system tests recreating an

inter-cell interference scenario identical of the one defined for

`lte-interference` test-suite. However, in this test the quantities to be

tested are represented by RSRP and RSRQ measurements performed by the UE in two

different points of the stack: the source, which is UE PHY layer, and the

destination, that is the eNB RRC.

The test vectors are obtained by the use of a dedicated octave script (available

in `src/lte/test/reference/lte-ue-measurements.m`), which does the link budget

calculations (including interference) corresponding to the topology of each

test case, and outputs the resulting RSRP and RSRQ. The obtained values are then

used for checking the correctness of the UE Measurements at PHY layer. After

that, they have to be converted according to 3GPP formatting for the purpose of

checking their correctness at eNB RRC level.

UE measurement configuration tests

----------------------------------

Besides the previously mentioned test suite, there are 3 other test suites for

testing UE measurements: `lte-ue-measurements-piecewise-1`,

`lte-ue-measurements-piecewise-2`, and `lte-ue-measurements-handover`. These

test suites are more focused on the reporting trigger procedure, i.e. the

correctness of the implementation of the event-based triggering criteria is

verified here.

In more specific, the tests verify the *timing* and the *content* of each

measurement reports received by eNodeB. Each test case is an stand-alone LTE

simulation and the test case will pass if measurement report(s) only occurs at

the prescribed time and shows the correct level of RSRP (RSRQ is not verified at

the moment).

Piecewise configuration

#######################

The piecewise configuration aims to test a particular UE measurements

configuration. The simulation script will setup the corresponding measurements

configuration to the UE, which will be active throughout the simulation.

Since the reference values are precalculated by hands, several assumptions are

made to simplify the simulation. Firstly, the channel is only affected by path

loss model (in this case, Friis model is used). Secondly, the ideal RRC protocol

is used, and layer 3 filtering is disabled. Finally, the UE moves in a

predefined motion pattern between 4 distinct spots, as depicted in Figure

:ref:`fig-ue-meas-piecewise-motion` below. Therefore the fluctuation of the

measured RSRP can be determined more easily.

.. _fig-ue-meas-piecewise-motion:

.. figure:: figures/ue-meas-piecewise-motion.*

   :scale: 80 %

   :align: center

   UE movement trace throughout the simulation in piecewise configuration

The motivation behind the *"teleport"* between the predefined spots is to

introduce drastic change of RSRP level, which will guarantee the triggering of

entering or leaving condition of the tested event. By performing drastic

changes, the test can be run within shorter amount of time.

Figure :ref:`fig-ue-meas-piecewise-a1` below shows the measured RSRP after

layer 1 filtering by the PHY layer during the simulation with a piecewise

configuration. Because layer 3 filtering is disabled, these are the exact values

used by the UE RRC instance to evaluate reporting trigger procedure. Notice that

the values are refreshed every 200 ms, which is the default filtering period of

PHY layer measurements report. The figure also shows the time when entering and

leaving conditions of an example instance of Event A1 (serving cell becomes

better than threshold) occur during the simulation.

.. _fig-ue-meas-piecewise-a1:

.. figure:: figures/ue-meas-piecewise-a1.*

   :scale: 80 %

   :align: center

   Measured RSRP trace of an example Event A1 test case in piecewise

   configuration

Each reporting criterion is tested several times with different threshold/offset

parameters. Some test scenarios also take hysteresis and time-to-trigger into

account. Figure :ref:`fig-ue-meas-piecewise-a1-hys` depicts the effect of

hysteresis in another example of Event A1 test.

.. _fig-ue-meas-piecewise-a1-hys:

.. figure:: figures/ue-meas-piecewise-a1-hys.*

   :scale: 80 %

   :align: center

   Measured RSRP trace of an example Event A1 with hysteresis test case in

   piecewise configuration

Piecewise configuration is used in two test suites of UE measurements. The first

one is `lte-ue-measurements-piecewise-1`, henceforth Piecewise test #1, which

simulates 1 UE and 1 eNodeB. The other one is `lte-ue-measurements-piecewise-2`,

which has 1 UE and 2 eNodeBs in the simulation.

Piecewise test #1 is intended to test the event-based criteria which are not

dependent on the existence of a neighbouring cell. These criteria include Event

A1 and A2. The other events are also briefly tested to verify that they are

still working correctly (albeit not reporting anything) in the absence of any

neighbouring cell. Table :ref:`tab-ue-meas-piecewise-1` below lists the

scenarios tested in piecewise test #1. 

.. _tab-ue-meas-piecewise-1:

.. table:: UE measurements test scenarios using piecewise configuration #1

   ====== ================== ================ ========== ===============

   Test # Reporting Criteria Threshold/Offset Hysteresis Time-to-Trigger

   ====== ================== ================ ========== ===============

   1      Event A1           Low              No         No

   2      Event A1           Normal           No         No

   3      Event A1           Normal           No         Short

   4      Event A1           Normal           No         Long

   5      Event A1           Normal           No         Super

   6      Event A1           Normal           Yes        No

   7      Event A1           High             No         No

   8      Event A2           Low              No         No

   9      Event A2           Normal           No         No

   10     Event A2           Normal           No         Short

   11     Event A2           Normal           No         Long

   12     Event A2           Normal           No         Super

   13     Event A2           Normal           Yes        No

   14     Event A2           High             No         No

   15     Event A3           Zero             No         No

   16     Event A4           Normal           No         No

   17     Event A5           Normal-Normal    No         No

   ====== ================== ================ ========== ===============

Other events such as Event A3, A4, and A5 depend on measurements of neighbouring

cell, so they are more thoroughly tested in Piecewise test #2. The simulation

places the nodes on a straight line and instruct the UE to *"jump"* in a similar

manner as in Piecewise test #1. Handover is disabled in the simulation, so the

role of serving and neighbouring cells do not switch during the simulation.

Table :ref:`tab-ue-meas-piecewise-2` below lists the scenarios tested in

Piecewise test #2.

.. _tab-ue-meas-piecewise-2:

.. table:: UE measurements test scenarios using piecewise configuration #2

   ====== ================== ================ ========== ===============

   Test # Reporting Criteria Threshold/Offset Hysteresis Time-to-Trigger

   ====== ================== ================ ========== ===============

   1      Event A1           Low              No         No

   2      Event A1           Normal           No         No

   3      Event A1           Normal           Yes        No

   4      Event A1           High             No         No

   5      Event A2           Low              No         No

   6      Event A2           Normal           No         No

   7      Event A2           Normal           Yes        No

   8      Event A2           High             No         No

   9      Event A3           Positive         No         No

   10     Event A3           Zero             No         No

   11     Event A3           Zero             No         Short

   12     Event A3           Zero             No         Super

   13     Event A3           Zero             Yes        No

   14     Event A3           Negative         No         No

   15     Event A4           Low              No         No

   16     Event A4           Normal           No         No

   17     Event A4           Normal           No         Short

   18     Event A4           Normal           No         Super

   19     Event A4           Normal           Yes        No

   20     Event A4           High             No         No

   21     Event A5           Low-Low          No         No

   22     Event A5           Low-Normal       No         No

   23     Event A5           Low-High         No         No

   24     Event A5           Normal-Low       No         No

   25     Event A5           Normal-Normal    No         No

   26     Event A5           Normal-Normal    No         Short

   27     Event A5           Normal-Normal    No         Super

   28     Event A5           Normal-Normal    Yes        No

   29     Event A5           Normal-High      No         No

   30     Event A5           High-Low         No         No

   31     Event A5           High-Normal      No         No

   32     Event A5           High-High        No         No

   ====== ================== ================ ========== ===============

One note about the tests with time-to-trigger, they are tested using 3 different

values of time-to-trigger: *short* (shorter than report interval), *long*

(shorter than the filter measurement period of 200 ms), and *super* (longer than

200 ms). The first two ensure that time-to-trigger evaluation always use the

latest measurement reports received from PHY layer. While the last one is

responsible for verifying time-to-trigger cancellation, for example when a

measurement report from PHY shows that the entering/leaving condition is no

longer true before the first trigger is fired.

Handover configuration

######################

The purpose of the handover configuration is to verify whether UE measurement

configuration is updated properly after a succesful handover takes place. For

this purpose, the simulation will construct 2 eNodeBs with different UE

measurement configuration, and the UE will perform handover from one cell to

another. The UE will be located on a straight line between the 2 eNodeBs, and

the handover will be invoked manually. The duration of each simulation is

2 seconds (except the last test case) and the handover is triggered exactly at

halfway of simulation.

The `lte-ue-measurements-handover` test suite covers various types of

configuration differences. The first one is the difference in report interval,

e.g. the first eNodeB is configured with 480 ms report interval, while the

second eNodeB is configured with 240 ms report interval. Therefore, when the UE

performed handover to the second cell, the new report interval must take effect.

As in piecewise configuration, the timing and the content of each measurement

report received by the eNodeB will be verified.

Other types of differences covered by the test suite are differences in event

and differences in threshold/offset. Table :ref:`tab-ue-meas-handover` below

lists the tested scenarios. 

.. _tab-ue-meas-handover:

.. table:: UE measurements test scenarios using handover configuration

   ====== ================ =========================== ===========================

   Test # Test Subject     Initial Configuration       Post-Handover Configuration

   ====== ================ =========================== ===========================

   1      Report interval  480 ms                      240 ms

   2      Report interval  120 ms                      640 ms

   3      Event            Event A1                    Event A2

   4      Event            Event A2                    Event A1

   5      Event            Event A3                    Event A4

   6      Event            Event A4                    Event A3

   7      Event            Event A2                    Event A3

   8      Event            Event A3                    Event A2

   9      Event            Event A4                    Event A5

   10     Event            Event A5                    Event A4

   11     Threshold/offset RSRP range 52 (Event A1)    RSRP range 56 (Event A1)

   12     Threshold/offset RSRP range 52 (Event A2)    RSRP range 56 (Event A2)

   13     Threshold/offset A3 offset -30 (Event A3)    A3 offset +30 (Event A3)

   14     Threshold/offset RSRP range 52 (Event A4)    RSRP range 56 (Event A4)

   15     Threshold/offset RSRP range 52-52 (Event A5) RSRP range 56-56 (Event A5)

   16     Time-to-trigger  1024 ms                     100 ms

   17     Time-to-trigger  1024 ms                     640 ms

   ====== ================ =========================== ===========================

Round Robin scheduler performance

---------------------------------

The test suite ``lte-rr-ff-mac-scheduler`` creates different test cases with

a single eNB and several UEs, all having the same Radio Bearer specification. In

each test case, the UEs see the same SINR from the eNB; different test cases are

implemented by using different distance among UEs and the eNB (i.e., therefore

having different SINR values) and different numbers of UEs. The test consists on

checking that the obtained throughput performance is equal among users and

matches a reference throughput value obtained according to the SINR perceived

within a given tolerance. 

The test vector is obtained according to the values of transport block

size reported in table 7.1.7.2.1-1 of [TS36213]_, considering an

equal distribution of the physical resource block among the users

using Resource Allocation Type 0 as defined in Section 7.1.6.1 of

[TS36213]_.  Let :math:`\tau` be the TTI duration, :math:`N` be the

number of UEs, :math:`B` the transmission bandwidth configuration in

number of RBs, :math:`G` the RBG size, :math:`M` the modulation and

coding scheme in use at the given SINR and :math:`S(M, B)` be the

transport block size in bits as defined by 3GPP TS 36.213. We first

calculate the number :math:`L` of RBGs allocated to each user as 

.. math::

   L = \left\lfloor \frac{B}{NG} \right\rfloor 

The reference throughput :math:`T` in bit/s achieved by each UE is then calculated as

.. math::

   T =  \frac{S(M, L G)}{8 \; \tau}

The test passes if the measured throughput matches with the reference throughput

within a relative tolerance of 0.1. This tolerance is needed to account for the

transient behavior at the beginning of the simulation (e.g., CQI feedback is

only available after a few subframes) as well as for the accuracy of the

estimator of the average throughput performance over the chosen simulation time

(0.4s). This choice of the simulation time is justified by the need to

follow the ns-3 guidelines of keeping the total execution time of the test

suite low, in spite of the high number of test cases. In any case, we note that

a lower value of the tolerance can be used when longer simulations are

run. 

In Figure `fig-lenaThrTestCase1`_, the curves labeled "RR" represent the test values

calculated for the RR scheduler test, as a function of the number of UEs and of

the MCS index being used in each test case. 

.. _fig-lenaThrTestCase1: