Bug 1789 - missing test condition for sigma in buildings-shadowing-test
authorNicola Baldo <nbaldo@cttc.es>
Thu, 12 Dec 2013 18:50:12 +0100
changeset 10510 34299825e664
parent 10509 e03b086478eb
child 10511 3df0e5c2eed5
Bug 1789 - missing test condition for sigma in buildings-shadowing-test
RELEASE_NOTES
src/buildings/doc/source/buildings-design.rst
src/buildings/doc/source/buildings-testing.rst
src/buildings/test/buildings-shadowing-test.cc
--- a/RELEASE_NOTES	Thu Dec 12 17:00:09 2013 +0100
+++ b/RELEASE_NOTES	Thu Dec 12 18:50:12 2013 +0100
@@ -46,6 +46,7 @@
 - Bug 1778 - Implement TapBridge::IsLinkUp() function
 - Bug 1777 - Implement the more direct way of "using" configuration of existing tap interface
 - Bug 1776 - Improve CRC performance for CsmaNetDevice in emulation modes
+- Bug 1789 - missing test condition for sigma in buildings-shadowing-test
 - Bug 1798 - Changing the rate of onOffApplication might stop transmission
 - Bug 1802 - FlowMon header deserialization problem with IPv4 fragments
 - Bug 1807 - Multiple bugs in Ipv4L3Protocol::LocalDeliver
--- a/src/buildings/doc/source/buildings-design.rst	Thu Dec 12 17:00:09 2013 +0100
+++ b/src/buildings/doc/source/buildings-design.rst	Thu Dec 12 18:50:12 2013 +0100
@@ -145,7 +145,9 @@
 Shadowing Model
 ---------------
 
-The shadowing is modeled according to a log-normal distribution with variable standard deviation as function of the connection characteristics. In the implementation we considered three main possible scenarios which correspond to three standard deviations (i.e., the mean is always 0), in detail:
+The shadowing is modeled according to a log-normal distribution with variable standard deviation as function of the relative position (indoor or outdoor) of the MobilityModel instances involved. One random value is drawn for each pair of MobilityModels, and stays constant for that pair during the whole simulation. Thus, the model is appropriate for static nodes only. 
+
+The model considers that the mean of the shadowing loss in dB is always 0. For the variance, the model considers three possible values of standard deviation, in detail:
 
  * outdoor (``m_shadowingSigmaOutdoor``, defaul value of 7 dB) :math:`\rightarrow X_\mathrm{O} \sim N(\mu_\mathrm{O}, \sigma_\mathrm{O}^2)`.
  * indoor (``m_shadowingSigmaIndoor``, defaul value of 10 dB) :math:`\rightarrow X_\mathrm{I} \sim N(\mu_\mathrm{I}, \sigma_\mathrm{I}^2)`.
--- a/src/buildings/doc/source/buildings-testing.rst	Thu Dec 12 17:00:09 2013 +0100
+++ b/src/buildings/doc/source/buildings-testing.rst	Thu Dec 12 18:50:12 2013 +0100
@@ -107,6 +107,6 @@
 Buildings Shadowing Test
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-The test suite ``buildings-shadowing-test`` is a unit test intended to verify the statistics distribution characteristics of the shadowing are the one expected. The shadowing is modeled according to a normal distribution with mean :math:`\mu = 0` and variable standard deviation :math:`\sigma`, according to models commonly used in literature.
-The test generates 10,000 samples of shadowing by subtracting the deterministic component from the total loss returned by the ``BuildingPathlossModel``. The mean and variance of the shadowing samples are then used to verify whether the 99% confidence interval is respected by the sequence generated by the simulator.
+The test suite ``buildings-shadowing-test`` is a unit test intended to verify the statistical distribution of the shadowing model implemented by ``BuildingsPathlossModel``. The shadowing is modeled according to a normal distribution with mean :math:`\mu = 0` and variable standard deviation :math:`\sigma`, according to models commonly used in literature. Three test cases are provided, which cover the cases of indoor, outdoor and indoor-to-outdoor communications. 
+Each test case generates 1000 different samples of shadowing for different pairs of MobilityModel instances in a given scenario. Shadowing values are obtained by subtracting from the total loss value returned by ``HybridBuildingsPathlossModel`` the path loss component which is constant and pre-determined for each test case. The test verifies that the sample mean and sample variance of the shadowing values fall within the 99% confidence interval of the sample mean and sample variance. The test also verifies that the shadowing values returned at successive times for the same pair of MobilityModel instances is constant.
 
--- a/src/buildings/test/buildings-shadowing-test.cc	Thu Dec 12 17:00:09 2013 +0100
+++ b/src/buildings/test/buildings-shadowing-test.cc	Thu Dec 12 18:50:12 2013 +0100
@@ -100,28 +100,43 @@
   building1->SetExtWallsType (Building::ConcreteWithWindows);
   building1->SetNFloors (3);
   
-  Ptr<MobilityModel> mma = CreateMobilityModel (m_mobilityModelIndex1);
-  Ptr<MobilityModel> mmb = CreateMobilityModel (m_mobilityModelIndex2);
-
+  Ptr<HybridBuildingsPropagationLossModel> propagationLossModel = CreateObject<HybridBuildingsPropagationLossModel> ();
+  
   std::vector<double> loss;
   double sum = 0.0;
   double sumSquared = 0.0;
-  int samples = 10000;
+  int samples = 1000;
   for (int i = 0; i < samples; i++)
     {
-      Ptr<HybridBuildingsPropagationLossModel> propagationLossModel = CreateObject<HybridBuildingsPropagationLossModel> ();
-      loss.push_back (propagationLossModel->DoCalcRxPower (0.0, mma, mmb) + m_lossRef);
-      sum += loss.at (loss.size () - 1);
-      sumSquared += (loss.at (loss.size () - 1) * loss.at (loss.size () - 1));
+      Ptr<MobilityModel> mma = CreateMobilityModel (m_mobilityModelIndex1);
+      Ptr<MobilityModel> mmb = CreateMobilityModel (m_mobilityModelIndex2);
+      double shadowingLoss = propagationLossModel->DoCalcRxPower (0.0, mma, mmb) + m_lossRef;
+      double shadowingLoss2 = propagationLossModel->DoCalcRxPower (0.0, mma, mmb) + m_lossRef;
+      NS_TEST_ASSERT_MSG_EQ_TOL (shadowingLoss, shadowingLoss2, 0.001, 
+                                 "Shadowing is not constant for the same mobility model pair!");
+      loss.push_back (shadowingLoss);
+      sum += shadowingLoss;
+      sumSquared += (shadowingLoss * shadowingLoss);
     }
-  double mean = sum / samples;
-  double sigma = std::sqrt (sumSquared / samples - (mean * mean));
-  // test whether the distribution falls in the 99% confidence interval, as expected with a nornal distribution
-  double ci = (2.575829303549 * sigma) / std::sqrt (samples);
+  double sampleMean = sum / samples;
+  double sampleVariance = (sumSquared - (sum * sum / samples)) / (samples - 1);
+  double sampleStd = std::sqrt (sampleVariance);
+
+  // test whether the sample mean falls in the 99% confidence interval
+  const double zn995 = 2.575829303549; // 99.5 quantile of the normal distribution
+  double ci = (zn995 * sampleStd) / std::sqrt (samples);
+  NS_LOG_INFO ("SampleMean from simulation " << sampleMean << ", sampleStd " << sampleStd << ", reference value " << m_sigmaRef << ", CI(99%) " << ci);
+  NS_TEST_ASSERT_MSG_EQ_TOL (std::fabs (sampleMean), 0.0, ci, "Wrong shadowing distribution !");
 
-  NS_LOG_INFO ("Mean from simulation " << mean << ", sigma " << sigma << ", reference value " << m_sigmaRef << ", CI(99%) " << ci);
+  // test whether the sample variance falls in the 99% confidence interval
+  // since the shadowing is gaussian, its sample variance follows the 
+  // chi2 distribution with samples-1 degrees of freedom
+  double chi2 = (samples - 1) *  sampleVariance / (m_sigmaRef*m_sigmaRef);
+  const double zchi2_005 = 887.621135217515;  //  0.5% quantile of the chi2 distribution 
+  const double zchi2_995 = 1117.89045267865;  // 99.5% quantile of the chi2 distribution
+  NS_TEST_ASSERT_MSG_GT (chi2, zchi2_005, "sample variance lesser than expected");
+  NS_TEST_ASSERT_MSG_LT (chi2, zchi2_995, "sample variance greater than expected");
 
-  NS_TEST_ASSERT_MSG_EQ_TOL (std::fabs (mean), 0.0, ci, "Wrong shadowing distribution !");
   Simulator::Destroy ();
 }