Towards Precise Performance Measurements in Live Air Traffic Validation Exercises

Day-to-day air traffic is exposed to a large number of influencing factors, which are steadily changing and causing significant volatility. Examples would be the daily weather situation, charter flights, flight cancellations, changes in airspace configuration, or the temporary non-availability of supporting infrastructure such as navigation aids, runways, or taxiways. When conducting validation activities with live air traffic, as it is currently done in SESARs xStream project, this circumstance complicates meaningful performance measurements. Especially when the volatility of the air traffic under consideration is within the same magnitude than the expected benefit of the system under test, the positive effects may be invisible similar to a poor signal-to-noise ratio. In order to statistically determine a small effect, one must either evaluate a very large data basis or, if this is not possible, find and eliminate disturbances before the actual evaluation. Within the xStream project, a methodology has been developed to do this elimination in a reasonable and systematic way and filter out those datasets that, because of a totally different weather situation, traffic mix etc., make a meaningful baseline-solution comparison impossible. This is done by performing multiple systematic pair-wise comparability checks between dedicated reference and exercise data sets. Beside others, these checks cover local and area weather conditions, airspace and air traffic control service configurations as well as the current traffic constellation. Based on the methodology, algorithms were developed by the Institute of Flight Guidance of the German Aerospace Center (DLR) in Braunschweig and have been implemented in several in-house software modules, allowing the conduction of these checks in an almost fully automatic way. This software has then been tested with several different air traffic recordings from a European hub airport by calculating selected performance metrics. Obtained results were then compared with performance measurements without previous comparability checks. The results show that the developed methodology was able to compensate approximately 60% of the air traffic volatility of the used radar tracks when reducing the sample size to one quarter containing the best comparable datasets. This paper provides comprehensive information about the developed comparability checks itself and describes the observed effects, compared to unfiltered performance measurements. It closes with a discussion and an outlook with special regard to the upcoming application of this methodology in the xStream project.