Effectiveness of computer-based preflight planning training on SA

Bolstad et al assessed the effectiveness of computer-based training of contingency planning and preflight planning on situation awareness and flight skills. They did an experiment with general aviation (GA) pilots (around 2003?) and published it in an article in 20101.

They used GA pilots already holding a private pilot's licence and with experience averaging 270 flight hours2. These pilots received theoretical computer training on contigency planning and preflight planning (the latter also including practical exercises).

The performance of the pilots on a 7-item flight plan knowledge questionnaire, as well as their performance on a flight simulator, were compared before and after receiving the training. A different group of pilots was used as control group2. Some 25 (independent) variables were measured, with training acting as the research (dependent) variable of interest. The main research hypothesis was that training would improve the specific skills being trained as well as overall situation awareness (measured with SAGAT3) and flight skills (measured in the flight simulator).

The research results are summarized in table 1. Training seems to improve (significantly) some preflight knowledge. It also appears to improve (significantly) some aspects of situation awareness.

However, these results need to be interpreted with care. On the one hand, only 2 out of 7 knowledge items and 3 out of 15 situation awareness variables were significant, which is not a great deal of improvement if basic skills and situation awareness were, indeed, improving. On the other hand, the researchers opted for a one-tailed level of significance with a probability of 0.05, and carried some 25 t-tests. We could expect at least one of those tests to be significant by chance alone if a probability of 0.05 was used. Yet, the evidence suggests that the researchers were, indeed, using a probability of 0.10 (or two-tailed tests with a probability of 0.05 for each tail), in which case we could expect up to two of those tests to be significant by chance alone. (The table shows results which are coherent with one-tailed tests and a significance level of 0.05).

Given that situation awareness has not been trained directly, the improvement in awareness may be simply incorrect, that is, a chance result. In this case, is safer to assume that the training of contingency planning and preflight planning seems to partially improve preflight planning but not actually much. Furthermore, it does not seem to have a clear effect neither on situation awareness nor on flight skills performance (in a simulator).

Table 1
Effect of training on flight plan knowledge questionnaire
item significant improvement?
Glide speed yes
Engine failure yes
Undisclosed item (3/7) no
Undisclosed item (4/7) no
Undisclosed item (5/7) no
Undisclosed item (6/7) no
Undisclosed item (7/7) no
Effect of training on flight skills and SA
skill significant improvement?
Flight skills performance (overall) no
SAGAT: awareness of current heading no
SAGAT: awareness of current altitude no
SAGAT: awareness of current vs planned altitude no
SAGAT: awareness of current airspeed no
SAGAT: awareness of current vs planned heading no
SAGAT: awareness of current winds no
SAGAT: awareness of aircraft level no
SAGAT: awareness of current airspace no
SAGAT: awareness of current course yes
SAGAT: awareness of special use airspace yes
SAGAT: awareness of current distance to destination no
SAGAT: awareness of current nearest airport no
SAGAT: awareness of current altimeter setting no
SAGAT: awareness of current system degrades no4
SAGAT: awareness of current ATC organization no
References
1. BOLSTAD Cheryl A, Mica R ENDSLEY, Anthony M COSTELLO & Cass HOWELL (2008). Evaluation of computer-based situation awareness training for general aviation pilots. The International Journal of Aviation Psychology, 2010, number 20, volume 3, pages 269-294. ISSN 1050-8414.
+++ Footnotes +++
2. Effective sample size of the experimental group = 11 pilots; effective sample size of the control group = 10 pilots.
3. SAGAT here stands for the GA aircraft version of the Situation Awareness Global Assessement Technique, used to assess the degree of situational awareness that a pilot has over several flight parameters at different times during a flight. The researchers substituted 8 of the original 15 items in the questionnaire for other items more appropriate to the research.
4. The authors report that the control group performed "significanlty" better than the experimental group for this item. However, the interpretation is incoherent with the one-tailed statistical significance approach opted for. That is, in principle, they can only support significant results whereby the experimental group achieves better results than the control group, not viceversa. In order to support the researchers' conclusions, we have to assume two-tailed tests with a significance level of 0.10, which do not seem to be the intended significance level.

Want to know more?

BOLSTAD Cheryl A, Mica R ENDSLEY, Anthony M COSTELLO & Cass HOWELL (2010). Evaluation of computer-based situation awareness training for general aviation pilots. The International Journal of Aviation Psychology, 2010, number 20, volume 3, pages 269-294. ISSN 1050-8414.
This is the original article, with above research described as experiment 2. (People with online access to the publisher can find the original article here.)
AviationKnowledge - Situational awareness
You can find information on situation awareness in this page.
AviationKnowledge - Effectiveness of computer-based training on SA
There are two other experiments releated to the one described here: the effectiveness of computer-based attention sharing training on SA and the effectiveness of computer-based basic skills training on SA.

Editor

Jose D PEREZGONZALEZ (2011). School of Aviation, Massey University, New Zealand (JDPerezgonzalezJDPerezgonzalez).


Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License