Effectiveness of computer-based basic skills training on SA

Bolstad et al assessed the effectiveness of computer-based basic skills training on both situation awareness and flight skills performance. They did an experiment with a group of ab-initio students pilots (around 2003?) and published their results in an article in 20101.

They used students2 already holding a private pilot licence but with experience of lesser than 150 flight hours. These students received computer training on three things:

  • Checklist completion - a theoretical training of the importance of completing checklists fully, and techniques on how to do so under different flight conditions
  • ATC comprehension - a practical training of ATC speech comprehension, advancing from comprehension at normal speech rates to fast speach rates
  • Basic psychomotor skills - a practical training of hand-eye coordination, advancing from low difficulty levels to higher difficulty levels of hand-eye coordination

The performance of the students on above tasks, in particular, as well as their general performance on a flight simulator (which assessed changes in situation awareness and flight skills), were compared before and after receiving the training. A different group of students was used as a control group2. A number of some 36 (independent) variables were measured, with training acting as the research (dependent) variable of interest.

The main research hypothesis was that training would improve the specific skills being trained as well as overall situation awareness (measured with SAGAT3) and flight skills (measured in the flight simulator).

The research results are summarized in table 1. Training seems to improve (significantly) some of the particular skills being trained. It also appears to improve (significantly) some aspects of situation awareness.

However, these results need to be interpreted with care. On the one hand, only 2 out of 15 situation awareness variables were significant, which is not a great deal of improvement if situation awareness were, indeed, improving. On the other hand, the researchers opted for a one-tailed level of significance, and carried some 36 t-tests. We could expect almost two of those tests to be significant by chance alone if a probability of 0.05 was used. (We could further expect three of those tests to be significant by chance alone if a probability of 0.10 was used, and there is suspicions that the researchers may have been using the latter probability instead4. Given that situation awareness has not been trained directly, the improvement in such awareness may be simply incorrect, that is, a chance result. In this case, is safer to assume that the training of particular skills seems to partially improve those skills, but does not seem to have a clear effect neither on situation awareness nor on flight skills performance (in a simulator).

Table 1
Effect of training on particular skills
skill significant improvement?
Checklist completion (knowledge) yes
Reaction time to ATC communications no
Correct responses to ATC no
Psychomotor skills difficulty yes
Reduction of psychomotor errors no
Effect of training on flight skills and SA
skill significant improvement?
Flight skills performance (overall) no
SAGAT: awareness of current heading no
SAGAT: awareness of current altitude yes
SAGAT: awareness of current vs planned altitude no
SAGAT: awareness of current airspeed no
SAGAT: awareness of current attitude no
SAGAT: awareness of current winds yes
SAGAT: awareness of current clearance no
SAGAT: awareness of current vs planned time no
SAGAT: awareness of current fuel no
SAGAT: awareness of special airspace no
SAGAT: awareness of current obstacles no
SAGAT: awareness of conflict with restricted airspace no
SAGAT: awareness of current altimeter setting no
SAGAT: awareness of current vs planned groundspeed no
SAGAT: awareness of current ATC organization no
References
1. BOLSTAD Cheryl A, Mica R ENDSLEY, Anthony M COSTELLO & Cass HOWELL (2008). Evaluation of computer-based situation awareness training for general aviation pilots. The International Journal of Aviation Psychology, 2010, number 20, volume 3, pages 269-294. ISSN 1050-8414.
+++ Footnotes +++
2. Sample size of the experimental group = 9 students; sample size of the control group = 11 students.
3. SAGAT here stands for the GA aircraft version of the Situation Awareness Global Assessement Technique, a 15-item questionnaire used to assess the degree of situational awareness that a pilot has over 15 flight parameters at different times during a flight.
4. See, for example, footnotes 4 and 5 in this document.

Want to know more?

BOLSTAD Cheryl A, Mica R ENDSLEY, Anthony M COSTELLO & Cass HOWELL (2010). Evaluation of computer-based situation awareness training for general aviation pilots. The International Journal of Aviation Psychology, 2010, number 20, volume 3, pages 269-294. ISSN 1050-8414.
This is the original article, with above research described as experiment 1. (People with online access to the publisher can find the original article here.)
AviationKnowledge - Situational awareness
You can find information on situation awareness in this page.
AviationKnowledge - Effectiveness of computer-based training on SA
There are two other experiments releated to the one described here: the effectiveness of computer-based attention sharing training on SA and the effectiveness of computer-based preflight planning training on SA.

Editor

Jose D PEREZGONZALEZ (2011). School of Aviation, Massey University, New Zealand (JDPerezgonzalezJDPerezgonzalez).


Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License