CRM Training and Teamwork Skills in the Cockpit

Table of Contents

Crew resource management (CRM) is commonly utilised within the aviation industry as an approach to augment teamwork skills. However, few studies have shown that the use of CRM methods do indeed lead to the development and use of these skills. Salas, Fowlkes, Stout, Milanovich & Prince (1999) conducted two evaluation studies on the effects of CRM training on naval aviators using a multiple assessment approach. Two studies were conducted due to identified extraneous variables, and the unavailability of a high fidelity simulator during Study One. This article provides an overview and analysis of the original findings.

Assessment Approaches

APPROACH 1: CREW REACTIONS

Illustration 1 shows how the participants of both studies viewed the usefulness of CRM training in the aviation environment, after having received CRM training. When considering the standard deviation of each mean score, overall, approximately 95 percent of the cohorts held at least positive views of the benefits of CRM training, whilst approximately 68 percent held moderate to strong views that endorsed CRM training as being an essential factor for Flight Safety and Mission Accomplishment.

Illustration 1. Crew Reactions to CRM Training: Study One & Two
	Study One		Study Two
Measure	Mean	Std Dev	Mean	Std Dev
Flight Safety	4.73	0.48	3.93	0.88
Mission Accomplishment	4.24	0.74	4.33	0.48

APPROACH 2: CREW ATTITUDES

Illustration 2 shows how crew attitudes changed during the course of both studies. In Study One, Pilot and Aircrew attitudes to CRM training improved considerably after the conduct of the training. Upon assessing the effect size of the differences between the pre-training and post-training means, a large delta of 1.04, and a medium delta of 0.37 were determined for the Pilot and Aircrew participants respectively; showing that CRM training had a high to medium positive influence on Crew Attitudes. However, Illustration 2 also shows that there was no significant change in Pilot attitudes to CRM training in Study Two. Salas, Fowlkes, Stout, Milanovich & Prince, (1999) suggest that this is likely to have occurred due to a ceiling effect that is inherent in more experienced crews, such as those used during Study Two.

Illustration 2. Overall Crew Attitudes To CRM Training
Study One
Participants	Pretraining Mean	Posttraining Mean	Difference in Mean Scores	Effect Size (d)	Effect Interpretation
Pilots	3.97	4.23	+0.26	1.04	High Influence
Aircrew	3.81	3.92	+0.11	0.37	Medium Influence
Study Two
Participants	Pretraining Means	Postraining Means	Difference in Mean Scores	Effect Size (d)	Effect Interpretation
Control	3.34	3.36	+0.02	0.08	Small Influence
Trained	3.33	3.35	+0.02	0.15	Small Influence

APPROACH 3: KNOWLEDGE OF CRM CONCEPTS

Illustration 3 shows how the studies cohorts varied in their ability to recall and apply major CRM concepts relating to teamwork skills. It can be seen in both studies that this training produced a significant effect (P<0.05) on the participants knowledge of CRM concepts, and in particular, for the Pilots this training has shown to have a very significant positive effect.

Illustration 3. Crew Knowledge of CRM concepts
Study One
Crew	Baseline Mean	Trained Mean	Difference in Mean Scores	P Value	Level of Significance
Pilots	9.80	12.63	+2.83	<0.0001	Very Highly Significant
Aircrew	8.64	10.47	+1.83	0.0005	Highly Significant
Study Two
Crew	Control Mean	Trained Mean	Difference in Mean Scores	P Value	Level of Significance
Pilots	14.42	19.13	+4.71	<0.0001	Very Highly Significant

APPROACH 4: CREW BEHAVIOUR

Illustration 4 shows how the productivity of aircrew undertaking a simulated flight scenario varied as a result of exposure to CRM training. In Study One, the trained teams did achieve higher targets than the control group during the pre-brief and high workload segments of the behavioural assessment. However, these results can only be suggestive of training effectiveness, given that no analysis was carried out to ensure that the cohorts were equal prior to the training (Salas, Fowlkes, Stout, Milanovich & Prince, 1999). This issue presented a significant extraneous variable, and became one of the major issues of Study One. However, in Study Two a counterbalanced, repeated measures design was used to improve the external validity of the research. The results of this study, which are also shown in Illustration 4, identifies that trained teams performed better than the control group, confirming what was concluded in Study One.

Illustration 4. Crew Behaviour
Study One
Segment Tested	Control Group Score %	Trained Group Score %	% Change
Prebrief	52	67	+29
High Workload	59	68	+15
Study Two
Segment Tested	Control Group Post-test Score %	Trained Group Post-test Score %	% Change
Prebrief	51	71	+39
High Workload	58	73	+26

GENERALISATION POTENTIAL

The sample of aviators used during both Study One and Two utilised individuals sourced from naval institutions. Thus the findings of the study may only be generalised to similar populations due to potential differences in the quality of CRM training provided at different military institutions, and to the training provided to civillian aviators. However, it is possible to suggest that the findings may extend to aviators within the wider aviation community, and that further future research should be conducted in order to further generalise the study’s findings.

Methods

Sample

STUDY ONE
A sample of 35 Pilots and 34 enlisted Aircrew were selected from a Navy transport helicopter squadron for Study One. The median pilot gender was male, and had an average of 690.36 hours of flight experience. The enlisted Aircrew were all men and had an average of 809.23 hours of flight experience.

STUDY TWO
A sample of 27 Pilots were selected from another Navy helicopter Squadron in Study Two. This group of pilots were split up into two separate groups; Trained and Control. The Trained Group consisted of 15 Pilots that had an average of 889.4 hours of flight experience. The Control Group included 12 Pilots that had accumulated an average of 1136.3 hours of flight experience.

Measurement Approach

A multi-measurement approach utilising a combination of between-group and within-group assessments were utilised to evaluate the effectiveness of the CRM training delivered. Data was collected on four levels: crew reactions to CRM training; crew attitudes to CRM training; knowledge of teamwork principles; and the use of CRM skills during simulated flight activities.

1. CREW REACTIONS
To assess Crew reactions to CRM training a reactions questionnaire was developed that utilised a five point Likert rating scale (5 = essential).

2. CREW ATTITUDES
An Aircrew Coordination Attitudes Questionnaire (ACAQ) with a five point Likert-type rating scale (5 =strongly agree) was used to to assess crew attitudes to CRM training.

3. KNOWLEDGE OF CRM CONCEPTS
A multiple-choice knowledge test was developed to assess the participants recall of major concepts related to the teamwork skills taught during the CRM course.

4. CREW BEHAVIOUR
Evaluation scenario's and support materials were developed to evaluate the participants performance using the Target Acceptable Response to Generated Events or Tasks (TARGETs) performance methodology.

Study One
During Study One a full-mission simulator was not available, as such, a PC-based simulator utilising Microsoft Flight Simulator software was utilised for the assessment. A single scenario was developed to assess the Crew's ability to achieve set tasks.

Study Two
During Study Two a full flight simulator was made available. Additionally two different scenarios were developed so a repeated-measures design could be employed to evaluate the performance of the experimental group and control group prior to any training, and the experimental group after having received training.

Data Analysis

The original research article provided data on central tendency, dispersion, frequencies, and percentages. This analysis, however provides further descriptive statistics, including effect size and percentage change.

References

SALAS Eduardo, Jennifer E FOWLKES, Renée J STOUT, Dana M MILANOVICH & Carolyn PRINCE (1999). Does CRM Training Improve Teamwork Skills in the Cockpit?: Two Evaluation Studies. The Journal of the Human Factors and Ergonomics Society. Volume 41, pages 326-343.

Federal Aviation Authority. Human Factors Research and Engineering Group. Coordinating human factors considerations in Federal Aviation Administration (FAA) programs and activities to enhance aviation safety, capability, efficiency, and productivity. Retrieved Jan 2012 from http://www.hf.faa.gov/

European Aviation Safety Agency Rulemaking Directorate Notice of Proposed Amendment 2014-17 Crew Resource Management. Retrieved Jan 2015 from www.globalairtraining.com