Method for measuring factors that affect the performance of pilots

Recebido: 08 de maio de 2017 Aceito para publicação: 08 de julho de 2017 Publicado: 31 de agosto de 2017 Editor de área: Li Weigang, UnB ABSTRACT This paper presents the development of a model of accident analysis according to the principal factors which influence aeronautical accidents that are able to assess any aircraft accident, taking into account human, organizational, environmental and airport infrastructure factors. The methodology of data collection of this research was through the literature, analysis of aircraft accident reports, technical visits to the center of certification of commercial aircraft pilots and interviews with industry experts. From this model, it is possible to evaluate the influence of these factors and identify the dependence and relationship existing, and how they influence the system. With the aid of Bayesian Networks technique, it is also possible to quantify the factors and assess which ones have more impact in the system. The results show the relationship between the factors that can influence the performance of the pilots and therefore can indicate how it may impact the success or failure of tasks related to flight procedures. The results also may indicate subsidies for mitigating actions, collaborating in the management of operational safety of air transport and assessing the overall impact of the factors that determine any accident.


INTRODUCTION
Safety is an intrinsic component of aviation. In addition to the key aspects of technical and human performance, the concept of organizational accidents which was developed in the 90s, must also be taken into account in the efforts to contemporary safety. This concept does not only consider the active faults of individuals developed in front line operations but also latent conditions inevitably present in any system. One of the oldest models of accident causation is "Heinrich's Domino Theory" proposed by Heinrich in 1940, which describes an accident as a chain of discrete events that occur in a particular temporal order (Heinrich et al., 1980). According Leveson (2003), this theory belongs to the class of sequential accident models or models based on accident events that gave allowances for most of the models of accident analyses introduced subsequently. These models became known as Failure Mode and Effect Analysis (FMEA), Fault Tree Analysis (FTA), Event Tree Analysis (ETA), and Cause-Effect Analysis. Large proportion of this approach has been subjected to fierce criticism for being based on a causal relationship between events (Rasmussen, 1997;Hollnagel, 2004;Leveson, 2004).
A series of new methods has been developed in the recent years to better meet the research needs of socio-technical systems and to respond to the introduction of new technological systems (Aven and Zio, 2011). Many of the introduced methods allow greater levels of detail and accuracy in modeling phenomena and processes covering physical phenomena. Human and organizational factors, as well as the use of software, made it a more dynamic analysis (Luxhøj and Coit, 2006;Mohaghegh et al., 2008;Ale et al., 2009;Røed et al., 2009).
Since the 50s, many efforts on research have been made to document the precise location of aircraft accidents, so that it could be possible, through these data, to obtain an effective planning of airport operational safety and its surroundings.
The highlights are "The Airport and Its Neighbors" (President's Airport Commission, 1952) who conducted one of the first studies on the impact of Safety relations and noise effect with neighboring communities. Despite the limited data, this report led to the establishment of "Clear Zones" which are currently known as "Runway Protection Zone". Another important study was "Air Installation Compatible Use Zone (AICUZ) Program" of the US Department of Defense in 1973. This study served to define significant areas of potential accidents for military aircraft, known as "Accidents Potential Zones (APZs)". Additionally, Ashford and Wright (1992), in surveys conducted by the Airline Pilots Association, in the period from 1967 to 1992, indicated that 5% of accidents occur on route and 15% occur in the vicinity of airports. The remaining 80% occur in landing and taking off areas or clear zones.
Such studies cited highlight the positive effects generated post-analysis, in order to mitigate or reduce the impacts caused by aircraft accidents. Moreover, the positive effects also caused several other studies on accidents arise by means of safety agencies and organizations worldwide.
It has been noted that studies are being conducted with an increasing number of occurrences (accidents or incidents) to reduce these negative effects. As an example, it can be cited the accident analysis studies developed by IATA (2014), Caltrans (2002), ACRP 3 (Hall et al., 2008), FSF (2009), ACRP 50 (Ayres et al., 2011), Boeing (2010 among others. Despite the important contribution with a large number of occurrences, they present limited conclusions because they do not evaluate the relationship between such occurrences and human performance factors or organizational factors. They were concerned only with accidents and/or aeronautical incidents analysis based on spatial location.
A systemic view requires much more than to list a given number of factors which caused a certain event, it must explain how the factors are interrelated and what factors are related. This answer comes before an analysis of an accident. The nature of these factors must be clarified before starting the analysis.
Within a systemic view, it can be mentioned the studies of Greenberg (2007), Roelen et al. (2011) andMaturana (2013). Such studies have emphasized a probabilistic approach analyzing human factors, organizational factors and other factors associated with the environment and/or local infrastructure.
The three studies developed a general model of quantitative analysis related to aviation accidents using Bayesian networks (BN) based on factors that influence the performance of the operators. To this purpose, the objective of this study is to present a probabilistic approach, using Bayesian Network (BN) to analyze accidents involving human and organizational factors within the framework of commercial aviation passengers.

METHOD OF ANALYSIS
To achieve the goal of this research and develop a general model for analyzing accidents with aircraft based on human factors, it was considered a systemic vision in which different kinds of factors may be involved in a given event. Then, to develop a general model, the study was based on three pillars: literature review, field research and interviews with experts and analysis method with Bayesian Networks (BN). The literature review was structured based on the following organizational factors: human factors, airport infrastructure and environmental factors. It is considered that there is an interrelationship among these factors, such as the organizational factors influencing human factors for example. At any given accident, some factors respond with greater intensity on the causes, others with less intensity. Therefore, this study suggests the use of Bayesian Networks to show the interrelationship between the factors associated with the system and the event. In addition to these factors, it is considered the pilot's skills in performing tasks for flight procedures. More details of this analysis, field research and expert consultation are described below.

Fieldwork
It is necessary to understand how human error can negatively affect the operations leading to the accident. Thus, to assess which variables should be selected, it is necessary to understand which procedures or tasks that are performed by the flight team or the company that can influence the flight team activities, and how environmental conditions may impact their abilities. Figure 1 shows the relationship between the curves of the demanding tasks that must be performed during the approach and landing and the pilot's skill (AOPA, 2008). At this point, an emergency or distraction may overload the flight crew and result in an accident. The curve designated by "Task requirements" represents the amount of tasks that the pilots must carry along operating phases shown on the x axis. A designated curve "pilot capabilities" is the skills required to perform these tasks. The difference between both curves indicates the margin of safety of operation, represented by the y axis. First, a search about human factors analysis was conducted in the literature. In order to familiarize with the theme, it was made visits to a training center for pilots -South America Flight Training located in Guarulhos, Sa o Paulo. The visits aimed to follow the tasks of the pilot, simulated in the Airbus 320, with tests on various procedures and different unfavorable situations of the aircraft and airport infrastructure to evaluate the skills and performance of the professionals. During these visits, commanders and first officers of commercial flights were interviewed about the skills required for the landing and takeoff procedures, the importance of communication between both and the difficulties perceived factor associated with infrastructure and climate change.
Four of the factors found in the literature were also evidenced during field observations. These factors were: authority gradient between the pilot and the co-pilot; there was a greater difference in authority between the pilot and the pilot; knowledge of technical standards, this factor was very important in the analysis, since the pilots were six months of the last training. The technical knowledge requires time to be incorporated by the operator, characteristic inherent to the human condition, therefore, it is an important human factor for the evaluation process. Another factor verified was the fatigue, clearly observed in the fatigue of some pilots during the flight procedures, who later confirmed the reasons for which they felt fatigued. Finally, it was observed that pilots who performed tasks with some degree of uncertainty had less experience and training than pilots who assertively performed the tasks without hesitation. Neapolitan (2004) defined Bayesian Networks as a graphical structure to represent the probabilistic relationships among a large number of variables and to make probabilistic inferences with those variables. A BN is a Directed Acyclic Graph (DAG), which is defined as G = (V, E), where V is the nodes representing discrete or continuous variables and E is a set of ordered pairs of distinct elements of V, which are called arcs (or edges) and represent the dependencies between the nodes. The conditional probabilities associated with the variables are the quantitative components.

Analysis Method with Bayesian Networks (BN)
The nodes and arcs are the qualitative components of the networks and provide a set of conditional independence assumptions that may be represented through a graph notion called d-separation, which means that each arc built from variable X to variable Y is a direct dependence, such as a cause-effect relationship.
If the variables are discrete, the probabilistic relationship of each node X with their respective parents P(X) is defined by its Conditional Probability Table (CPT) while for continuous variables, this probabilistic relationship is defined by its Conditional Probability Distribution (CPD), which represents conditional probability density functions. The quantitative analysis is based on the conditional independence assumption. Consider three random variables X, Y and Z, in this case X is said to be conditionally independent of Y given Z, if P(X,Y|Z)=P(X|Z)P(Y|Z). The joint probability distribution of set of variables, based on conditional independence, can be factorized as shown in Eq.1 and Eq.2.
The graphical representation is the bridging of the gap between the (high level) conditional independence statements that must be encoded in the model and the (low level) constraints, which enforce the CPD, Langseth and Portinale (2007).
The possibility of using evidences of the system to reassess the probabilities of network events is another important feature of the BNs. Given some evidence; beliefs are recalculated to indicate their impact on the network. In many cases, it is interesting to determine critical points in the system. Classical methods of inference of a BN for this purpose involve computation of the posterior marginal probability distribution of each component, computation of the posterior joint probability distribution of subsets of components and computation of the posterior joint probability distribution of the set of all nodes.
Bayesian belief nets (BBNs) (Pearl, 1988;Jensen, 1996;Jensen and Nielsen, 2007;Cowell et al., 1999) are currently the method of choice for decision modeling in complex systems. Ale et al. (2009) mention the most important reasons for this are: (i) they provide a high level graphical representation of the system in terms of "influences" in which the problem owner easily recognizes his/her problem; (ii) this graphical representation also serves as a user-interface with which the user can perform recalculations and updates; (iii) they incorporate the effects of proposed decisions in a natural and transparent way.
The submission of the Bayesian Networks (BN) explains how these factors can be analyzed qualitatively or quantitatively. The general model is focused on the application of human reliability analysis (HRA), which aims at assessing the human contribution in aeronautical systems operations.

DEVELOPMENT OF GENERAL MODEL OF ACCIDENTS ANALYSIS
This paper proposes a generic model to analyze aircraft accidents considering the factors that contribute to the pilot's performance based on studies of Greenberg (2005;2007), Roelen et al. (2011) and Martins and Maturana (2013).
Such studies have emphasized a probabilistic approach analyzing human factors, organizational factors and other factors associated with the environment and/or local infrastructure. Greenberg (2007) developed a general model of quantitative analysis related to aviation accidents using Bayesian networks, but only factors that increase the threats of an accident. Roelen et al. (2011) developed a hybrid model for risk analysis jointly by Eurocontrol and the Federal Aviation Administration (FAA). In summary the model shows the conjunction of the technical fault tree and event tree to the method of Bayesian networks. Martins and Maturana (2013) analyzed the human error contribution for ship collision taking into consideration the activities performed by the crew and is focused on the operation of an oil tanker in the Brazilian coast. The authors searched evidence related to the occurrences of the possible human errors (not found in literature) according to the routine of the ship thus obtaining the necessary data to build the network.
The proposed model is focused on the influence of the contributing factors in the pilot's and co-pilot's performance -the flight crew's performance. The proposed model considers that there are five main types of factors that can influence the pilot's performance during their task execution. Thus, four groups of factors were considered for the model construction:  Figure 1. It is considered that the organizational factors (MOF) directly influence human factors. In other words, it is considered that before performing the tasks relevant to their profession the pilots have the MOF established a priori because they are related to their home institution. Thus, such influence occurs during the period in which the pilot is in the organization and not only during a particular flight task. Soon, the group "human factors" is the one which is able to change the pilot status.
For the marginal probabilities of the model variables, some hypotheses were assumed:  It was assumed that 90% of the weather conditions is adequate for the flight operation and only 10% is inadequate. It should be noted that environmental factors do not influence all required skills from flight crew. The environmental factor highlighted was the visibility, which can be caused by bad weather, heavy rain, fog and others. It was considered that the visibility influences the "monitoring" and "judgment" of the pilots. It can be noted that environmental factors do not influence all required skills from flight crew. For example, it may be considered that there is not influence of visibility into the "knowledge of procedures" of the decision maker;  It was considered the actions of the first officer only if the commander did not perform the verification of the parameters;  The nodes shown in the model contain two "states" (high/low, adequate/inadequate among others, depending on the node concept);  It was assumed that 95% of positive states of all the MOF was adequate only 5% was not adequate;  For conditional probability tables (CPT), each PSF, the probability of the positive and negative states of their parent nodes were considered proportional. For example, a PSF with four parent nodes may have 0% to 0 parent node in the negative state, 25% to 1, 50% for two and 75% for 3 and 100% for the 4 nodes-parents in the negative state (the same logic to the positive state);  For conditional probability tables (CPT) of the skills used the linear interpolation, where the limits ranged from [0.4; 1]. This is because for a particular skill, even if the probability of all nodeparents is presenting themselves in the negative state, yet it is not possible to predict that this skill of the flight crew will be 100% in the negative state. When all parent nodes of a particular factor is the positive state, the probability of this factor will be [0; 1], 1 (100%) chance of the positive state and 0 (0%) chance of the negative state.

Management and Organizational Factors (MOF)
Little is known about the types of organizational factors that directly contribute to aeronautical accidents. In contrast, there is a growing number of studies in the literature about the role that the error of the flight crew plays in the etiology of accidents. Studies suggest that up to 80% of all injuries are caused by unsafe pilot's actions (Dismukes et al., 1999). This discrepancy on understanding the organizational factors is not surprising, given the fact that the pilot actions are more easily linked to the occurrence of an accident, while organizational factors are most of the times, temporally very distant from the event, making it difficult to connect them to an accident during an investigation (Wiegmann and Shappell, 2001). Some authors have argued that despite a growing awareness of organizational factors, they are often overlooked or are not identified by aeronautical accidents investigators (Heinrich et al., 1980;Yacavone, 1993;Maurino et al., 1995). For this research, it was used the study "Measuring Organizational Factors in Airline Safety" conducted by Thaden et al. (2004) which was supported by the Federal Aviation Administration (FAA). The main reason for choosing this reference was to analyze the errors of pilots which had their roots in organizational factors.

Human Factors or Performance Shaping Factor (PSF)
PSF known as the "performance shaping factors," or "human factors" or "internal factors" influence the perception of a situation of an operator and modulate their diagnosis and their decisions. Thus, it is important to understand and model the PSF and its influence (Sundaramurthi and Smidts, 2013). Generally, human factors can be classified into physiological, personal and psychological among others. It is important to emphasize that it is not necessary that all MOF are related to each established human factor. Another issue that must be pointed out is that interrelationships among PSF may occur.
The references used in obtaining the PSF are: Swain and Guttman (1983), NUREG 1792 (Kolaczkowski et al., 2005), and Chang and Mosleh (2007), Greenberg (2007), Roelen (2008) and Sundaramurthi and Smidts (2013). The first three studies approach to the nuclear industry and are references to several other studies. The NUREG 1792(Kolaczkowski et al., 2005 is cited in various applications in other industries. The last three studies bring a focused approach to the aviation industry. Greenberg (2007) selected the PSF subjectively based on the analysis of occurrences of aircraft accidents. Roelen (2008) used NUREG 1792 as a starting point to list the PSF directed it to the flight crew. Sundaramurthi and Smidts (2013) compared the same model of PSF they used in a Bayesian network for nuclear accident in aviation accidents.

Environmental Factors
It was considered as environmental factors the ones related to weather conditions. Report 3 from ACRP, particularly its Appendix 3, identifies the most relevant weather characteristics in overruns analysis and undershoots (Hall et al., 2008). For the presentation of the current study, it was considered only external environmental factors.

Pilot's Abilities
To select factors about ability required for pilots, it was made a selected study on human performance factors in the literature. The required abilities take time to be developed by the professionals, while human factors are related to the current pilot's condition at the time the task will be executed. Among the literature review, it was selected Martins and Maturana (2010), CENIPA report (2013) and Cabral et al. (2014). In addition to the literature, the field research already mentioned, contributed to the selection of these factors (see 2.1).

FINDINGS
This topic presents an application of the concepts presented in this study considering the model proposed and the identified factors. This way, it was made a relationship between these factors. This relationship was identified considering the judgment of the authors of this study, based on literature review, analysis of accident reports and research with experts in the industry. In this topic an example of application was made to show the relationship between these factors and how they influence the pilots' actions.

An Example of Application: Verification of Aircraft Parameters
To exemplify the analysis, it was considered a common verification or checking of the aircraft parameters which is performed just before the landing procedure. So, for this task to be completed, it is understood that the commander must check all the parameters required for the aircraft landing procedure. The first officer can check the data and intervene to conduct the procedure appropriately if the commander has not made it. Thus, analyzing the possible factors that may influence this task, it was identified the relationship presented in Table 1. For the development of Bayesian network, the following factors were considered in the analysis: the required skills to perform each task; the related human performance factors (PSF); the management and organizational factors (MOF) that can influence the related PSF and its visibility as an environmental factor.
Considering the factors listed in Table 1 a relationship among the factors was identified according to the proposal of a generic model for analyzing air accidents. Figure 2 gives an overview of the relaionship among the fators of each category; an arrow on this figure represents the influence of one factor on another. This relationships was modeled using a BN, as described in the previous section. It should be noted that this paper does not intend to introduce the probabilities of the factors in this example, but to present the general model of accident analysis and the relationship between the factors in a simple example of a task performed by pilots who may be visually understood. The general model for accident analysis must include all tasks performed by the crew considering the factors influencing their execution and the relationship among them.
The results show the relationship between the factors that can influence the performance of the pilots and, therefore, it can indicate how this may impact on the success or failure of tasks related to flight procedures. The results also may indicate subsidies for mitigating actions, collaborating in the management of operational safety of air transport and to assess the overall impact of the factors that determine any accident.

Impact of the Factors to Verification of Aircraft Parameters
To identify the critical factors of General Model (Figure 3), that is, the factors that most impact on the failure of the execution of the checking. All parameters were considered that the negative state of this activity has a probability of 100%, and hence 0% success, as shown in Figure 4. It is observed that in the general model, considering the assumptions, there is a likelihood of 2.92% of failure occurrence in the execution of the verification of aircraft parameters and 97.08% of success.
The Conditional Probability Tables (CPTs) resulting from the network developed for the verification task of the aircraft parameters were obtained according to the result of the interactions among the factors presented on Tables 2 to 8 according to Figure 3.         To obtain the factors that most impact the task of checking all parameters -the factors behavior was observed for the variable "check all parameters" and were at 100% in the negative state. That is, when this task was not performed correctly by the pilots. The comparison between: M1 -without evidence ( Figure 3) and M2 -with evidence of failure ( Figure 4) are presented in Table 9.
The results showed that "safety culture" is the MOF that has the highest impact on the implementation of successful task by pilots. The "knowledge technical standards" was the PSF with the greatest impact on the achievement of successful work by the pilots, followed by fatigue. The skills required by pilots, the highest impact factors in task accomplishment, are: attitude, followed by judgment.
The ranking of priorities of the factors selected in the accident analysis model was obtained according to the results (ranking column - Table 9). In order to improve the safety of air transport in the short and medium term, the factors that had the highest increase in its negative state should be the first to be prioritized, since the improvement of these factors have a greater impact on safety -safety culture (1º), knowledge technical standards (2º) and attitude (3º).
This study does not intend to delimit the factors linked to the error, but to present a model of probabilistic analysis using the factors that have a relation with the event in question. For other activities of pilots, new factors can be addressed, however the modeling presented in this study serves to any analysis of the sector. Such approach promotes a wide vision regarding accidents caused by human error, either by the pilot or the air company's operator.

CONCLUSIONS
Human factors are the most important source of uncertainties of any model, though many techniques and computational tools have arisen in recent decades to deal with the complexity of socio-technical systems. To be able to get a representative analysis of the real system, a systemic vision of the process is required.
However to model operational procedures of a system, its main tasks are not an easy step. So it is most important to know the system that is intended to model, and then analyzes the factors (and their relationships) that can contribute to an occurrence. For such information, a survey of the literature and a research with pilots and accident investigators become extremely important. It is important to note that factors linked to component failures in aircraft systems are not being considered in the general model. This is because such failures have a low probability of occurring. Therefore, the emphasis was given to human factors, environmental factors and management and organizational factors.
It was possible to observe that the tasks performed by pilots suffer the impact of the organization to which it belongs. Such influence is indirect, as observed in this study, the influence of organizational factors are of human factors, that is, on operators. This holistic view of the process highlights that aeronautical accidents start long before the implementation of the pilot's task. Another highlight of this work is that human factors influence the skills of the pilots, this means that even if the pilots in question has the best possible performance, it cannot control the impact the human factors have on their skills. An example of this is the human fatigue factor that influences the judgment ability. Thus, this study, by a probabilistic analysis, allows us to present a new way to evaluate the aeronautical accidents as well as contribute to improve the management of airlines and their flight crew and agencies of the sector.