The Pilot Passed Every Check. The Psychometric Test Said No. Who Tests the Test?
- 23 hours ago
- 10 min read
Psychometric testing has become a common feature of pilot recruitment, helping airlines identify cognitive, behavioural, and aptitude-related characteristics that may predict training and operational success. Yet while aviation places considerable focus on preventing unsuitable candidates from entering flight training, far less attention is given to the opposite risk: experienced, competent pilots who may be incorrectly screened out by the very systems designed to identify talent.
Published: 4 September 2026
Written by: Todd Skaggs
Consider an experienced captain with 12,000 flying hours. The pilot holds a current licence and medical, has repeatedly passed simulator checks, line checks and recurrent training, and may even be a Type Rating Instructor or Type Rating Examiner authorised to assess the competence of other professional pilots. He or she applies to another airline, completes a computerised psychometric assessment before reaching the simulator, fails to meet the required threshold and is removed from the recruitment process.
There is nothing inherently contradictory about that outcome. Experience does not guarantee judgement, adaptability, teamwork or suitability for every airline, and an examiner's regulatory privileges are not the same thing as an employer's recruitment criteria. But it raises a legitimate question: what if the recruitment system has rejected someone who would have performed successfully in the job?
That is a false negative, and aviation does not appear to know how many it creates.
Aviation Is Designed to Fear the False Positive
Selection science distinguishes between false positives and false negatives. A false positive occurs when an organisation accepts someone who later proves unsuitable. A false negative occurs when it rejects someone who would have succeeded.
Aviation understandably worries about the first. Training a pilot is expensive, unsuccessful training consumes scarce simulator and instructor capacity, and poor selection can eventually have operational or safety consequences. Rigorous selection therefore makes sense.
But increasing selectivity does not eliminate error. Raising a cut-off score may reduce unsuitable candidates getting through while simultaneously increasing the number of suitable candidates rejected. A recruitment process with an impressive failure rate is therefore not automatically an accurate recruitment process.
If 80 per cent of applicants fail an assessment, that tells us 80 per cent were rejected. It does not tell us that all 80 per cent should have been rejected.
The objective should not be to reject as many candidates as possible. It should be to make as many correct decisions as possible.
Psychometric Testing Can Work
There is strong evidence that appropriately designed aptitude, cognitive, psychomotor and multitasking assessments can help predict pilot-training outcomes.
A 2025 study involving 123 ab-initio pilot candidates found that elements of a psychometric battery predicted ATPL theory performance and additional training requirements. Spatial orientation and working memory were associated with extra training needs, while time-sharing performance related to both measures.
The European Association for Aviation Psychology, EAAP, also points to a striking historical example. In 1943, normal US Army Air Force selection-score requirements were waived for 1,311 pilot cadets admitted largely on a first-come basis. Only 23 per cent subsequently graduated from advanced flight training. After traditional selection standards were restored, graduation rose to 63.4 per cent. None of the 150 candidates admitted with the lowest scores graduated.
The lesson is not that psychometric testing is bad. Quite the opposite. Good selection can be extremely valuable.
The difficulty is assuming that a test shown to predict who can successfully become a pilot is equally effective at distinguishing between people who have already spent thousands of hours successfully operating as pilots.
A 10,000-Hour Captain Is a Different Population
An FAA review examining airline pilot age and performance made this distinction decades ago. Researchers warned against assuming that tests developed and validated for flight-training selection could automatically distinguish between levels of performance among highly trained commercial pilots without validation on that population.
Experienced airline pilots also present a statistical problem called range restriction. They are already highly selected. Before becoming captains, they have passed licensing requirements, medical examinations, type ratings, simulator checks, line checks, recurrent training and years of operational scrutiny. The differences between people within that population may therefore be narrower than the differences among hundreds of aspiring pilots beginning their careers.
An experienced captain also brings information that an ab-initio applicant cannot possess: thousands of real operational decisions, abnormal situations, crew interactions and recurrent professional assessments.
Psychometric testing may still add valuable information, but the question becomes more demanding: what additional information does the assessment provide, and has its ability to predict something important been demonstrated among pilots like the person being tested?
Age can complicate this further. Research presented at the 2025 International Symposium on Aviation Psychology found that performance on perceptual-speed measures tended to peak in the late twenties and early thirties before gradually declining into the early sixties. That does not mean older pilots are worse pilots. It means age can influence performance on some speeded cognitive tasks.
If a 58-year-old examiner performs more slowly than a 28-year-old applicant on an abstract computer task, the relevant question is not simply who completed it faster. It is whether that difference predicts meaningful performance in the aviation role being filled.
The Numbers Raise Questions
The EAAP's survey of aviation-selection practitioners provides useful insight into how psychometric systems are applied.
Among civilian-pilot respondents, 67.4 per cent reported conducting in-house validation of a selection instrument or battery. This does not mean the remainder used unvalidated tests, because some may have relied on external evidence or assessment providers, but it demonstrates why understanding how a test was validated matters.
Norm groups varied as well. For personality assessment in civilian-pilot selection, 23.2 per cent reported using general-population norms. For ability testing, 16.3 per cent reported doing so. General-population norms are not inherently inappropriate, but employers need to know whether the comparison population is relevant to experienced professional pilots.
Cut-off scores introduce another issue. Across respondents using psychometric tools, 71.1 per cent reported using some form of psychometric cut-off. Among civilian-pilot respondents, 34.9 per cent cited expected success as one basis for setting cut-offs, 25.6 per cent cited score distribution, 32.6 per cent assessment-provider advice, and 18.6 per cent said applicant numbers and available vacancies were among the considerations. These categories were not mutually exclusive.
That last point is particularly important. There is a considerable difference between asking whether a candidate is capable of performing a job successfully and asking whether the candidate scored highly enough to survive competition for a limited number of vacancies. Both can result in the applicant being told that they failed, but they are not necessarily the same judgement.
When One Result Ends the Process
EAAP also found that 27.5 per cent of civilian-pilot respondents used a hurdling approach, where one critical assessment result can become a knock-out condition. The remaining 72.5 per cent reported compensatory decision-making, where results can be considered together.
The difference matters. Under a compensatory system, a weaker result in one area might be considered alongside simulator performance, command experience, a competency interview and a strong professional record. Under a hard hurdle, that later evidence may never be collected because the candidate has already been eliminated.
A historical Air India recruitment exercise demonstrates how dramatically different assessment stages can produce different results. In 2015, around 260 candidates were reportedly called for simulator proficiency assessment and 160 progressed to interview. Of 69 candidates ultimately rejected, contemporary reporting said 56 were rejected predominantly following psychological assessment, despite the candidates being type-rated A320 pilots and some having already passed the simulator stage.
That does not establish that the decisions were wrong. There is no comprehensive public record showing what subsequently happened to those pilots. But that missing information is precisely where the false-negative problem becomes difficult to measure.
The Number Nobody Publishes
There is an important limitation to this argument. There is no credible public industry-wide dataset showing how many experienced pilots rejected during psychometric selection would subsequently have performed successfully in the role.
The evidence therefore does not justify claiming that aviation has a proven large-scale false-negative crisis. Nor is there reliable public evidence demonstrating that psychometric failure rates among experienced pilots are rising globally.
What exists is a well-established selection-science risk combined with a significant aviation data gap.
Airlines can measure how many applicants completed an assessment, how many passed, how many reached the simulator, how many entered training and how many eventually succeeded. The hardest statistic concerns the candidates who disappeared from the process. How many of the people rejected would actually have succeeded?
An airline may never know. The rejected captain could join another operator, pass its simulator and fly successfully for another decade. The instructor might train another airline's pilots. The examiner may continue checking crews elsewhere. The original recruitment system sees none of those outcomes.
That creates an unusual blind spot. Organisations can become very good at studying the people they selected while knowing remarkably little about the quality of the people they rejected.
The FAA Says Selection Systems Need Validation
This is not simply an academic concern. In 2019, the FAA reviewed 15 civilian pilot-selection batteries and processes against recognised best practices and found inconsistent adherence. Relatively few were based on a recent or current job analysis.
Its recommendations included defining measurable job-performance criteria, using reliable and valid predictors, conducting appropriate validation studies, evaluating fairness and documenting the analysis. Crucially, the FAA said cut-scores should have a rational relationship with predicted job performance.
That principle becomes particularly important with experienced pilots. If an assessment is being used to reject captains, its relevance should ideally be demonstrated against outcomes that matter for captain performance. If it is used to select instructors, its validity should relate to instructional performance. If it is being applied to examiners, the relevant competencies should reflect what examiners actually do.
Validation should follow the job.
Instructors and Examiners Raise the Stakes
The question becomes more significant when the candidate is an instructor or examiner because regulators already assess these people against detailed aviation-specific competencies.
EASA instructor standards include preparation, creating a learning environment, presenting knowledge, facilitating learning, assessing trainee performance and evaluating instructional effectiveness.
Examiner privileges require additional experience, competence and authorisation because the examiner is entrusted to determine whether other pilots meet defined standards.
IATA likewise defines observable competencies for instructors and evaluators, including threat and error management, continuously assessing trainee performance, intervening appropriately and adapting instruction when circumstances change.
None of this means an instructor or examiner should be exempt from recruitment assessment. It does mean substantial evidence about that person's professional competence already exists.
It also means that rejecting the wrong instructor can have a larger workforce effect than rejecting one line pilot. Instructors and examiners are capacity multipliers. Airlines can recruit large numbers of new pilots, but somebody must train them, conduct simulator sessions, complete line training, perform checks and assess command upgrades.
Aviation Still Needs 674,000 Pilots
This matters because aviation continues to forecast enormous workforce requirements.
Boeing's 2026 Pilot and Technician Outlook estimates that commercial aviation will require 674,000 new pilots by 2045. Across Boeing's wider personnel forecast, roughly two-thirds of demand comes from replacing people leaving the workforce rather than simply supporting fleet growth.
The retirement pipeline is already visible. Figures supplied to the UK Parliament in April 2026 showed 1,315 commercial pilots aged 60 to 64 in Civil Aviation Authority records, meaning they were expected to reach the mandatory airline retirement age of 65 within five years. The same parliamentary discussion raised the availability and retention of qualified flight instructors.
That creates a legitimate workforce question. Aviation could spend billions attracting and training new pilots while making it unnecessarily difficult to recruit some of the experienced people required to train and check them.
There is no evidence proving this is happening at significant scale. There is equally no comprehensive public dataset demonstrating how frequently suitable experienced candidates are being screened out.
Psychological Assessment Is Broader Than a Psychometric Test
The regulatory position also requires careful distinction. EASA requires flight crew in European commercial air transport to undergo a psychological assessment before commencing line flying. It does not prescribe one particular online psychometric recruitment test.
Its guidance is much broader. Assessment should reflect the complexity of the operating environment, be based on a job analysis of safety-critical dimensions and can include cognitive abilities, personality traits, operational competencies and social competencies. EASA also emphasises appropriate validation and aviation psychology expertise.
ICAO has itself raised questions about how some of these tools are used. At its 42nd Assembly in 2025, a proposal was made for standardised global guidance on psychometric assessment in pilot licensing. ICAO's Technical Commission agreed the issue deserved further consideration while highlighting concerns about personality-trait assessment, particularly its subjectivity and situations where there is no clear relationship with required competencies.
That is not an argument against psychological assessment. It is an argument for aviation doing what it normally does extremely well. Demand evidence.
Who Tests the Test?
Aviation would not install an aircraft component simply because somebody said it worked. The requirement would be defined, the component tested under relevant conditions, its limitations understood and its performance monitored after entering service. Pilot-selection systems deserve similar discipline.
A psychometric assessment should have a clear purpose. Its norms should suit the population being assessed. Its pass mark should have a defensible relationship with job performance, and the system should be interested in false negatives as well as false positives.
Not every experienced pilot who fails an assessment should have passed. Thousands of flying hours do not make someone immune from scrutiny, and properly validated psychometric testing can add valuable information that logbooks and simulator checks cannot provide. But a good safety system should want to know when it is wrong.
The difficulty is that a false-negative pilot generates no incident report and no training failure. The person simply disappears from one airline's recruitment statistics and may later become another airline's captain, instructor or examiner.
That leaves aviation with a question far more important than how many applicants failed a test: How many of those failures were actually correct?
Perhaps the next development in pilot psychometric assessment should not be another test. Perhaps aviation needs to become much better at testing the tests themselves.
Key Facts
Boeing forecasts demand for 674,000 new commercial pilots by 2045.
EAAP found 67.4% of civilian-pilot respondents reported in-house validation of an assessment instrument or battery.
71.1% of respondents using psychometric tools reported using cut-off scores.
27.5% of civilian-pilot respondents reported using a hurdling approach, where one critical result could eliminate a candidate.
23.2% reported using general-population norms for personality assessment, and 16.3% for ability testing.
FAA guidance says pilot-selection cut-scores should have a rational relationship with predicted job performance.
ICAO raised concerns in 2025 about personality assessment where measured traits cannot be clearly related to required competencies.
UK CAA records showed 1,315 commercial pilots aged 60 to 64 in March 2026.
EASA requires psychological assessment before line flying, but this is broader than a specific online psychometric recruitment test.
No credible public global dataset currently establishes the false-negative rate for psychometric selection of experienced commercial pilots.
Related Articles
Planning growth, fleet changes or seasonal operations in 2026? Contact Brookfield to discuss your staffing and consultancy needs. Email: info@brookfieldav.com
Explore our full range of recruitment services, connecting aviation businesses with skilled pilots, aircraft engineers and industry professionals worldwide.
Author: Todd Skaggs Aviation staffing and consultancy insights LinkedIn
