Programme of Study: Assessment of the Admissions Enrollment Processes

Original Editor - Stacy Schiurring based on the course by Larisa Hoffman

Top Contributors - Stacy Schiurring

Introduction

Professional programmes are expected to perform a self-evaluation on the success of the programme.  This is often done using a quality improvement process:  Planning, Doing, Studying, and Acting (Brown, 2008).    Applying this process to admissions, means reflecting on the characteristics of the successful graduate, using tools that will select applicants who will most likely become the successful graduate, evaluating the effectiveness of the criteria, and modifying the criteria.  

Characteristics of the Successful Graduate

In the planning phase, where the admissions criteria are initially selected, success must be defined.  Desirable characteristics of a successful graduate of the programme include:

• success as a practitioner
• success in the licensure exam
• success in the programme

Tools for Selecting the Graduate

Tools that are most often used as admissions criteria include (1) grade point average, (2) aptitude tests, (3) interviews, (4) personal statements, (5) letters of reference, and (6) personality testing (Siu, 2009) (Choi, 2018).  The predictive validity of these tools has been evaluated for the selection of students who will be successful in their future performance (Siu, 2009) (Hall, 2017).  In general, these are described as cognitive and non-cognitive assessments.

1. Pre-professional GPA is a strong predictor of GPA in the professional programme for physical therapy (Payton, 1997) (Thieman, 2003), (Doctor, 2001) (Nuciforo, 2014) (Utzeman, 2007) (Zipp, 2010) (Shiyko, 2009).  Pre-professional GPA only weakly correlates with scores on licensure exams (Thieman, 2003), (Doctor, 2001).  
2. Aptitude Tests
   • For physical therapy programmes, the verbal GRE and Quantitative GRE are predictive of the first year GPA in a PT programme (Shiyko 2009), although this relationship is not consistently observed (Kirchner, 1994).   Verbal and Quantitative GRE scores are the best predictor of National Physical Therapy Examination (NPTE) Scores (Kume, 2018).  Scoring low on the GRE, particularly the verbal component of the GRE predicts failure on the physical therapy licensure examination (Hollman, 2008) (Utzman, 2007).
   • Noncognitive assessments have been proposed to identify behaviors and values consistent with healthcare professions.  The purpose of noncognitive assessments is to identify characteristics in an applicant that are challenging to obtain by other means (Hollman, 2007) (Albanese, 2003).  Non-cognitive assessments used in the admissions process include interviews, personal statements, and letters of reference (Siu, 2009).  
3. Interviews
   • The inter-rater reliability of interviews is quite variable (Wilding, 1998) (Richards, 1988) (Youdas, 1992) (Albanese, 2003), but interviews can predict performance on clinical skills (Balogun, 1988).  To improve reliability and validity, Hollman suggests creating a semi-structured interview emphasizing behaviors expected in the practicing clinician (Hollman, 2007).  The behaviors include decision making and problem solving, interpersonal skills, focus on the patient or client, communication, and teamwork.  The interviewer rates the applicant’s description of experiences in these domains (Hollman, 2007).  Using this type of structured format improves inter-rater reliability (Hollman, 2007), and improves ability of the interview to predict performance on a national licensure examination (Hollman, 2007).  The multi-mini interview has similar outcomes when a structured format is applied (Cameron, 2012) comprised of multiple brief interview encounters that have a standard format and scored with a rubric (Choi, 2018).  
4. Personal Statements
   • The reliability and validity of a written essay is variable (Youdas, 1992) (Brown, 1991) (Hull, 1996) (Kirchner, 1997), and is likely dependent on the guidelines provided to the applicant. Concerns regarding the authors of these statements has been highlighted.  To address this concern, some authors recommend the performance of a writing task based on previously provided materials to assess communication skills (Jones, 2000).  
5. Letters of Reference
   • The use of letters of reference as a tool for applicant selection has been scrutinized (Siu, 2009).  While some authors have found that use of objective criteria to evaluate letters of reference is correlated with clinical performance (Brothers, 2007) (Peskun, 2006), the writing ability of the author of the letter (rather than the applicant) can influence the selection process.  
6. Personality Testing
   • Emotional intelligence is a collection of characteristics that enable a person to be aware of their own emotional state and those of others (Pindar, 2020) (Goleman, 1996).  Healthcare providers are expected to be self-aware, empathetic and effective at influencing others, which are essential components of emotional intelligence.  Pindar et al recommends the use of a tool to measure emotional intelligence (such as the Trait Meta-Mood Scale) during the admissions process (Pindar, 2020).  Other authors recognize the value of a measurement of emotional intelligence, but raise concerns about the lack of predictive validity (Siu, 2009).  

Recognizing that addition of structure to non-cognitive measures improves the validity of the measures (Brothers, 2007) (Peskun, 2006), a holistic process applied to student admissions has been proposed (Choi, 2018).  It is based on these criteria:  Selection criteria are broad based and often linked to promoting diversity, Criteria is based on experience, attributes, and metrics, Individualized assessment is heavily weighted, along with race and ethnicity (Choi, 2018).

The tools with the highest predictive validity for medical student’s performance include grade point average (Krieter, 2007), aptitude tests (Krieter, 2007) (Donnan, 2007), and mini-interviews (Siu, 2009).   An undergraduate grade point average predicts graduate level grade point average (Choi, 2018).  Aptitude tests tend to predict outcomes on licensure exams.  Mini-interviews maybe a way to promote structure to evaluate communication and interpersonal skills.  

Pre-requisite Courses

Pre-requisite courses are often listed in the application criteria for healthcare programmes (Hull, 2016) and lack of successful completion of prerequisite courses can create a barrier to admission for pre-occupational therapy (Freeman, 2011) (Hull, 2016), pre-physical therapy students (Freeman 2011) (Hull, 2016), and pre-speech therapy students (Sylvan, 2020).   The purpose of pre-requisite courses has been described as foundational knowledge in a subject (Sylvan, 2020) and as an indicator for success in subsequent science-based courses (Cor, 2018) (Wireman, 2019).   The impact of prerequisite courses on student learning has been questioned (Shaffer, 2016).  Topics that are revisited at a higher level of Bloom’s taxonomy or the contextual framework applied in a new way is one way to improve student’s familiarity with a topic (Shaffer, 2016). Without revisiting the topic, students are unlikely to be familiar with the topic (Shaffer, 2016).      

Evaluating Criteria

The admissions criteria should admit individuals who will be successful in the programme. 

1. Student Attrition Rate
   • The simplest assessment of this process is to review the retention rate of the programme and compared to other programmes.  Student attrition in healthcare has been attributed to pressure related to workload, clinical placement, and personal circumstances (Hamshire, 2018).  Poorer student entry qualifications have been associated with student dropout or attrition rates in medical students (O’Neill, 2011) and physical therapy students (Andrews, 2006).   Attrition rates for healthcare students range from 6% (Maher, 2013) to 17% (Ryan, 2021).   It follows that low attrition rates within a programme would suggest the admissions process is selecting students who are more likely to be successful.  
2. Student Demographics
   • Another factor to consider in evaluating admissions criteria is demographics of the admitted group of students.   The demographics of a student group should relatively closely match the demographics of the society the students will service.   This is important to ensure that the demographics of the graduating healthcare students match the demographics of the clients they will serve.  
   • Cultural differences may be a threat to creating a therapeutic alliance between healthcare providers and patients (Vasquez, 2007) (Sue, 1998).  Individuals from similar cultural and ethnic backgrounds tend to remain in therapy and have less premature termination and better outcomes than when the backgrounds are not aligned (Sue, 1998) (Schouten, 2006) (Anderson, 2018) (Paternotte, 2015). 
   • Admissions criteria can create a barrier for specific subgroups within a culture.  Intentionally evaluating the outcome of the admissions process can help identify these barriers, such that they can be revised for future cohorts.  

The most common way to investigate admissions criteria is through regression analysis.  Regression analysis allows programmes to understand the relationship been successful graduate outcomes and admissions criteria (Kingsley, 2007).  While predictive validity of admissions criteria has been published, it is very likely that these correlations are dependent on the programme itself.  Therefore, evaluating the predictors of success within the programme is critical.  

Correlations between outcomes (such as student performance and graduate performance) and admissions criteria can be measured using a Pearson Correlation Coefficient (Kingsley, 2007).  

Kingsley recommends using the following definitions:  

• -1 to -0.7 = Strong Negative Correlation
• -0.7 to -0.3 = Negative Correlation
• -0.3 to 0.3 = No Association
• 0.3 to 0.7 = Positive Correlation
• 0.7 to 1.9 = Strong Positive Correlation

It is important to look at both the strength of relationship, as well as the statistical significance of the finding.  

Finally, the number of students needs to be defined.  The number of students admitted will be constrained by two competing factors:  the workforce needs in the community and the resource needs of the programme.  In the assessment of the workforce needs, an assessment of the number and type of local professionals in the community, the age of the professionals, and distribution of professionals (urban compared rural providers and public compared to private sectors) (Conradie, 2022).    Understanding the workforce needs can highlight important growth areas for an educational programme, however, the growth must be constrained by the available resources in the programme.  One of the most valuable resources in a programme are the faculty themselves.  Student to faculty ratios have increased overtime, particularly in low to middle income countries (Buckner, 2021).  While a large number of students to a small number of faculty does not necessarily limit student learning, it does imply limited practice opportunities.  This is particularly true in the clinical setting, where clinical supervision and mentorship are necessary for student success (Nadoo, 2022).   Limiting the number of students such as the student to faculty ratios (including clinical faculty), while simultaneously achieving workforce development goals strikes a balance in terms of numbers of students admitted.  

Resources

• bulleted list
• x

or

1. numbered list
2. x