Exam Validity

How do we ensure that the PCE is a good exam?

All of the physiotherapy regulators in Canada, except the Ordre in Quebec[1], rely on CAPR to accurately assess candidates to ensure they are competent and safe and ready to practise. The Canadian public, in turn, relies on the Regulators to know that the care they receive from a physiotherapist is good care – safe, high quality, evidence-based care. This means that CAPR needs to be very sure that its exam is doing its job in a way we can trust. We do this by constantly reviewing the reliability and validity of the exam.

[1] The Ordre in Quebec has its own credentialling and assessment process.

What is Reliability and Validity?

Reliability:

For exams, reliability means consistency or repeatability. It means that if you take the exam on one day, and then retake the exam a week later without any significant changes in learning or experience during the course of that week, you will achieve a similar score on the exam to that which you received previously. In other words, the exam indicates more or less the same level of ability for an individual candidate each time it is administered. Reliability in the context of the PCE can also mean that different examiners score candidates in roughly the same way.

For both the written and clinical exams, there are several statistical methods used to calculate reliability. CAPR measures and monitors the Cronbach’s alpha, which indicates if an examination contains acceptable levels of reliability. CAPR staff and committee members review the alpha statistic, as well as other appropriate measures, for each exam prior to releasing exam scores.

In addition, for the clinical exam, it is important to ensure that exams are administered and scored the same way. We do this through:

  • standardized examiner training
  • standardized training of all of our “standardized clients”
  • creation and distribution of identical test sheets and answer sheets
  • centralized marking/scoring of the written stations with all of the markers in one room at one time
  • centralized decision making about safety and professionalism incidents, and the creation of a ‘decision rules’ handbook to ensure consistency from one exam to the next.

Validity:

While reliability of an exam focuses on consistency, validity of an exam is about its accuracy – is it measuring what it says it is measuring?   In the case of the PCE, the exam must be able to correctly establish that a candidate has the necessary competency (knowledge, abilities and skills) for entry-level physiotherapy practice. If the exam is not measuring what it is supposed to be measuring, CAPR might be passing candidates who are not actually safe and competent physiotherapists. Without validity, it does not matter if an exam is reliable, because it is not measuring the correct skill sets.

While reliability is measured through a statistic, validity is established through a process, or many processes – it is built over time by following internationally-accepted standards for the development and maintenance of a sound exam and it is built by following best practice procedures for each step of the exam life cycle.

All stages of the assessment life cycle must be followed to promote exam validity.

Step 1:  Development and design of the exam

  1. Clear and specific purposes: The first step in establishing a valid exam is specifying what exactly the purpose of the exam is. In the case of CAPR, the purpose of the exam is to determine that a candidate has the necessary knowledge, skills and abilities of a physiotherapist at entry-to-practice. This is a specific purpose for a specific time period in the physiotherapist career path – not a specialist physiotherapist, not a physiotherapist with 5 year’s of experience in practice, but a physiotherapist at the point of entry to practise. 
  2. Exam is based on a job (practice) analysis that is conducted and updated on a periodic basis: The Practice Analysis is based on physiotherapy practice information (through a survey) from hundreds of currently-practising physiotherapists from across Canada.
  3. The Physiotherapy Competency Exam is “multi-modal”: That is, it includes both a written (knowledge) component and a clinical (performance) component to assess different types of skills and abilities of the entry-level physiotherapist.

Step 2:  Item writing for inclusion in the written and clinical exam

  1. Exam items are linked to an exam “blueprint” that mirrors:
    1. the findings in the Practice Analysis which tell us the frequency and importance of different skills in physiotherapy practice and
    2. best practices in assessment.
  2. All exam items are developed by individual physiotherapy subject-matter experts across the country. Subject-matter experts are chosen because of their reputation as practice leaders in their regions. Subject-matter experts are trained in item writing.
  3. Draft exam questions written by regional experts are reviewed by a regional committee and then by a national committee of additional respected physiotherapy experts – from academic and community settings – to ensure relevance and accuracy. No single physiotherapist can develop a question on their own without external review and input. Questions written by multiple content experts are stronger than questions written by a single content expert, because no one expert can have all of the depth of knowledge for all of the concepts being examined by the exam[2]. This is why CAPR uses the two-step process of regional experts plus a national-level review of questions.
  4. All exam items are reviewed after each administration of the exam to gauge their performance. Items identified as being statistically problematic are removed entirely from the exam item bank or are re-written; For example, the item is very difficult, does not differentiate between higher or lower performing candidates and the like.
  5. New items are pretested – introduced into each exam with their performance reviewed before being included in actual exam scoring. This is a “blinded” process – meaning neither the examiners nor the candidates know which items are new and are being pre-tested.
  6. Best practices are followed to rate the difficulty of each question.
  7. Exam forms are put together ensuring they align with the exam blueprint and are standardized to make them more or less the same level of difficulty as other exams.

Step 3:  Administration of the exam

  1. Strict security procedures are followed for both the written and clinical exams to ensure items are not memorized and passed on to other candidates – this would decrease the item’s ability to accurately assess a candidate’s competence.
  2. CAPR conducts rigorous quality control steps to ensure that the exams are being administered in an appropriate manner.
  3. CAPR adheres to very strict consequences for anyone caught cheating on the exams.
  4. In the context of the clinical exam, the following additional steps are followed:
    1. training is conducted prior to each clinical exam to ensure that all examiners across the country are scoring exam performance in the same way.
    2. standardized client receive rigorous training supervised by physiotherapists to ensure clinical scenarios are presented in the correct way.

Step 4:  Exam scoring

  1. Best practices are followed for exam scoring and passmark setting.
  2. Regular reviews are conducted to introduce new scoring methods as new best practices emerge.

Step 5:  Continuous Quality Improvement

  1. The PCE is reviewed on a regular basis by independent, third-party reviewers to assess its compliance with international standards for the administration of certification exams. These international standards are developed by the International Standards Organization (ISO)/Standards Council of Canada, or the National Commission for Certifying Agencies – internationally-recognized accreditation agencies.
  2. CAPR conducts its own quality checks and quality improvement initiatives on a regular basis to ensure validity and reliability.
[2] From “ISO/IEC 17024 Conformity Assessment: general requirements for bodies operating certification of persons”  Cynthia D. Woodley. CLEAR Exam Review. Volume XXV (2).  Winter 2015, page 18.

External reviews

The CAPR exam program, its development and administration and its continuous quality improvement work are designed and implemented to be in compliance with relevant international standards for licensure and certification examinations – Standards for Educational and Psychological Testing (AERA, APA, NCME) and the NCCA Standards for the Accreditation of Certification Programs.

The CAPR exam program has been regularly reviewed by external experts since its inception. The most recent review in 2010 summarized “CAPR examination program revealed an extremely strong program, essentially in compliance with industry testing standards and responsive to the needs of key stakeholders.”[3] Another review of the exam program is currently underway.

[3] “An Evaluation of the Canadian Alliance of Physiotherapy Regulators’ Physiotherapy Competency Exam.” Department of Research and Development, Professional Examination Services, New York, New York. April 9, 2010.