Consistency is our Goal

Consistency

The results from the Physiotherapy Competency Exams are used by Canadian Regulators to make licensing decisions that affect the professional lives of over a thousand physiotherapists a year.

The stakes are high and it is important that these decisions are made with reliable data. To be reliable, an exam must be consistent and repeatable from one exam administration to the next and from one examiner to the next. We moni­tor overall exam reliability with every exam administration, both written and clinical, to ensure the reliability scores meet best practic­es.

Inter-rater Reliability

In addition, we recently completed two studies to test the inter-rater reliability of written-station markers of the June 2017 clinical component of the PCE. The first study was conducted at the Written Station Marking session, while the marking was in process. This study showed that markers had an overall percentage of agreement of 80%. The second study was a blinded double-marking of candidate test sheets done after marking was completed. In this study, the reliability described by the Pear­son co-efficient was 0.96.  Both of these scores demonstrate a high level of inter-rater reliability.

How do we do it?

At every marking session, our staff train all written station mark­ers, using practice questions and providing feedback to markers, with the goal of standardizing the marking process. On an ongoing basis, we review our marker training approaches to maintain and increase exam reliability and provide regulators with the best information possible so that they can make good decisions that protect the public.