College of American Pathologists
Printable Version

  Statistically speaking, federally mandated
  PT is unreliable


CAP Today



May 2008
PAP/NGC Programs Review

B. Lane Jr., MD

Seventeen years after proficiency testing in cytopathology was mandated by the Clinical Laboratory Improvement Amendments of 1988, annual nationwide testing began, in 2005, as required by the Centers for Medicare and Medicaid Services. There is widespread agreement among medical professional organizations that the federally mandated cytopathology proficiency testing program is ineffective, and legislative efforts to change the program have been introduced in Congress. Meanwhile, annual proficiency testing in cytopathology continues unabated.

In a recent article, George K. Nagy and Sonya Naryshkin review basic statistical principles of testing theory and evaluate the validity and reliability of the current CLIA ’88-mandated cytopathology proficiency testing format (Nagy GK, Narysh­kin S. The dysfunctional federally mandated proficiency test in cytopathology: a statistical analysis. Cancer Cytopathology. 2007; 111: 467–476). The authors explain basic principles of dichotomous and non-dichotomous testing. They illustrate a simple binomial error model and show how it can be applied to cytopathology proficiency testing. They show how the size of the test set and the competence of individual examinees directly affect the accuracy of a testing system.

The authors provide statistical evidence that the current cytopathology proficiency testing system misclassifies substantial numbers of cytopathologists. Using basic statistics they show why most first-attempt failures are not caused by insufficient cytopathology skills but instead are due to the poor validity and reliability of a 10-slide test. They also show through basic statistical principles why a 10-slide test with three attempts allowed is an ineffective method to identify less-than-competent cytopathologists. They conclude that the current federally mandated proficiency testing system cannot reliably measure the expertise of cytopathologists, and they advise that if cytopathology proficiency testing is to achieve its intended purpose it needs to be completely redesigned, with the participation of experts in modern test theory.

Dr. Lane, a member of the CAP Cytopathology Committee, is a pathologist with Southeastern Pathology Associates, Brunswick, Ga.

Related Links Related Links