Andrew H. Fischer, MD
The CAP makes the results of its 2006 gynecologic cytology proficiency testing program transparent in an analysis published this month (Bentz JS, Hughes J, Fatheree L, et al. Summary of the 2006 College of American Pathologists gynecologic cytology proficiency testing program. Arch Pathol Lab Med. 2008; 132:788-794). Proficiency testing has turned Pap test interpretation into a high-risk activity. If you want to read Pap tests, you have to take proficiency testing. If you fail the first PT, you can simply opt out of reading Pap tests and find other work, and the lab will face no obligations other than to replace you. Deciding to take the test a second time raises the professional and monetary stakes considerably, and you have only 45 days to decide. If you fail the second test, all the work you have done after being notified of the failure has to be reviewed by someone else at the lab’s expense, with documentation, and you may find yourself out of a job unless the laboratory pays for your remedial training. For the 2006 CAP PT, all of the more than 9,000 individuals who stuck with it eventually passed, and no one was forced to quit interpreting Pap tests. However, 40 individuals (0.4 percent) opted to stop reading Pap tests after failing one or more tries. This “culling” of 0.4 percent of individuals from the cytology work force in 2006 is statistically much lower (p<.001 by chi square) than the 2005 dropout rate of 0.8 percent. The overall pass rate for the first attempt at CAP’s 2006 proficiency test was considerably higher (94 percent) than the comparable rate (91 percent) for the 2005 proficiency test administered by the Midwest Institute for Medical Education, or MIME.
This paper is loaded with information, and the authors ask tough questions, such as: “Is culling good for the herd?” The 0.8 percent or 0.4 percent loss of individuals from the PT program cannot account numerically for the improved passing rates between 2005 and 2006, even if the 259 people who did not take the examination in 2006 but took it in 2005 had all been “failures” [http://www.cms.hhs.gov/CLIA/downloads/2006FinalTestingResults103007(MDASCPCAP).pdf]. Persistent failure rates from year to year (though the rate did improve in 2006) and from test to retest is in keeping with the idea (Nagy GK, Naryshkin S. Cancer Cytopathology. 2007;111:467–476) that PT failure is as much a statistical inevitability as it is a measure of competence. Cytotechnologists had a higher passing rate than pathologists in the first of the 2006 proficiency tests (95 percent versus 94 percent), and pathologists who functioned as primary screeners did the worst (84 percent). Bentz, et al., show that these differences can be attributed partly to the different scoring systems used to grade cytotechnologists and pathologists. They conclude: “Failure rates between cytotechnologists and MDs cannot be directly compared as an assessment of overall accuracy of one group versus the other.” However, as expected, the data analysis shows that pathologists added to the passing rate of a case examined by a cytotechnologist, when accounting for the differences in the scoring systems. A surprising finding that remains difficult to understand in the face of other studies is that the passing rates were largely independent of the type of slide preparation (liquid-based versus conventional).
Since culling cannot explain persistent failure rates, and everyone who did not get culled eventually passed, wouldn’t it be more efficient and cost-effective (not to mention less stressful) if PT were replaced with constructive CME activities? The article by Bentz, et al., adds more data and ideas to address this and many other questions.
Dr. Fischer, a member of the CAP Cytopathology Committee, is director of cytopathology, University of Massachusetts Medical School-Worcester.