Proficiency testing: A Winkle in time

October 2005
PAP/NGC Programs Review

Ann Moriarty, MD
George Birdsong, MD

Dr. Rip Van Winkle1 arouses from a 10-year slumber after dozing at his desk while reading an excellent editorial about proficiency testing in cytopathology.2 He yawned, stretched, and thought to himself: “I agree with the Clinical Laboratory Improvement Advisory Committee. Cytology PT on glass slides is totally infeasible. Look, no one has been able to come up with a glass-slide PT test yet, and it’s already six years past CLIA ’88. Besides, all other PT is laboratory based. I hope we can get changes to those regulations. But I’m not worried—I can tell HSIL from LSIL.” He rose to find a cup of coffee brewing in the lab.

He saunters out and stops dead in his tracks.

“What has happened here?” he practically shouts. Computers are on every desk. Technologists are at microscopes connected to computers. Everyone is looking at slides with round cellular areas. Stains are automated. Coverslips are being applied automatically. The slide numbers are in the 100,000 range. All the technologists have a degree from an accredited school. Everyone is keeping track of their workload. There are bulletin boards with ASC-US rates (“What is with the dash?” he thinks to himself, “and what the heck is NILM?”) and variance from a national benchmark hanging in the lab for all to see. Cytotechnologists are performing HPV testing in a separate room. Dr. Rip thinks he must be dreaming.

“Well, you finally awoke,” says an unfamiliar face with a familiar voice. “We gave up trying to wake you after several months. I am afraid you have no password or electronic signature for signout. You will need to take a proficiency test in cytology if you expect to sign out Pap tests. You will also need to be trained in the new liquid-based technologies and automated screening algorithms.”

“Sue, is that you?” Dr. Van Winkle asks. “Please tell me what is going on here.”

“You are in big trouble, Dr. Van Winkle. Not only have you slept for 10 years, and have had no continuing education, but you awoke in time to take the PT test.”

Dr. Van Winkle remembers the editorial he had finished. He asks Sue, his longtime manager, to fill him in. Certainly there have been vast improvements in required PT over 10 years. He is astounded when Sue tells him that the 2005 test is the first PT test the lab has taken.

“What happened to New York, Maryland, and Wisconsin?” he asks, “and what’s the scoring system?”

“Wisconsin left in 1995, New York is not CMS-approved, and Maryland only tests labs with Maryland patients,” Sue replies. “The company for 2005 is Midwest Institute for Medical Education, or MIME. You have a lot to learn about the testing situation.”

Dr. Van Winkle is recovering slowly. “What’s the problem? We’re a good lab. We can all pass the test.”

“Dr. Van Winkle, are you still asleep? There has been extensive literature out recently outlining the difficulties of precision in cytology as well as difficulty in the validation of slides used for interlaboratory comparison programs. Let me fill you in,” Sue says.

“First,” she begins, “the good news is that cytologic screening is still the best screening tool in medicine, with a 70 percent decrease in the rate of cervical cancer since the 1950s. The CAP has had an interlaboratory comparison program since 1989. You recall that the CAP program sends out glass slides to labs, we answer as individuals (for educational purposes), and we send the ‘lab answer’ back to the College. Remember that this is an educational program. Participants send the CAP ‘classic’ cases to be used in the Interlaboratory Comparison Program in Cervico vaginal Cytology. Over time the CAP has been able to field validate a portion of its slides that are used for graded challenges.

“Over the last 10 years,” Sue continues, “the CAP has identified major problems with precision and validation of seemingly straightforward cases. The CAP resource committee uses CMS—that’s the new HCFA, Dr. Van Winkle—guidelines for its choice of slides in the program. To be accepted in PT, CMS requires that any diagnosis of SIL or cancer must be biopsy proven. Three expert pathologists must agree independently on the diagnosis. The CAP has examples in its educational programs of unsatisfactory, negative for intraepithelial lesion or malignancy (NILM), LSIL, and HSIL or higher, all categories used for grading PT tests. The educational program is designed to be as rigorous in choosing slides as is expected by CMS proficiency testing.”

The CAP has gone further than what is required by CMS for PT, she says. “They ‘field validate’ their slides. For a slide to be field validated, there must be 20 responses, and 90 percent of the responses to that slide must be in the correct series, with a standard error of <0.05. The results reported in the NILM category must show at least 50 percent to the exact reference interpretation, LSIL must have 70 percent of the answers to LSIL, and HSIL must show 70 percent concordance to HSIL or worse.

“Despite these precautions, the CAP has discovered several issues. When Dr. Renshaw3 looked at exact match rates for slides examined between five and 24 times in the CAP program, with over 40,000 responses, only 29.7 percent of validated HSIL slides had a 100 percent exact match rate. 11.8 percent of field validated HSIL had a less than 50 percent exact match rate. HSIL was one of the least reproducible interpretations. Furthermore, 19 percent of conventional smears and 15 percent of ThinPrep smears failed field validation.”4

Dr. Van Winkle is clearly shaken. “Does that mean that even though the slide is thought to be a classic case by a local pathologist, and three CAP pathologists agree, that 19 percent of the slides can’t get 70 percent agreement?”

“Precisely,” Sue says. “The other issue is that while you were asleep, several studies have been reported that have tried to link performance on PT to work performance. Remember, PT was invoked to assess cytologist performance. Unfortunately, the rank correlation between PT performance and rescreening performance has been reported at .24.5 A study reported from the United Kingdom discussed their findings with PT since 1989. Of 63 cytologists taking the test seven times, seven failed one round despite scoring highly on the remaining rounds.”6

“So, really, my score on proficiency testing may not have anything to do with my diagnostic ability?” asks Dr. Van Winkle.

“Maybe, maybe not, is as close as I can figure,” Sue sighs.

“Well what about the grading system?” he asks. “At least we can look at the clinical relevance of separating low grade from high grade. It is important, after all, to try to separate LSIL from HSIL on Pap smears. Oh, excuse me Pap test.”

“Once again, Dr. Van Winkle, you snooze and lose,” Sue explains. “While we attempt to separate LSIL from HSIL in daily practice, the reproducibility is poor, as Dr. Renshaw has shown. But added to poor reproducibility are the ASCCP consensus guidelines on management of women with cytologic abnormalities.7 The ASCCP guidelines clearly indicate that colposcopy is the next step in patient management for a Pap test result of LSIL or HSIL. The very next step clinically is to perform colposcopy. In the long run, a woman with HSIL is treated differently from a woman with only LSIL, but for purposes of screening, the immediate followup of a person with either LSIL or HSIL is usually the same.”

“So,” Dr. Van Winkle says, “I will be tested on non-field-validated slides and have to distinguish between LSIL and HSIL at least twice on each test, and clinically it makes no difference to the patient? And the test may not really weed out poor performers?”

“Now you can share our consternation,” Sue commiserates. “As you can see, we have made the switch to computer-assisted screening and are totally converted to liquid-based preparations. The PT will not evaluate our practice because we don’t practice the old-fashioned way. Dr. Van Winkle, cytology has come a long way while you were gone. We’ve made strides in quality improvement, understand the weakness of a good test, and have tried to move toward a more reliable Pap test by implementing personnel standards, workload limits, quality control, and outcome assessments. And furthermore, the automated instruments may help diminish human error.”

“Obviously, cytology has changed, Sue,” he says. “But it appears that the regulations on PT have been dormant along with me.”

Notes and references

1. Apologies to Washington Irving for use of the concepts of dormant objects from Rip Van Winkle, Matthews, Brander. 1907

2. Feichter M. Cytology proficiency testing: a look at what happened and what’s ahead. Laboratory Medicine. 1994;25: 213–214.

3. Renshaw AA, Davey DD, Birdsong GG, et al. Precision in gynecologic cytologic interpretation: a study from the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology. Arch Pathol Lab Med. 2003;127:1413–1420.

4. Renshaw AA, et al. Measuring the significance of field validation in the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology: how good are the experts? Arch Pathol Lab Med. 2005;129:609–613.

5. Keenlyside RA, Collins CL, Hancock JS, et al. Do proficiency test results correlate with the work performance of screeners who screen Papanicolaou smears? Am J Clin Pathol. 1999;112: 769–776.

6. Gifford C, Green J, Coleman DV. Evaluation of proficiency testing as a method of assessing competence to screen cervical smears. Cytopathology. 1997;8:96–102.

7. http://www.asccp.org/consensus/cytological.shtml