PAP/NGC Programs Review
Ann Moriarty, MD
George Birdsong, MD
Dr. Rip Van Winkle1 arouses from a 10-year slumber after dozing
at his desk while reading an excellent editorial about proficiency testing in
cytopathology.2 He yawned, stretched, and thought to himself:
“I agree with the Clinical Laboratory Improvement Advisory Committee.
Cytology PT on glass slides is totally infeasible. Look, no one has been able
to come up with a glass-slide PT test yet, and it’s already six years
past CLIA ’88. Besides, all other PT is laboratory based. I hope we can
get changes to those regulations. But I’m not worried—I can tell
HSIL from LSIL.” He rose to find a cup of coffee brewing in the lab.
He saunters out and stops dead in his tracks.
“What has happened here?” he practically shouts. Computers are
on every desk. Technologists are at microscopes connected to computers. Everyone
is looking at slides with round cellular areas. Stains are automated. Coverslips
are being applied automatically. The slide numbers are in the 100,000 range.
All the technologists have a degree from an accredited school. Everyone is keeping
track of their workload. There are bulletin boards with ASC-US rates (“What
is with the dash?” he thinks to himself, “and what the heck is NILM?”)
and variance from a national benchmark hanging in the lab for all to see. Cytotechnologists
are performing HPV testing in a separate room. Dr. Rip thinks he must be dreaming.
“Well, you finally awoke,” says an unfamiliar face with a familiar
voice. “We gave up trying to wake you after several months. I am afraid
you have no password or electronic signature for signout. You will need to take
a proficiency test in cytology if you expect to sign out Pap tests. You will
also need to be trained in the new liquid-based technologies and automated screening
“Sue, is that you?” Dr. Van Winkle asks. “Please tell me
what is going on here.”
“You are in big trouble, Dr. Van Winkle. Not only have you slept for
10 years, and have had no continuing education, but you awoke in time to take
the PT test.”
Dr. Van Winkle remembers the editorial he had finished. He asks Sue, his longtime
manager, to fill him in. Certainly there have been vast improvements in required
PT over 10 years. He is astounded when Sue tells him that the 2005 test is the
first PT test the lab has taken.
“What happened to New York, Maryland, and Wisconsin?” he asks,
“and what’s the scoring system?”
“Wisconsin left in 1995, New York is not CMS-approved, and Maryland only
tests labs with Maryland patients,” Sue replies. “The company for
2005 is Midwest Institute for Medical Education, or MIME. You have a lot to
learn about the testing situation.”
Dr. Van Winkle is recovering slowly. “What’s the problem? We’re
a good lab. We can all pass the test.”
“Dr. Van Winkle, are you still asleep? There has been extensive literature
out recently outlining the difficulties of precision in cytology as well as
difficulty in the validation of slides used for interlaboratory comparison programs.
Let me fill you in,” Sue says.
“First,” she begins, “the good news is that cytologic screening
is still the best screening tool in medicine, with a 70 percent decrease in
the rate of cervical cancer since the 1950s. The CAP has had an interlaboratory
comparison program since 1989. You recall that the CAP program sends out glass
slides to labs, we answer as individuals (for educational purposes), and we
send the ‘lab answer’ back to the College. Remember that this is
an educational program. Participants send the CAP ‘classic’ cases
to be used in the Interlaboratory Comparison Program in Cervico vaginal Cytology.
Over time the CAP has been able to field validate a portion of its slides that
are used for graded challenges.
“Over the last 10 years,” Sue continues, “the CAP has identified
major problems with precision and validation of seemingly straightforward cases.
The CAP resource committee uses CMS—that’s the new HCFA, Dr. Van
Winkle—guidelines for its choice of slides in the program. To be accepted
in PT, CMS requires that any diagnosis of SIL or cancer must be biopsy proven.
Three expert pathologists must agree independently on the diagnosis. The CAP
has examples in its educational programs of unsatisfactory, negative for intraepithelial
lesion or malignancy (NILM), LSIL, and HSIL or higher, all categories used for
grading PT tests. The educational program is designed to be as rigorous in choosing
slides as is expected by CMS proficiency testing.”
The CAP has gone further than what is required by CMS for PT, she says. “They
‘field validate’ their slides. For a slide to be field validated,
there must be 20 responses, and 90 percent of the responses to that slide must
be in the correct series, with a standard error of <0.05. The results reported
in the NILM category must show at least 50 percent to the exact reference interpretation,
LSIL must have 70 percent of the answers to LSIL, and HSIL must show 70 percent
concordance to HSIL or worse.
“Despite these precautions, the CAP has discovered several issues. When
Dr. Renshaw3 looked at exact match rates for slides examined
between five and 24 times in the CAP program, with over 40,000 responses, only
29.7 percent of validated HSIL slides had a 100 percent exact match rate. 11.8
percent of field validated HSIL had a less than 50 percent exact match rate.
HSIL was one of the least reproducible interpretations. Furthermore, 19 percent
of conventional smears and 15 percent of ThinPrep smears failed field validation.”4
Dr. Van Winkle is clearly shaken. “Does that mean that even though the
slide is thought to be a classic case by a local pathologist, and three CAP
pathologists agree, that 19 percent of the slides can’t get 70 percent
“Precisely,” Sue says. “The other issue is that while you
were asleep, several studies have been reported that have tried to link performance
on PT to work performance. Remember, PT was invoked to assess cytologist performance.
Unfortunately, the rank correlation between PT performance and rescreening performance
has been reported at .24.5 A study reported from the United
Kingdom discussed their findings with PT since 1989. Of 63 cytologists taking
the test seven times, seven failed one round despite scoring highly on the remaining
“So, really, my score on proficiency testing may not have anything to
do with my diagnostic ability?” asks Dr. Van Winkle.
“Maybe, maybe not, is as close as I can figure,” Sue sighs.
“Well what about the grading system?” he asks. “At least
we can look at the clinical relevance of separating low grade from high grade.
It is important, after all, to try to separate LSIL from HSIL on Pap smears.
Oh, excuse me Pap test.”
“Once again, Dr. Van Winkle, you snooze and lose,” Sue explains.
“While we attempt to separate LSIL from HSIL in daily practice, the reproducibility
is poor, as Dr. Renshaw has shown. But added to poor reproducibility are the
ASCCP consensus guidelines on management of women with cytologic abnormalities.7
The ASCCP guidelines clearly indicate that colposcopy is the next step in patient
management for a Pap test result of LSIL or HSIL. The very next step clinically
is to perform colposcopy. In the long run, a woman with HSIL is treated differently
from a woman with only LSIL, but for purposes of screening, the immediate followup
of a person with either LSIL or HSIL is usually the same.”
“So,” Dr. Van Winkle says, “I will be tested on non-field-validated
slides and have to distinguish between LSIL and HSIL at least twice on each
test, and clinically it makes no difference to the patient? And the test may
not really weed out poor performers?”
“Now you can share our consternation,” Sue commiserates. “As
you can see, we have made the switch to computer-assisted screening and are
totally converted to liquid-based preparations. The PT will not evaluate our
practice because we don’t practice the old-fashioned way. Dr. Van Winkle,
cytology has come a long way while you were gone. We’ve made strides in
quality improvement, understand the weakness of a good test, and have tried
to move toward a more reliable Pap test by implementing personnel standards,
workload limits, quality control, and outcome assessments. And furthermore,
the automated instruments may help diminish human error.”
“Obviously, cytology has changed, Sue,” he says. “But it
appears that the regulations on PT have been dormant along with me.”
Notes and references
1. Apologies to Washington Irving for use of the concepts of
dormant objects from Rip Van Winkle, Matthews, Brander. 1907
2. Feichter M. Cytology proficiency testing: a look at what
happened and what’s ahead. Laboratory Medicine. 1994;25: 213–214.
3. Renshaw AA, Davey DD, Birdsong GG, et al. Precision in gynecologic
cytologic interpretation: a study from the College of American Pathologists
Interlaboratory Comparison Program in Cervicovaginal Cytology. Arch Pathol Lab
4. Renshaw AA, et al. Measuring the significance of field validation in the
College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal
Cytology: how good are the experts? Arch Pathol Lab Med. 2005;129:609–613.
5. Keenlyside RA, Collins CL, Hancock JS, et al. Do proficiency test results
correlate with the work performance of screeners who screen Papanicolaou smears?
Am J Clin Pathol. 1999;112: 769–776.
6. Gifford C, Green J, Coleman DV. Evaluation of proficiency testing as a method
of assessing competence to screen cervical smears. Cytopathology. 1997;8:96–102.
Drs. Moriarty and Birdsong are members of the CAP Cytopathology Committee.
Dr. Moriarty is with AmeriPath, Indianapolis. Dr. Birdsong is with Grady Health