College of American Pathologists
Printable Version

  CAP program takes competency
  assessment paperless


CAP Today




February 2009
Feature Story

Anne Paxton

Surrounded by wireless instant communication and powerful computers, we may sometimes forget that relatively primitive, paper-and-pencil technologies still abound. Until recently, written assessments have been a dominant format in many laboratories charged with evaluating the competencies of lab personnel. But labs in growing numbers are taking advantage of a flexible, systematic, 100 percent online Competency Assessment Program, developed by the College, to ensure CLIA-mandated competencies of medical technologists, laboratory technicians, and other personnel.

The Competency Assessment Program is an online package of assessment courses that come with customized training and CE credit, reassessment courses, a library of independent stand-alone “Pro Courses” with CE credit, instrument-specific observation checklists, course-building and modifying tools, management tracking and reporting, and individual transcripts. With some customization by the laboratory, the Competency Assessment Program will help laboratories satisfy all six elements of competency in Laboratory General Checklist question GEN.55500.

“The program has the potential to be a massive timesaver if you utilize it the way it’s designed to be used,” says Paul R. Labbe, vice president of operations for the reference lab facility of CompuNet Clinical Laboratories, LLC, near Dayton, Ohio. “I think CAP is ahead of the curve in saying: Let’s develop an easy template for everybody to try to standardize as much as possible, but still have that flexibility of customization.”

Now in its third year (version 2.0), the CAP’s Competency Assessment Program has the numbers to demonstrate its usefulness and popularity. Four hundred fifteen laboratories subscribed in 2007, the first year; by 2008 the subscriber count was at 950—and climbing.

It’s a program that has won consistent praise for its focus on hands-on, bench-level, usable knowledge, crisp color photographs and graphics, comprehensive coverage, ease of navigation, immediate scoring and feedback, and customized training for areas needing improvement. One user says: “As a manager I like the tracking and documentation. As a tech I like the content.” Another anonymous evaluator comments: “The program is very thorough¬—but in a fun way.” Even more common is this reaction: “These questions pertained to the actual work we do in this laboratory. Finally.”

For the CAP, the area of competency assessment is a new enterprise. “The CAP checklists for lab accreditation require competency assessment at least annually for regular employees, every six months for new employees,” says Keri Gonzalez, MT (ASCP), program manager in the CAP’s education division. “But we didn’t offer a program to assist labs in doing that. This program stemmed from our Quality Practices Committee, which did a Q-Probes study, published in the Archives in 2000, and recommended improvement in the process.”

Strong word of mouth is part of the reason the program has grown. “I think a lot of CAP programs get more exposure when they’re in PT [proficiency testing] renewal, and the first year our competency assessment program wasn’t part of that. Now that it’s part of the order renewal process, people are getting more information and hearing about it from their colleagues as well as the College’s marketing efforts,” Gonzalez says.

The user-friendliness of the Competency Assessment Program is mentioned frequently in evaluations: “I liked the fact that the correct answers were given and explained after each question rather than with the grade at the end of the assessment,” says one commenter. Another says: “The feedback at the end of each question helps me understand more and refresh my memory—especially in ‘grey’ areas.” The program also suits the atmosphere of busy laboratories, a user points out: “I was able to work on it while time permitted. I was not ‘logged off’ when I had to step away for a few minutes.”

In its latest incarnation, the program has 18 assessment courses, with nine released in January and nine in July, plus 18 stand-alone Pro Courses in major disciplines. For example, the assessment course in chemistry for January–June 2008 covered lipids, while for July–December 2008 it focused on endocrinology. The self-paced, interactive, and CE credit-bearing Pro Course in blood banking for January–June 2008 was on ABO typing discrepancies; the July–De­cem­ber 2008 blood banking Pro Course was on antibody screening and identification.

“The schedule of courses cycles over three or four years, so by changing topics every six months we will end up covering all of them,” explains pathologist Paul Fekete, MD, president and CEO of MediaLab Inc. MediaLab contracts with the CAP to develop and administer the Competency Assessment Program. CAP staff members review and edit all materials before they are included in the program.

The CAP expects that the Competency Assessment Program will eventually become a tool for physicians, nurses, and other hospital personnel to keep their laboratory knowledge current, though Dr. Fekete says this expanded use may take a while to develop.

The University of Colorado Hospital Clinical Laboratory in Aurora gave the program a trial run two years ago and now relies heavily on it, says Karin Ingle, MT(ASCP), administrative laboratory coordinator. “We ordered the smallest package you could get and asked for volunteers, dangling the carrot that they could earn free continuing education credits.”

The lab didn’t make a final decision, though, until Ingle tried out the CourseBuilder. Then she was hooked. “Having our own courses is the thing that’s most attractive to us.”

“We started out with administrative kinds of things. For example, I’m the compliance officer for the lab, and we have to have annual compliance training. I built the compliance course and assigned it to everybody. They really liked it; it’s a lot nicer than having to read dull documents. And it’s great to be able to pull up reports.”

The hospital program used across all departments to test competencies annually, called HealthStream, has a similar mode of operation. “People were already used to that format, but since the CAP program is tailored to the lab, it made a lot of sense for us to do it. The hospital program is geared more to nursing, but everyone has to be tested on safety issues.”

“In the lab, safety is more specific than in other parts of the hospital, so we have our own program instead of HealthStream. That is something we are gradually moving to the CAP program.”

There have been a few snags along the way. “We still have an old laboratory information system, which has what are called dumb terminals,” Ingle notes. That has meant limitations on the number of available computer terminals, but the LIS is scheduled for replacement with a PC-driven system, which will ease the problem.

The University of Colorado actually has an online testing program through its laboratory intranet. “People can read procedures we post on the intranet and sign off on them electronically. We’re trying to phase that out because it’s too confusing to people to have so many places to go to address different competencies, whether they were assigned by the hospital HR department or the clinical lab. We’re using the CAP program as something that can bring everything together in one place.”

Only a handful of courses are mandatory at this time, Ingle says. “When we load new people to the program, we assign all of the CAP courses to them. Then they can pick and choose among them to earn continuing education credits. We have quite a few people who have been certified since 2004 and need certification maintenance credits.” The CE credit feature is one of the most valuable for laboratory personnel, she adds. “Staff really like this because they don’t have to pay for it.”

Admittedly, she says, building custom courses can be confusing at times. “For new people it’s a little challenging, and I think you have to practice with it for a while before you get accustomed to it. But once the course is created, it takes care of itself. You just assign courses by clicking a few buttons, and e-mails get sent out to remind people.”

During CAP inspections, the laboratory needs to show evidence of employee competency. “In many cases staff have online competency documentation in addition to paper competencies. Eventually we hope to get most of it online.”

In fact, managing those records has historically been one of the chief challenges of competency assessment, Ingle says. “Several years back we didn’t get a deficiency cited but CAP inspectors pointed out that our competency records were really variable between lab sections, and it was hard for them to tell who had done what, and when, or whether anything was missing.”

The cost of the three-tiered program has proved very reasonable, she notes. “It’s based on how many people you want to load to the program—one to 20 or 20 to 50, and so on. We purchase an annual subscription for 51 to 250 users.” (For one to 50 users, the price is $595; for 51 to 250, $1,295; and for 251 to 1,000, $1,995.)

CompuNet Clinical Laboratories has 111 of its 550 employees registered in the Competency Assessment Program, Paul Labbe says. “We tried it out for 60 days in 2007, using it with a very small number of folks here, then sign­ed up for a bigger package for 2008.” The phlebotomy department is still using a separate Web-based phlebotomy program, “but they’re taking a hard look at the CAP program first quarter and will probably convert to that.”

CompuNet is a joint venture between Miami Valley Hospital in Dayton, Valley Pathologists, and Quest Diagnostics. “Basically we’ve grown to be the reference lab for the Greater Dayton area, and we’ve essentially cap­tured the outreach market. We have 65 to 75 percent of the physician office market in a 15-county area.”

The beauty of the Competency Assessment Program is the customization, Labbe says. “We like the overall template. The layout of the Web site could use a little improvement, but the test itself and being able to customize and generate your own assessments is really quite nice.”

Technical managers Mark Shearer and Brian Schrand, and other department managers, coordinate the customization process at CompuNet. Loading images, Shearer says, is “just a breeze. The only difficulty has been finding enough images to put in there.”

The old system, though custom­ized, did not have imaging capability. “That was actually one of the stumbling blocks. You can imagine there was a little reluctance to make a change from something that was working, so we hired a college student during the summer to work with the various department managers to transcribe all the information they had into the CAP system, and customize it with photographs from microscopies,” he says.

“I developed a test on the automated QC package where I was able to take a screen capture of specific out-of-control situations and say, This is what you see, what do you do?” Connecting such questions with instant feedback is a hit, Shearer adds. “If you choose the wrong answer, it immediately pops up on the screen and explains why.”

The assessment is basically what you would do with all lab personnel, he says. “They’re broken down by instrument. Tech A may work with instrument one, two, and four, and Tech B with two, three, and six, so you assign the appropriate test to them for what they typically do, you set up a due date, and once they have completed the assessment, each department manager is notified by e-mail. Anyone with administrative capabilities can go out and pull a report.”

The College-supplied materials change every six months, but “a nice thing about the program is we can ‘abscond’ with those modules that are fully developed. They’re a godsend, because we’re able to take them and make them our own.” For example, with the CAP phlebotomy module, one rule of specimen handling was the requirement to spin blood within two hours. “But our phlebotomy people said they teach everybody one hour just to give themselves a buffer zone, so we were able to modify the module to our specific needs.”

In another instance, the Competency Assessment Program reporting helped with a fix of a problematic test item. “There was concern about one of the tests that was a homebrew test, and I was able to go in and pull up a report that listed each question, what the answer was, and what percent got it right. There was one question that only two people got right, and when we went back to it, it turns out the way it was written it could be interpreted two ways. So we were able to very quickly pinpoint why people were not doing as well on the test as we thought.” With another item that people were getting wrong, they identified an area where more training was needed.

Labbe says the only downside at this point is the cumbersome user permissions process. “CAP is already aware of this and is working on it. But when we have to make changes from an administrative standpoint, the system can load only four individuals at a time, so it is extremely slow for user permissions.”

A number of other assessment programs have been developed for schools and colleges, such as Exam Manager, which CompuNet was using before, and most are open enough systems that they can be adapted. “But the College’s program is the only one I’m aware of that’s electronic and specifically for the clinical laboratory.”

Labbe attributes the growth in subscribers to positive buzz about the Competency Assessment Program’s value. Presentations at CLMA meetings and e-mail outreach from the CAP have been part of that. “But I think you will see even more people taking advantage of the program as CAP inspections take place. There’s always a lot of benchmarking and brainstorming that takes place during inspections, and people say, ‘Hey, do you know about this program?’ It’s something every laboratory struggles with, so any resource to help out is generally considered very positively.”

Why is competency assessment such a struggle? “There’s very little standardization in the laboratory industry, and you’ve got multiple vendors, multiple instrument types, multiple reagents, just a lot of different customized processes,” Labbe points out. “But there’s been more recognition, due to the audits and inspections, that we’ve got to document more of our competency assessment.”

The program’s set of standard questions is a real help because customization can be a tedious process. Shearer says, “Coming up with reasonable wrong answers is really difficult. So rather than have every lab in the world out there reinventing the wheel for themselves, CAP has come up with these standard questions. For the most part, what they have is specific enough for people to learn something but generic enough to apply to most situations. So it saves me spending hours and hours every year coming up with tests—or giving the same test to people every year because I don’t have time to make a new one. It’s a huge timesaver and better quality than I was producing before.”

The Pro Courses are something new—an assessment program that has a lot more interactive feedback. “You answer a series of questions, then they provide all the educational material to fill in the gaps, and it’s usually a full page of text such as a small article,” Shearer says.

This kind of simultaneous training-assessment framework is not all that common, Labbe believes. “Competency assessment basically means are you still able to accurately perform this test, and under CLIA it can be done through direct observation,” Labbe says. “So some people watch a run, and if the person gets it, they call that enough. We prefer to use a test, because you can throw questions in there on things people tend to forget such as nuances in procedures or policies. Plus you can throw in questions as regular reminders.”

Since CompuNet, through its human resources department, requires 13 hours of continuing education by all technologists and technicians each year, the company is considering using the program there as well. “I’d like to take the Pro Courses and make them available to HR, and these courses would count toward the techs’ assessment. Since CAP will track it all for you, HR wouldn’t have to maintain a separate database.”

The program sends out reminders when personnel get close to their due dates for an assessment, “so it becomes a nice, efficient management tool for the department manager. Once they get through all the intensity of getting what they need into the system and customizing it, it then makes the process a whole lot easier.”

Not every medical technologist is a cheerleader for the program, of course. “It’s a bit of a change, and some question the need, and there are some grumbles, but once we got through the access issues and got people used to it, the response has been relatively positive. Our challenge has been making sure we have a lot of PCs available here that have Web access and e-mail network access.”

There are a couple of important keys to successfully adopting the program, Labbe says. First, don’t try to roll it out in a single month, as CompuNet did. “We probably didn’t give it the concentration and fanfare it needed, and daily lab life got in the way.”

And as for any new program, “you have to have at least one target individual who is going to lead the charge on this new initiative, who understands the capabilities of what’s being offered, and who can manage individual permissions and identify leaders within each department to help customize the program.”

Subscribers to the program this year will encounter another useful feature: the new ChecklistBuilder tool. “This tool is even easier than our CourseBuilder tool,” the CAP’s Gonzalez says. “ChecklistBuilder allows the user to create checklists from scratch as well as modify CAP-approved checklists.”

“With all these components offered—the assessment courses, Pro Courses, instrument-specific observation checklists, CourseBuilder, ChecklistBuilder, and the reporting features, as well as the convenience of an online product, the Competency Assessment Program provides users with a comprehensive package to assist laboratories in meeting their accreditation and CLIA requirements.”

In addition to the clear benefits of online competency assessment, data from the program are likely to prove useful to researchers, says Paul Valenstein, MD, vice chair of the CAP’s Council on Scientific Affairs.

“The CAP Competency Assessment Program is, of course, designed to help institutions periodically assess the competence of employees and identify individuals who need additional training,” he says. “But hidden within the program’s database is a treasure trove of multi-institutional data waiting to be mined. We can use these data to identify skills that our nation’s laboratory workers seem to have difficulty maintaining, as well as areas where laboratory staff tend to do well. I look forward to some insightful analyses and publications in the years ahead.”

Anne Paxton is a writer in Seattle.