October 12, 2000
Dockets Management Branch (HFA-305)
Food and Drug Administration
5630 Fishers Lane, Room 1061
Rockville, MD 20852
The College of American Pathologists (CAP) is providing written response to the Federal Food and Drug Administration’s (FDA) request for public comment on issues related to the classification and criteria used for the categorization of laboratory tests as waived under the Clinical Laboratory Improvement Amendments (CLIA) of 1988.
The College is a national medical specialty society representing over 16,000 pathologists who practice clinical and/or anatomic pathology in laboratories across the country. The College’s Laboratory Accreditation Program is an Health Care Finance Administration (HCFA) approved accrediting organization as specified in CLIA regulations. The College’s Commission on Laboratory Accreditation is responsible for the accreditation of over 6,000 laboratories worldwide. In addition, the College conducts a HCFA approved proficiency testing program involving more than 20,000 laboratories. These test surveys provide an opportunity to evaluate the accuracy of laboratory testing and to enhance quality in the clinical laboratory thereby contributing to higher quality patient care. CAP members have extensive expertise in providing and directing laboratory services and serve as inspectors in the accreditation program. These programs are designed to improve the quality of laboratory services and to ensure the accuracy and reliability of test results. Therefore, the College has a profound interest and extensive experience in this topic.
In testimony given at the August 14th and 15th public workshop on the topic, the College reiterated its position that all laboratory tests used for diagnosis, treatment, and monitoring of human disease should be subject to quality control and proficiency testing. The College, as well as other presenters at the Workshop, expressed concern over the lack of oversight and follow-up in waived laboratories, which contributes to problems with the conduct of waived tests. Study data was presented by HCFA from two pilot studies in Colorado and Ohio where waived and physician provider microscopy (PPM) laboratories underwent voluntary inspections. These studies indicated that as many as 50% of waived/PPM laboratories have compliance problems that may result in inaccurate and erroneous test results. In view of the problems identified, the College cautions FDA against the rapid expansion of waived testing.
The following list of published articles provide further support for the potential for problems to develop in the conduct of a waived test and the need for oversight and follow-up of waived testing:
- Searching for Inaccuracy in Clinical Laboratory Testing Using Medicare Data: Evidence for Prothrombin Time (Mennemeyer ST, Winkelman JW. JAMA 1993; 269:1030-1033)
- Quality Control in Patient Self-Monitoring or Blood Glucose (Greyson, J. Diabetes Care 1993 Vol. 16, No.9; 1306-1308)
- Variation in Proficiency Testing Performance by Testing Site (Stull TM, Hearn TL, Hancock JS, Hansfield JH, Collins CL. JAMA 1998; 279:463-467)
- Are Physicians' Office Laboratory Results of Comparable Quality to Those Produced in Other Laboratory Settings? (Hurst J, Nickel K, Hilborne LH. JAMA 1998; 269: 468-471)
- Is It Time to Turn the Page on CLIA 1988? (Bachner, P. JAMA 1998; 269: 473-475)
- Point-of-Care Glucose Testing - Effects of Critical Care Variables, Influence of Reference Instruments, and a Modular Glucose Meter Design (Louie RF, Zuping Tang BS, Sutton DV, Lee JH, Kost GJ. Arch Pathol Lab Med 2000; 124: 257-266)
- Analytical error of home glucose monitors: a comparison of 18 systems (Johnson RN, Baker JR. Ann Clin Biochem 1999; 36: 72-79)
- Accuracy of International Normalized Ratio determined by portable whole-blood coagulation monitor versus central laboratory (Reed C, Rickman H. Am J Health-Syst Pharm 1999; 56: 1619-1623)
It is clear to the College that evolving technology will expand the scope of tests that can be considered as waived well beyond that which was ever originally envisioned. It is also clear that any laboratory test if performed incorrectly, or that generates an inaccurate result is not acceptable for patient care. It is therefore incumbent on the FDA to develop new and innovative approaches to quality control (QC), proficiency testing (PT), performer competence and test/instrument performance in the field, which will ensure that waived tests are accurate and reliable over the life of the instrument/kit. The College has the following suggestions for this type of an approach to quality control and proficiency testing for waived tests:
- FDA should hold manufacturers accountable for incorporating the necessary QC and PT into waived test device design and instructions for use so as to ensure that the performance of the test is reliable and accurate over the life of the instrument and/or reagents.
- HCFA has the responsibility to ensure that providers performing waived testing comply with manufacturer’s instructions. HCFA should develop mechanisms to identify and address non-compliance issues.
- Manufacturer’s should develop self-assessment tools for waived devices to assist testing facilities determine their ongoing compliance with manufacturer instructions for QC and appropriate PT.
- Programs should be developed for non-laboratorians that would educate them on the value and importance of QC and PT for accuracy and reliability of tests.
- Performer competence needs to be assessed and assured. This could be accomplished via PT and competency review by the laboratory director. The manufacturer should provide directions for this.
- Manufacturers are required to collect and report data on test system/instrument failure after a waived test has been approved. Clearly defined instructions for the reporting of test system/instrument failures should be included in the manufacturer’s user procedure manual. The FDA should enforce the reporting of this information and make it available to the users.
In addition and to provide further clarification of the points made above, the College offers the following specific response to the questions posed by the FDA in the Federal Register notice issued July 21, 2000.
- What criteria should be used to demonstrate that a waived test is a simple laboratory examination and procedure with “an insignificant risk of erroneous result?” For example:
- Should a waived test, when performed by untrained users, provide an accurate result with no significant clinical or statistical error when compared to a measure of truth? This requires availability of well-characterized reference methods and/or materials as part of the waived test assessment. The current threshold for waiver as established by CDC is no significant inaccuracy and no significant imprecision.
A waived test when performed by an untrained user should provide as high a degree of accuracy as testing performed in a moderate or high complexity laboratory by trained personnel in the context of appropriate clinical relevance and medical decision limits.
- Should a waived test, when performed by untrained users, provide a test result that shows no user error when compared to the same test performed in a CLIA certified lab by a trained user? This requires comparison of the test in a lay-user setting with performance of the test in a CLIA certified lab by a trained user. The threshold for waiver would be no difference in performance in the two settings.
A waived test should be evaluated in the setting of intended use with a representative population of intended users. Specimens used for testing should represent the expected range of clinical specimens and should include some specimens near the thresholds of assay sensitivity or near important medical decision levels. A waived test should not exhibit any performance variability when performed by an untrained user in that setting when compared to the performance of the same test in a moderate or high complexity setting by a trained user.
- Should the FDA apply a different model to determine the waived status of a test?
There should be an evaluation of expected sources of problems, failures, and/or interference with a specific test. The recent NCCLS document Guideline on Quality Management for Unit-Use Testing (EP18P) provides valuable information to manufacturers and users alike on identification of “Source of Error” analysis. After identification of potential problems, there should be an analysis of how a potential problem affects a test. Necessary resources for support of users such as manufacturer hotlines should be identified to address expected user problems. It would be desirable for test kits to have internal controls or indicators that would identify when a test had been stored improperly, sustained packaging leaks, or when test reagents no longer have full reactivity. For example:
- A moisture indicator on bottles of test strips for glucose that would indicate whether test strips had been exposed to excessive humidity due to leaving a bottle open.
- Internal procedural controls that test for reactivity of reagents in qualitative test kits.
- Indicators of interference such as an indicator of hemolysis in a test that measures potassium.
- What criteria should FDA use to determine if a methodology is “so simple and accurate to render the likelihood of erroneous results by the user negligible?”
The College generally agrees with the test system properties that illustrate simplicity and ease of use as endorsed by CLIAC.
- Should a waived test be so accurate when performed by untrained users that inaccurate results will not occur?
A waived test should be so accurate that there is minimal likelihood that inaccurate results will occur when performed by untrained personnel. To ensure accuracy of results, principles of quality assurance such as quality control and proficiency testing at an appropriate interval should be used to assess the accuracy of these methods in the hands of untrained personnel. Minimally, waived test when performed by an untrained user should provide as high a degree of accuracy as testing performed in a moderate or high complexity laboratory by trained personnel in the context of appropriate clinical relevance and medical decision limits.
- Should a waived test have variable accuracy if used adjunctively? Is it acceptable to waive tests that have inaccurate results but do not have any major negative clinical impact? How should FDA make this assessment?
The College does not believe that it is appropriate to waive tests that have inaccurate results even if it might be perceived that inaccurate results would not have a negative clinical impact. We would again like to emphasize that incorrectly performed waived tests, reagent or device failures carry the potential for very real risks of patient harm.
- What criteria should FDA use in determining that a test will “pose no unreasonable risk of harm to the patient if performed incorrectly?”
The College believes that all tests performed incorrectly can result in risk to the patient. Therefore, CAP does not think that FDA can develop criteria to determine that a test will “pose no unreasonable risk if performed incorrectly.” For those tests that are waived by FDA, CAP would ask FDA to develop new and innovative approaches to QC, PT, performer competence and test/instrument performance in the field, which will ensure that waived tests are accurate and reliable over the life of the instrument/kit. The College’s suggestions for this type of an approach to QC and PT for waived tests were provided earlier in this document.
The automatic categorization of “approved for home use” tests as waived is inappropriate. The same stringent criteria should be applied for evaluation and approval of OTC/home use testing as are employed for review of requests for test/instrument waiver. While the College supports the FDA approval for home use tests for patient use, there is concern over the use of these tests in multiple sites by a variety of unskilled users with no oversight or quality assurance. Literature supports the need for continued assessment of performer competence and device accuracy for home use testing such as glucose meters.
- Should the waiver process be different for screening tests that require a second test for confirmation? Because there are no CLIA standards for performance of waived testing, except instructions to follow the manufacturer’s package insert, what is the assurance that confirmatory testing will be performed? Should the need for confirmatory testing raise, lower, or have no impact on the threshold for a waiver decision?
The waiver process should be no different for screening tests that require a second test for confirmation. There can be no assurance that confirmatory testing will be performed when indicated. The need for a confirmatory test should have no impact on the threshold for a waiver decision.
- Should accuracy be determined using comparison of the waived test to a well-characterized reference method and/or materials, to a designated comparative method and/or materials, to a working laboratory method and/or materials, to a clinical algorithm for diagnosis, and/or to other endpoints?
Accuracy should be determined by comparison to well-characterized reference methods and/or materials and to appropriate clinical endpoints.
- How many samples what types of samples (real or artificial), by how many users and how many sites are appropriate to evaluate accuracy? (Current guidelines being followed by FDA are for performance to be demonstrated by laboratory users at a minimum of one site.)
The number of samples needed to evaluate accuracy of waived tests should be at a minimum the same as those for moderate or high complexity tests. Samples should be evenly dispersed over the clinically relevant range of the test. Specimens near the threshold or near medical decision points should be used. It is desirable to use real patient samples or samples free from significant matrix effects. Evaluation should occur in multiple settings in which this testing will be performed. The number of samples tested should be determined by statistical methods in order to detect clinically relevant inaccuracy.
NCCLS Publication EP9-A Method Comparison and Bias Estimation Using Patient Samples; Approved Guideline,(December 1995) Experiments presented in this publication can be used to determine and evaluate accuracy of method or device against a reference method or comparative method. The recommendation for method comparison is at least 40 patient samples analyzed over at least 5 operating days. Analyze each patient sample in duplicate for both reference/comparative method and waived test. If possible, at least 50% of the sample run should be outside the reference intervals. Evaluation of bias may be of limited value depending on specimen type for reference vs. waived test method.
- What should be the background of these users?
Users should have a high school education at minimum and appropriate training to perform the test including quality control and external monitoring when appropriate. Variation in user technique and competence represents one of the common problems associated with waived tests. There should be evaluation of variation of test performance by non-laboratory personnel with specimens at or near assay threshold or medical decision levels. It is important to evaluate the effects of color blindness on the ability to obtain an accurate result. Users should also consist of trained laboratory personnel to provide a benchmark in which to compare non-laboratory user results.
- What performance criteria (statistical or clinical) should FDA apply to the accuracy threshold for a waived test (e.g., t-test or McNemar test at key decision points, description of performance with confidence intervals at key decision points, use of set performance standards using a receiver operator curve-80 percent, 90 percent, 95 percent, or other-at key decision points, and/or others)?
The FDA should minimally require the same performance criteria as those for moderate complexity tests. T-test, McNemar test, regression analysis, and linearity should be used. The threshold should be at 90%. Performance should be at key medical decision points and include confidence intervals as well as sensitivity and specificity.
- How should FDA define precision for purposes of waiver determination? What types of samples, how many and what types of operators/sites are appropriate? Current CDC recommendation is for 20 samples at three levels representing appropriate decision points to be tested at three sites by lay users using material in either artificial and/or real matrices depending on availability and biohazard issues.
At a minimum, the CDC recommendations should be used. Other guidelines such as NCCLS should be consulted for recommendations.
NCCLS Publication EP5A Evaluation of Precision Performance of Clinical Chemistry Devices; Approved Guideline,(February 1999) Total precision is defined as the variability of the device when used over an indefinitely long period. The recommendation for precision evaluation is a minimum of 20 operating days to ensure that total precision is adequately estimated. A preliminary precision test of within-run precision is recommended. Twenty aliquots of an appropriate test material should be assayed in sequence. The standard deviation and coefficient of variation of the results should be calculated. The purpose of this is to identify problems that should be solved before continuing the evaluation. The experiment that this publication then describes is designed to provide estimates of the total precision and within run precision of the device during operation. It does not incorporate other possibly significant sources of variability such as operator differences and multiple instruments, which must be considered when evaluating a device for purposes of waiver determination.
- What performances thresholds should FDA use to determine whether the precision studies are appropriate for waiver status (e.g., ANOVA (analysis of variance) analysis, use of a predefined performance goal, such as Tonks’ formula, or percent agreement out of total repeat runs)?
The same criteria as those used for moderate and high complexity tests. The performance threshold should consider the allowable medical error at key decision points.
- What interference studies are appropriate to establish performance of waived tests (e.g., effects of hemolysis, lipemia, etc.)?
Interference studies should be determined by evaluating the sample matrix and methodology involved in the test. The criteria should be the same as those used for moderate or high complexity tests. When significant interference’s or matrix issues are identified, strong consideration should be given to denying waiver.
- What environmental studies or flex (stress) studies are appropriate to establish performance of waived tests (e.g., temperature or humidity stresses, short fills)?
Environmental or stress studies should be determined by evaluating the methodology used in the test and the type of sites where the test might be used. Again, NCCLS document EP18P outlines the use of a “Source of Matrix Error”which addresses identification of potential environmental issues.
- What additional studies (if any) should be submitted for evaluation of qualitative tests for waiver?
The sensitivity of tests to variations in technique should be evaluated. As an example, if two drops of specimen need to be added to the test device or the procedure needs to be timed for a defined interval, how sensitive would the test be to the addition of a different specimen volume or to an error in timing? Does a readout fade or change if not read at an appropriate time interval?
- What additional studies (if any) should be submitted for evaluation of quantitative tests for waiver?
The same as used for moderate and high complexity testing. Linearity at a minimum of seven points should be considered. Points should be carefully chosen to assure proper weighting (usually evenly distributed).
NCCLS Publication EP10-A Preliminary Evaluation of Quantitative Clinical Laboratory Methods; Approved Guideline (May 1998): Describes a procedure for the preliminary evaluation of linearity, bias, linear drift, and precision of a clinical laboratory method. It is intended primarily for evaluating automated instruments but may be appropriate for kits and manual procedures.
Thank you for the opportunity to present the College’s views. Please feel free to contact me or David Mongillo, Director of Public Health and Scientific Affairs at (202) 354-7110with any comments or questions.
Paul Bachner, MD, FCAP