Laboratory tests with single results are one thing. Tests with multiple results may be more complicated. But DNA sequencing goes orders of magnitude beyond: It’s a laboratory test with millions of results. How can laboratory accreditation inspectors grapple with the quality issues DNA sequencing raises—especially as increasing numbers of clinical labs turn to next-generation sequencing?
To answer that question, last year the CAP appointed a special next-generation sequencing working group to develop a new section of the College’s Laboratory Accreditation Program molecular pathology checklist. The group completed its work last January, the accreditation program quickly won approval from the Centers for Medicare and Medicaid Services for a dedicated NGS section, and the deadline was met to add the new section to the 2012 checklist. As a result, on July 31, the nation’s first standards for accrediting labs that perform next-generation sequencing for clinical purposes were published and took effect.
Adoption of the 18 new checklist requirements dedicated to NGS is a significant achievement for the College. “The College convened a special group of experts to write these checklist requirements before our LAP customers would actually need them. And that, I don’t think, has ever happened before,” says Gerald Hoeltge, MD, chair of the CAP Checklists Committee and a pathologist at the Cleveland Clinic. “Usually we’ve waited until we had a demonstrated need. In this case, the need was anticipated and the requirements were written ahead of time.”
“The publication of this checklist demonstrates CAP’s leadership in advancing standards of practice in genomic medicine,” says Nazneen Aziz, PhD, CAP director of molecular medicine.
The checklist requirements cover the two parts of the NGS workflow: the analytical wet bench process (which includes DNA handling, fragmentation, library preparation, and sequencing), and the bioinformatics analysis, which is needed for sequence alignment, mapping, variant calling, and annotation. Laboratories that wish to be inspected with the new NGS checklist section will need to put the scope-of-service code for next-generation sequencing on the CAP activities menu. “There’s a custom checklist generated based on the activities menu, so if you check off that activity, you’ll get the 18 NGS requirements,” Dr. Hoeltge says.
The NGS checklist items aren’t the only changes made in the checklist this round. Also part of the 2012 edition of the checklists are new instructions to make inspections more efficient. And reagent and quality control requirements have been added to the all-common checklist, Dr. Hoeltge notes. “There are still items about reagents and quality control in individual checklists where there are special requirements for that specialty, but the generic ones have been moved as part of a continuing effort to place more aspects in the all-common area.” Further elimination of redundancy was one of the aims of the latest checklist revision.
Another change: Because of the tremendous interest in the accreditation program outside the U.S., the Checklists Committee has “globalized” the language. “This was done to generalize things that aren’t U.S.-specific and don’t need to be,” Dr. Hoeltge says. “So, whenever possible, we don’t cite specific regulatory agencies in the U.S. Instead we’ll talk about occupational safety, for example, without mentioning OSHA. There are some places where we flag requirements as ‘U.S. only’ or ‘international only,’ but for the most part we’ve globalized the language to make the requirements fully interpretable anyplace in the world.”
It was only a year ago that the NGS working group took on, as its first issue, the fact that NGS was being used as a clinical test in CAP-accredited labs, yet there were no accreditation standards, says Dr. Aziz. She spearheaded formation of the working group, which consists of a mix of staff and CAP members representing accreditation, proficiency testing, and the CAP Transformation program—the College’s multi-year campaign to transform the specialty.
“The era of NGS has been undergoing a translation from use as a research tool to a tool to generate diagnostic results, so there certainly was, and is, a need for checklist requirements, standards, and guidance from the appropriate professional organization,” says Karl V. Voelkerding, MD, the CAP member lead of the NGS working group and medical director for genomics and bioinformatics, ARUP Laboratories, Salt Lake City.
In forming the NGS working group, “We wanted to create requirements that would apply across the spectrum of different NGS technologies, using general principles that would be relevant but independent of which technology one was using in the laboratory,” Dr. Voelkerding says. “Second, we wanted to write requirements that would not be eclipsed or supplanted but would stand the test of time—at least for several years. And that’s a challenge when you’re working with technology that’s undergoing a continuous evolution. The third goal was to add an appropriate level of quality control and assurance but not inordinately impede the adoption and development of these technologies by being overly prescriptive.”
But the task facing the NGS working group was not a simple one. Says Dr. Hoeltge, “Since you’re screening for millions of mutations when you sequence someone’s DNA, and the sensitivity and specificity of each of those individual results will be different, that means there are unique aspects of test validation we’ve never faced before in any other kind of lab test.”
The new NGS checklist requirements are divided into the two workflow sections: wet bench processing and bioinformatics. While many of the items are similar to routine requirements throughout clinical diagnostic testing, several are particularly important in NGS, Dr. Aziz points out.
For example, validation is a standard feature of the accreditation program checklists, requiring that the laboratory run a series of controls to ensure the performance characteristics of the test. “Pathologists often say we are in a constant state of validation and revalidation, and that’s particularly true of NGS,” she says. “Let’s say the laboratory changes a parameter because a new software comes in and it has upgraded its version or downloaded a patch. The lab needs to revalidate the processes to make sure things haven’t changed significantly so that the false-positive rates are altered.”
A separate checklist item (MOL.34948) asks laboratories to have a policy for monitoring and implementing upgrades to instruments, sequencing chemistries, and reagents or kits used to generate NGS data. Monitoring is essential, Dr. Aziz believes. “You have to be aware of upgrades, and be sure you are not using obsolete methods. Because things are being improved on a monthly basis, and new bioinformatics algorithms are appearing, laboratories almost have a duty, if they are running NGS, to scout for upgrades.”
Part of the wet bench process validation requirement (MOL.34936) is that sequencing error rates (that is, false-positives and false-negatives) for variants assayed must be determined and documented using an alternative method. “We know that each NGS platform has its own biases for false-positives. With false-negatives we’ll never know, but Illumina, Ion Torrent, Complete Genomics—all have certain false-positive biases. So the same gene, if sequenced by three different technologies, may not come up with the same variant. That doesn’t mean the variant doesn’t truly exist; it’s just being labeled differently,” Dr. Aziz explains.
“So while setting up the test, the laboratory needs to do what is called confirmation by an alternative method like Sanger sequencing, or it could be it uses even a different NGS platform—say Illumina as an alternative method for Ion Torrent.” The NGS working group deliberately made the requirement nonspecific as to the method.
But discussion of further refinements of the validation requirements is already underway, Dr. Voelkerding says. “The checklist requirements include different parameters, but there are always nuances and additional levels of detail in terms of how to approach validation, so our working group is actively considering how we can provide additional information to laboratories that will facilitate this.”
NGS opens up one particularly thorny area that will require careful thought: what to do with unrelated clinically significant genetic findings, Dr. Aziz says. “When you are conducting large-scale sequencing, you might find out for someone being tested for, say, a genetic hearing disability that they have Huntington’s disease or a risk mutation in one of the BRCA genes—even though the patient was not being tested for that. It is likely that incidental findings like these will happen because you are looking at far more than just a focused gene mutation when you do NGS testing.”
The informed consent implications are considerable. Some patients may not want to know about a non-actionable disease gene, or they may not even want their physicians to know about it. But the accreditation program does not typically get involved in informed consent, so the committee decided not to stipulate a policy, but to require that the laboratory have a policy for dealing with such findings (MOL.34954). “It means that you have to have a documented policy that you thought about how to deal with incidental findings,” Dr. Aziz says. Laboratories conducting NGS are urged to develop a policy with the assistance of a genetic bioethicist, says Dr. Hoeltge. “And it logically would have to include a discussion with patients ahead of time, as to what they would want the lab to do if a finding wasn’t part of a test.”
The bioinformatics elements on the NGS checklist are also important and illustrate the unique challenges NGS poses. In checklist item MOL.34958, the laboratory is required to document the bioinformatics process or pipelines it uses to support the analysis, interpretation, and reporting of NGS-based results. That includes all the algorithms, software, scripts, database packages, reference sequences, and databases, whether in-house developed, vendor-developed or vendor-supported, or open source.
In the required validation of the bioinformatics processes, pseudogenes are listed as an example of a potential source of error that laboratories must try to stem. Under checklist item MOL.34960, interference by clinically relevant pseudogenes and other sequences highly homologous to the target must be determined and documented. “Pseudogenes are closely related to the real gene, so if, for example, you are offering a 50-gene panel for a certain disease and one of the genes within this panel has a pseudogene, the laboratory needs to be aware of it and deal with it in a certain way, or exclude it from the analysis. If you’re not aware some genes have a pseudogene counterpart, then you could accidentally create problems by analyzing the pseudogene and find variants that are not relevant,” Dr. Aziz explains.
Because of the enormous volume of data NGS creates, data storage is another bioinformatics issue for which laboratories must have a policy. “We’re not prescribing, but we’re saying you have to have a policy for how long the data are retained, in what files or formats, and when they can be purged,” Dr. Aziz says. As impractical as it may seem, some states may still require that laboratories keep their raw data for 20 years, so laboratories need to be aware of state, local, and national requirements. Data storage is an area that she expects to evolve as the state of the art advances. “As we get more knowledgeable over time and confidence in our data, we may determine that only the variant file could be stored without needing to store the short-read files.”
Still, labs performing NGS are finding they just don’t have the storage capacity, so they may consider storing data with Amazon and other cloud computing vendors. Many of the confidentiality issues raised by storage of data in the cloud, which is becoming more and more routine, remain to be resolved, especially since the Health Insurance Portability and Accountability Act of 1996 far predates the emergence of cloud computing. For the time being, however, the NGS checklist includes a data transfer confidentiality policy requirement, asking that labs have procedures in place to ensure reasonable confidentiality and security of internal and external storage and transfer of sequencing data.
How many laboratories are conducting next-generation sequencing for clinical diagnostics? No firm figures are available. “We believe there are about 40 or 50 doing NGS clinical testing and many more for research purposes,” Dr. Aziz says. But that number is likely to grow. Because supplies or disposables for NGS cost only a few thousand dollars, Dr. Hoeltge says, “I’ve been told that the number of labs that are likely to be using this is going to be very large.”
“Everyone agrees it’s going to be a huge part of diagnostic medicine,” Dr. Aziz says. “So I am very proud that we took the lead and created a framework by which NGS can be used for clinical testing without being too prescriptive; we created the first set of standards in the absence of any standards, and we were very judicious in not making it too prescriptive.”
The NGS working group is continuing to work on other NGS-related issues such as proficiency testing. The plan is to update the NGS checklist frequently, perhaps starting with more items in oncology or infectious disease. “I expect we’ll update every year because the field is evolving rapidly and so much change is happening,” Dr. Aziz says. “As the checklist is applied in the field and being used by inspectors and laboratories, we will probably see the need for new requirements and modifications.”
Amid the flurry of standard-setting activity, it’s easy to forget just how recently NGS arrived on the scene. “The first publication using NGS that occurred in basic biomedical research literature is in 2005,” Dr. Voelkerding points out. “So we are seven years out from the first basic science publication, and five years ago the technology was still too young to seriously consider it for diagnostic application. So I think the effort by CAP to establish a working group whose first charge was to establish NGS checklist requirements was very timely, and shows a good element of foresight as more laboratories move into this area.”
Anne Paxton is a writer in Seattle.