Literature Evaluations for Biological Safety

A Whitepaper by Dr. Nancy J Stark

During the course of product development there are three times when you should do a literature evaluation: 1) to justify animal use, 2) to justify a clinical trial design, and 3) to apply for certification and CE Mark or Medicare reimbursement. I've written previous whitepapers about doing literature evaluations as a part of clinical evaluations. In this whitepaper I'll do a brief overview and then look at literature evaluations to justify animal use.

Three literature evaluations
Selective Clinical Evaluations
ISO 14155 requires a clinical evaluation to justify the design of a clinical trial. 0. ISO 14155. A European colleague showed me several literature evaluations that were done for this purpose. The most important point I noted was that they focused only on the selected indication being investigated, and resulted in a relatively short, compact literature review utilizing only 5-10 publications.

 

Comprehensive clinical evaluations
The literature evaluations intended for certification and CE Mark are supposed to be comprehensive, "The clinical evaluation is based on a comprehensive analysis of available pre- and post-market clinical data relevant to the intended use of the device in question…." 1. MEDDEV Clause 5.5.1. However, CDG has been successful with Notified Bodies by averaging ~40 articles per review for Class IIa and IIb devices. The number of articles reviewed depends on the number available, the quality of the articles, the age of the technology, the volume of competition, and other similar factors. I observed one European Class III device that got certification and CE Mark based on a review of only 9 case series and 7 publications.

Literature reviews for Medicare reimbursement application should be larger and will be more persuasive if they follow the examples found at www.ahrq.gov. The Agency for Healthcare Research and Quality (AHRQ) has been publishing technology assessment literature reviews for years. If you look at the technology assessment for negative pressure wound therapy (NPWT) devices, you'll see that 1078 citations were identified, 288 full-text articles retrieved, and 137 articles reviewed. There were also more than 1400 items submitted for review by independent "interested parties". The authors point out that none of the studies compared an NPWT system head-to-head to another system. 2. Negative Pressure.

Similarly, AHRQ recently released for comment a comparative effectiveness review of devices used in diagnosing cardiac disease in women, including stress electrocardiography, echocardiography, single proton emission computed tomography, cardiac magnetic resonance, and computed tomography angiography. A total of 7163 citations were identified, 1452 full-text articles retrieved, and 101 articles (98 studies) reviewed. The authors note that none of the studies were randomized controlled trials. 3. Diagnosing CAD. This review is currently open for public comment and device manufacturers should absolutely submit publications and data based on research they have done.

In 2010 AHRQ received $500,000,000 from the federal government to prepare comparative effectiveness reviews on a wide variety of products. These reviews will surely affect the availability of new and old technologies to the American patient as Medicare revisits its reimbursement decisions based on these reviews. The message to device manufacturers is that we will have to rethink how we design clinical trials and be prepared to submit our own clinical evaluations if we are to obtain reimbursement in the American marketplace.

Literature evaluations for biological safety
I haven't yet written about literature evaluations to justify the use of animals for biological safety testing or preclinical animal testing. The beauty is that the way you un-justify the use of animals is by demonstrating the work has already been done and does not bear repeating—saving you costly resourses and the unnecessary use of animals. Let's look more closely at literature evaluations to justify (or un-justify) animal tests.

How to Write A Clinical Evaluation—CD Workshop

A new CD workshop is now available on "How to Write a Clinical Evaluation". This well-researched presentation takes you step-by-step through the complicated process of writing a clinical evaluation by making it organized and understandable. The 100+ content slides and supporting handouts explain the logistics of clinical evaluations, how to define objectives and key questions, identify and retrieve abstracts, filter and screen abstracts to identify articles, filter again to zone in on the articles that are applicable, assess and weight the literature, and write the evaluation to address the key questions originally posed. The presentation discusses the three major purposes for evaluations, discusses their regulatory foundation, and proposes a strategy to allow each evaluation to build upon the previous ones. Finally, some average metrics are disclosed.

If you want a learning guide to help you write a clinical or literature evaluation, this workshop is the place to turn. A procedure, protocol, and template—meeting the requirements of MEDDEV 2.7.1 r3—are available for a small, additional cost. Click here to purchase or more information.

Authority—who says so
Take out your current copy of ISO 10993-1 "Biological evaluation of medical devices—Part 1: Evaluation and testing within a risk management process" (2009) and turn to Annex C, "Suggested procedure for literature review." 4. ISO 10993-1.

The introduction tells us, "A review and evaluation of the literature is essential for justification and planning of any biological evaluation of a material or a medical device. The aim of such a review is to determine scientific background for the biological evaluation. It also provides essential information for assessing risks/benefits and achieving the ethical conduct of the planned evaluation as required by ISO 10993-2.

NOTE Such a literature review can be helpful to assess whether the relevant data available in the literature are sufficient to demonstrate biological safety of the device in question without the need to generate further data from actual testing or to conclude that the available data are not sufficient.

Performing a literature review is a scientific activity that should be done with rigor and objectivity, and
should allow verification by third parties."

Methodology
[1] Establish a plan
The methodology, taken from Annex C, sounds a lot like the methodology described by GHTF Study Group 5 and MEDDEV 2.7.1 r3. The first step is to establish a plan or protocol for the identification, selection, collation and review of all available studies and data. The plan should be documented in writing and based on recognized practices for the systematic review of the scientific literature.

[2] Define objectives or key questions
The objectives of the literature evaluation should be clearly defined up-front. Sometimes it is easier to phrase the objectives in the form of key questions, for example, which, if any, animal safety tests from the G95-1 Matrix should be performed? 5. FDA BlueBook G95-1. Does the chemical characterization of the materials justify not doing animal safety tests? Do literature data satisfactorily explain the negative outcomes of certain animal tests? Do literature data justify the design of preclinical animal tests, selection of species, route of exposure, duration of exposure or follow-up, or other design parameters?

[3] Document sources and selection criteria
The next step is to identify the sources of literature and other databases that will be searched. The selection will depend on what information you need. For example, you might need evidence regarding the safety of chemicals such as additives or adulterants, information about chemical properties such as glass transition points or vaporization, data about adverse effects related to the device or technology, published studies about in vivo animal use, or information about 'active water' concentration. The type of data you need will influence where you look for it.

You will probably need to contact suppliers directly (called manual searching) to ask for specific information. For example you might need, say, formulation or processing information that isn't published.

In addition, you'll need to specify criteria for the data. For example, you might accept data from rodents but not from fish, you might require data from certain bacterial strains, large mammals, or ex vivo studies. You might need data with a certain sensitivity, such as parts per billion rather than parts per million. I find it easier to specify what I need than what I don't need until I have a feeling for what kind of information is out there.

[4] Assessment of documents
The documents are then valued in such a way as to group them by low, moderate, and high quality, depending on pre-determined assessment criteria; and then the values for each criteria are summed to give the document a weight. It may be that some high quality documents are unfavorable to the objectives, but these documents must be included in the evaluation nonetheless.

A 'document' may be anything from a MAUDE report of an adverse event to a full-text published article to information provided directly by a supplier. Documents might be assigned a value on: 1) the similarity of the device in the document to the device under consideration, based on technology, critical performance, design, and principles of operation; 2) the relevance of the particular experimental animals used in the selected studies for the biological evaluation of the device under consideration; and 3) conditions of use of the material or device in the selected documents and the intended use of the device in question.

Documents might also be valued on, 4) whether the author's conclusions are substantiated by the data, 5) whether the document reflects the current medical practice and state of the art technology, 6) whether documents are taken from recognized scientific publications and whether or not they have been reported in peer reviewed journals, and 7) the extent to which the data were acquired through established scientific literature.

[5] Critical evaluation of the literature
Note that valuation means the act of setting the value of something, while evaluation means a diagnosis or diagnostic study based on those values. 6. Dictionary.com.

Applying these definitions to literature reviews, we can say that we assess a document through the process of assigning it a separate value for each of several criteria, we weight the document by adding up the values, and we evaluate the body of documents (the literature) by determining if they address the key questions, placing the most emphasis on the document with the highest weights. The idea is to evaluate high quality data whether or not it is favorable to our technology.

The hoped-for outcome is to conclude than no new animal testing is justified, but it may be that we identify that new animal testing is indeed needed.

[6] Format and content
The literature evaluation itself may follow the standard format and content of other literature reviews.

[a] A short description of the material and device, including its intended use. I like to include the European and US classification, if known.
[b] Discussion of whether or not the literature supports the objectives or key questions.
[c] Evaluation of the hazards, risks, and appropriate safety measures that should be taken by the manufacturer.
[d] Description of the method of document selection, reasons for omitting documents, and methods of weighting documents.
[e] List of documents (full citations) used in the evaluation and appropriate cross-references.
[f] Conclusion with a justification for whether additional animal testing should be performed or is not necessary. The conclusion should include a discussion of probable risks and benefits, considering the 'state of the art' of the technology. And the conclusion should identify any gaps in evidence regarding safety or performance that should be addressed by future testing.
[g] The printed names, signatures, and responsibilities of the reviewers, and the dates of signature. I recommend attaching a biosketch or curriculum vitae of the reviewers as evidence of their experience.

Conclusion
Literature evaluations for biocompatibility are used to determine the need for animal biological safety testing or preclinical testing. Documents both favorable and unfavorable to the technology should be reviewed, but only those documents of the greatest assessed value need be evaluated. Naturally, the hoped-for outcome is that no further testing is needed.

The more common situation, however, is for the manufacturer to do a bunch of tests and then turn to the literature to defend an unfavorable outcome. While I'm not a proponent of this strategy, it is wiser to turn to the literature to explain an unwanted test result than to do even more animal testing. If you have unfavorable animal test results, find a knowledgable medical device toxicologist to assess what is currently known about your material, technology, or device, and then do whatever testing is still needed.

References
0. ISO 14155 "Clinical investigation of medical devices for human subjects—good clinical practice" (2011).
1. MEDDEV 2.7.1 r3 "Clinical Evaluation: A guide for manufacturers and Notified Bodies," (2009); clause 5.5.1.
2. AHRQ "Negative Pressure Wound Therapy Devices", 2009, http://www.ahrq.gov/Clinic/ta/negpresswtd/references.htm#ref65.
3. AHRQ "Comparative Effectiveness of Noninvasive Technologies for the Diagnosis of Coronary Artery Disease in Women", 2011, http://effectivehealthcare.ahrq.gov/index.cfm/research-available-for-comment/comment-draft-reports/?pageaction=displaydraftcommentform&topicid=202&productid=770&documenttype=draftReport.
4. ISO 10993-1 "Biological evaluation of medical devices—Part 1: Evaluation and testing within a risk management process" (2009).
5. FDA BlueBook Memo G95-1, "Use of International Standard ISO-10993, 'Biological Evaluation of Medical Devices Part 1: Evaluation and Testing'" (1995).
6. Dictionary.com, http://dictionary.reference.com/browse/evaluation.

Best Regards,
Nancy J Stark, PhD
President, Clinical Device Group Inc

 

Comments

[Bjorn Sowden]

An interesting article, but I have one problem with ISO 14155:2011, as it apparently combines parts 1 & 2 of the original 14155 standard, but fails to mention Clinical EVALUATION even once (indeed the only reference - in section 7.3 - was an error which was corrected in a subsequent Technical Corrigendum).
Whilst the need for, and process of, a Clincial Evaluation was described in section 4 and Annex A of ISO 14155-1:2003, this is no longer metioned in the 2011 version. If one was being pedantic, one might say that this leaves MEDDEV 2.7.1 r3 stranded, as it references a defunct standard, and a requirement that no longer exists in the current standard.
Lacking any preamble, ISO 14155 dives straight into the process of Clinical Investigation, starting with Ethical Considerations, apparently not requiring any justification to do, or not to do, such an investigation . . .
Or have I missed something ?
Some insight on this matter would be welcome.

[Nancy J Stark]

Hello Bjorn,
This is a good question, I feel the discussion of clinical evaluations is much too well hidden--many people have trouble finding it.

Look at clause 5.3 in ISO 14155 (2011). It reads, "The justification for the design of the clinical investigation shall be based on the evaluation of pre-clinical data and the results of a clinical evaluation...."

Hope this helps, Nancy

[Marge Walls-Walker]

Nancy,
Nice summary of principle of 10993-1; however we need to remain mindful that many APAC Regulatory bodies and a few in South America AND on ocassion our own FDA do not fully subscribe to the literature/risk assessment/risk management approach to biological safety even though it has been the underlying principle since the first edition. As long as the Table exists and its "easy", the efforts to reduce/eliminate animal testing based on fidings in the literature and toxicological databases is going to remain an uphill struggle.
Best