Registry Studies: Why and How

There is only one difference between registry studies and clinical studies: registry studies are observational and clinical studies are investigational. (When clinical studies are randomized they are called randomized clinical studies or RCTs.) To put it another way, in a registry study we tell the physician to treat the condition however they want—as sponsors, we are passive observers; in a clinical study we instruct the investigator to treat the condition in a certain manner—we are active researchers.

There is another, inevitable, feature of registry studies that I want to point out. The words "effectiveness" and "efficacy" are often misused, even by FDA. Effectiveness refers to how well a device performs as intended in the general population of patients and the general chaos of clinical practice. Effectiveness is measured in registry studies. Efficacy refers to how well a device performs in a setting of carefully selected patients and a carefully controlled protocol. Efficacy is measured in clinical studies.

 

Why do a registry study?
[1] Reimbursement data
One common reason for doing a registry study is to obtain data for reimbursement purposes. While CMS prefers comparative effectiveness data obtained from randomized clinical trials, such studies aren't always possible. 1. Definition of Comparative Effectiveness. There may not be a direct comparator for your new technology: the comparators might be an office procedure versus a surgical procedure or a device versus a drug. Imagine the complexities of comparing a device that emits a magnetic field intended to lower the viral load of patients with AIDS or hepatitis C to a multi-drug regimen. Or co-pay policies may be different for the comparators. Martin et. al. described an issue where a new (possibly more effective) drug costing $2000 per month was to be compared to an older drug being used off-label and costing $50 per month. There was concern that patients assigned to receive the expensive drug would drop out of the study. 2. Martin, et. al. In such a case, the sponsor would pick up the copay for every subject, no matter what their third-party coverage might be, in order to level the playing field.

[2] Post-approval effectiveness publications
Before adopting your technology most clinicians will ask about its effectiveness in the real world. Registry studies are ideal for obtaining effectiveness data. By allowing wide patient selection criteria you will include patients with multiple confounding complications, wide age ranges, various socioeconomic backgrounds, and differing healthcare attitudes. Learning how your technology behaves in these complex scenarios can provide valuable information to clinicians and important data for publications.

[3] Section 522
In certain circumstances FDA may require a post-approval study under Section 522 of the Food, Drug, and Cosmetic Act. The so-called Section 522 Postmarket Surveillance authority is limited to Class II or Class III devices the failure of which might lead to serious adverse health consequences, devices implanted for more than one year, life-sustaining or life-supporting devices used outside a device user facility, or devices used in pediatric populations. 3. Section 522. Registry studies are an ideal way to collect the broad surveillance data required.

[4] Off-label uses
Devices aren't always used the way we think they will be or in the populations we anticipate. In registry studies we—as sponsors—are not dictating how our device will be used, so there is always the possibility it will be used off label. Off-label use in a registry study is not a protocol violation since we don't specify in the protocol how to use the device in the first place. A review of how our device is actually used in real-world practice can provide valuable marketing information, hypothesis development for future studies, and indications for use development for future regulatory submissions.

 

How to do a registry study
Implementing a registry study is not much different than implementing a clinical study. All the basic elements of design, planning, and project management are present. There is a lack of consensus standards for registry studies so it is difficult to find guidance on how to do them. Your primary resource for information will be "Registries for Evaluating Patient Outcomes: A User's Guide, Second Edition" from the Agency for Healthcare Research and Quality. 4. Registries.

[1] Planning
In the planning phase, identify your stakeholders, the scope of data required, define the core data set (what do you need to know?), identify the patient outcomes or endpoints, define the target population (i.e. inclusion and exclusion criteria), and most importantly, get your funding. Funding may come from top management, venture capital firms, government grants, private grants, or other resources, but in the end we are all accountable to someone. Next you'll set up the registry team, determine if safety monitoring boards, IRBs, or other committees are necessary, and finally plan an exit strategy so you'll know when the study is completed.

[2] Registry design
In the design phase, the details of the registry study are worked out and a protocol is written. There are only a few options for study design:

a) Cohort designs follow over time a group of people who possess a characteristic to see if they develop a particular endpoint or outcome. 5. Registries, p38.

b) In case-control designs you gather 'cases' of patients who have a particular outcome or who have had a particular adverse event and 'controls' who have not, and then you look backwards to see what proportion had an exposure or characteristic of interest. For example, in the evaluation of re-stenosis after coronary angioplasty in patients with end-stage renal disease, investigators found both cases and controls from an existing PTCA registry. Alternatively, cases could come from the PTCA registry and controls from outside the registry (say, Medicare data). 5. Registries, p38 and p46.

For example, in 2004 Cordis began a registry designed to assess stenting outcomes in relation to the outcomes of their SAPPHIRE trial, which was used as the historic comparison group. The research question was to see if non-academic physicians would achieve the same level of success as the academic investigators used in the clinical study. The registry was conducted because of concerns by FDA and the Centers for Medicare and Medicaid Services (CMS), and involved 74 sites and 1493 patients. The large number of sites and subjects are characteristic of registry studies. 5. Registries, p38.

c) A case-cohort design is a statistical variant of a case-control study. Controls are sampled from a list of people, with each person having an equal probability of being sampled. 5. Registries, p38.

Help with Registry Studies

CDG is pleased to offer a five-hour workshop on designing and implementing registry studies for medical devices. Designed and recorded by Dr. Nancy J Stark, the workshop is a focused presentation of the AHRQ User Guide, adapted to medical device registries. You can find more information on our website. Scroll down to Registry Studies for Medical Devices.

If you're short on human resources and need to outsource the planning and implementation of a device registry study, please phone or email us at 773-489-5721 or cdginc@clinicaldevice.com. Dr. Stark will be happy to discuss a proposal.

[3] Selecting subjects and comparison groups
The target population is all the patients with a common disease or condition or a common exposure. For example, the target population might be all people with cataracts, all women with urinary incontinence, or all people who have been exposed to radiation for cancer treatment. Then broad inclusion/exclusion criteria are used to select a representative population of patients. One common feature of registries is that they have few inclusion and exclusion criteria, thus enhancing their applicability to broader populations.

Selecting comparison groups is more critical in observational studies than in clinical studies, because subjects have a choice as to which intervention they receive. The sickest patients may choose your technology, while less-ill patients may choose the comparator. The result will be an unfair imbalance in adverse events for the new technology. Key demographic factors—such as age, lifestyle, and disease advancement—are collected and statistically applied to help achieve equipoise.

Comparison groups may be "internal" (data collected simultaneously), "external" (data were collected outside of the registry, such as Medicare data), or "historical" (data collected under the registry protocol but not simultaneously). Comparison groups are essential when you want to distinguish between alternative procedures, assess the magnitude of differences, or determine the strength of associations between groups.

Registries do not need comparison groups when the purpose is to characterize the "natural history" of an intervention.

[4] What data should be collected?
Registry enthusiasts have their own language for many of the concepts we are already familiar with from RCTs. For example, they talk about "domains" of data, and by that they mean data should be collected from the personal domain (patient demographics, medical history, health status, and patient identifiers), the exposure domain (patient's experience with the technology or device), and the outcomes domain (primary endpoints, secondary endpoints, adverse events, and technology deficiencies.) In addition, you should collect information about potential confounders (say a drug being taken to treat the same condition as the study device). The collected data should relate directly to the purpose of the registry.

"Data elements" refers to the exact data that will be collected. Currently there are few, if any, broadly accepted sets of standard data elements for most disease areas, making it difficult to use external data as a source of comparison data. Look to the specialty societies to see if they have created clinical data standards that you can use as a guide for selecting data for collection. For example, the American College of Cardiology has created clinical data standards for acute coronary syndromes, heart failure, and atrial fibrillation. 5. Registries, p53. Whenever possible, tie your data elements to established terminology, such as Current Procedural Terminology (CPT) codes, International Classification of Disease (ICD-10), or events related to device deficiencies. 6. ISO 19218-1.

[5] Data Sources for Registries
Depending on the data sources required, registries may utilize certain personal identifiers for patients to locate the specific patients and link the data. For example, Social Security numbers (SSN), as well as a combination of other personal identifiers, can be utilized to identify individuals in the National Death Index (NDI). What peaks my interest is that data may come from many different sources: outpatient clinic records, inpatient hospital records, laboratory records, billing records, and even payer claims data! Data may come from medical chart abstraction, electronic medical records, institutional or organizational databases, administrative databases, death and birth records, census databases, or existing registry databases. For example, if you are developing a thermoebolization technology for treating liver cancer, you may want to access data from the Registry of Liver Diseases.

[6] Ethics, Data Ownership, and Privacy
The principles of ethics, data ownership and privacy are the same for registry studies as they are for clinical studies. You need IRB approval to conduct the study, HIPAA waiver to access patient medical records, a financial agreement with the institution regarding payments, data ownership and publication rights, and assurances of patient privacy.

Consider the case study of the National Oncologic PET Registry, a registry developed to collect data about PET scans in cancer management with the goal of obtaining expanded CMS coverage for PET scans. The registry was to be conducted at hundreds of hospitals and free-standing PET facilities. The sponsor's believed the registry was not subject to IRB approval because it was being "conducted by or subject to the approval of Department or Agency heads" for the purpose of evaluating a "public benefits or services program." CMS agreed. One week before starting operation the Office of Human Research Protections (OHRP) issued a letter of disagreement. The study was put on hold while the sponsors contemplated the difficulty of obtaining approval from hundreds of IRBs. Ultimately OHRP conceded that only the registry was engaged in research and study needed to be approved by only a single IRB. 5. Registries, p84.

[7] Recruitment
Recruitment of sites becomes a major issue in studies the breadth of registries. Sites must be paid fair-market value for their time and must see a benefit to their operations if they are to join and actively participate in a registry. This is especially true if the registry study is to include community physicians or high-volume specialty centers, as well as academic centers. Community physicians are more likely to participate if the registry is viewed as a scientific endeavor, is endorsed by leading organizations, led by a respected opinion-leader, provides useful self-assessment data to the physician, or helps meet other physician needs such as maintenance of certification, credentialing, or pay-for-performance programs.

Patient recruitment presents the same challenges as clinical studies. The best success comes from recruitment by the patient's own physician. It also helps to communicate that registry participation may help improve care for future patients, to provide written materials in language easily understood by the lay public, keep survey forms short and simple, and provide incentives such as newsletters, reports, and modest monetary compensation.

[8] Data collection and quality assurance
Three sets of documents, together, form the system for data collection. The first is the case report forms, be they paper or electronic. These are the forms whereby data is gathered in the field, entered into coded fields, and transmitted to a data management center. The second is a data dictionary which contains a detailed description of each variable used in the registry. For example, the question may be: "Do you smoke?" And smoking may be defined has having smoked tobacco within the last year. The third is the set of data validation rules. These are logical checks on data entered to look for inconsistencies such as males taking birth control pills.

A data management manual should be developed to define how missing data will be handled, how invalid entries will be handled, how data will be cleaned, and what level of error will be accepted. The manual should describe how data will be tracked and coded, how query reports will be generated and resolved, and how it will be stored and secured. Finally, the data management manual should describe a quality assurance system for data entry and registry procedures.

[9] Adverse event reporting
For device and device procedure registries, adverse event detection, collection, and reporting is the same as adverse event reporting for any other post-approval setting. It begins with the "becoming aware" principle; i.e. the clock for reporting adverse events starts at the moment the investigator becomes aware of symptoms or events reported by the patient or signs such as out-of-range laboratory results reported by a lab, or from the moment the manufacturer learns of an event from an investigator.

Investigators are responsible to report serious injuries to manufacturers within 10 days and to FDA within 10 days if the manufacturer is not known. Investigators are responsible to report deaths to both the manufacturer and FDA within 10 days. 7. 21 CFR 803. Interestingly, if an adverse event occurs with a comparator device the investigator must report the event to the comparator's manufacturer. Manufacturers have 30 days to report deaths, serious injuries and malfunctions to FDA, and 5 days to report events that require remedial action to prevent an unreasonable risk of substantial harm to the public health. Events are logged into the Manufacturer and User Facility Device Experience Database (MAUDE).

[10] Analysis and Interpretation
Statistical analysis of registry data is no different than statistical analysis of clinical data. There are a couple of points that deserve mentioning, though. First, you'll need to determine how closely the actual study population represents the target population. Second, there should exist a statistical analysis plan for how the data are to be analyzed and interpreted. And third, there should exist a plan for how to handle missing data.

Conclusion
Don't be misled, registry studies are not cheap, second-rate clinical studies. They are easily as complex and costly than the exalted RCT. What they are is different. They are observational studies that asses a technology's ability to achieve its intended use in the real world. They are used when alternative technologies don't exist, are outdated, or perhaps unethical.

References
1. Draft definition, prioritization Criteria, and Strategic Framework for Public Comment.
2. Identifying and Eliminating the Roadblocks to Comparative-Effectiveness Research, Martin et.al., New England Journal of Medicine, June 2, 2010.
3. Food, Drug, and Cosmetic Act, Section 522 Postmarket Surveillance.
4. Registries for Evaluating Patient Outcomes: A User's Guide, Second Edition. Agency for Healthcare Research and Quality, 2010.
5. Registries for Evaluating Patient Outcomes: A User's Guide, Draft. Agency for Healthcare Research and Quality, 2006.
6. ISO/DTS 19218-1 Medical devices—Hierarchical coding for adverse events—Event type codes (2010).
7. 21 CFR 803 Medical Device Reporting.

Best Regards,
Nancy J Stark, PhD
President, Clinical Device Group Inc