Medical Device Link.

Originally Published MX March/April 2004

GOVERNMENTAL & LEGAL AFFAIRS

Using a benchmarking metric, medtech companies may be able to measure their level of FDA compliance.

Bradley Merrill Thompson

Lately, as companies work to appraise their performance on a variety of levels, status metrics in the form of executive dashboards have received a lot of attention. Companies well aware of the adage "What gets measured gets done" are increasingly looking for quantitative rather than qualitative metrics by which to measure performance in various areas.

Drug, medical device, and biological product manufacturers, however, have had to struggle to find a way to assess broadly and quantitatively their general compliance with the laws and regulations administered by FDA. Historically, FDA-regulated enterprises could measure their compliance only in simple terms of high, medium, or low; and assessments often have been restricted to whether a specific operation was in compliance with a specific regulatory requirement. These limitations have caused medtech company regulatory and quality assurance personnel to have to ask management to invest in compliance-related systems without a way to gauge whether the investment ever pays off.

Clearly, a standardized, quantitative metric for determining degrees of compliance in FDA-regulated industries is long overdue. Such a metric can be formulated, but companies will have to work together to achieve this goal.

The Uses of a Metric

The need for FDA-regulated companies to comply with applicable laws and regulations is well documented and understood. Studies have shown that compliant companies tend to have higher market valuations than noncompliant companies.^1,2 Moreover, multimillion-dollar settlements resulting from violations of FDA regulations and the news of corporate executives facing jail time cast the need in harsher light. Indeed, drug and device companies have paid billions of dollars in fines over the past three years.³ It goes without saying that an enforcement action can hugely disrupt a company's business.

A metric that enabled a company to measure its overall FDA compliance status could help it institute preemptive corrective measures to avoid susceptibility to costly and reputation-damaging enforcement actions. As part of the mitigation of that risk, company managers must decide whether to invest in compliance programs and technology, and what type of investment to make. It is important that executives be able to assess the degree to which a particular investment would or does achieve its intended purpose so that they can determine the advisability of related future expenditures and even of continuing the original program.

For example, a quantitative metric might be particularly useful in helping companies to determine which information technology systems are most beneficial for attaining compliance. Executives often must decide whether to invest perhaps millions of dollars in process validation systems, new corrective and preventive action (CAPA) systems, training programs, and design control processes. A compliance performance standard that shows the impact of those investments over time and also identifies the areas where the company most needs to improve can make it less likely that manufacturers will spend money on the wrong technology.

Moreover, under state and federal law, corporate boards have a legal responsibility to assess and oversee their company's overall level of compliance. In this regard, the corporation statutes of most states include provisions taken from, or have been significantly influenced by, the Model Business Corporation Act of 1984 and its amendments.⁴

The well-publicized financial disasters and management misdeeds of a few prominent corporations in recent years, and the ensuing public outcry crystallized in the adoption of the Sarbanes-Oxley Act of 2002, have delivered companies—especially those that are publicly held—into a corporate governance environment that focuses unprecedented scrutiny on, and raises expectations for, board accountability. More than ever, therefore, corporate boards could use an effective, reliable compliance measurement tool to satisfy their fiduciary obligations and reduce the potential for fines being levied against their organizations.

Finally, from a management standpoint, a valid compliance metric would equip the CEO with a motivational tool for staff members who have a role in the compliance function. Companies routinely use profitability figures and other measures to encourage employees to pull together as a team to achieve a common goal. A compliance metric would similarly give managers a tool for making sure that employees give compliance its fair share of attention.

Qualities of a Useful Metric

Any compliance metric devised to address these needs must provide a comprehensive and accurate assessment while allowing for updating to maintain its effectiveness as company circumstances evolve. Specifically, it must be:

• Quantitative.
• Broad and comprehensive.
• Benchmarked to other firms.
• Weighted by the severity of the noncompliance.
• Easy to update.
• Accurate.
• Validated.

Quantitative, Broad, and Comprehensive. The metric must be quantitative; that is, it must give numerical values to specific measurable operations. It should also be broad enough to cover all aspects of compliance with FDA requirements. In addition to particular regulatory requirements, the metric should measure the adoption of best practices pertinent to FDA regulations. These best practices are not those that are specifically required by the regulations, but rather those that have garnered industrywide recognition as being likely to enhance a company's ability to comply with FDA requirements. An example would be whether a company has adopted Web-based training systems for new employees who need to understand the quality system requirements.

The metric should also measure the degree to which the compliance system adopted by a company is effective beyond merely the regulatory requirements specific to FDA. For example, it would indicate whether the CEO has created an environment favorable to compliance generally by taking all available opportunities to reinforce the corporate expectation that no employee should engage in or tolerate noncompliant practices of any sort. Most companies these days have a comprehensive compliance program that addresses all of the legal requirements imposed on the company.

Compared to Similarly Situated Companies. A measurement of compliance, like a school exam, can either be based on an absolute scale or reflect a curve of relative performance.

An absolute scale offers the advantage of sharing the nature of the law, another absolute system. From the standpoint of law, a company's performance is assessed strictly according to whether it meets the statutory test. Yet, in many instances, the FDA regulatory neighborhood lacks exactly defined speed limits, and, to use another metaphor, there is nothing in the nature of a radar gun that can measure compliance with great precision. An absolute scale depends on having a very precise statute that establishes concrete rules and a performance measurement tool that can precisely measure company compliance.

The fuzziness at the edges of FDA regulation makes comparison with the conduct and performance of other companies an important benchmarking measure. Moreover, from a policy standpoint, a metric analogous to grading on a curve would give companies an incentive to improve continually over time. Those enterprises finding themselves below the average would be induced to raise their level of compliance and thereby lift the industry average. Meanwhile, companies near or even well above the average level of compliance would not be complacent as other companies worked to improve their compliance performance; they would likely strive to maintain their position as very compliant enterprises.

Elements of Noncompliance Weighted. The metric should also mathematically reflect the perspective of FDA in distinguishing a serious case of noncompliance from one that is less serious. This is essential if a single metric is to encompass different types of noncompliance. It is like using a common denominator to add fractions. For example, using the database of historical FDA enforcement actions to establish the weighting, the metric could enable company leaders to arrive at a meaningful measure of their company's cumulative degree of noncompliance across all operations.

Easily Updated. So that the metric will remain timely and indicative of a company's current state of compliance, it should allow updating with only a modest amount of effort. Regular updating will enable the corporate board and upper management to use the metric to make decisions that are grounded in current data.

Accurate and Validated. Finally, the metric must be accurate enough to be useful for these purposes and merit the confidence of those who employ it. The weight attributed to various elements of the metric should be validated through analysis of data available from FDA and from entities in the business of assessing company compliance. Any metric for application by medtech companies should be developed by means of a consensus process within the industry, and presented to FDA.

A Compliance Index

To meet the need just outlined, the medtech industry can import a concept used in the social sciences—that of an index. To understand how an index works requires looking no further than the Dow Jones Industrial Average. The Dow is useful in large part because it has been measured over a long period and changes in it can be observed. While it does not directly measure the performance of all stocks, it has proved to be a valuable barometer of overall stock market performance.

An index of this type does not try to establish an absolute or true value. Rather, its purpose is to establish a gauge that makes possible meaningful comparison of the performance of companies over time and at a high level. Indexes carry a potential for bias (see sidebar, above). However, so long as the index is not read as more than it is, the possibility of bias should not frustrate its purpose.

An FDA-compliance index could seek to identify a variety of surrogate markers designed to estimate the degree of a company's compliance over time. Like the various stock market and other economic indices, this compliance index would be only an educated estimation of reality. But, as with the others, the elements that form its basis could be carefully chosen and weighted to be representative of overall compliance. Through standardization of those elements, the index could serve as a reliable barometer of a company's compliance over time and as a basis for comparing the level of compliance among companies in a particular medtech sector.

Drug, device, and biologic companies thus should consider using an index to gauge their overall compliance with FDA regulations and their associated risk of susceptibility to FDA enforcement action.

The formula used to derive the compliance metric would have to be quite detailed. However, in a nutshell, the index should be a composite of disparate data from four general sources: prior company audits, company compliance interviews, FDA inspection assessments, and regulatory quality data.

Prior Company Audits. By regulation, companies are required to audit key functions periodically.⁵ These audits are conducted using internal personnel, outside consultants, or both, usually on an annual basis. They might cover such areas as design controls, clinical trials, corrective and preventive actions, complaints, medical device reporting, and management controls. The index should catalog audit reports by their scope and quality and categorize audit observations by type and significance.

Company Interviews. Company audits, while important, do not tell the whole story by any means. For example, most auditing focuses on the quality system and leaves unexamined such other important regulatory compliance matters as data integrity and marketing claims. To capture these other data, as well as softer issues such as the company's incorporation of best practices and its overall compliance culture, the index should rely on surveys of individuals within the company who have compliance responsibilities, including lower-level employees. Each survey should consist of a written component followed by an oral interview. All data collected through the survey should then be averaged and combined according to the formula (see sidebar).

FDA Inspection Assessments. The third area from which the index should gather data is prior FDA inspections. In theory, because FDA has the responsibility to enforce applicable laws, this could be more important to the issue of compliance than any information that comes from within the company. But FDA's inspectional experience of a given company is usually far narrower than the company's examination of itself, and usually represents only a small sliver of the company's compliance picture.

The index should examine this area by collecting the results of FDA inspections for the company, dividing the inspectional observations into major and minor ones, and aggregating the observations via a mathematical formula. It is particularly important to assess FDA's industrywide practice in order to determine whether the company was the recipient of more or fewer observations than other similarly situated companies.

Regulatory Quality Data. The final data resource category should be comprised of assessments of the company's device complaint record, its adverse-incident-reporting experience, and the recalls and other corrective actions that have been its obligation. In this area, the index should collect quantitative data not just from the company but also from FDA's databases in order to produce industry averages. The index should be designed to acquire sufficient sales and other data to create an appropriate context for these quality data at a high level.

Scoring

Once data from the four areas described here have been examined, subscores for those areas should be calculated and then combined into an overall base score. A reasonable weighting of the subscores as components of the base score is prior company audits, 35%; company interviews, 25%; FDA inspection assessments, 25%; and regulatory quality data, 15%. Data from the company's own audits should be given the greatest weight because those audits should be the broadest and most systematic investigations into the company's state of compliance. The employee surveys and FDA inspectional experience should receive slightly less weight because both of these sources are limited in terms of scope and reliability. Regulatory quality data should carry the least weight because, notwithstanding the probative nature of those data and the fact that the index attempts to put the data in context, there are clear limits on their meaningfulness.

In addition to the base score, which provides an overall qualitative indication of the company's compliance, the index should incorporate an examination of certain compliance risk factors such as past FDA enforcement actions, the nature of the company's products, and certain business practices such as acquisition activity.

Moreover, when the results are presented, the company should also receive an assessment of the areas with the greatest potential for improvement. The index results should also include a detailed statistical analysis and a database, organized by the four areas that contribute to the base score, that can be mined for more information (see sidebar).

Index Side Benefits

The index would consist of more than a detailed, weighted formula. It would also incorporate features designed to facilitate efficient collection and meaningful evaluation of data.

One is a standardized report form for internal audits that companies should adopt to speed up the assessment process and make detailed comparisons easier. Such a form would enable an audit's scope to be evaluated by means of a standard code, and would help to link audit observations to subsequent corrective actions. Observations could be coded by type and identified as critical, major, or minor with reference to a guidance.

This standardization would support updating of the index quarterly, an interval that would allow changes in compliance status to be discerned in a timely manner.

As just mentioned, to be truly useful a compliance index should be a tool by which to create a database that should enable benchmarking among companies in the industry. Each company's data generated by index-based analysis should be sanitized to remove the company's identity and then added to the database. Over time, as more and more companies participated, the database would become increasingly robust and the benchmarking comparisons more meaningful.

Cautions

This FDA-compliance index gets into some very sensitive areas. Therefore, the index program should include a confidentiality agreement that spells out the limitations on use of the data.

Loss of Confidentiality. There is, of course, a risk that FDA or a private plaintiff might seek access to a company's analytical results. A company wishing to ensure the confidentiality of the process could ask its attorneys to conduct the analysis under privilege, but even that approach is not entirely risk free. For example, even though FDA has said that it will not seek access to compliance auditing reports in most circumstances, the agency has occasionally required a company to waive the privilege in the context of settling an enforcement action.⁶

Some company executives may wonder whether they would be better off not knowing about their company's compliance levels. Ignorance could be a valid policy, as it were, for companies unprepared to work to correct whatever the index may uncover. However, managers who want to control their company's regulatory fate will probably want to risk exposing deficiencies.

Cost. Clearly, any corporate assessment involving the amount of information envisioned here would require a great deal of time to evaluate, and thus impose a considerable cost burden on the company. Developing a standard auditing report format, as described above, and automating certain elements of the data calculation are ways to reduce the expense. The cost would be greatest for the first assessment, of course. After that, much of the information would simply be updated.

Costs to the company would include claims on the time of certain company personnel as well as out-of-pocket expenses. Employees would need to devote effort to gathering some of the assessment data, completing the written questionnaire, and participating in oral surveys. Several hours might be taken from a few employees, while the time investment for others might be an hour or less.

Conclusion

To some, treading in these waters may sound like a risky proposition. The issues are sensitive, and companies may not like what emerges from the process of analysis. But on balance, good things can be achieved.

A compliance index would enable company executives to make informed investment decisions and address risk, enable the board of directors to meet its fiduciary duty to monitor compliance, provide a metric for employees to rally around, and make personal performance more accountable. Such an index, albeit clearly imperfect, could be designed to supply the most accurate possible overall picture of any company's compliance status, and to do so in a way that allows meaningful comparison with industry benchmarks.

References

1. Global Investor Opinion Survey, 2002 [on-line] (Washington, DC: McKinsey & Co., 2002 [cited 16 November 2003]); available from Internet: www.mckinsey.com/practices/corporategovernance/index.asp.

2. Paul A. Gompers et al., "Corporate Governance and Equity Prices," Working Paper No. 8449 (Cambridge, MA: National Bureau of Economic Research, 2001).

3. "Protecting the Public Health: FDA Pursues an Aggressive Enforcement Strategy," FDA White Paper (Rockville, MD: Office of Compliance, FDA, 2003).

4. Model Business Corporation Act of 1984, as amended (2000), sects. 8.01 and 8.30.

5. Code of Federal Regulations, 21 CFR 230.34 and 21 CFR 820.22.

6. "FDA Access to Results of Quality Assurance Program Audits and Inspections," in Compliance Policy Guide (CPG 7151.02) (Rockville, MD: Office of Regulatory Affairs, FDA, 1996), sect. 130.300.

Bradley Merrill Thompson, Esq., is a partner and chair of the drug and medical device law practice at the law firm of Baker & Daniels (Indianapolis). He is also chairman of the board of Aventor, a global, multidisciplinary consulting firm that helps medical technology companies and investors navigate regulatory and reimbursement requirements worldwide.

Copyright ©2004 MX