Medical Device Link.

Originally Published MX September/October 2003

BUSINESS PLANNING & TECHNOLOGY DEVELOPMENT

An audit that examines complaints deemed not worth investigating can expose system weaknesses and improve compliance. 

Mark Allen

No medtech company enjoys receiving product complaints, of course. Nevertheless, besides being necessary for maintaining compliance with industry regulations, complaint tracking is an essential tool for improving a manufacturing company’s performance. It is critical that the system for receiving and evaluating complaints be effective. But appraising the effectiveness of the complaint-handling system while continuing to meet the various expectations of company stakeholders —customers, employees, and regulators—can be a challenge.

The performance of a complaint system should be judged against stated objectives. These performance objectives can be formal or informal, but they should be well defined and documented. 

Over the short term, complaint system objectives may seem not to be in sync with other of the company’s quality metrics. For example, making improvements in the complaint-handling system may cause the volume of complaints to go up incidentally because the process by which complaints are evaluated has become more sensitive. That is a good thing. What is important is that product quality and performance continually improve over the long run.

This article outlines a method that medtech company managers can use to determine how well their complaint-handling systems process incident reports into formal complaints and then, as necessary, into corrective action. A self-appraisal audit focused on key performance areas can optimize system compliance and thus lower the likelihood of regulators citing complaint-management inadequacies.

The Effective Medtech Complaint System

A complaint-handling system ought to:

• Align with the company’s strategic plan.
• Be compliant.
• Provide benchmarks for improvement.
• Serve operational purposes.
• Evolve with changing circumstances.

If a company is client-focused, its complaint system should also be client-focused. Likewise, if it compliance-focused, its complaint system should be too. Executives should ensure the effectiveness of the complaint system by designing it to help fulfill the strategic vision of the company.

Any medtech company must comply with applicable domestic and foreign regulations. Its complaint-handling system is subject to specific regulatory and quality requirements.

Whenever management undertakes to measure some aspect of company performance, it should use the results to effect improvement. This holds true whether the beneficiaries would be employees, customers, or shareholders. The measure arrived at is simply the benchmark for improvement, and should not be used for the nonproductive purposes of assigning blame or distributing punishment.

Quality managers are obliged to maintain control over the measurement system so the performance metrics are relevant to business operations. Undertaking to determine the effectiveness of a complaint-handling system is not much good if the business does not derive some additional value from the effort beyond having an assessment of system effectiveness.

Company leaders should not allow one method of complaint handling to become enshrined. Things important today may not be important a year from now. Old benchmarks that are no longer relevant should be discarded. In creating a corporate clinical affairs department to move a new product to market, the company might have overlooked establishing a mechanism for reporting problems encountered in clinical research to the complaint-handling system. Similarly inadequate linkage can follow the integration of another company through merger or acquisition.

These are important points to consider when setting out to develop a quality audit to determine complaint system effectiveness.

Approaching Complaint System Appraisal

Designing a complaint-handling system is one thing, but appraising its effectiveness is something else. Because the number of complaints received over a given period of time is an easy measure to obtain, some companies use it as the primary measure. That can be a mistake. Complaint rate variation is not always directly related to an issue of product quality or performance, nor is it within the control of those who manage the complaint-handling system.

The complaint rate is a mirror of the health of the company; the complaint system is merely a monitoring device. If a patient has a fever, the physician does not adjust the thermometer to achieve a normal reading, but rather treats the patient. The complaint rate in itself does not meaningfully measure complaint system effectiveness. This is because complaint volume is affected by many variables. Some are internal; they include, for example, information collection systems such as the complaint-handling system itself. Others are external variables, such as sales volume. If a company improves its complaint-tracking system, its reward is likely to be a short-term increase in complaint volume.

Thus, company leaders would do well to focus on how they can smooth out the complaint rate and obtain useful knowledge from the complaint data. They typically compare the number of complaints received with the number of units sold during a defined period, but this relationship is more complex than it seems. Dividing the former number by the latter to obtain a complaint rate for the month is not necessarily meaningful. 

The time when a complaint is registered often has little or nothing to do with when the product was sold. Increases in complaint volume generally follow increases in sales volume by some period. More important is the mean time to complaint, or, in cases where there is an actual failure, mean time to failure. A meaningful calculation of complaint rate requires much more complex mathematics than dividing complaints by units sold. Companies should be careful to appreciate these dynamics.

The audit continues to be the best tool for checking system effectiveness. Large organizations often enjoy the luxury of independent audits performed by outside agents. Smaller organizations, on the other hand, sometimes must perform self-audits. A self-audit is better than no audit, but an independent audit is usually more valuable than a self-audit. Some small companies make reciprocal arrangements with similar companies to conduct audits of each other. This is a good way to gather feedback on complaint system effectiveness.

Complaint-handling processes should be accurate and consistent, most would agree. The system also should be compliant and well documented, with written standard operating procedures that are consistent and verifiable. In addition, everyone in the organization should have at least a basic understanding of how the system works, because complaints can come from anywhere. Although complaints most commonly are funneled into the system from the customer service and support functions, or come as direct communications to the complaint-handling unit, they may also be forwarded by sales representatives and service and repair staff. More-dramatic sources of information about performance inadequacies are lawsuits and regulatory enforcement actions.

From Input to Corrective Action

Any audit of a system for tracking complaints from input to possible corrective action should focus on three key thresholds of the complaint-handling process, each with its own vulnerability.

Filtering All Input. At the first threshold, reports of product problems are examined to determine which should be treated as complaints. FDA regulations require that “[e]ach manufacturer . . . establish and maintain procedures for receiving, reviewing, and evaluating complaints by a formally designated unit.”¹ Company leaders should make sure that input from all sources is filtered such that all reports that qualify as complaints are channeled into a formal complaint-handling system.

Some product performance issues turn into grounds for regulatory action because they were not evaluated properly initially. This may have happened because the filtering mechanism used by customer service or some other specific department kept too many reports from entering the complaint system. Any process for filtering input and identifying complaints should be designed to address the question of timeliness. That is, product reports should be evaluated promptly so that potentially serious issues can be recognized quickly, enabling the company to report complaints in accordance with the medical device reporting (MDR) rule.²

The filtering process also should allow documentation of the rationale for not turning a particular case of negative product input into a formal complaint. In addition, a means of classifying reported experiences is crucial to the system, especially with respect to trending, and can help the company to comply with the regulation on keeping a record of the investigation.³

Auditors of this first system threshold should pay the greatest attention to those reports that were determined not to be complaints. Chances are that if the event report was made a formal complaint, it was handled appropriately. But if the issue was deemed not to be a complaint, it escapes the radar of most standard compliance activities and can linger as potential trouble without notice.

For example, imagine a company receives a complaint from a consumer using its blood glucose test strips from lot number 28A4517. The report states that a series of tests involving strips with this lot number produced unusually high readings. But because the consumer discarded the test strips and in follow-up communications with the company provided no further information, the manufacturer decides to exclude this report from its complaint system.

Strict interpretation of the complaint files rule would compel this report to be included in the complaint system, because it involves the performance of the device.⁴ Certain investigations could be conducted even without return of the product. The manufacturer could perform complaint trend analyses of the product lot to uncover possible related reports. The device history record for lot 28A4517 could be analyzed to see whether any anomalies in the manufacturing process were observed that could have contributed to the problem. Also, by not processing the report as a complaint, the company loses the opportunity to document its attempts to contact the consumer for more information.

This hypothetical company’s failure to include this report in its complaint system could indicate to an auditor that the company’s policies and procedures for complaint handling may be inadequate. Whenever an exclusive filter like this is applied, it attracts the attention of auditors. Its unfortunate effect is a good indicator of how very important this initial assessment process is. In auditing a complaint-handling system, the first things to check are reports of problems that have not been deemed complaints. There should be strong objective evidence showing why the issues raised did not qualify as complaints. Likely sources of noncomplaints that perhaps should have been input to the complaint system are customer service call logs, service and warranty records, and return authorization records.

Filtering Complaints. The second complaint-processing threshold occurs when all complaints are filtered in order to identify reportable events in accordance with the MDR rule. Because this rule has important reporting timelines, this filtering stage should be equipped with a mechanism to address timeliness and be able to document rationale and assess potential risk.
Not all complaints are reportable under the rule. On the other hand, those that are need not involve an injury directly associated with the event. The regulation declares events reportable if “manufacturers or importers have received or become aware of information that reasonably suggests that one of their marketed devices has malfunctioned and that the device or a similar device marketed by the manufacturer or importer would be likely to cause or contribute to a death or serious injury if the malfunction were to recur.”⁵

At this threshold again, the auditor might focus not on the complaints that were reported, but rather on those that were not. FDA and other regulatory agencies do not typically cite organizations for overreporting events; however, they assuredly do take exception to underreporting. If a company’s practices in this area are deficient, an audit of the MDR decision record should identify the areas of weakness.

Assessing the Need for Corrective Action. The final threshold on which the audit should focus is the process used to assess which complaints may necessitate corrective action. Those responsible for managing the complaint system should monitor the effectiveness of the mechanism that categorizes some complaints as grounds for corrective action. Companies are directed by FDA to have procedures for “analyzing . . . complaints . . . to identify existing and potential causes of nonconforming product, or other quality problems.”⁶ In order to comply with this part of the regulation, system managers should find ways to identify and group similar root causes. They should be prepared to explain why some complaints are not identified as cause for corrective action, and to monitor others that may not require immediate corrective action but could justify attention down the road if circumstances change.

Again, the complaints that did not result in corrective action are the ones the auditor will want to review. The file should make clear the reason corrective action seemed unwarranted, and should indicate the information on which the decision not to act was based. Something to consider here is that, if the complaint investigation is unable to determine the root cause of the problem, it will be more difficult to make a valid decision about corrective action.

Electronic Complaint Systems

Many medtech companies are converting to electronic complaint systems. Whether their systems are partially or completely electronic, they must meet the validation requirements imposed on electronic records by regulation, making an audit trail and system security very important considerations.⁷

A company’s electronic systems are only one-third of what is involved in compliance with 21 CFR Part 11. An adequate computing infrastructure must additionally be in place, including network controls, configuration management, and application security. And only when the systems and infrastructure are combined with policies and procedures to prevent such security breaches as the sharing of passwords will a company fall into compliance.

Electronic systems established since the 1997 introduction of this rule should at the very least be validated for their intended use.⁸ But auditors of electronic systems also ought to focus on the other controls mentioned in the same section of the regulation.

Most electronic record systems put in place today are interfaced with other systems. As companies migrate data from one system to the next, they must pay attention to the elements of change control. Computerized control of changes is beneficial, of course; however, the system manager should make sure that someone examines and authorizes every change, especially those caused by external factors, to determine whether the system needs to be revalidated. Any audit of the complaint system should address areas related to 21 CFR Part 11.

Areas of Scrutiny

Two areas of complaint-handling practice have the greatest potential for drawing regulatory scrutiny. Both conspicuously involve the absence of action taken.

First is the practice of conducting no investigation if the product that generated the complaint is not returned. As noted in connection with the hypothetical case, this policy is almost certain to draw the attention of an auditor because the regulation specifies the circumstances under which an investigation must be conducted.4 Investigations can and should be conducted even when no returned product is available for study. The focus of the investigation has to be different in this case, but important information regarding the cause of the reported problem can still be acquired. The manufacturer might consider reviewing the device history record to look for evidence of circumstances that could have contributed to the situation described in the complaint. Alternatively, retained specimens of the product can be examined to see if a similar situation is present.

The greatest benefit derived from investigating even products that were not returned is the contribution to trend analysis. Accounts of field experience, too, are helpful in an investigation, along with information on other similar reported problems. The company can analyze how users were applying the product in light of its intended use. Much investigational progress can be made even with no reported product in hand. An audit of complaint files where no product was returned will reveal any deficiencies in this area.

The other area of potential scrutiny involves the determination of root cause. Perceptive auditors are likely to review areas of the complaint record in which a relatively high percentage of complaints appear to have the same root cause but no corrective action was initiated.6 Root cause determination is typically considered part of the complaint investigation.

A common complaint-handling fault is when use error is too readily determined to be the root cause for a type of complaint. Use error may be the cause of a problem in some cases, but not always. A company that reports a large number of use errors for a given product or use case may not have identified the actual root cause of a problem.

One simple mechanism used to avoid an incomplete process of root cause determination is the so-called rule of five whys. If, like the curious child, an investigator has asked why through five levels of inquiry, the root cause has probably been reached. An audit finding of a single root cause too often assigned may pinpoint a complaint-handling practice needing remediation.

Conclusion

The true measure of complaint system effectiveness is whether more time is spent processing complaints or analyzing them for trends. Medtech companies that minimize the time they require for processing in order to devote resources to examining and analyzing complaint-related data can be said to have an effective system.

Executives with responsibility for managing complaint systems should focus attention on system areas where process or practice weaknesses are likely to be revealed. Then the company can take steps to improve the system, dramatically reducing the potential for compliance issues to arise during future audits.

References

1. Code of Federal Regulations, 21 CFR 820.198(a).
2. Code of Federal Regulations, 21 CFR 803.
3. Code of Federal Regulations, 21 CFR 820.198(e).
4. Code of Federal Regulations, 21 CFR 820.198(c).
5. Code of Federal Regulations, 21 CFR 803.3(r)(2)(ii).
6. Code of Federal Regulations, 21 CFR 820.100(a)(1).
7. Code of Federal Regulations, 21 CFR 11.
8. Code of Federal Regulations, 21 CFR 11.10(a).

Mark Allen is director of quality assurance and regulatory affairs at NetRegulus (Centennial, CO). 

Copyright ©2003 MX