Formative Versus Summative Usability Testing

A user struggling to attach a patient lead, interpret an alarm, or prepare a catheter is not simply a design concern. It can represent a use-related hazard with consequences for patient safety, product performance, and regulatory review. Formative versus summative usability testing distinguishes the two evidence-building activities medical device teams need to address that risk: one improves the design while it is still flexible; the other demonstrates that the finished design can be used safely and effectively.

For manufacturers, treating these activities as interchangeable creates avoidable delays. A formative study cannot substitute for usability validation, and a summative test cannot efficiently uncover issues that should have been resolved months earlier. The difference is primarily one of purpose, timing, and evidentiary rigor.

Formative usability testing is conducted to learn. It identifies where intended users encounter confusion, make errors, take inefficient actions, or misunderstand information. The results guide design decisions. A team may test an early user interface, a physical prototype, draft instructions for use, training concepts, packaging, or a near-final system. The design is expected to change in response to the findings.

Summative usability testing, more commonly described in U.S. medical device regulatory work as usability validation, is conducted to demonstrate. It evaluates whether representative intended users can complete critical tasks safely and effectively with the final or production-equivalent user interface under realistic conditions of use. Its purpose is not to generate a list of preferred design improvements. Its purpose is to provide objective evidence that use-related risks have been adequately controlled.

This distinction matters because the same observation can carry a different meaning in each setting. In a formative session, a participant’s confusion about a touchscreen symbol may prompt the team to revise the icon, screen flow, or labeling. In a summative study, the same confusion may indicate that the device does not yet meet validation readiness if it contributes to failure or difficulty on a critical task.

What Formative Testing Should Accomplish

Formative work belongs throughout development, not only at the beginning. Early studies can reveal fundamental problems with workflow, control placement, terminology, display hierarchy, and physical handling. Later studies can test whether specific risk controls are understandable and whether revisions introduced new use problems.

For a diagnostic ultrasound system, formative evaluation may examine how sonographers select presets, adjust output-related parameters, recognize image quality indicators, or respond to system messages. For an endoscope, it may examine reprocessing steps, connector identification, accessory assembly, and interpretation of cleaning instructions. For a patient monitor, it may focus on alarm configuration, lead placement, display navigation, and response to high-priority notifications.

The appropriate method depends on the question. Direct observation and moderated task scenarios are useful when a team needs to understand why users behave in a particular way. Interviews may reveal terminology mismatches between the device and clinical practice. Heuristic reviews and expert evaluations can help identify apparent interface weaknesses before recruiting participants. Simulated-use sessions can test workflows that would be difficult or inappropriate to evaluate in a clinical environment.

Formative protocols do not need to replicate every final validation condition. Participants may be fewer in number, prototypes may be incomplete, and the moderator may ask probing questions. That flexibility is a strength. It allows the study to investigate uncertainty quickly and directs engineering effort toward the highest-value design changes.

However, flexibility does not mean informality. Formative testing should be planned against the use-related risk analysis. Each session should document the users represented, tasks performed, observations, use errors, close calls, root causes, and resulting design actions. This record supports traceability from identified use-related hazards to design mitigations and subsequent verification.

A formative finding is not automatically a validation failure

Not every observed difficulty represents an unacceptable residual risk. A user may pause because the prototype is unfamiliar, because a scenario uses an unusual clinical condition, or because the interface differs from a legacy product. Teams should assess the event in context: Was the task critical? Did the difficulty lead to an incorrect action? Could it result in harm or compromise performance? Was the participant within the intended user population and using the device in its intended environment?

The same disciplined analysis prevents teams from dismissing meaningful signals as user preference. Repeated hesitation, workarounds, or requests for clarification can expose an inadequate risk control well before a use error occurs.

What Makes Summative Testing Different

Summative testing evaluates the final user interface, including all elements needed for safe use. Depending on the device, this can include the device itself, accessories, software, packaging, labels, instructions for use, training, service tools, and environmental conditions that affect use.

The test design should be grounded in the human factors engineering file and use-related risk management process. It should identify intended user groups, use environments, critical tasks, foreseeable use errors, and the specific risk controls to be evaluated. Critical tasks are not merely common tasks. They are tasks that, if performed incorrectly or omitted, could cause serious harm or compromise medical care.

A credible usability validation study uses representative participants and realistic scenarios. For many FDA submissions, a common expectation is 15 participants per distinct user group, but that figure is not a universal substitute for sound justification. The appropriate participant plan depends on differences in user characteristics, task responsibilities, training, and use environments. Combining groups without a rationale can obscure meaningful differences in performance.

The moderator’s role is also more constrained than in formative work. Participants must use the available interface and information as they would in actual use. Coaching, leading prompts, and immediate corrective instruction can invalidate the observation. If a participant makes an error, the study should capture what occurred, the context, whether recovery was possible, and the potential harm pathway.

A successful summative result is not simply a high task-completion rate. Teams must analyze failures, close calls, difficulties, and observed deviations to determine whether unacceptable use-related risk remains. A participant may complete a task but use an unsafe workaround. Conversely, a task interruption may be harmless if the participant recognizes the issue and recovers through an effective control. The analysis must be clinically and technically defensible.

Planning the Evidence Across Development

The most efficient programs connect formative and summative activities through a deliberate evidence path. Begin by defining the intended users, environments, and use scenarios. Then identify hazards and hazardous situations associated with normal use and reasonably foreseeable misuse. From there, specify the user interface characteristics and information needed to reduce those risks.

Formative studies should test the assumptions behind those controls early enough for changes to be practical. If users misinterpret a cleaning step, the response may involve a redesigned component, clearer visual cues, revised instructions, or a different workflow. Training alone may be insufficient when the risk control depends on a step users must remember under time pressure.

As the design stabilizes, the team should review validation readiness. The production-equivalent interface should be available, critical tasks should be finalized, risk controls should be traceable, and the intended-use claims should align with the test scenarios. Late changes to labeling, software, accessories, or training can require reassessment because each change may affect the validated interface.

ISO 62366-1 provides the human factors engineering framework that connects usability engineering to safety, while ISO 14971 supports the broader medical device risk management process. For U.S. submissions, FDA human factors expectations should also shape the test strategy and documentation. The goal is not merely to conduct a study. It is to maintain a coherent record showing how use-related risks were identified, controlled, evaluated, and accepted.

Common Failure Modes in Human Factors Programs

A frequent mistake is scheduling the summative study before the device is truly final. This can create a costly cycle in which validation identifies a design deficiency, the team changes the interface, and the revised configuration requires additional evaluation. Formative work is less expensive than repeating a late-stage validation study.

Another problem is testing tasks that are easy to observe rather than tasks tied directly to safety risk. A polished study can still provide weak evidence if it does not evaluate the actions most likely to cause harm. Teams should use the risk analysis to drive scenario selection, not rely solely on standard workflow demonstrations.

Teams can also underestimate the effect of the complete system. A device may be straightforward to operate in isolation but difficult to use when packaging, accessories, labeling, environmental constraints, and clinical interruptions are considered together. Usability validation should reflect the actual conditions under which the product will be prepared, operated, maintained, or reprocessed.

Finally, documentation quality matters. Regulatory reviewers need to understand who was tested, what was tested, why each task was selected, how findings were assessed, and how conclusions were reached. Clear protocols, complete observation records, and traceability to risk controls are as important as the final report.

For medical device teams, the practical question is not whether to choose formative or summative testing. It is whether each study is being used for its proper purpose. Build learning into development, reserve validation for a stable design, and let use-related risk determine what evidence the program must produce.