Upper-limb Prosthetics: Pattern Recognition Shows Practical Promise
September 2018 Issue
Pattern recognition technology has shown promise in providing easier, more intuitive control of myoelectric prosthetic devices over the past two decades in research settings, though the concept and some research goes back to the 1960s and 1970s. However, the technology has only relatively recently emerged from the laboratory into practical clinical application as two companies launched commercially available systems to provide more intuitive and accurate prosthetic control. Coapt, Chicago, launched its Coapt Complete Control system in late 2013 and received Class II medical device clearance from the U.S. Food and Drug Administration (FDA) in 2017. Infinite Biomedical Technologies (IBT), Baltimore, began offering its Sense control system in Europe in November 2017 and is expecting FDA clearance for sale in the United States soon.
Pattern Recognition Versus Direct Control
Pattern recognition (PR) differs substantially from the conventional direct control (DC) of myoelectric prostheses and offers the potential for intuitive and simultaneous, as well as sequential and semi-sequential, control with multiple degrees of freedom.
PR aims for more intuitive control in two ways: The first differentiator is that the classifier algorithms take on some of the user's learning curve by identifying the muscle pattern generated when the user makes a particular movement. The classifier links the specific pattern with the movement and thus recognizes that pattern when it occurs and controls the prosthesis to make the intended movement. As noted in a 2018 study by Enzo et al., "It has been widely shown that machine learning algorithms can be trained to recognize the patterns of muscle activations enclosed in the EMG signals, and to decode the motor intention of the user. This approach brings the advantage of providing a more intuitive control, where the learning burden is shared between the user and the machine."1
A paper presented during the 2016 Institute of Electrical and Electronic Engineers (IEEE) conference tested cognitive workload with pupillography measured with an eye-tracking system in using a two degree-of-freedom (DOF) prosthesis for a clothespin task. The study provided evidence of performance and cognitive workload advantages of PR control over DC. PR control was more intuitive to the prosthesis user and, therefore, required less cognitive effort.2
The second way PR differs from DC is that by using an array of muscle sites PR gathers much more data to create control algorithms that enable smoother, more intuitive control.
Instead of one or two electrodes as in conventional DC, there are a number of electrodes surrounding the limb, Coapt CEO Blair Lock explains. He likens it to an orchestra where not only the sound of individual instruments is heard, but also the music of the entire group. "Where that comes into play is in the total level of control. It feels more natural in using wrist rotation, hand open and close, etc.; there's more of the sense of natural movements in activities. It's more like a concert of activity; muscles work together in the trained system to create the action the user wants."
Conventional dual-site DC utilizes an agonist-antagonist muscle pair in the residual limb to generate needed myoelectric signals that can efficiently control one DOF. However, when two or more DOFs are needed to control additional joints, such as an externally powered wrist and elbow, a slow, inconvenient sequential switching mechanism requiring non-intuitive muscle co-contractions or pulses is required.
Although there are several specific DC strategies, such as differential, first-come-first-served, maximum-wins, etc., each works primarily by using the flexor EMG signal to control flexor movements and the extensor EMG signal to control extensor movements, according to a 2016 study by Smith et al.3 The authors add, "Patients must use co-contractions to switch modes in order to sequentially control different DOFs, thus disrupting the continuous control of the device."3
However, the most technologically advanced prosthetic hands allow for different hand grips that the user selects sequentially or semi-sequentially by performing certain pre-defined muscle contractions, such as co-contraction or multiple pulses over the selected pair of channels for DC. "This way of controlling the prosthesis is not intuitive and demands training and practice for the user to become familiar with the switching mechanisms, possibly explaining why efficacy outcomes strongly vary from subject to subject."3
Pattern Recognition Benefits
More intuitive control that is easier to learn and the capability to control two or more DOFs using simultaneous control, as well as sequential or semi-sequential control, are important benefits of PR, bringing the prosthesis closer to natural arm function.
"Typically, the Coapt system is set up to control all of the prosthetic degrees of freedom that an individual is choosing to use with their prosthesis, and that is greatly varied," Lock says. "For some, that is only open/close terminal devices; for others it would include powered elbow flexion/extension, powered wrist pronation/supination, and terminal device. Control of up to four or five hand-grasp postures is also possible for those individuals who choose that."
"The number of degrees of freedom that a person can access with Sense is dependent on their anatomy, their ability to differentiate muscle activity patterns, and the number of degrees of freedom present in their prosthesis," IBT CEO Rahul Kaliki says. "I've personally seen individuals range using a single DOF up to six DOFs: elbow joint, wrist rotation, four grips. Typically, most people are in the range of two-three DOFs."
Although simultaneous control is important, other benefits are even more important in Lock's view, including the following: It allows the user to elicit intuitive contractions for control; it eliminates cumbersome mode switching; it does not require isolated myosites and reduces extensive myotesting; it provides an expanded range of access to proportional control; and it allows the wearer a constant and convenient way to adapt and modify the system control to his or her liking.
"We find it is helpful to let people control the speed at which they can move their prostheses with Sense proportional control," Kaliki says. "It allows them to move quickly when making gross movements and slowly when they are doing precision-oriented tasks."
TMR Surgery Enhances Pattern Recognition Control
Among other benefits, targeted muscle reinnervation (TMR) surgery provides a richer array of myosites for individuals with transhumeral and shoulder disarticulation amputations to enhance use of PR control, explains Todd Kuiken, MD, PhD. Kuiken, a physical medicine and rehabilitation (PMR) physician at Shirley Ryan AbilityLab (formerly the Rehabilitation Institute of Chicago), and Gregory A. Dumanian, MD, of Northwestern Memorial Hospital, Chicago, developed the innovative surgical procedure that provides easier, more intuitive control for individuals with upper-limb amputations.
A 2017 paper Kuiken co-authored points out that recently commercial-ized powered prosthetic arm systems have great function-restoring potential but are limited by conventional (direct) control relying on EMG signals from a limited set of muscles.4
TMR surgery involves transferring residual-limb motor nerves that previously controlled arm and hand function to denervated target muscles for reinnervation. The reinnervated target muscles serve as biological amplifiers of the motor nerve commands intended for the missing arm and thus provide physiologically appropriate EMG control signals related to arm/hand function, not only making prosthesis control more intuitive, but also providing a rich source of EMG sites for PR control. The Smith et al. paper mentions that TMR may allow some patients using DC to also simultaneously control up to two DOFs.
The Hargrove study that Kuiken co-authored was the first home trial large enough to establish clinical and statistical significance in comparing PR with DC. The study included eight participants with transhumeral amputations who had undergone TMR surgery. "Results demonstrate that pattern recognition is a viable option and has functional advantages over direct control," according to the authors.4
There are several challenges to overcome in PR control, including the following three.
A recent study notes, "[Pattern recognition-based control systems] commonly fail when the user attempts to use the limb in a different position from which it was trained, resulting in significantly reduced functionality. Robust models for decoding EMG signals, accounting for specific changes that occur with positional variation, are needed to reduce this negative effect."5
For the study, participants were fitted with surface EMG electrodes as well as a network of inertial measurement units (IMUs) to monitor limb position during tasks. Positional covariates including elbow angle, hand height, and forearm angle were analyzed for impact on EMG signal features to drive the generation of unique linear discriminant analysis classifier algorithms.
Elbow angle and hand height demonstrated the greatest impact on the EMG signal.
Incorporating these specific covariates into classifier algorithms improved performance compared to classifiers trained in the conventional fashion (single-position EMG). Training in random positions as well as single positions also improved performance and function. "As position-tracking hardware becomes smaller and can be implemented into socket designs, incorporating this information into classifier algorithms can dramatically reduce the limb-position effect," according to the authors. "Current users can experience reduction of the limb-position effect through training in multiple random positions."5
Kaliki comments, "There have been some research groups that have looked at improving pattern recognition performance using inertial sensors, though no commercial systems offer position-tracking for pattern recognition yet. It is possible that the inclusion of motion data can improve classification; however, if this approach requires additional hardware or any added burden on the end-user, it may be faced with challenges commercially." He adds, "The inertial sensors are now small enough that they could be located within a pattern recognition controller, which may overcome some of these challenges."
Electrode shift is a well-recognized problem for conventional control, Lock says. "In conventional direct control, placement of electrodes is very, very important to get exactly right." Clinical testing is done to identify one or two muscle hotspots, he explains. "If the electrode shifts, control can go away, so adjustments have to be made to the socket to make sure shift is minimized."
However, PR uses more EMG information collected as a whole, "so where the electrodes are and how much they shift can be a little more flexible. Whenever users recalibrate the system, which only takes a minute or so, the system learns more about them and their signals," Lock says. "Users can recalibrate the system whenever they need or like, which may be once a day to once a month.
The control system is somewhat similar to a voice recognition system —the more you talk to it, the more you use it, the better it functions for you." Over time, electrode shifts funnel more information to the system and can actually make the control more robust, he adds.
"The biggest challenge, I think, is education in the field—what this technology is and does," Lock says. "We tend to encounter a lot of misconceptions about what it is and how it works. Through workshops, seminars, [and] websites, we hope to help practitioners want to get involved with it. The technology is going to evolve, and that's exciting. We hope to bring it into the practitioners' purview—that's our biggest challenge."
Lock points out that PR control works well with the majority of current prosthetic systems without needing to reengineer the prosthesis, although a new socket may be needed to contain the PR control unit. PR works very well as a complementary technology and can be retrofitted to most users' existing prosthetic configurations.
Pattern Recognition in the Real World
Although most of the studies The O&P EDGE found demonstrate better clinical function with PR control compared to conventional DC, some studies have shown very little difference or slightly better function with DC. Researchers have recognized the need for more research, with a larger number of participants, to focus on clinical, home, and workplace function and advantages of upper-limb control strategies as well as other promising advances.
For example, Resnik et al. state in a 2018 paper that although EMG PR for multifunctional upper-limb prosthesis control has been reported for decades, the clinical benefits have rarely been examined.6 Outcomes, including measures of dexterity with and without cognitive load, activity performance, self-reported function, and prosthetic satisfaction were administered immediately and one week after training. One subject completed PR training only. The other subject completed both PR and DC training and thus was able to serve as the subject in the crossover study. "Outcomes of PR and DC for operating a two-DOF prosthesis in a single-subject crossover study were similar for 74 percent of metrics and favored DC in 26 percent of metrics," according to the authors.6 Findings related to rate of skill acquisition varied considerably by subject, the authors note.
There are two examples of clinically focused studies with a somewhat larger sample sizes worth mention. Although Coapt's Complete Control system was used in both, results likely would be similar with other commercially available systems.
Over a two-year span, 13 patients were fitted with PR technology by Handspring Prosthetic Rehabilitation Services, headquartered in Middle-town, New York, according to a paper presented during the MEC 17 Myoelectric Controls Symposium in Fredericton, New Brunswick, Canada.7
Although four patients discontinued using PR for various reasons, nine patients continued using it. The authors note that the study demonstrates PR technology can be successfully used in externally powered prostheses for patients with all levels of upper-limb differences. They add, "It was our anecdotal experience that patients fit with the Coapt system were able to progress faster in their OT [occupational therapy] training than other patients."7
Another study involving multiple participants was conducted across multiple centers comprising a diverse patient population. The observational interview study of patients' and prosthetists' experiences included 14 patients with various levels of amputations and ten prosthetists in Hanger's Upper Extremity Prosthetics Program.
Uellendahl et al. report, "This case series study provides initial evidence that pattern recognition can be implemented in standard prosthetic practices and is equal to or superior to conventional myoelectric control."8 Five subjects had TMR procedures. "In all cases, pattern recognition was found to have advantages over two-site conventional control," the authors note. User comments included: Faster response time, better proportional control, shorter learning curve, and no need for co-contraction.
In addressing clinical applications, the case series concludes, "This review of our first pattern recognition control patients supports the hypothesis that pattern recognition control is easier to learn and more intuitive than conventional control methods."8
The study by Smith et al. quoted earlier details areas of research to improve myoelectric control as well as mentioning research into other control strategies. "There has…been a large emphasis in the research community on improving the controllability of myoelectric devices, focusing on providing simultaneous control of multiple DOFs in a way that also allows for proportional control.3 The recent development of implantable EMG-recording devices allowed conventional [direct control] methods to provide simultaneous control by configuring multiple dual-site strategies in parallel."
An individual with transradial amputation and implanted myolectric sensors in the forearm demonstrated simultaneous control of wrist rotation and a two-DOF hand, the study notes. Additionally, parallel dual-site control has been investigated for individual finger control and for a three-DOF wrist/hand system, the authors add.3 Among other research initiatives, the study notes that PR has been used to provide accurate predictions of intended simultaneous DOF motion classes, but with limited methods to allow for proportional independent velocity control of the DOFs. Neural networks have been used to predict joint kinematics or kinetics of the wrist. Blind source separation algorithms for extracting muscle synergies have been used to control a two-DOF wrist system.
Looking Toward the Future
In looking at what lies ahead, Kaliki's comments sum up those of Lock and Kuiken as well. "There's osseointegration where the socket doesn't have to be the critical part of the prosthesis. [Along the same lines,] in the future, we're looking at being able to implant sensors directly into the muscle, so we don't have to worry about the sensor or the electrode shifting on the skin. The PR control systems will not need recalibration often. There also is a lot of work going on to provide sensory feedback so the prosthesis can be used in a more functional way. That's really exciting as well. I think the technology is going to grow; within a five- to ten-year range we should see these technologies coming on the market."
Miki Fairley is a freelance writer based in southwest Colorado. She can be contacted at firstname.lastname@example.org.
1. Enzo, M., J. Ahlberg, E. Lendaro, L. Hermansson, B. Håkansson, and M. Ortiz-Catalan. 2018. An alternative myoelectric pattern recognition approach for the control of hand prostheses: A case study of use in daily life by a dysmelia subject. IEEE Journal of Translational Engineering in Health and Medicine 6: 2600112.
2. Zhang, W., M. White, and M. Zahabi, et al. 2016. Cognitive workload in conventional direct control vs. pattern recognition control of an upper-limb prosthesis. 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC) DOI 10.1109/SMC.2016.7844587.
3. Smith, L., T. Kuiken, and L. Hargrove. 2016. Myoelectric control system and task-specific characteristics affect voluntary use of simultaneous control. IEEE Transactions on Neural Systems and Rehabilitation Engineering 24(1): 109–116.
4. Hargrove, L., L. Miller, K. Turner, and T. Kuiken. 2017. Myoelectric pattern recognition outperforms direct control for transhumeral amputees with targeted muscle reinnervation: A randomized clinical trial. Scientific Reports 7: 13840.
5. Beaulieu R, M. Masters, and J. Betthauser et al. 2017. Multi-position training improves robustness of pattern recognition and reduces limb-position effect in prosthetic control. Journal of Prosthetics and Orthotics (2):54-62.
6. Resnik, L., H. Huang, A. Winslow, D. Crouch, F. Zhang, and N. Wolk. 2018. Evaluation of EMG pattern recognition for upper limb prosthesis control: A case study in comparison with direct myoelectric control. Journal of Neuroengineering and Rehabilitation 15(1):23.
7. Baschuk, C., L. Katzenberger, D. Latour, T. Passero, and E. Tompkins. 2017. Outcomes of the clinical application of pattern recognition in upper-limb prosthetics: A two-year retrospective. Proceedings of MEC 17 Myoelectric Controls and Upper Limb Prosthetics Symposium. Fredericton, New Brunswick, Canada: University of New Brunswick.
8. Uellendahl, J., J. Tyler, and K. Hung. 2016. A case series study of pattern recognition for upper-limb prosthesis control. Proceedings of American Academy of Orthotists & Prosthetists 42nd Academy Annual Meeting & Scientific Symposium. Orlando, Florida.