The following results describe the pooled data of all three subjects to point to general trends we observed and to make for easier reading. Wherever necessary or interesting, we further clarify how this applied to or deviates from the individual results. Owing to the small number of subjects, we only made statistical comparisons within subjects, not between. All individual results are available in the text for Experiment 1 and in the Supplementary materials for Experiment 2.
Experiments started with an extensive fitting session aimed to find each subject’s personal stimulation settings to elicit sensory perception. The electrode contact used for stimulation and the stimulation parameters were selected empirically, prioritizing the contact that required the lowest charge to produce perception (Table 1 and Fig. 2). The resulting percepts were located on the middle and distal phalanxes of the index and middle fingers for S1 and S3, and also on the distal phalanx of the thumb for S3. These locations corresponded to the somatotopic arrangement of the median nerve where the selected electrode was located. For S1, and to a lower extent also for S3, an increase in pulse current amplitude was perceived as an increased area of stimulation on the phantom fingers. For S2, whose selected electrode was placed around the ulnar nerve, percepts were located on the palmar side below the fifth finger. The perceived area increased towards the center of the phantom palm when increasing the pulse current amplitude. Higher pulse amplitudes resulted in higher perceived intensities in all subjects. Furthermore, all subjects perceived the phantom hand in the same location as the robotic hand.
Experiment 1: motor coordination under certainty
At the beginning of this experiment, subjects were fitted with a custom prosthetic setup. Among other components, this comprised the IH2 Azzurra research robotic hand (Prensilia SRL, Italy), controlled via direct proportional speed control to perform a tridigital grasp.
For each feedback condition, experiments started with a familiarization stage in which the subjects grasped and relocated fragile objects with different weights (200 g, 300 g, 400 g), akin to the virtual eggs test21. Subjects then completed two sessions of the PLT (20 repetitions each) to assess motor coordination under certainty, i.e., with a non-breakable object of known and constant weight of 200 g, similar to experiments by Cipriani et al.29 who used non-invasive interfaces.
Subjects showed high motor coordination with hybrid feedback
Motor coordination can be visualized as the temporal correlation of the grip and load forces (Fig. 3). High temporal correlation, that is, a more linear relationship between the temporal evolution of the grasp and lift forces, indicates a more mature and natural grasping behavior30. Figure 3 shows qualitatively that all subjects displayed higher motor coordination with Hybrid feedback. Notably, S3 generally exhibited high motor coordination even without feedback, which was hindered when provided with Continuous feedback.
Motor coordination can also be quantified as the temporal delay between the instants when the grip force and load force reach 50% of the load force at lift-off21. Looking at the pooled data, we found that this delay was considerably reduced from 226:347 ms (median:IQR) in the no-feedback condition, to 176:122 ms in the Hybrid feedback condition, which represents a 22% reduction of the median delay (Fig. 4). However, this trend was mostly represented by, and statistically significant for, S1. Individually, the delay was lowest with Hybrid feedback for S1 and second lowest for S2 and S3. The grip-load delay with Discrete feedback was 230:215 ms and thus comparable to no-feedback, but with Continuous feedback it actually increased by 62% (368:344 ms). S2 had the shortest (149:108 ms) and longest (597:802 ms) grip-load forces delays of all subjects, with Discrete and Continuous feedback, respectively. Remarkably, S3 had the second-shortest result, with a delay of only 160:79 ms, using no-feedback.
Supplementary feedback did not affect the maximum grip force
No statistically significant difference was found regarding the maximum grip force applied to the test object when using different feedback modes (Figure S1—Supplementary Materials). The peak forces applied were contained within similar ranges (3.9:0.8 N). The grip force measured at the instant of lift-off was 3.6:0.7 N (median:IQR of all subjects and all feedback conditions).
Temporal metrics showed benefit of hybrid but not of continuous feedback
The load and release phase duration, as well as the trial duration, exhibited a similar trend to that seen for the delay between grip and load forces: Discrete and Hybrid feedback allowed for faster executions in subjects S1 and S2 compared to no-feedback, although this trend was not always significant (Fig. 4 and Movie S1). On average, Hybrid compared to no-feedback resulted in a 30% reduction of the median load phase, from 460:450 ms to 320:170 ms, in a 5% reduction of the median release phase, from 580:485 ms to 550:450 ms, and in a 16% reduction of the median trial duration, from 2.61:0.91 s to 2.19:0.49 s. Discrete feedback compared to no-feedback resulted in a 17% longer median load phase (540:675 ms), in a 17% shorter median release phase (480:365 ms), and in a 8% shorter trial duration (2.39:1.08 s). Continuous feedback consistently increased these temporal metrics to 665:660 ms (+45%), 800:485 ms (+38%), and 3.23:1.38 s (+24%) for median load phase, release phase and trial duration, respectively. Individually, these findings were all significant for S1, but for S2 the difference was only significant for Discrete feedback. Nonetheless, the results with Hybrid feedback were still lower for all three metrics for S2. As before, S3 did not profit from feedback, but, for this subject, Hybrid feedback was the only one not significantly increasing trial duration compared to no-feedback.
Summary of experiment 1
The PLT results under certainty showed that the Hybrid and Discrete feedback generally improved the temporal metrics for S1 and S2. Hybrid feedback also allowed for a more linear relation between grip and load forces for S1 and S2, who benefited most from sensory feedback. S3 showed highly coordinated control without feedback and did not improve in any of the considered metrics when provided with feedback. Instead, feedback worsened coordination for S3 in most cases, but Hybrid seemed to interfere the least. Notably, both S2 and S3 were slowed down by Continuous feedback, and S1 was significantly faster using any kind of feedback.
Experiment 2: motor coordination under uncertainty
In order to examine the effect of sensory feedback under uncertainty, subjects performed a second experiment with the PLT, using the same non-breakable object, for three sessions of 20 repetitions each. However, in this experiment the weight of the object could change randomly and unexpectedly between 200 g, 300 g, and 400 g, from one repetition to the next. Subjects were informed that after a weight change, the object would stay the same weight for at least two further repetitions.
We analyzed variations on grasping behavior owing to unexpected weight changes by contrasting (1) the trial preceding the weight change (old weight), (2) the trial in which the weight changed (new weight), (3) the trial immediately following the weight change, and (4) the last trial in the series of consecutive lifts with the new weight. Clearly, each trial preceding a weight change is at the same time the last of the previous series of lifts, but the context may change. For the analysis, the weight changes were pooled for changes from lighter to heavier weights (from 200 g to 300 g or 400 g, and 300 g to 400 g), and for changes from heavier to lighter weights (from 400 g to 300 g or 200 g, and 300 g to 200 g). Considering previous findings by Jenmalm et al.26, our analysis focused on the load phase duration and the maximum grip force rate (Fig. 5). The subjects’ individual results are included in the Supplementary Materials as Figures S2, S3, and S4.
Feedback increased speed of lifting under uncertainty
It has previously been observed in non-amputees that a sudden increase in weight significantly increases the duration of the load phase in a PLT20,26. Indeed, we observed that the load phases became longer in the weight change trials for all feedback modes and then decreased again towards the last lift in the series as the same weight was lifted repeatedly (Fig. 5A; see also Figure S6—Supplementary Materials). This trend was statistically significant only for S1 using Discrete feedback (p=0.02 for preceding vs weight change; Figure S2A—Supplementary Materials). When considering the pooled data in no-feedback mode for changes from lighter to heavier weights, the load phase duration increased from 710:783 ms to 1240:938 ms (+75%), and then decreased to 940:770 ms (− 24% compared to weight change) in the last trial in the series with the same weight. Subjects exhibited generally shorter load phases when provided with tactile feedback, with the shortest observed using Hybrid feedback. When the weight was unexpectedly increased, the load phase duration with Hybrid increased from 360:145 ms to 500:250 ms (+39%) and increased again to 600:520 ms (+20% compared to weight change) in the following trial. Towards the last trial in the series, it decreased to 360:460 ms (− 28% compared to weight change).
Sudden weight decreases had no strong effect on maximum grip force rate
Previous work predicted a significant decrease in the grip force rate during the load phase between trials with an unexpected decrease of weight, and those following the weight change26. Even though this trend was confirmed with Continuous and Hybrid feedback, it was statistically significant only for S3 with Hybrid (p<0.001 for weight change vs following; Figure S4D—Supplementary Materials). In general, when using Discrete feedback, the grip force rate already decreased during the weight change trial and stayed low until the last trial in the series (Fig. 5). We also found a general trend for higher grip force rates with Hybrid and Discrete than with no-feedback (e.g., Hybrid 34.0:16.0 N/s and Discrete 32.3:12.0 N/s compared to no-feedback 24.0:8.0 N/s, for all trials preceding a sudden weight decrease). This complements the longer load phase durations observed with no-feedback.
As in Experiment 1, the maximum grip forces applied to the object were contained with all feedback modes (Figure S5—Supplementary Materials). As expected, the grip force rescaled according to the object weight similarly for all modes, except for Discrete.
Summary of Experiment 2
The PLT results from Experiment 2 showed that our subjects were faster when provided with any sensory feedback mode, but particularly with Hybrid mode. We found shorter load phase durations and higher grip force rates with tactile feedback than with no-feedback (Fig. 5 and Movie S1). Moreover, the subjects generally used a slower control approach under uncertainty, more feedback-based than what observed in Experiment 1. On average, the load phase, release phase and trial durations were 34% (496:488 vs 664:775), 16% (603:446 vs 697:573) and 10% (2.60:0.96 vs 2.86:1.45) longer, respectively.
Qualitative evaluation of sensory perception
Subjects assigned qualitative descriptors to the elicited sensations using a questionnaire in which they could choose up to 16 descriptors for each stimulation mode (Table 2). The Continuous mode was described by all three subjects as an “electrical” sensation. This mode was also associated with the terms “pressure” (S1, S2), “buzzing” and “vibration” (S1, S3), “tingling” (S2), as well as “touch”, “needle prick”, “numbness” and “movement” (S1). The Discrete mode was described as “electrical” (S1, S2) and “pressure” (S2, S3), whereas terms like “tingling” (S2), “touch” and “tapping” (S3), and “needle prick” (S1) were each used once. The Hybrid mode was described as “electrical” and “buzzing” by all three subjects. Two times “pressure” (S2, S3) and “movement” (S1, S2) were used to further describe it, whereas “tingling” (S2), “vibration” (S3), “needle prick”, “tickling” and “itch” (S1) where each used once. Regarding the perceived intensity, interestingly the Continuous mode was rated as the most intense by all three subjects. Hybrid was rated equally as intense as Discrete by two out of three subjects and more intense than Discrete by one subject.
Moreover, subjects rated the naturalness and the pleasantness of each sensory feedback mode on a freely chosen, continuous scale (Fig. 6). The subsequently normalized results of the subjective naturalness rating (on a scale of 0 to 10, with 10 being perfectly natural) were: 3.5±1.3 (mean ± standard deviation) for Continuous, 3.1±2.6 for Discrete, and 3.1±0.9 for Hybrid feedback. Regarding the pleasantness of the stimulation, the subjective ratings were (on a scale from 0 to 10, with 10 being extremely pleasant): 7.9±2.6 for Continuous, 7.2±1.6 for Discrete, and 7.2±1.6 for Hybrid.