Explore the vast range of titles available.

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers. Though wearable gloves are widely used for gesture associated applications (e.g. sign language recognition), sentence identification of sign language remains a challenge. Here, the authors report AI-enabled recognition system helps barrier-free communication between signers and non-signers.

Our ability to grab, hold, and manipulate objects involves our dexterous hands, our sense of touch, and feedback from our eyes and muscles that allows us to maintain a controlled grip. Billard and Kragic review the progress made in robotics to emulate these functions. Systems have developed from simple, pinching grippers operating in a fully defined environment, to robots that can identify, select, and manipulate objects from a random collection. Further developments are emerging from advances in computer vision, computer processing capabilities, and tactile materials that give feedback to the robot. Science , this issue p. eaat8414 Dexterous manipulation is one of the primary goals in robotics. Robots with this capability could sort and package objects, chop vegetables, and fold clothes. As robots come to work side by side with humans, they must also become human-aware. Over the past decade, research has made strides toward these goals. Progress has come from advances in visual and haptic perception and in mechanics in the form of soft actuators that offer a natural compliance. Most notably, immense progress in machine learning has been leveraged to encapsulate models of uncertainty and to support improvements in adaptive and robust control. Open questions remain in terms of how to enable robots to deal with the most unpredictable agent of all, the human.

The design of dexterous robotic hands remains unresolved, with open questions on whether to prioritize general-purpose manipulation or task-specific control, and how much anthropomorphism to adopt. This perspective revisits the principle of cheap design from embodied intelligence and introduces the concept of ecological balance in hand designs. Effective hand designs emerge not from maximizing either anthropomorphism or task-specificity alone, but from a balanced integration. We frame ecological balance as a corridor that reflects uncertainties in tasks, sensing and control, and emphasize enlarging task-family coverage without excessive complexity. A case study of a piano-playing hand under this principle shows how simplified kinematics, passive compliance and parsimonious sensing allow for refined but efficient performance.

Cervical spinal cord injury (SCI) causes extensive impairments for individuals which may include dextrous hand function. Although prior work has focused on the recovery at the person-level, the factors determining the recovery of individual muscles are poorly understood. Here, we investigate the muscle-specific recovery after cervical spinal cord injury in a retrospective analysis of 748 individuals from the European Multicenter Study about Spinal Cord Injury (NCT01571531). We show associations between corticospinal tract (CST) sparing and upper extremity recovery in SCI, which improves the prediction of hand muscle strength recovery. Our findings suggest that assessment strategies for muscle-specific motor recovery in acute spinal cord injury are improved by accounting for CST sparing, and complement person-level predictions. How the segmental innervation of upper limb muscles recovers after spinal cord injury is not fully understood. Here the authors show associations between corticospinal tract sparing and upper extremity recovery in spinal cord injury.

Humans constantly interact with objects to accomplish tasks. To understand such interactions, computers need to reconstruct these in 3D from images of whole bodies manipulating objects, e.g., for grasping, moving and using the latter. This involves key challenges, such as occlusion between the body and objects, motion blur, depth ambiguities, and the low image resolution of hands and graspable object parts. To make the problem tractable, the community has followed a divide-and-conquer approach, focusing either only on interacting hands, ignoring the body, or on interacting bodies, ignoring the hands. However, these are only parts of the problem. On the contrary, recent work focuses on the whole problem. The GRAB dataset addresses whole-body interaction with dexterous hands but captures motion via markers and lacks video, while the BEHAVE dataset captures video of body-object interaction but lacks hand detail. We address the limitations of prior work with InterCap, a novel method that reconstructs interacting whole-bodies and objects from multi-view RGB-D data, using the parametric whole-body SMPL-X model and known object meshes. To tackle the above challenges, InterCap uses two key observations: (i) Contact between the body and object can be used to improve the pose estimation of both. (ii) Consumer-level Azure Kinect cameras let us set up a simple and flexible multi-view RGB-D system for reducing occlusions, with spatially calibrated and temporally synchronized cameras. With our InterCap method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 daily objects of various sizes and affordances, including contact with the hands or feet. To this end, we introduce a new data-driven hand motion prior, as well as explore simple ways for automatic contact detection based on 2D and 3D cues. In total, InterCap has 223 RGB-D videos, resulting in 67,357 multi-view frames, each containing 6 RGB-D images, paired with pseudo ground-truth 3D body and object meshes. Our InterCap method and dataset fill an important gap in the literature and support many research directions. Data and code are available at https://intercap.is.tue.mpg.de.

Accurate real-time tracking of dexterous hand movements has numerous applications in human–computer interaction, the metaverse, robotics and tele-health. Capturing realistic hand movements is challenging because of the large number of articulations and degrees of freedom. Here we report accurate and dynamic tracking of articulated hand and finger movements using stretchable, washable smart gloves with embedded helical sensor yarns and inertial measurement units. The sensor yarns have a high dynamic range, responding to strains as low as 0.005% and as high as 155%, and show stability during extensive use and washing cycles. We use multi-stage machine learning to report average joint-angle estimation root mean square errors of 1.21° and 1.45° for intra- and inter-participant cross-validation, respectively, matching the accuracy of costly motion-capture cameras without occlusion or field-of-view limitations. We report a data augmentation technique that enhances robustness to noise and variations of sensors. We demonstrate accurate tracking of dexterous hand movements during object interactions, opening new avenues of applications, including accurate typing on a mock paper keyboard, recognition of complex dynamic and static gestures adapted from American Sign Language, and object identification. Accurate real-time tracking of dexterous hand movements and interactions has applications in human–computer interaction, the metaverse, robotics and tele-health. Capturing realistic hand movements is challenging due to the large number of articulations and degrees of freedom. Tashakori and colleagues report accurate and dynamic tracking of articulated hand and finger movements using machine-learning powered stretchable, washable smart gloves.

The anthropomorphic hand plays a crucial role in human-machine interaction tasks. However, there are very few hands that realize multimodal perception with high degrees of freedom (DOF) in a low-cost way. Here, we present a dexterous hand that achieves multimodal sensing solely through a camera. The hand has 18 DOF but does not require any position or force sensors, making it cost-effective and easy to manufacture. We develop an integrated forearm for the hand that provides both actuation and multimodal sensing information simultaneously. This includes the 18 joint angles, 5 fingertip positions and contact forces, and information on object softness and contour. The core principle of perception is that the camera can track the displacement and tension of all tendons simultaneously. The multimodal perception model is developed by characterizing tendon properties and coupling them with the hand dynamics. Experiments indicate that our hand has potential in multimodal sensing and dexterity. Achieving high-dexterity, multimodal perception at low cost remains a challenge for anthropomorphic hands. Here, authors present an 18-DOF dexterous hand that relies solely on a monocular camera to sense joint angles, contact forces, and object properties without the need for additional sensors.

We present a method for enabling Reinforcement Learning of motor control policies for complex skills such as dexterous manipulation. We posit that a key difficulty for training such policies is the difficulty of exploring the problem state space, as the accessible and useful regions of this space form a complex structure along manifolds of the original high-dimensional state space. This work presents a method to enable and support exploration with Sampling-based Planning. We use a generally applicable non-holonomic Rapidly-exploring Random Trees algorithm and present multiple methods to use the resulting structure to bootstrap model-free Reinforcement Learning. Our method is effective at learning various challenging dexterous motor control skills of higher difficulty than previously shown. In particular, we achieve dexterous in-hand manipulation of complex objects while simultaneously securing the object without the use of passive support surfaces. These policies also transfer effectively to real robots. A number of example videos can also be found on the project website: sbrl.cs.columbia.edu

With the advancement of technology, research on humanoid robots has rapidly expanded. To address the challenge of regular object grasping in assembly scenarios, this paper proposes a 6-degree-of-freedom (6DoF) grasping method for regular objects tailored to humanoid robots. The method integrates semantic segmentation, depth imaging, and position prediction to obtain the 3D position of a target object, which is then used to design an appropriate grasping posture for the dexterous hand. First, the YOLOv8n-seg model is employed to detect and segment a steel pipe, and the depth image is combined with this segmented image to estimate the object’s 3D position using the FoundationPose model. Second, based on hand-eye calibration results and internal coordinate transformations of the robot, the gripping position of the robotic arm is calculated, and corresponding control commands are issued to execute the grasping task. Several strategies, such as constraining the grasping axis to remain parallel to the horizontal plane, are implemented to enhance the stability of the grasping process. Experimental results demonstrate that the proposed method enables the robot to achieve stable and reliable grasping across various pipe positions and orientations.

Robotic avatar systems can enable immersive telepresence with locomotion, manipulation, and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic upper body with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing omnidirectional locomotion on flat floors, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR display worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying an animated image of the operator’s face, enabling direct interaction with remote persons. Our system won the $10 M ANA Avatar XPRIZE competition, which challenged teams to develop intuitive and immersive avatar systems that could be operated by briefly trained judges. We analyze our successful participation in the semifinals and finals and provide insight into our operator training and lessons learned. In addition, we evaluate our system in a user study that demonstrates its intuitive and easy usability.