In existing surgery process, surgeons need to manually adjust the laparoscopes to provide a better field of view during operation, which may distract surgeons and slow down the surgery process. This paper presents a data-driven control method that uses a continuum laparoscope to adjust the field of view by tracking the surgical instruments. A Koopman-based system identification method is firstly applied to linearize the nonlinear system. Shifted Chebyshev polynomials are used to construct observation functions that transfer low-dimension observations to high-dimension ones. The Koopman operator is approximated using a finite-dimensional estimation method. An optimal controller is further developed according to the trained linear model. Furthermore, a learning-based pose estimation framework is designed to detect keypoints on surgical instruments and provide visual feedback for adjusting the laparoscope. Compared with other detection methods, the proposed scheme achieves a higher detection precision and provides more optional keypoints for tracking. Simulation and experiments validate the feasibility of the proposed control method. Experiment results show that the proposed method can automatically adjust the field of continuum laparoscope through tracking surgical instruments in a timely manner and the number of surgical tools is not limited.

While most deep learning approaches are developed for single images, in real world applications, images are often obtained as a series to inform decision making. Due to hardware (memory) and software (algorithm) limitations, few methods have been developed to integrate multiple images so far. In this study, we present an approach that seamlessly integrates deep learning and traditional machine learning models, to study multiple images and score joint damages in rheumatoid arthritis. This method allows the quantification of joining space narrowing to approach the clinical upper limit. Beyond predictive performance, we integrate the multilevel interconnections across joints and damage types into the machine learning model and reveal the cross-regulation map of joint damages in rheumatoid arthritis. Corresponding author(s) Email:   [email protected] or [email protected]

The cocktail party problem refers to a challenging process when the human sensory system tries to separate a specific voice from a loud mixture of background sound sources. The problem is much more demanding for machines and has become the holy grail in robotic hearing. Despite the many advances in noise suppression, the intrinsic information from the contaminated acoustic channel remains difficult to recover. Here we show a simple-yet-powerful laser-assisted audio system termed REAL (Robot Ear Accomplished by Laser) to probe the vibrations of sound-carrying surfaces (mask, throat and other nearby surfaces) in optical channel, which is intrinsically immune to acoustic background noises. Our results demonstrate that REAL can directly obtain the audio-frequency content from the laser without acoustic channel interference. The signals can be further transcribed into human-recognizable audio by exploiting the internal time and frequency correlations through memory-enabled neural networks. The REAL system would enable a new way in human-robot interaction. Xiaoping Hong  Email:    [email protected]

Two-dimensional metal-organic frameworks (2D-MOFs) have been extensively studied as promising materials in the fields of eletrocatalysis, drug delivery, electronic devicese, etc. However, few studies have explored the application potential of 2D-MOFs in novel neuromorphic computing devices. In this work, we report an optoelectronic neuromorphic transistor based on a 2D-MOFs/polymer charge-trapping layer. We found that, the large specific surface area, stable crystal structure, and highly accessible active sites in 2D-MOFs make them excellent charge-trapping materials for our devices, which are beneficial for mimicking the memory and learning functions observed in the organism's nervous systems. Different types of synaptic behaviors have been realized in our 2D-MOFs-based neuromorphic devices under stimuli signal, e.g., paired-pulse facilitation, excitatory post-synaptic current, short-term memory, and long-term memory. More interestingly, emotion-adjustable learning behavior was realized by changing the value of the source-drain voltage. This work can shed light on the application of 2D-MOFs in neuromorphic computing and will contribute to the further development of neuromorphic computing devices. Corresponding authors Email:  [email protected] (Jia Huang)                                                                [email protected] (Shilei Dai)

"AI & Drug Discovery" mode has significantly promoted drug development and achieved excellent performance, especially with the rapid development of deep learning, making remarkable contributions to protecting human physiological health. However, due to the "black-box" characteristic of the deep learning model, the decision route and predicted results in different research stages assisted by deep models are usually unexplainable, limiting their application in practice and more in-depth research of drug discovery. Focusing on the drug molecules, we propose an explainable fragment-based molecular property attribution technique for analyzing the influence of particular molecule fragments on properties and the relationship between the molecular properties in this paper. Quantitative experiments on 42 benchmark property tasks demonstrate that 325 attribution fragments, which account for 90% of the overall attribution results obtained by the proposed method, have positive relevance to the corresponding property tasks. More impressively, most of the attribution results randomly selected are consistent with the existing mechanism explanations. The discovery mentioned above provides a reference standard for assisting researchers in developing more specific and practical drug molecule studies, such as synthesizing molecular with the targeted property using a fragment obtained from the attribution method.Corresponding author(s) Email:   [email protected]  

The integration of an ingestible dosage form with sensing, actuation and drug delivery capabilities can enable a broad range of surgical-free diagnostic and treatment strategies. However, the gastrointestinal (GI) tract is a highly constrained and complex luminal construct that fundamentally limits the size of an ingestible system. Recent advancements in mesoscale magnetic crawlers have demonstrated the ability to effectively traverse complex and confined systems by leveraging magnetic fields to induce contraction and bending-based locomotion. However, the integration of functional components (e.g., electronics) in the proposed ingestible system remains fundamentally challenging. Here, we demonstrate the creation of a centralized compartment in a magnetic robot by imparting localized flexibility (MR-LF). The centralized compartment enables MR-LF to be readily integrated with modular functional components and payloads, such as commercial off-the-shelf electronics and medication, while preserving its bidirectionality in an ingestible form factor. We demonstrate the ability of MR-LF to incorporate electronics, perform drug delivery, guide continuum devices such as catheters, and navigate air-water environments in confined lumens. The MR-LF enables functional integration to create a highly-integrated ingestible system that can ultimately address a broad range of unmet clinical needs.  Keywords: Ingestible electronics; ingestible robots; soft robots; magnetic robots; magnetic crawlers; drug delivery.  Corresponding author email:   [email protected]    

The substantial increase in global population and climate change, among other factors have led to global food security and supply chain challenges. The United Nations has laid out an agenda to sustainably achieve zero hunger by 2030 as one of its sustainable development goals. However, sustainably achieving improved food yield has become a challenge as excessive use of fertilizers has also led to adverse environmental impact. To address the aforementioned challenges, WisDM Green, an artificial intelligence (AI)-based platform that aims to pinpoint and prioritize compound (e.g. biostimulants) combinations in peat moss, is harnessed to sustainably improve the yield of Amaranthus cruentus (red spinach). In this proof-of-concept study, from a pool of 8 compounds, WisDM Green-pinpointed combinations (6-Benzylaminopurine/Ethylenediaminetetraacetic Acid Iron (III) and Humic Acid/Seaweed Extract) achieve 26.34±15.80 and 33.59±14.60 increase in %Yield, respectively. The study also indicates that compound combinations may exhibit concentration-dependent synergies and thus, properly adjusting the concentration ratios of combinations may further improve plant yield in the context of sustainable farming. P. Wang and K. You contributed to this work equally.Corresponding author(s) Email:   [email protected], [email protected], [email protected] 

Deformability and self-adaptability are important for soft robots in order to deal with uncertain and varying situations and environments during movement and navigation. Droplet-based robots are great candidates to travel inside narrow and constrained spaces without damaging the interfaces due to their extreme deformability and liquid nature, which enables smooth contact between robots and target spaces. Here, we propose magnetic liquid metal droplet robots, comprising liquid metal and carbonyl iron, that can perform reversible telescopic deformation, bending, and on-demand locomotion. The magnetic liquid metal-based robots can perform on demand and reversible coalescence and splitting by intricately applying magnetic fields. Importantly, the liquid metal robot can perform phase transition to fix the desired shape after the programmable shape encoding. The liquid metal-based soft robots can serve as dynamic and recyclable switches for complex circuits, and are capable of repairing damaged sections of microcircuits by remote actuation, controllable coalescence, and on-demand circuit welding. The technology provides a new application scenario of droplet-based soft robots for on-demand circuit welding and transient recyclable electronics. Corresponding author(s) Email:  X.Z.: [email protected]; L.Z.: [email protected]; B.W.: [email protected]

Human receives and transmits various information from the outside world through different sensory systems. The sensory neurons integrate various sensory inputs into a synthetical perception to monitor complex environments, and this fundamentally determines the way how we perceive the world. Developing multifunctional artificial sensory elements that can integrate multisensory perception plays a vital role in future intelligent perception systems, whereas prior spiking neurons reported can only handle single-mode physical signals. Here, we present a bio-inspired haptic-temperature fusion spiking neuron based upon a serial connection of piezoresistive sensor and VO2 volatile memristor. The artificial sensory neuron is capable of detecting and encoding pressure and temperature inputs based on the voltage dividing effect and the intrinsic thermal sensitivity of metal-insulator transition in VO2. Recognition of Braille characters is achieved through multiple piezoresistive sensors, taking advantage of the spatial integration capabilities of such spiking neurons. Notably, the traditionally separate haptic and temperature signals can be fused physically in the sensory neuron when synchronizing the two sensory cues, which is able to recognize multimodal haptic/temperature patterns. The artificial multisensory neuron thus provides a promising approach towards e-skin, neuro-robotics and human-machine interaction technologies.

Avi Hazan1, Elishai Ezra Tsur1*1 Neuro-Biomorphic Engineering Lab, Department of Mathematics and Computer Science, The Open University of Israel, Ra’anana, Israel* Correspondence: [email protected] hardware designs realize neural principles in electronics to provide high-performing, energy-efficient frameworks for machine learning. Here, we propose a neuromorphic analog design for continuous real-time learning. Our hardware design realizes the underlying principles of the neural engineering framework (NEF). NEF brings forth a theoretical framework for the representation and transformation of mathematical constructs with spiking neurons. Thus, providing efficient means for neuromorphic machine learning and the design of intricate dynamical systems. Our analog circuit design implements the neuromorphic prescribed error sensitivity (PES) learning rule with OZ neurons. OZ is an analog implementation of a spiking neuron, which was shown to have complete correspondence with NEF across firing rates, encoding vectors, and intercepts. We demonstrate PES-based neuromorphic representation of mathematical constructs with varying neuron configurations, the transformation of mathematical constructs, and the construction of a dynamical system with the design of an inducible leaky oscillator. We further designed a circuit emulator, allowing the evaluation of our electrical designs on a large scale. We used the circuit emulator in conjunction with a robot simulator to demonstrate adaptive learning-based control of a robotic arm with six degrees of freedom.

A prominent problem in computer vision is occlusion, which occurs when an object’s key features temporarily disappear behind another crossing body, causing the computer to struggle with image detection. While the human brain is capable of compensating for the invisible parts of the blocked object, computers lack such scene interpretation skills. Cloud computing using convolutional neural networks is typically the method of choice for handling such a scenario. However, for mobile applications where energy consumption and computational costs are critical, cloud computing should be minimized. In this regard, we propose a computer vision sensor capable of efficiently detecting and tracking covered objects without heavy reliance on occlusion handling software. Our edge-computing sensor accomplishes this task by self-learning the object prior to the moment of occlusion and uses this information to “reconstruct” the blocked invisible features. Furthermore, the sensor is capable of tracking a moving object by predicting the path it will most likely take while travelling out of sight behind an obstructing body. Finally, sensor operation is demonstrated by exposing the device to various simulated occlusion events. Keywords:  Computer vision, occlusion handling, edge computing, object tracking, dye sensitized solar cell. Corresponding author Email: [email protected]  

The origami technique realizes unique mechanical properties of sheet materials without additional parts. In this study, a self-folded corrugated structure (SCS) is developed based on the reinforcing properties of the origami technique. The corrugated structures are employed as the core material for a high-strength, open-channel sandwich structure. Research on self-folded core materials is scarce; thus, a design concept is proposed, and the mechanical properties of the SCS are evaluated. First, the structural parameters of the SCS fabricated by changing the printing parameters (e.g., linewidth and number of lines/creases), to derive the structural model are determined. The model facilitates the design of an SCS with the desired structure. Thereafter, the mechanical properties of the SCSs are evaluated by conducting three-point bending tests to determine the essential design parameters corresponding to high stiffness. Moreover, SCSs can be stacked without occupying space, thus leading to improved strength. These SCSs fabricated using self-folding paper by ink-jet printing are low cost and eco-friendly. Moreover, they are specialized for rapid design and fabrication, depending on the application. This paper proposes the use of SCS as a novel smart core because it exhibits a high transportation efficiency and stiffness without additional components. Corresponding author(s) Email: [email protected] , [email protected]

Shape memory Nitinol has long been used for actuation. However, utilizing Nitinol to fabricate novel devices for various applications is a challenge, but has shown incredible promise and impacts. Bistable metal strips are widely adopted for shape morphing purposes (primarily in kid’s toys, e.g., snap bracelets) due to their easy and robust transformation between two states. In this paper, we combine Nitinol shape memory alloy and bistable metal strip to fabricate a swimming actuator with both slow moving and fast snapping capability, akin to an octopus swimming slowly in water, but quickly moving upon encountering a threat. The actuator developed here can also swim in multiple directions, all controlled by a wireless module. Furthermore, we demonstrate that an on-board sensor can be incorporated for potential environmental monitoring applications. Taken together, along with the fact that the device developed here has no mechanical parts, makes this  an interesting potential alternative to more expensive, and energy consuming boats.

Leukocyte differential test is a widely performed clinical procedure for screening infectious diseases. Existing hematology analyzers require labor-intensive work and a panel of expensive reagents. Here we report an artificial-intelligence enabled reagent-free imaging hematology analyzer (AIRFIHA) modality that can accurately classify subpopulations of leukocytes with minimal sample preparation. AIRFIHA is realized through training a two-step residual neural network using label-free images of isolated leukocytes acquired from a custom-built quantitative phase microscope. By leveraging the rich information contained in quantitative phase images, we not only achieved high accuracy in differentiating B and T lymphocytes, but also classified CD4 and CD8 cells, therefore outperforming the classification accuracy of most current hematology analyzers. We validated the performance of AIRFIHA in a randomly selected test set and cross-validated it across all blood donors. Owing to its easy operation, low cost, and accurate discerning capability of complex leukocyte subpopulations, we envision AIRFIHA is clinically translatable and can also be deployed in resource-limited settings, e.g., during pandemic situations for the rapid screening of infectious diseases.  Corresponding author(s) Email:    [email protected],  [email protected]

Soft actuators and robotic devices have been increasingly applied to the field of rehabilitation and assistance, where safe human and machine interaction is of particular importance. Compared with their widely used rigid counterparts, soft actuators and robotic devices can provide a range of significant advantages; these include safe interaction, a range of complex motions, ease of fabrication and resilience to a variety of environments. In recent decades, significant effort has been invested in the development of soft rehabilitation and assistive devices for improving a range of medical treatments and quality of life. This review provides an overview of the current state-of-the-art in soft actuators and robotic devices for rehabilitation and assistance, in particular systems that achieve actuation by pneumatic and hydraulic fluid-power, electrical motors, chemical reactions and soft active materials such as dielectric elastomers, shape memory alloys, magnetoactive elastomers, liquid crystal elastomers and piezoelectric materials. Current research on soft rehabilitation and assistive devices is in its infancy, and new device designs and control strategies for improved performance and safe human-machine interaction are identified as particularly untapped areas of research. Finally, insights into future research directions are outlined.Corresponding author(s) Email:   [email protected], [email protected]

Interacting with surrounding road users is a key feature of vehicles and is critical for intelligence testing of autonomous vehicles. The Existing interaction modalities in autonomous vehicle simulation and testing are not sufficiently smart and can hardly reflect human-like behaviors in real world driving scenarios. To further improve the technology, in this work we present a novel hierarchical game-theoretical framework to represent naturalistic multi-modal interactions among road users in simulation and testing, which is then validated by the Turing test. Given that human drivers have no access to the complete information of the surrounding road users, the Bayesian game theory is utilized to model the decision-making process. Then, a probing behavior is generated by the proposed game theoretic model, and is further applied to control the vehicle via Markov chain. To validate the feasibility and effectiveness, the proposed method is tested through a series of experiments and compared with existing approaches. In addition, Turing tests are conducted to quantify the human-likeness of the proposed algorithm. The experiment results show that the proposed Bayesian game theoretic framework can effectively generate representative scenes of human-like decision-making during autonomous vehicle interactions, demonstrating its feasibility and effectiveness. Corresponding author(s) Email:   [email protected]  

Scene flow tracks the three-dimensional (3D) motion of each point in adjacent point clouds. It provides fundamental 3D motion perception for autonomous driving and server robot. Although the Red Green Blue Depth (RGBD) camera or Light Detection and Ranging (LiDAR) capture discrete 3D points in space, the objects and motions usually are continuous in the macro world. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the Generative Adversarial Networks (GAN) is utilized to self-learn 3D scene flow with no need for ground truth. The fake point cloud of the second frame is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) scene flow dataset show that our method realizes promising results without ground truth. Just as human, the proposed method can identify the similar local structures of two adjacent frames even without knowing the ground truth scene flow. Then, the local correspondence can be correctly estimated, and further the scene flow is correctly estimated.   Corresponding author(s) Email: [email protected]

Single-use jumping robots that are mass-producible and biodegradable could be quickly released for environmental sensing applications. Such robots would be pre-loaded to perform a set number of jumps, in random directions and with random distances, removing the need for onboard energy and computation. Stochastic jumpers build on embodied randomness and large-scale deployments to perform useful work. This paper introduces simulation results showing how to construct a large group of stochastic jumpers to perform environmental sensing, and the first demonstration of robot prototypes that can perform a set number of sequential jumps, have full-body sensing, and are well suited to be made biodegradable. Corresponding author(s) Email: [email protected] [email protected]