The second unit is the visual feedback and optimal control unit, which provides control feedback for the surgical instruments tracking task. In this unit, a deep keypoints detection network is developed to predict the pixel positions of keypoints on the surgical instruments. Unlike the existing object detection methods, a pose estimation method is developed to detect the keypoints on surgical instruments. The pose estimation method can directly regress the pixel coordinates of the keypoints on surgical instruments, instead of detecting the whole area as the object detection method. This method can increase the precision of keypoints detection, and is beneficial to the subsequent control tasks. The keypoints needed as the tracking point can be selected flexibly according to different surgery stages in the following control system. In addition, weights of different surgical instruments can be set by surgeons when multiple instruments are used according to their requirements.
Experimental Section/Methods
We build an experiment platform based on the proposed automatic tracking system to validate the data-driven control method. A 2mm diameter pinhole camera with a resolution of 400×400 pixels is fixed at the end effector of the cable-driven continuum manipulator (Intuitive Surgical, California, USA). Sensing image can be obtained with a frequency of 30 Hz through a USB port. The continuum manipulator has four connected joints, which can be divided into two groups. Joint1 and Joint4 control movement in the X-axis direction, and both are actuated by a brushless motor (Maxon Group, Sachseln, Switzerland). Joint2 and Joint3 control movement in the Y-axis direction and are actuated by another brushless motor. The continuum manipulator is fixed in the Z-axis direction, which could ensure safety once the initial position is determined. Elmo drivers (Elmo Motion Control Ltd., Israel) are used to actuate motors precisely by receiving command from TwinCAT3 (Beckhoff Automation GmbH & Co. KG, Germany) through the EtherCAT bus. The Large Needle Driver and the Grasping Retractor (Intuitive Surgical, California, USA) are used in this work.[4] The Large Needle Driver holds the needle while forcing the tip through the tissue to complete the suturing task in RMIS. Grasping Retractor are used to retract the tissue to reveal the surgical scope so that the surgeon can explore the surgical area and perform surgery.
In order to calculate the infinite-dimensional approximation of the Koopman Operator, a set of system states and corresponding inputs were collected. We collected 100 trials with random initial inputs of the motors. Then the input of the two motors varies one cycle according to the trigonometric function to generate the states. Following this rule, the continuum laparoscope runs 500 steps a cycle in the workspace. Finally, we collected 50,000 pairs of data about system states and inputs. The Koopman operator of the continuum laparoscope system is estimated according to the system identification method.
Furthermore, we also evaluate the dynamic uncertainty of the continuum laparoscope system based on the data collection. Dynamical uncertainty means that the same inputs may lead to slightly different outputs. This uncertainty is caused by motor encoder error and system assembly error. Specifically, the system gets input that changes periodically.