Autonomous Collision-Free Navigation of Microvehicles in Complex

Subscriber access provided by Universiteit Utrecht

Article

Autonomous Collision-Free Navigation of Microvehicles in Complex and Dynamically Changing Environments Tianlong Li, Xiaocong Chang, Zhiguang Wu, Jinxing Li, Guangbin Shao, Xinghong Deng, Jianbin Qiu, Bin Guo, Guangyu Zhang, Qiang He, Longqiu Li, and Joseph Wang ACS Nano, Just Accepted Manuscript • DOI: 10.1021/acsnano.7b04525 • Publication Date (Web): 12 Aug 2017 Downloaded from http://pubs.acs.org on August 12, 2017

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a free service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are accessible to all readers and citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

ACS Nano is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

Autonomous Collision-Free Navigation of Microvehicles in Complex and Dynamically Changing Environments Tianlong Li,1, 2, § Xiaocong Chang,1, 2, § Zhiguang Wu,1, 2, § Jinxing Li,2, § Guangbin Shao,1 Xinghong Deng,1 Jianbin Qiu,1 Bin Guo,1 Guangyu Zhang,1 Qiang He,1 Longqiu Li,1, * and Joseph Wang2, * 1 State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, Heilongjiang 150001, China 2 Department of Nanoengineering, University of California San Diego, La Jolla, California

92093, United States §

T. L., X. C., Z. W. and J. L. contributed equally to this work.

* E-mail: [email protected], [email protected].

ABSTRACT Self-propelled micro- and nanoscale robots represent a rapidly emerging and fascinating robotics research area. However, designing autonomous and adaptive control systems for operating micro-/nanorobotics in complex and dynamically-changing environments, which is a highly demanding feature, is still an unmet challenge. Here we describe a smart microvehicle for precise autonomous navigation in complicated environments and traffic scenarios. The fully autonomous navigation system of the smart microvehicle is composed of a microscope-coupled CCD camera, an artificial intelligence planner, and a magnetic field generator. Microscopecoupled CCD camera provides real-time localization of the chemically-powered Janus microsphere vehicle and environmental detection for path planning to generate optimal collisionfree routes, while the moving direction of the microrobot towards a reference position is determined by the external electromagnetic torque. Real-time object detection offers adaptive path planning in response to dynamically changing environments. We demonstrate that the autonomous navigation system can guide the vehicle movement in complex patterns and in the presence of dynamically changing obstacles, and in complex biological environments. Such navigation system for micro-/nanoscale vehicles, relying on vision based close-loop control and 1 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

path planning, is highly promising for their autonomous operation in complex dynamic settings and unpredictable scenarios expected in variety of realistic nanoscale scenarios. KEYWORDS: micro-/nanorobot, artificial intelligence, targeted delivery, autonomous navigation, collision-free Artificial micro-/nanomachines that convert local fuels or external energies into kinetic movement for overcoming viscous drag at low Reynolds number regimes have markedly extended the reach of human beings in biomedical and nanomanipulation tasks.1-13 The operation of micro-/nanoscale machines and propellers have proved to be useful for drug delivery,14-19 biosensing,20,21 environmental remediation,22-24 active materials assembly,25-28 and nanoscale lithography and imaging.29,30 Although the fabrication, locomotion, and functionalization of micro-/nanomachines have been thoroughly investigated, the control and navigation of micro/nanomachines are restricted to simple closed-loop control.31-38 By using the closed-loop control system, the micromotors were commanded to move along a pre-defined route toward a fixed point in two-dimensional (2D) or three-dimensional (3D) spaces. However, moving micro/nanomotors on demand in complicated and dynamically changing environments without human assistance is still an unmet challenge. In addition, the lack of artificial intelligence (AI) impedes their operation for autonomous path planning in the fields of biomedicine and nanomanipulation. While considerable efforts have been devoted to advancing the motion control of nanoscale motors,39 limited attention has been given for achieving fully autonomous operation of micro/nanovehicles in complex settings and unpredictable traffic scenarios. Therefore, there are growing demands for exploiting an intelligent control system for micro-/nanomotor to navigate without colliding with obstructions in a complicated environment. While self-driving vehicles are at the heart of industrial and academic research activities, no efforts have been devoted for realizing such autonomous movement of micro-/nanoscale vehicles. Achieving autonomous operation of nanovehicles could benefit a wide range of relevance nanoscale movement challenges associated with unpredictable nanoscale ‘traffic’ settings, e.g. nanomotor traffic jams during active transport operation in narrow microchannels.40 2 ACS Paragon Plus Environment

Page 2 of 23

Page 3 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

Path planning problems arise in diverse fields such as robotics, manufacturing, and virtual prototyping.41,42 Such problems involve searching the unknown space for a collision-free path that connects a given start and target configurations, while satisfying the constraints imposed by complicated obstacles. The process of path planning can be classified into global path planning and local path planning. Global path planning methods are usually conducted in a completely known environment. Different attempts to address this problem have been carried out in connection to artificial potential field and search algorithm.43,44 On the other hand, local path planning techniques are more efficient in robot navigation in an unknown and dynamically changing settings. Neural network and fuzzy logic approaches, which offer an attractive alternative for building reactive systems, are the efficient local path planning methods.45-48 Hence, combining the global path planning with local path planning can offer effective navigation in complex dynamically changing environments. Herein, we introduce a smart microvehicle for precise autonomous navigation in complicated and dynamically changing environments through optimal path planning. The autonomous navigation system of microvehicle is composed of an artificial intelligence (AI) planner, a microscope-coupled CCD camera, and a magnetic field generator, and addresses the challenge of accurate path planning in unknown environments. The CCD camera provides the real time localization of the microvehicle and environmental detection. Based on artificial potential field, searching algorithms and fuzzy logic approach, AI planner can generate optimal collision-free path and offers high localization accuracy. The locomotion of the microvehicle along the proposed route is navigated by the electromagnets. Similar to their large vehicle counterparts, the autonomous navigation of microvehicles entails collision-free movement in dynamic environments. We demonstrate that such autonomous control diverts the vehicle route away from stationary and moving obstacles and navigates it in complicated environments. Such control system allows handling realistic nanoscale ‘traffic’ scenarios where the dynamics of other traffic participants or unexpected events must be considered. By adding the visual recognition ability to the path planner, we demonstrate that the control system is able to distinguish different 3 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

biological entities, such as cancer cells and red blood cells, and then further guide the microvehicle to a desired disease site. Such a microvehicle based on autonomous navigation system can open the door for a wide variety of applications ranging from nanomanipulation to precise medical treatment.

RESULTS AND DISCUSSION The setup of autonomous navigation system is shown in Fig. 1a, which is composed of a CCD camera, an AI planner, and a magnetic field generator. The microscope-coupled camera provides real-time visual detection for the AI planner. The AI planner is split into three functional modules: a) a computer vision module for tracking the microvehicle and detecting obstacles in its environment; b) a motion planner, which consists of an artificial potential field module, a search algorithm module and a fuzzy logic module to generate optimal obstacle-free path between starting point and destination; c) a magnetic motion controller to manipulate the microvehicle movement along a predesigned path. As shown in Fig. 1b, digital microscopy images from the camera are sent back to the feature extraction processor to identify features and compute the global occupancy map and local occupancy map. Thereafter, these maps are used as inputs to the AI planner which gives instruction to the magnetic field generator for local navigation of the microvehicle. Our control method thus consists of the following three steps (see Fig. 1c): (Ⅰ) sensing and detecting the unknown environment, followed by building a map representing the unoccupied and occupied spaces of the environment; ( Ⅱ ) reconstructing collision-free microvehicle trajectories in the global occupancy map through AI planner; (Ⅲ) optimizing the input signal for the magnetic coils generating a desired orientation of the magnetic field.

4 ACS Paragon Plus Environment

Page 4 of 23

Page 5 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

Figure 1. Schematic of Autonomous Navigation System for adaptive control of microvehicle. (a) Schematic of the autonomous navigation system for microvehicle. (b) Overall process flow of the closed-loop feedback autonomous navigation mechanism for the microvehicle. (c) Schematic of autonomous navigation mechanism for microvehicle: (I) sensing, (II) decision and (III) action.

The autonomous obstacle avoidance ability of microvehicle was demonstrated first in the presence of a stationary object (see Figure 2). The navigation strategy is shown in Fig. 2b. First, 5 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

the computer vision module of autonomous navigation system is triggered to sense the single object and then creates a global map, representing the unoccupied and occupied spaces of the environment surrounding the microvehicle. Subsequently, the motion planner reconstructs the most favorable collision-free trajectory for the microvehicle. Artificial potential field, expressed in Cartesian workplace of microvehicle, is employed to obtain the optimal path between the initial and target points, using repulsive potential and attractive potential for the obstacle and target, respectively. These potentials are combined to form a composite potential to generate collision-free trajectories between the initial point and target destination. Finally, the optimal control signal is applied to the magnetic field generator, steering the moving microvehicle along the selected collision-free pathway. To investigate the navigation behavior of microvehicle, a stationary obstacle, with dimension of 100×100×10 µm, has been fabricated by photolithography technique. A 4.74-µm-diameter Janus sphere micromotor, propelling in an aqueous solution containing 5% H2O2 fuel, has been used to demonstrate the concept. The Janus micromotors are prepared by coating silica spheres with thin hemispherical Ti/Ni/Pt metallic layers for magnetic steering and catalytic propulsion in the H2O2 solution. The real-time binary images (following image processing) and the corresponding path planning are presented in Figs. 2c-g. The manipulation aimed to transport this microvehicle autonomously to the target point. By using the proposed path planner, a collision-free path was obtained, guiding the microvehicle to bypass the single obstacle under the controllable magnetic field. Trajectories images from the CCD camera are shown in Figs. 2h-l. The estimated planning path and actual trajectory are displayed in Fig. 2m, where the red solid line and black circles show the planning path generated by the AI Planner and the tracked microvehicle position at 1s intervals, respectively (see Supplementary Video 1). Here, the average path error between the planning path and actual trajectory is found to be approximately 0.8 body length. Fig. 2n displays the dependence of the collision probability upon the safety distance between the microvehicle and the stationary obstacle, and the microvehicle velocity using safety distances between 0.2 to 1.0 body length and speeds ranging from 2 to 10 body 6 ACS Paragon Plus Environment

Page 6 of 23

Page 7 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

length/s. As expected, the collision probability increases upon increasing the velocity of microvehicle and decreasing the safety distance. For a microvehicle moving at a speed higher than 6 body length/s, and using a safety distance smaller than 0.25 vehicle body length, the collision probability is 1, indicating that the microvehicle cannot avoid the obstacle under the autonomous navigation system control.

Figure 2: Autonomous navigation in capability of microvehicle to avoid a stationary obstacle. (a-b) Schematic illustration of microvehicle locomotion in the presence of a stationary obstacle (a) without, and (b) with AI control. Trajectories locomoted by autonomous navigation of microvehicle in visualization of stationary obstacle map at 0 s (c), 8 s (d), 16 s (e), 24 s (f) and 32 s (g). Scale bar, 20 µm. Fuel level, 5% H2O2. Microvehicle location at 0 s (h), 8 s (i), 16 s (j), 24 s (k) and 32 s (l). Scale bar, 20 µm. (m) Comparison between the estimated route and actual path of autonomous navigation for microvehicle. (n) Collision probability as a function of the 7 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

microvehicle/stationary obstacle safety distance and the microvehicle velocity. Conditions, Janus microsphere vehicles (4.74-µm-diameter) propelling in aqueous solution containing 5% H2O2 fuel.

Path searching and planning are crucial for the successful operation of autonomous vehicles. To evaluate the autonomous navigation ability of the smart microvehicle, two different complex maze-like microstructures were fabricated using photolithography to serve as swimming circumstance. As shown in Fig. 3a, similar to the navigation in the presence of stationary obstacle, the autonomous navigation in a complicated environment consists of sensing, decision and action. However, due to the narrow corridor effect, the microvehicle guided by the potential field could behave erratically, oscillating in a sustained manner between the two walls of the corridor. Path planning in complicated environments thus requires different techniques that guide the movement through narrow corridors. Dijkstra algorithm is an attractive algorithm that is mainly used for determining the shortest and optimal paths.44 The algorithm modifies the breadth-first strategy by expanding the lowest-cost leaf node first, as measured by the path cost function. Figs. 3b-e demonstrate the actual navigation of vehicle in a chiral micromaze containing a single exit. The microvehicle is initially pre-located at the center of the chiral maze. Trajectories images from the CCD camera and after feature extraction of the autonomous microvehicle navigation are shown in Fig. 3d and Fig. 3e, respectively. Due to the presence of noise, the microvehicle can drift from the planning path. However, the AI planner is able to measure the error and steer the microvehicle simultaneously. Under the autonomous navigation, the microvehicle can be guided to the exit of the chiral micromaze without any collision (see Supplementary Video 2). The autonomous navigation system can also steer the microvehicle in a complex micromaze structure with multiple possible paths, with the Dijkstra algorithm based path planner selecting the most favorable route among all potential collision-free paths. Figs. 3f-i display the optimized autonomous navigation of a microvehicle passing thorough a micromaze with two possible paths. These images illustrate that the navigation control offers collision-free


Page 8 of 23

Page 9 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

movement to the target exit destination along the optimal and shorter path (see Supplementary Video 2).

Figure 3. Autonomous navigation of microvehicle in a complicated environment. (a) Schematic illustrating the autonomous navigation for microvehicle moving in a micromaze: (I) sensing, (II) path planning, (III) decision and (Ⅳ) action. (b) Chiral micromaze. Scale bar, 50 µm. (c) Initial position of microvehicle Scale bar, 50 µm. (d) Real-time path planning. Scale bar, 50 µm. (e) Trajectories steered by autonomous navigation of microvehicle through a chiral micromaze in a 5% H2O2 fuel solution. Scale bar, 20 µm. (f) Complex micromaze with multiple paths. Scale bar, 50 µm. (g) Initial position of the microvehicle. Scale bar, 50 µm. (h) Real-time path planning. Scale bar, 50 µm. (i) Trajectories steered by autonomous navigation of the microvehicle through the complex micromaze after optimized path planning. Scale bar, 50 µm. Conditions, Janus microsphere motors (4.74-µm-diameter) propelling in aqueous solution containing 5% H2O2 fuel. Avoiding dynamic obstacles is expected to be a major concern in realistic future nanoscale traffic settings. Our system can sense its surroundings and detect dynamic obstacles, such as other microvehicles. The real-time video tracking and decision making enable the microvehicle to avoid collision with such dynamically moving objects. Fuzzy logic approach is employed to 9 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 10 of 23

avoid the dynamic obstacle. This approach is credited with being an adequate methodology for designing robust controllers that can deliver a satisfactory performance in the presence of large amounts of variability in the parameters.45,46 The rule base of fuzzy logic approach is constructed based on human experience. Here, triangular membership functions were applied to change the performance of the system. The performance of fuzzy logic based path planner for avoiding dynamic moving objects is demonstrated using different working conditions. As shown in Fig 4a, the destination is on the right while the dynamic Janus motor obstacle locomotes from top (see Fig 4a). Trajectories images from CCD camera and the feature extraction of the vehicle motion are shown in Fig. 4b and Fig. 4c, respectively. When the microvehicle starts to move to the target point, the states of the obstacle-avoidance (OA) behaviors in front, right and left are far since the dynamic obstacle is distant (see details in Supplementary Information). When the dynamic obstacle is approaching the microvehicle, the states of OA behaviors in front, right and left become middle, far and near, respectively. The microvehicle will turn left to avoid the dynamic obstacle. The real-time turn angle can be calculated by the fuzzy logic based path planner. When the microvehicle is far from the dynamic obstacle, the states of OA behaviors in front, right and left become far. The microvehicle locomotes towards the target point under the control of AI planner (see Supplementary Video 3). Another example is presented in Fig 4d, with both the target point and the dynamic obstacle are on the top, and the microvehicle is moving from the bottom in the opposite direction. Under the real-time fuzzy logic inference, the microvehicle can avoid the dynamic obstacle and locomote towards the target point. Trajectories images (from the CCD camera) of the vehicle locomotion before and after feature extraction under the autonomous navigation system are shown in Fig. 4e and Fig. 4f, respectively (see Supplementary Video 3). The fuzzy logic based path planner can also be used for other dynamic obstacle avoidance using different obstacle shapes (e.g., nanowire vehicles). As illustrated in Fig. 4g, the target point is on the left, and nanowire dynamic obstacle vehicle locomotes from top. The fuzzy logic based path planner performs well towards real-time collision-free path planning (see Supplementary Video 3). Fig. 10 ACS Paragon Plus Environment

Page 11 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

4h and Fig. 4i display the real-time planned path and corresponding trajectories images of the microvehicle movement after such feature extraction, respectively. The microvehicles are able to respond to moving obstacles within 0.05s and avoid the collision with a speed of 6 body length/s. Further improvements in the nanoscale traffic safety under dynamically changing conditions (e.g., sudden stop of one of the vehicles) based on adding autonomous-braking or distancecontrol capabilities are currently being investigated.

Figure 4. Smart microvehicle avoiding collision with dynamically moving object. (a) Schematic illustrating dynamic obstacle avoidance movable for microvehicle in the condition of target point on the right and obstacle on the top. (b) Real-time path planning. Scale bar, 20 µm. (c) Trajectories steered by autonomous navigation of microvehicle, avoiding collision with dynamic obstacle. Scale bar, 20 µm. (d) Schematic illustrating dynamic obstacle avoidance movable for microvehicle in the condition of both target and dynamic obstacle on the top. (e) Real-time path planning. Scale bar, 20 µm. (f) Trajectories steered by autonomous navigation of microvehicle avoiding collision with dynamic obstacle. Scale bar, 20 µm. (g) Schematic 11 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 12 of 23

illustrating the microvehicle avoidance of the nanowire-motor dynamic obstacle. (h) Real-time path planning. Scale bar, 20 µm. (i) Trajectories steered by autonomous navigation of microvehicle avoiding collision with dynamic obstacle. Conditions, Janus microsphere motors (4.74-µm-diameter) propelling in aqueous solution containing 5% H2O2 fuel.

The growing sophistication of our autonomously moving microvehicles indicates considerable potential in the biomedicine field. However, the lack of visual recognition based autonomous navigation limits the utility of smart microvehicle for disease diagnostics, delivery of therapeutic agents, and precision surgery automatically. Besides autonomous navigation, our smart microvehicles are capable of identifying the feature of biological targets. By integrating the visual recognition capacity, the microvehicle is capable to distinguish different cells, such as cancer cells and red blood cells, thus enabling the precision control of the microvehicle towards desired biological target. As shown in Fig. 5a, our approach consists of three steps, including sensing the red blood cells and cancer cells through visual recognition, determining the collision-free trajectories, and instructing the magnetic field generator to steer the microvehicle along the proposed trajectory toward cancer cells. The original images from CCD of cancer cells, red blood cells, and mixed cells are shown in Figure 5b, c and d, respectively. The features of the red blood cells and cancer cells, particularly their size and morphology, were extracted using Canny edge detection.49 The first step of canny algorithm is to smooth image using two dimensional Gaussian function. The second step is to calculate the magnitude and direction of image gradient. After acquiring the gradient magnitude image, the system performs non-maximum suppression on the image to accurately define the edges. Finally, the edges are checked and connected. The binary image of the cancer cells, red blood cells, and mixed cells are shown in Figure 5b, c and d, respectively. The diameter of red blood cells and cancer cells are 5 and 20 micrometers, respectively. As shown in Fig. 5h, the red blood cells and cancer cells can be distinguished by the AI Planner by recognizing differences in their shape and size. The red blood cells and cancer cells are regarded as the obstacle and goal point, respectively, i.e., surrounding the cancer cells and red blood cells 12 ACS Paragon Plus Environment

Page 13 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

by attractive and repulsive potential functions, respectively. An optimal collision-free path is obtained between start and goal points. Motion controller performs magnetic generator to steer the microvehicle along the generated path. Fig. 5i displays the actual trajectory of the microvehicle steered by the autonomous navigation system towards the cancer cells. This image indicates that the smart microvehicle can successfully recognize the cancer cells and efficiently approach the target (see Supplementary Video 4).

Figure 5. Feature recognition ability of smart microvehicle for theranostic applications. (a) Schematic illustrating autonomous navigation for microvehicle in theranostic application: (I) sensing, (II) decision and (III) action. (b), (c) and (d) are original images of cancer cell, red blood cell and mixed cells, respectively. Scale bar, 50 µm. (e), (f) and (g) are after identification images of cancer cell, red blood cell and mixed cells, respectively. Scale bar, 50 µm. (h) Real-time path 13 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 14 of 23

planning by microvehicle in mixed cells solution. Scale bar, 50 µm. (i) Trajectories steered by autonomous navigation of microvehicle in the mixed cells solution. Final location of smart microvehicle in the cancer cell with red blood cell solution mixture; scale bar, 50 µm. Experiments were performed with 10-µm-diameter Janus micromotors propelling in aqueous solution with 2.5% H2O2 as chemical fuel.

CONCLUSIONS We introduced an advanced autonomous navigation system for microvehicle with high performance path planning in diverse environments and realistic settings. This system was composed of a camera, AI Planner, and a magnetic field generator. The microscope-coupled camera can provide real-time visual feedback. The AI Planner can detect dynamic obstacles, generate optimal obstacle-free path and drive the microvehicle along the generated path. This autonomous navigation system guides the microvehicle to locomote in complex and dynamically changing environments through optimal path planning. By visual recognition, the microvehicle is able to distinguish different biological targets and steer towards its predetermined target in biological environment. Future efforts should lead to more sophisticated autonomous vehicles, capable of interacting with the environment and to further improvements and minimization of trajectory errors. These will include features such as steering actuations, autonomous braking, adaptive ‘cruise’ (speed) control and lane-keeping. By further minimizing trajectory errors, these advances could address major challenges associated with nanoscale traffic jams and high-volume congestion. Such innovative autonomous navigation approach thus dramatically enhances the intelligence of micro/nanorobotic systems towards practical applications in diverse biomedical operations and nanoscale manipulation.

METHODS/EXPERIMENTAL Synthesis of smart microvehicles. The smart microvehicles were prepared by using silica microspheres (with diameters of 4.74 µm and 10 µm, Bangs Laboratories, Fishers, IN, USA) as the base particles. For the microvehicle, the 4.74 µm and 10 µm silica microspheres were placed 14 ACS Paragon Plus Environment

Page 15 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

onto glass slides and coated with a 20 nm titanium layer at 1 Å s−1, a 80 nm nickel layer at 2 Å s−1, and a 10 nm platinum layer at 1 Å s−1 using Temescal BJD 1800 E-beam Evaporator. After a brief sonication in ultrapure water, the microvehicles were released from the glass slide and dispersed into ultrapure water. The microvehicle were stored in the ultrapure water until use. Synthesis of dynamic obstacles. The Janus microvehicle based dynamic obstacle were prepared by using silica microspheres (10 µm mean diameter, Bangs Laboratories, Fishers, IN, USA) as the base particles. The 10 µm silica microspheres were placed onto glass slides and coated with a 20 nm titanium layer at 1 Å s−1, and a 10 nm platinum layer at 1 Å s−1 using Temescal BJD 1800 E-beam Evaporator. After a brief sonication in ultrapure water, the Janus microvehicle dynamic obstacles were released from the glass slide, dispersed into ultrapure water, and stored in the ultrapure water until use. The nanowire motor based dynamic obstacle composed of Au and Pt segments were prepared by electrodepositing the corresponding metals into 400 nm polycarbonate (PC) membrane template (Catalogue no. 800282; Whatman, New Jersey, USA). A thin silver film was first sputtered on the branched side of the alumina membrane or back side of the PC membrane to serve as a working electrode. The membrane was assembled in a plating cell with aluminum foil serving as an electrical contact for the subsequent electrodeposition. A sacrificial silver layer was electrodeposited into the branched area of the membrane using a silver plating solution (1025 [email protected] Troy per Gallon; Technic Inc., Anaheim, CA, USA) and a total charge of 3.0 C at 0.90V (versus Ag/AgCl, in connection to a Pt wire counter electrode). This was followed by an electrodeposition of gold for 1.5 C from a gold plating solution (Orotemp 24 RTU RACK; Technic Inc.) at -1.0 V. Subsequently, the Pt segment galvanostatically deposited at -2 mA for 120 min using a platinum plating solution (Platinum RTP; Technic Inc.). The sputtered and sacrificial silver layers were simultaneously removed by mechanical polishing using cotton tip applicators soaked with 35% HNO3. Then, the PC membrane was removed in methylene chloride for 15 min to completely release the nanowire vehicle based dynamic obstacles. The corresponding nanowire dynamic obstacle vehicles were collected by centrifugation at 6,000 15 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 16 of 23

r.p.m. for 1 min and washed repeatedly with methylene chloride, followed by ethanol (two times) and ultrapure water (three times). In vitro experiment. The culture of HeLa cells was carried out according to the Standard cell culture procedure. Briefly, the HeLa cells were cultured in dulbecco's modified eagle medium (DMEM), 1% penicillin and streptomycin, and 10% fetal calf serum at 37℃ in an atmosphere of 5% CO2. The cells were diverted to a 60mm polystyrene (PS) petri dish for test by the standard trypsin technique. The red blood cells were collected from Harbin Institute of Technology hospital and rinsed with PBS for 3 times. Prior to the movement of the microvehicle in the cell media, the HeLa cells, cultured in 60 mm PS petri dish and the red blood cells, were washed twice with PBS to remove the DMEM medium and plasms, respectively, followed by the addition of the red blood cells into the HeLa media. Then the PBS containing 2.5% H2O2 and the microvehicle were incubated with the HeLa/red blood cells media. The viability of HeLa cells was evaluated by staining the cells with propidium iodide, which indicates that the most HeLa cells are viable after 30 mins. Therefore, the petri dish was transferred directly to the setup of the autonomous navigation system and the relative experiments were performed at 37oC by using a miniheater. Magnetic field generator. The magnetic field setup consisted of a laptop computer, a power amplifier and a set of eight inductors (see Supplementary Figure 1). The output signal from the laptop computer was an analog voltage. A 50 W audio amplifier was used to amplify the analog signal. The amplified signal was used to drive a set of eight inductors, which generated a uniform magnetic field in the plane of the space between inductors. Autonomous navigation process. The CCD camera connected with the microscope can provide visual feedback. Images from the CCD camera are inputs to the feature extraction process which recognizes features and computes global occupancy map and local occupancy map. The Canny edge detection is employed for feature extraction. Thereafter, these maps are 16 ACS Paragon Plus Environment

Page 17 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

used as inputs to the AI planner for path planning. The artificial potential field, searching algorithms and fuzzy logic approach are employed to generate collision-free trajectory in stationary obstacle, complicated environment and dynamic obstacle, respectively. Finally, the AI planner selects new waypoints and passes these waypoints to the field generator which rotated the microvehicle from its current direction to the new direction. Equipment. Template electrochemical deposition of the microvehicles was carried out with a CHI 660D potentiostat (CH Instruments, Austin, TX, USA). Videos were captured at 45 frames per second by an upright optical microscope (Nikon Eclipse Ni-U), coupled with a 40 objective (CFI Plan Fluor, Nikon, N.A. 0.75) and a Hamamatsu digital camera C11440. Autonomous navigation system is running with a Thinkpad P50 (Lenovo group Beijing, China). A compact 50W audio amplifier (SMSL, SA-S3) was used to amplify the signal. The magnetic field is generated by 10 mH iron core inductors (Erse Audio, 266-580).

ACKNOWLEDGMENT This project received support from National Natural Science Foundation of China (51521003 and 51175129), Key Laboratory of Micro-systems and Micro-structures Manufacturing of Ministry of Education (2016KM004), Self-Planned Task of State Key Laboratory of Robotics and System (HIT) (SKLRS 201706A), Program of Introducing Talents of Discipline to Universities (Grant No. B07018) and the U.S. Defense Threat Reduction Agency Joint Science and Technology Office for Chemical and Biological Defense (Grant Number HDTRA1-14-10064). Competing interests: The authors declare no competing financial interests. Supporting Information Available: Additional figures and videos as described in the text. This material is available free of charge via the Internet at http://pubs.acs.org.


ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

REFERENCES [1]

Li, J.; de Avila, B. E. F.; Gao, W.; Zhang, L.; Wang, J. Micro/Nanorobots for

Biomedicine: Delivery, Surgery, Sensing, and Detoxification. Sci. Robot. 2017, 2, eaam6431. [2]

Mei, Y.; Solovev, A. A.; Sanchez, S.; Schmidt, O. G. Rolled-up Nanotech on Polymers:

from Basic Perception to Self-Propelled Catalytic Microengines. Chem. Soc. Rev. 2011, 40, 2109-2119. [3]

Wang, H.; Pumera, M. Fabrication of Micro/Nanoscale Motors. Chem. Rev. 2015, 115,

8704-8735. [4]

Mallouk, T. E.; Sen, A. Powering Nanorobots. Sci. Am. 2009, 300, 72-77.

[5]

Chen, C.; Mou, F.; Xu, L.; Wang, S.; Guan, J.; Feng, Z.; Wang, Q.; Kong, L.; Li, W.;

Wang, J.; Zhang, Q. Light-Steered Isotropic Semiconductor Micromotors. Adv. Mater. 2017, 29, 1603374. [6]

Sanchez, S.; Soler, L.; Katuri, J. Chemically Powered Micro- and Nanomotors. Angew.

Chem. Int. Edit. 2015, 54, 1414-1444. [7]

Qiu, T.; Lee, T. C.; Mark, A. G.; Morozov, K. I.; Munster, R.; Mierka, O.; Turek, S.;

Leshansky, A. M.; Fischer, P. Swimming by Reciprocal Motion at Low Reynolds Number. Nat. Commun. 2014, 5, 5119. [8]

Xu, T.; Gao, W.; Xu, L.; Zhang, X.; Wang, S. Fuel-Free Synthetic Micro-/Nanomachines.

Adv. Mater. 2017, 29, 1603250. [9]

Dong, R.; Hu, Y.; Wu, Y.; Gao, W.; Ren, B.; Wang, Q.; Cai, Y. Visible-Light-Driven Bioi-

based Janus Micromotor in Pure Water. J. Am. Chem. Soc. 2017, 139, 1722-1725. [10]

Jang, B.; Hong, A.; Kang, H. E.; Alcantara, C.; Charreyron, S.; Mushtaq, F.; Pellicer, E.;

Buchel, R.; Sort, J.; Lee, S. S.; Nelson, B. J.; Pane, S. Multi-Wavelength Light Responsive Au/B-TiO2 Janus Micromotors. ACS Nano 2017, 11, 6146-6154.


Page 18 of 23

Page 19 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

[11]

Jang, B.; Wang, W.; Wiget, S.; Petruska, A. J.; Chen, X.; Hu, C.; Hong, A.; Folio, D.;

Ferreira, A.; Pane, S.; Nelson, B. J. Catalytic Locomotion of Core–Shell Nanowire Motors. ACS Nano 2016, 10, 9983-9991. [12]

Kherzi, B.; Pumera, M. Self-Propelled Autonomous Nanomotors Meet Microfluidics.

Nanoscale 2016, 8, 17415-17421. [13]

Chen, X.; Hoop, M.; Mushtaq, F.; Siringil, E.; Hu, C.; Nelson, B. J.; Pane, S. Recent

Developments in Magnetically Driven Micro- and Nanorobots. Appl. Mater. Today. 2017, 9, 37−48. [14]

Schuerle, S.; Vizcarra, I. A.; Moeller, J.; Sakar, M. S.; Ozkale, B.; Lindo, A. M.;

Mushtaq, F.; Schoen, I.; Pane, S.; Vogel, V.; Nelson, B. J. Robotically Controlled Microprey to Resolve Initial Attack Modes Preceding Phagocytosis. Sci. Robot. 2017, 2, eaah6094. [15]

Wu, Z.; Li, T.; Li, J.; Gao, W.; Xu, T.; Christianson, C.; Gao, W.; Galarnyk, M.; He, Q.;

Zhang, L.; Wang, J. Turning Erythrocytes into Functional Micromotors. ACS Nano 2014, 8, 12041-12048. [16]

Li, J.; Angsantikul, P.; Liu, W.; de Avila, B. E. F.; Thamphiwatana, S.; Xu, M.; Sandraz,

E.; Wang, X.; Delezuk, J.; Gao, W.; Zhang, L.; Wang, J. Micromotors Spontaneously Neutralize Gastric Acid for pH-Responsive Payload Release. Angew. Chem. Int. Edit. 2017, 56, 2156-2161. [17]

Ma, X.; Hahn, K.; Sanchez, S. Catalytic Mesoporous Janus Nanomotors for Active Cargo

Delivery. J. Am. Chem. Soc. 2015, 137, 4976-4979. [18]

Wang, W.; Li, S.; Mair, L.; Ahmed, S.; Huang, T.; Mallouk, T. E. Acoustic Propulsion of

Nanorod Motors Inside Living Cells. Angew. Chem. Int. Edit. 2014, 53, 3201-3204. [19]

Wu, Z.; Li, T.; Gao, W.; Xu, T.; Jurado-Sanchez, B.; Li, J.; Gao, W.; He, Q.; Zhang, L.;

Wang, J. Cell-Membrane-Coated Synthetic Nanomotors for Effective Biodetoxification. Adv. Funct. Mater. 2015, 25, 3881-3887. [20]

Ma, X.; Wang, X.; Hahn, K.; Sanchez, S. Motion Control of Urea-Powered 19 ACS Paragon Plus Environment

ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 20 of 23

Biocompatible Hollow Microcapsules. ACS Nano 2016, 10, 3597-3605. [21]

Dong, R.; Li, J.; Rozen, I.; Ezhilan, B.; Xu, T.; Christianson, C.; Gao, W.; Saintillan, D.;

Ren, B.; Wang, J. Vapor-Driven Propulsion of Catalytic Micromotors. Sci. Rep. 2015, 5, 13226. [22]

Li, J.; Singh, V.; Sattayasamitsathit, S.; Orozco, J.; Kaufmann, K.; Dong, R.; Gao, W.;

Jurado-Sanchez, B.; Fedorak, Y.; Wang, J. Water-Driven Micromotors for Rapid Photocatalytic Degradation of Biological and Chemical Warfare Agents. ACS Nano 2014, 8, 11118-11125. [23]

Li, T.; Li, L.; Song, W.; Wang, L.; Shao, G.; Zhang, G. Self-Propelled Multilayered

Microrockets for Pollutants Purification. ECS J. Solid State Sc. 2015, 4, S3016-S3019. [24]

Gao, W.; Feng, X.; Pei, A.; Gu, Y.; Li, J.; Wang, J. Seawater-Driven Magnesium based

Janus Micromotors for Environmental Remediation. Nanoscale 2013, 5, 4696-4700. [25]

Xu, T.; Soto, F.; Gao, W.; Dong, R.; Garcia-Gradilla, V.; Magana, E.; Zhang, X.; Wang, J.

Reversible Swarming and Separation of Self-Propelled Chemically Powered Nanomotors under Acoustic Fields. J. Am. Chem. Soc. 2015, 137, 2163-2166. [26]

Palacci, J.; Sacanna, S.; Steinberg, A. P.; Pine, D. J.; Chaikin, P. M. Living Crystals of

Light-Activated Colloidal Surfers. Science 2013, 339, 936-940. [27]

Gao, W.; Pei, A.; Feng, X.; Hennessy, C.; Wang, J. Organized Self-Assembly of Janus

Micromotors with Hydrophobic Hemispheres. J. Am. Chem. Soc. 2013, 135, 998-1001. [28]

Wang, W.; Castro, L. A.; Hoyos, M.; Mallouk, T. E. Autonomous Motion of Metallic

Microrods Propelled by Ultrasound. ACS Nano 2012, 6, 6122-6132. [29]

Li, J.; Liu, W.; Li, T.; Rozen, I.; Zhao, J.; Bahari, B.; Kante, B.; Wang, J. Swimming

Microrobot Optical Nanoscopy. Nano. Lett. 2016, 16, 6604-6609. [30]

Li, J.; Gao, W.; Dong, R.; Pei, A.; Sattayasamitsathit, S.; Wang, J. Nanomotor

Lithography. Nat. Commun. 2014, 5, 5026. 20 ACS Paragon Plus Environment

Page 21 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

[31]

Li, T.; Li, J.; Zhang, H.; Chang, X.; Song, W.; Hu, Y.; Shao, G.; Sandraz, E.; Zhang, G.;

Li, L.; Wang, J. Magnetically Propelled Fish-Like Nanoswimmers. Small 2016, 12, 60986105. [32]

Khalil, I. S. M.; Magdanz, V.; Sanchez, S.; Schmidt, O. G.; Misra, S. Wireless Magnetic-

Based Closed-Loop Control of Self-Propelled Microjets. PloS one 2014, 9, e83053. [33]

Marino, H.; Bergeles, C.; Nelson, B. J. Robust Electromagnetic Control of Microrobots

under Force and Localization Uncertainties. IEEE T. Autom. Sci. Eng. 2014, 11, 310-316. [34]

Khalil, I. S. M.; Magdanz, V.; Sanchez, S.; Schmidt, O. G.; Misra, S. Three-Dimensional

Closed-Loop Control of Self-Propelled Microjets. Appl. Phys. Lett. 2013, 103, 172404. [35]

Hong, A.; Zeydan, B.; Charreyron, S.; Ergeneman, O.; Pane, S.; Toy, M. F.; Petruska, A.

J.; Nelson, B. J. Real-Time Holographic Tracking and Control of Microrobots. IEEE Robot. Autom. Lett. 2017, 2, 143-148. [36]

Khalil, I. S. M.; Magdanz, V.; Sanchez, S.; Schmidt, O. G.; Misra, S. The Control of Self-

Propelled Microjets inside a Microchannel with Time-Varying Flow Rates. IEEE T. Robot 2014, 30, 49-58. [37]

Zeydan, B.; Petruska, A. J.; Somm, L.; Pieters, R.; Fang, Y.; Sargent, D. F.; Nelson, B. J.

Automated Particle Collection for Protein Crystal Harvesting. IEEE Robot. Autom. Mag. 2017, 2, 1391-1396. [38]

Charreyron, S.; Pieters, R. S.; Tung, H. W.; Gonzenbach, M.; Nelson, B. J. Navigation of

a Rolling Microrobot in Cluttered Environments for Automated Crystal Harvesting. In Intelligent Robots and Systems, Proceedings of the IEEE/RSJ International Conference, Hamburg, Sept 28-Oct 2, 2015. [39]

Wang, J.; Manesh, K. M. Motion Control at the Nanoscale. Small 2010, 6, 338-345.

[40]

Ross, J. L. The Impacts of Molecular Motor Traffic Jams. Proc. Natl. Acad. Sci. USA.

2012, 109, 5911-5912.


ACS Nano

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

[41]

Page 22 of 23

Bruce, J.; Veloso, M. Real-Time Randomized Path Planning for Robot Navigation. In

Intelligent Robots and Systems, Proceedings of the IEEE/RSJ International Conference, Lausanne, Switzerland, Sept 30-Oct 4, 2002. [42]

Ersson, T.; Hu, X. Path Planning and Navigation of Mobile Robots in Unknown

Environments. In Intelligent Robots and Systems, Proceedings of the IEEE/RSJ International Conference, Maui, HI, USA, Oct 29-Nov 3, 2002. [43]

Khatib, O. Real-Time Obstacle Avoidance for Manipulators and Mobile Robots. Int. J.

Robot Res. 1986, 5, 90-98. [44]

Eklund, P. W.; Kirkby, S.; Pollitt, S. A Dynamic Multi-Source Dijkstra's Algorithm for

Vehicle Routing. In Intelligent Robots and Systems, Proceedings of the IEEE/RSJ International Conference, Adelaide, SA, Australia, Nov 18-20, 1996. [45]

Saffiotti, A. The Uses of Fuzzy Logic in Autonomous Robot Navigation. Soft Comput.

1997, 1, 180-197. [46]

Wang, M.; Liu. Fuzzy Logic Based Robot Path Planning in Unknown Environment.

Proceedings of the 2005 International Conference on Machine Learning and Cybernetics, Guangzhou, China, Aug 18-21, 2005. [47]

Martinez-Alfaro, H.; Gomez-Garcia, S. Mobile Robot Path Planning and Tracking using

Simulated Annealing and Fuzzy Logic Control. Expert Syst. Appl. 1988, 15, 421-429. [48]

Lee, S.; Park, J. Cellular Robotic Collision-Free Path Planning. In Advanced Robotics,

Proceedings of the ICAR Fifth International Conference on Robots in Unstructured Environments, Pisa, Italy, June 19-22, 1991. [49]

Canny, J. A Computational Approach to Edge-Detection. IEEE T. Pattern Anal. 1986, 8,

679-698.


Page 23 of 23

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Nano

TOC


Autonomous Collision-Free Navigation of Microvehicles in Complex

Recommend Documents