A deep-learning technique to convert a crude ... - ACS Publications

Subscriber access provided by the Henry Madden Library | California State University, Fresno

Applications of Polymer, Composite, and Coating Materials

A deep-learning technique to convert a crude piezoresistive CNT-Ecoflex composite sheet into a smart, portable, disposable, and extremely flexible keypad Jin-Woong Lee, Jiyong Chung, Min-Young Cho, Suman Timilsina, Keemin Sohn, Ji Sik Kim, and Kee-Sun Sohn ACS Appl. Mater. Interfaces, Just Accepted Manuscript • DOI: 10.1021/acsami.8b04914 • Publication Date (Web): 04 Jun 2018 Downloaded from http://pubs.acs.org on June 4, 2018

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Applied Materials & Interfaces

A deep-learning technique to convert a crude piezoresistive CNT-Ecoflex composite sheet into a smart, portable, disposable, and extremely flexible keypad

Jin-Woong Lee,1 Jiyong Chung,2 Min-Young Cho,3 Suman Timilsina,3 Keemin Sohn,2,* Ji Sik Kim,3,* and Kee-Sun Sohn1,*

1

Faculty of Nanotechnology and Advanced Materials Engineering, Sejong University, Seoul 143-

747, Republic of Korea

2

Laboratory of Big-data applications for public sector, Chung-Ang University, 221, Heukseok-dong,

Dongjak-gu, Seoul 156-756, Republic of Korea

3

School of Nano & Advanced Materials Engineering, Kyungpook National University, Kyeongbuk

742-711, Republic of Korea

* Corresponding Authors: [email protected]; [email protected]; [email protected]

KEYWORDS: piezoresistive, deep neural network, carbon nanotube, tactile sensing, portable keypad

1

ACS Paragon Plus Environment

ACS Applied Materials & Interfaces 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Abstract

An extremely simple bulk sheet made of a piezoresistive CNT-Ecoflex composite can act as a smart keypad that is portable, disposable, and flexible enough to be carried crushed inside the pocket of a pair of trousers. Both a rigid-button-imbedded, rollable (or foldable) pad and a patterned flexible pad have been introduced for use as portable keyboards. Herein, we suggest a bare, bulk, macro-scale piezoresistive sheet as a replacement for these complex devices that are achievable only through high-cost fabrication processes such as patterning-based coating, printing, deposition, and mounting. A deep-learning technique based on deep neural networks (DNN) enables this extremely simple bulk sheet to play the role of a smart keypad without the use of complicated fabrication processes. To develop this keypad, instantaneous electrical resistance change was recorded at several locations on the edge of the sheet along with the exact information of the touch position and pressure for a huge number of random touches. The recorded data were used for training a DNN model that could eventually act as a brain for a simple sheet-type keypad. This simple sheet-type keypad worked perfectly and outperformed all of the existing portable keypads in terms of functionality, flexibility, disposability, and cost.

2


Page 2 of 32

Page 3 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


1. Introduction

Until now, the possibility of paying 1 dollar for a portable keypad that can be carried crushed inside the pocket of a pair of trousers would have amounted to little more than an unrealistic dream. The only option for a conventional, flexible, portable keypad has been the introduction of either rigid push buttons imbedded on a rollable sheet or tactile sensor array patterns on a multi-layered soft sheet.1-5 The flexibility level for such conventional keypads is restricted to only a slight amount of either bending or rolling. Rather than a manipulated warp, however, the flexibility level that is commonly pursued in normal daily life often involves complete folding and harsh crumpling. To accomplish this level of flexibility in a real sense, neither rigid push buttons nor tactile sensor array patterns are acceptable. Conventional tactile sensor devices consist of array patterns that are resistive, capacitive, inductive, piezoresistive, optical, magnetic, binary, piezoelectric, and hydraulic. 6-10

These patterns all involve brittle components.

Simple, homogeneous sheets made of soft polymers such as Polydimethylsiloxane (PDMS), Polymethyl methacrylate (PMMA), and Ecoflex, etc., could meet the realistic flexibility requirements of a real world. A major concern, however, is to give such a monotonous sheet sophisticated keypad functions without losing realistic flexibility. Once 3



either rigid push buttons or tactile sensor array patterns are introduced on the sheet to realize a keypad, the flexibility deteriorates significantly. The trade-off between functionality and flexibility was so stern that it seemed there was no way to meet both simultaneously.

The above-described conventional on-sheet-mounted devices were not used to develop the flexible and portable keypad that is proposed in this study. Instead, the proposed keypad was realized by training a monotonous sheet via deep-learning techniques. We recently reported a pattern-free tactile sensor sheet achieved via a deeplearning technique.11 This revolutionary concept changed the existing tactile sensor paradigm from micro-(or macro-) patterns for signal addressing and readout to the deep learning of a crude bulky piezoresistive sheet. Deep learning enabled a simple bulky piezoresistive material to act as a smart device with no need for high-cost fabrication processes. Based on our success with the pattern-free tactile sensor sheet, we applied the same techniques to developing a flexible, portable keypad in the present study.

The piezoresistive traits of composites that consist of soft polymeric matrix materials and nano-scale conducting materials12-15 were utilized in the development of the proposed concept. We adopted a carbon nano tube (CNT)-dispersed Ecoflex matrix to realize the piezoresistive nature of flexible, portable keypads, where previously we used a CNT4


Page 4 of 32

Page 5 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


dispersed PDMS sheet for the tactile sensor (so-called e-skin).11 Electrical resistance signals instantaneously detected at several locations on the sheet edges (the voltage drop at a reference resister (100 k) connected to each probe terminal) were collected throughout the duration of a number of repetitive pushes on every key site on the keypad sheet. This procedure is referred to as training data acquisition. Thereafter, an enormous amount of collected data was used to set up a robust deep neural network (DNN) model. When the amount of collected data for the DNN model training was sufficient to train the DNN, the DNN worked properly. Eventually, the DNN recognized a key site under pressure with a high degree of accuracy by reading the instantaneous resistance signals at several locations on the sheet edge.

2. A Brief Description of Deep Learning

Deep learning, which is a technique for learning in artificial neural networks (ANNs), has recently played a promising and effective role in many areas by outperforming traditional rule-based methods. In particular, deep learning is a powerful method for image classification, pattern recognition, speech recognition, and natural language processing. Deep learning is responsible for considerable levels of recent progress in the fields of 5



biological and pharmaceutical research.16-18 In addition, we have noticeably succeeded in introducing deep-learning techniques into work on inorganic functional materials by achieving powder XRD pattern classification that is based on deep learning.19

Machine learning based on ANNs with a single hidden layer was developed in the 1940s and continued to prosper throughout the 80s and 90s.20–27 However, ANNs with such a shallow structure were almost considered obsolete after the new millennium, since their performance was not sufficient to exceed that produced from conventional rule-based engineering. The term “deep learning” was born from such a failure of previous shallowlearning structures with a single hidden layer. Deep neural networks (DNNs) were devised by simply employing multiple hidden layers in an ANN. In parallel with DNNs exhibiting a simple increase in the number of hidden layers, more advanced forms of ANNs such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) were also developed as iconic representatives of deep learning.28,29

In the initial stages of developing training methods for DNNs based on the well-known back-propagation algorithm, researchers in the field suffered from an identification problem due to the need to evaluate so many weighted parameters. Training a DNN with small-sample datasets definitely led to a fatal variance-and-bias problem, the so-called 6


Page 6 of 32

Page 7 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


over-fitting problem. Hinton et al.30 proposed a robust way to avoid this problem. They suggested a pre-training model with unlabeled data, which provided an initial solution for weighted parameters. Stacking multiple restricted Boltzmann machines (RBMs) made it possible to provide a plausible initial solution for a DNN. This pre-training method clearly triggered a boom in the use of deep learning. Furthermore, recent efforts to secure labeled big data and improvements in computing speed are more responsible for enhancing the performance of DNNs.31 Deng and Yu32 argued that judiciously designed random initial parameters could sort out the over-fitting problem without the need for pre-training, but only if a large amount of training data were available. In this context, we established a deep architecture in the development of our novel keypad using large-scale data that incorporated 960,000 input vectors.

3. Keypad Materials & design

The piezoresistive function denotes a change in electrical resistance by loading, which was adopted by using a macroscale piezoresistive composite sheet that consisted of homogeneously

dispersed

multi-wall

carbon

nanotubes

(MWCNTs)

in

a

polydimethylsiloxane(PDMS) matrix.11−15 The piezoresistive resistance of MWCNT-PDMS 7



material is known to be strain-rate independent.15 This means that the piezoresistive resistance is only dependent on the instantaneous state of the conductive CNT distribution in an insulating PDMS matrix, but not on the strain rate. The piezoresistive resistance exhibited a complete linear relationship with the strain (displacement or pressure) irrespective of the loading frequency.15 Thus, unless the matrix material is highly anelastic or viscous, the expression “pressure” can be used along with “strain” and “displacement.” In fact, the terms “pressure,” “displacement,” and “strain” should be treated equally as output signals with no distinction among them from a practical point of view, since they all exist in completely elastic relationships under either static or dynamic conditions. Consequently, it makes no difference which term is used in this instance.

In contrast to the previous case where PDMS was used as the matrix material for a piezoresistive composite sheet, we adopted Ecoflex in the present investigation. While other possible soft polymeric materials such as PMMA, PDMS, epoxy resin, and urethane were tested, Ecoflex was the most suitable matrix material to detect typical tactile pressures ranging from zero to 50 kPa. Figure S1 in the supporting Information. shows the results of three different weights (10, 50, 100 g) placed on a conventional keyboard. The lightest weight that would never be sufficient pressure to push a button 8


Page 8 of 32

Page 9 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


simulated a non-pressurized touch. The recognition of such a low level of pressure as a non-intentional push is important when considering the fact that most keypad users keep their fingers in contact with key buttons while the keypad is in use. This sort of slight pressure should never be recognized as intentional pressing. The intermediate weight and the heaviest weight represent the actual action of pushing a button. The pressure exerted on a pushed button is represented by three weights: 0.68, 3.4, and 6.8 kPa. A clear discrimination between the heaviest and the intermediate weight is crucial so that a single pushed button can designate two functions by choosing the correct pressure. For instance, if pressed hard, uppercase is recognized, but, otherwise, the press is interpreted as lowercase. This means that the ‘Shift’ key can be removed from the keypad. It should be noted that our training process executed the pressure range corresponding to these weights when pressing the key buttons on the keypad. This means that the overall pressure level adopted for the training process was far lower than a conventional tactile range (~ 50 kPa).33 Since the keypad was successful over such a generously chosen pressure range in the present investigation, 100% successful operation is guaranteed for any higherpressure range.

Figure 1a and 1b shows the arrangement of keys on the keypad and of electrodes that 9



serve as so-called prove terminals. The keypad consists of a full set (60) of virtual keys on the pad surface along with 30 prove terminals along the edge of the pad. It should be noted that there is no physical device underneath the keypad design but only a virtual partitioning was designed. The number of prove terminals should be minimized to simplify the keypad structure, and a smaller number of prove terminals would deteriorate the DNN model performance so that a reasonable tradeoff must be made prior to the point of diminishing returns. At the initial stage of development, we had to split the keypad into two pieces: one for the left hand, the other for the right hand, and both were connected by extremely flexible bands with no electrical bridge. The split deteriorated neither the functionality nor the flexibility, but we soon developed one pad with a full set of keys. The half pad outperformed the full pad in terms of accuracy, however, and the details of its DNN architecture and training process are presented in the supporting Information.

4. Keypad Training and DNN Architecture

4.1. Data Structure

The dataset structure for every key location on the keypad is fully described for the training datasets shown in Figure 1a and for the test datasets shown in Figure 1b. The 10


Page 10 of 32

Page 11 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


datasets pertaining to each of the keys are represented as boxes. The green boxes represent electric resistance datasets as input data and the red slabs display the pressure datasets as labels. The width (800) indicates electrical resistance data sequentially collected from a prove terminal for 0.16 sec, which means that a touch (or a pressing) exerted on a key of concern for 0.16 sec produces 800 sequential electrical resistance values per each probe terminal. The depth (30) designates the number of prove terminals from which the electrical resistance signals were collected, and the height (20 for the training dataset and 2 for the test dataset) represents the number of touches.

The input feature was vectorized into 30-dimensional vectors (blue bars inside the green box), the component of which denotes the actual resistance number measured from each of the 30 probe terminal electrodes. A certain level of pressure (from zero to 6.7 kPa) was applied to a key location of concern for 0.16 second. As a result, we collected 800 30dimensional input vectors along with the corresponding 800 pressure values per each key location, and 20 independent measurements were implemented at each key location, so that the total training dataset for 60 keys consisted of 960,000 (800 X 60 X 20) 30dimensional input vectors. The label (output) data were simply alphabet letters, numbers, and some function keys corresponding to the key location of concern, and the label data 11



for displacement (pressure) were real numbers represented by red slabs. Although it was smaller, the structure of the test dataset was similar to that of the training dataset.

4.2. DNN architecture for Pressure Regression

Since the static bias (10 V) was applied to every probe terminal in parallel and the center of the keypad was grounded, a touch on a certain location of the keypad gives rise to electric resistance changes at some of the prove terminals. In fact, however, no significant change arises in most of the prove terminals that are distant from the touch position. Regardless of the location of a touch, the change in the electrical resistance signal is somehow in a consistent relationship with the applied pressure. In this regard, we integrated all the electrical resistance data instantaneously collected at all prove terminals upon every touch on different locations and used them as input while the pressure value was output. The DNN model for the pressure prediction worked properly irrespective of the position of a touch. This means regardless of which key position is touched the exact pressure of the touch is identified in a real timeframe. This simplified pressure regression process is a conspicuous improvement from the previous e-skin case where the pressure regression was executed separately for each touch location.11 Figure 1c shows the DNN 12


Page 12 of 32

Page 13 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


architecture of pressure regression and Figure 1d shows its promising performance, wherein quite a coincidence was obtained between the experimental data and that predicted by the pressure-regression DNN model.

Although we secured the regression for the real values of pressure in a real timeframe by monitoring the electrical signal data collected at 30 prove terminals, such a learning process resulted in over-functioning caused by an overwhelming amount of data. A much simpler learning process would have been sufficient to attain a reasonable keypad operation. We didn't need the exact pressure values during a keypad touch. We decided that the aforementioned three-step recognition of pressure would be sufficient to endow the DNN-driven keypad with further functionality by comparison with the conventional keypad that we adopted for computers and smart phones. We divided the full range of pressure into three regions: the lowest level defined non-intentional touches that lead to no function; the intermediate level defined major key functions; and, the highest level defined key functions combined with a ‘Shift’ command. For instance, if a key was pressed above a certain threshold it would recognize uppercase, but otherwise the lowercase function would be recognized. Consequently, the DNN-driven keypad led to the removal of the ‘Shift’ key from the keypad. The three-step pressure-division scheme that we 13



adopted is depicted in Figure 1c. Figure 1c also shows the DNN architecture for the threestep pressure classification and exhibits almost the same architecture as the pressure regression except for the last two layers. The last-activation and loss functions of the pressure regression DNN model were linear and mean-squared error functions, respectively, but those for the three-step pressure classification DNN model represented softmax and cross-entropy functions. The former three layers are commonly operated via relu and dropout actions.

The pressure regression gives a prompt real value for pressure during a touch action, the accuracy of which is acceptable in terms of both root mean square error (%RMSE) and mean absolute error (MAE), as shown in Table 1. As shown in Figure 1d, the low-pressure region in the initial stage of a touch action initially looked very noisy, but the scattering was reduced soon after in this region (Region 0). In fact, the percentages of RMSE and MAE were poor when these noisy data were included. In particular, Regions I and II are important from a practical point of view, since these two particular regions were only utilized as key functions that designated either soft or hard touches. The fitting qualities (%RMSE and MAE) for these regions were very promising and allowed for an acceptable level of accuracy. Although the fitting quality for the initial, noisy region and that for the 14


Page 14 of 32

Page 15 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


total region encompassing this noisy region looks unacceptable, this would never be problematic since these data would not be actually used for the keypad operation.

Rather than the regression of a real pressure value, the step classification was more practical and achieved realistic keypad functionality. As mentioned earlier, although the real-value pressure regression seemed like an overwhelming amount of data, the result was brilliant. The DNN architecture for the three-label classification model worked very nicely and the performance returned test and training accuracy values of 94.95 and 96.21%, respectively. The accuracy was obtained on the basis of individual data points. If the most frequently hit label emerged from a data group collected during a certain time period such as several milliseconds, 100% accuracy was guaranteed, and we adopted this type of group data-based classification scheme for the actual keypad system.

4.3. DNN architecture for Key Recognition

Now that the three-step pressure recognition was completed, the next task was to achieve the recognition of a touched key. The arrangement of the 60 keys appears in Figure 1. The DNN architecture for key-position recognition (key identification) shown in Figure 1 (c) enabled the classification of 60 labels. The DNN architecture is similar to both the 15



Page 16 of 32

pressure-regression and the three-step pressure-classification models with the exception of the last two layers. The last-activation and loss functions of the key-position recognition DNN model were softmax and cross-entropy functions, respectively.

Both the Individual data-based and group data-based position recognition manners showed brilliant accuracy for both the test and training datasets.

Both individual data-

based and group data-based position recognition are described well in a previous report. 11

In brief, individual data-based position recognition could lead to an exorbitant degree

of performance. Individual data points indicate a signal integrated for 200 microseconds and such real-time data processing can lead to overwhelming amounts of data that could fall completely out of the normal keypad operation timeframe for a human's daily life. Nonetheless, individual data-based position recognition gives almost 100% real-time accuracy, as shown in Figure 2a-d. The overall accuracy for a hard touch (Region II) is more promising than that for a soft touch (Region I). It must also be noted that the test accuracy was slightly lower than the training accuracy, which is a trend for many machine-learning tasks.

The training and test accuracies for the recognition of both hard and soft touches for 60 keys approximated 100%. The test accuracy of the letter ‘s’ for a soft touch, however, 16


Page 17 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


dropped to 90.76%. Based on these results, individual data-based position recognition is yet to be used for actual keypad operation. However, further improvements in materials would definitely enhance the accuracy to 100% for all instances. Even at the current level of materials development, when using group data-based position recognition instead of the Individual data-based alternative, 100% accuracy is possible for every key site under all circumstances. Group data-based position recognition simply considers an independent touch action taking place for 0.16 seconds as a unit of data — so-called group data. In addition, the timeframe required for reliable group data-based position recognition could be further reduced to a few milliseconds, which no doubt would support even the world's speediest typists. We have never failed to identify any key by touch under any circumstances when using a group data-based position recognition scheme. The group data-based position recognition scheme meets the practical requirements for actual keypad function, which makes this scheme the practical choice if the DNN-driven keypad were marketed as a commercially available application.

4.4. Tremendous Flexibility Achievement

Some of the most remarkable advantages of the DNN-driven keypad over conventional 17



keypads are extreme flexibility, foldability and simplicity. Extreme flexibility and foldability were the most important goals targeted in the present investigation that began with the pursuit of a DNN-driven simple keypad at very low cost — this version is only 1 dollar — that would be portable, disposable, and flexible enough to carry crushed inside the pocket of a pair of trousers, yet could exhibit smarter functions than conventional complex keypads.

We tested the performance of a keypad before and after severe squeezing, which should have definitely crumpled a conventional keypad. A severe squeeze, as shown in Figure 2e, did not alter the electrical signals emanated from 30 probe terminals when pressing a letter ‘s’ and thereby the same DNN model could be used even after multiple severe squeezes. In fact, the keypad that we used for the previous training and test process was subjected to a series of severe squeezes, and thereafter we skipped the training processes. As a result, 100% test accuracy was observed in the group data-based position recognition scheme even after the severe squeezing with no re-training. Although the Individual data-based position recognition test showed a slight bit of deterioration in accuracy for the recognition of each key after being severely squeezed, neither the group data-based pressure regional classification nor the key identification deteriorated.

18


Page 18 of 32

Page 19 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


5. Conclusions

With the assistance of deep learning, we developed an extremely flexible keypad with a simple structure and no device pattern. Neither a pattern fabrication nor the corresponding logic circuit design concept was involved. Instead, a DNN and an extremely simple, flexible, macro-scale, piezoresistive sheet functions as a versatile keypad. A CNTdispersed Ecoflex pad was selected for its excellent performance and brevity as a result of testing many other combinations of matrix materials. Sixty key positions were located on a homogeneous CNT-dispersed Ecoflex pad by virtually compartmentalizing the pad. Actual training was executed on an actuator equipped with an x-y position stage, wherein precise pressure values were read from the load cell by repeatedly pushing on each key position, and the electric resistance change was simultaneously recorded from 30 probe terminals.

The DNN architectures for pressure regression, three-step-pressure classification, and touch-position recognition (key identification on touch) were developed to withstand rigorous testing. Accordingly, the DNN-driven keypad was verified to operate in the same manner as conventional keypads. In addition, the DNN-driven keypad has two outstanding advantages over conventional keypads; first, the three-step-pressure-induced key 19



recognition doubles the function of a key, leading to the discrimination of soft and hard touches, and, second, extreme flexibility was secured by removing all the brittle materialsbased patterns.

20


Page 20 of 32

Page 21 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


6. Experimental Procedures: Fabrication of the Ecoflex/CNT piezoresistive sheet

An ultra-soft piezoresistive sheet was fabricated from a homogenous mixture of multiwalled carbon nanotubes (MWCNTs) (Carbon Nano-material Technology Co., Ltd.) and silicone rubber matrix (EcoflexTM, platinum-catalyzed silicone). The detailed process adopted to fabricate the sample is highlighted in Figure 3. The Ecoflex that we adopted consists of two different components: A and B at a mixture ratio of 1:1 by weight. Additionally, a (Thinning EcoflexTM Silicones) lowering of the viscosity was also incorporated to facilitate the dispersion of CNT in the matrix and also to enhance the softness of the cured material. CNT with an average length of 5 μm and an average diameter of 20 nm was mixed with Ecoflex A and an amount of thinner equal to 1 wt% of the total weight of the matrix (i.e. Ecoflex A + Ecoflex B + Thinner). To ensure the enhancement of the homogeneous dispersion of CNT in Ecoflex and reduce the agglomeration, a few 10 mm diameter alumina grinding balls were added and the container was transferred to a planetary shear mixer at a mixing speed of 400 rpm for 2 hr. Then, a quantity of Ecoflex-B equal to Ecoflex-A was added, and again the container was transferred to a planetary shear mixture for 10 min. Eventually to free the trapped air bubbles from the nanocomposite, the container was placed in a vacuum for 10 min. 21



In advance, molds were prepared with dimensions of 160 X 59 X X

Page 22 of 32

0.2 mm and 80 X

59

0.2 mm with a base made up of a 5 mm thick Ecoflex matrix with edges made of

removable 0.2 mm thick Acrylic film. In addition to the through-hole from the center for the purpose of grounding, the molds also had several extensions from each of the sides with dimension of 5 X 3 X 5 mm to facilitate wiring. Using the Doctor’s Blade Technique, the piezoresistive CNT/Ecoflex nanocomposite was cast and wired from each of the extended terminals. The wired samples were then let stand for 10 hr at room temperature to solidify, and were finally transferred to a vacuum oven for 2 hr at 60 oC to ensure complete solidification. All the reference steps used to fabricate the PDMS/CNT piezoresistive sheet are listed in the steps mentioned above, with the exception of the liquid PDMS (Sylgard® 184 A), which was mixed with a curer (Sylgard® 184 B) at a 10:1 wt %.

ASSOCIATED CONTENT

Supporting Information. DNN Architecture and test performance for the half pad. “This material is available free of charge via the Internet at http://pubs.acs.org.”

22


Page 23 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


AUTHOR INFORMATION

Corresponding Author

* Email: [email protected] and [email protected]

Notes

The authors declare no competing financial interest.

ACKNOWLEDGMENT

This research was supported by the Creative Materials Discovery Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT, and Future Planning (2015M3D1A1069705).

ABBREVIATIONS

DNN, deep neural network; MWCNT, multi-wall carbon nano tube; ANN, artificial neural network; PDMS, Polydimethylsiloxane; PMMA, Polymethyl methacrylate.

23



Test

Train

Page 24 of 32

All

Region 0

Region I

Region II

%RMSE

11.2476

22.3562

4.0898

3.0758

MAE

0.0418

0.0558

0.0230

0.0219

%RMSE

10.1265

20.2280

3.7217

2.4612

MAE

0.0368

0.0506

0.0197

0.0166

Table 1. root mean square error (%RMSE) and mean absolute error (MAE) for pressure regression

24


Page 25 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Figure Captions

Figure 1 the arrangement of 60 keys on the CNT-Ecoflex keypad and the distribution of 30 electrodes (probe terminals) located on the edge of the CNT-Ecoflex keypad. The dataset structure was schematically well described by placing a box-type dataset representation on the key location of the keypad; a for the training dataset; b for the test dataset; c the DNN architecture for pressure regression (bottom), three-step-pressure classification (middle), and touch-position recognition (top); d a plot between the experimental data and that predicted by the pressure regression DNN model, wherein three pressure regions are clearly divided. The pressure is replaced by the displacement for this plot, and the maximum pressure is 6.8 kPa (1 N).

Figure 2a-d real-time accuracy for Individual data-based position recognition, which approximated 100% at most key positions. Although the number is not shown here, the group data-based position recognition (key recognition) achieved 100% accuracy for all positions; e Non-variant electrical signals from 30 probe terminals even after severe squeezing of the CNT-Ecoflex keypad. The graphs denote the time evolution of 30 signals while a letter ‘s’ is under a pressure.

Figure 3 Detailed process adopted to fabricate the CNT+Ecoflex sample. 25



References

1. Cho, J.; Moon, J.; Sung, M.; Jeong, K.; Cho, G. In Human-Computer Interaction. Interaction Platforms

and Techniques; Jacko, J. A., Eds; Springer, Heidelberg, 2007; pp 1078-1085. 2. Parkova, I.; Vališevskis, A.; Kašurins, A.; Viļumsone, A. In Environment Technology Resources:

Proceedings of the 8th International Scientific and Practical Conference, Rezekne, 2011; pp 173180. 3. Charlier, M. L.; Steuer, P.; Schirtzinger, T.; Koo, S. Method and apparatus for a holster having a user interface. U.S. Patent 7,386,325, Jun 10, 2008. 4. Chen, C.; Rak, R. P.; Griffin J. T. Keypads for mobile devices and method of manufacturing the same. U.S. Patent 9,075,576, Jul 7, 2015. 5. Ichikawa, Y. Key sheet, press switch and electronic device provided with the press switch. U.S. Patent 7,978,467, Jul 12, 2011. 6. Zou, L.; Ge, C.; Wang, Z. J.; Cretu, E.; Li, X. Novel Tactile Sensor Technology and Smart Tactile Sensing Systems: A Review. Sensors 2017, 17, 2653. 7. Najarian, S.; Dargahi, J.; Mehrizi, A. A. Artificial Tactile Sensing in Biomedical Engineering; McGrawHill Biophotonics: New York, 2009. 8. De Rossi, D. Artificial tactile sensing and haptic perception. Meas. Sci. Technol. 1991, 2, 10031016. 9. Yousef, H.; Boukallel, M.; Althoefer, K. Tactile sensing for dexterous in-hand manipulation in robotics—A review. Sens. Actuators A: Phys. 2011, 167, 171-187. 10. Tiwana, M. I.; Redmond, S. J.; Lovell, N. H. A review of tactile sensing technologies with applications in biomedical engineering. Sens. Actuators A: Phys. 2012, 179, 17-31. 11. Sohn, K.-S.; Chung, J.; Cho, M.-Y; Timilsina, S.; Park, W. B.; Pyo, M.; Shin, N.; Sohn, K.; Kim, J. S. An extremely simple macroscale electronic skin realized by deep machine learning. Sci. Rep. 2017, 7, 11601. 12. Jung, H.-C.; Moon, J.-H.; Baek, D.-H.; Lee, J.-H.; Choi, Y.-Y.; Hong, J.-S.; Lee, S.-H. CNT/PDMS Composite Flexible Dry Electrodes for Long-Term ECG Monitoring. IEEE Trans. Biomed. Eng. 2012,

59, 1472-1479. 13. Yamada, T.; Hayamizu, Y.; Yamamoto, Y.; Yomogida, Y.; Izadi-Najafabadi, A.; Futaba, D. N.; Hata, K. 26


Page 26 of 32

Page 27 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


A Stretchable Carbon Nanotube Strain Sensor for Human-motion Detection. Nat. Nanotechnol. 2011, 6, 296-301. 14. Lu, N.; Lu, C.; Yang, S.; Rogers, J. Highly Sensitive Skin-Mountable Strain Gauges Based Entirely on Elastomers. Adv. Funct. Mater. 2012, 22, 4044-4050. 15. Sohn, K.-S.; Timilsina, S.; Singh, S. P.; Lee, J.-W.; Kim, J. S. A Mechanoluminescent ZnS:Cu/Rhodamine/SiO2/PDMS and Piezoresistive CNT/PDMS Hybrid Sensor: Red-Light Emission and a Standardized Strain Quantification. ACS Appl. Mater. Interfaces 2016, 8, 34777-34783. 16. Spencer, M.; Eickholt, J.; Cheng, J. A Deep Learning Network Approach to ab initio Protein Secondary Structure Prediction. IEEE/ACM Trans. Comput. Biol. Bioinform. 2015, 12, 103-112. 17. Heffernan, R.; Paliwal, K.; Lyons, J.; Dehzangi, A.; Sharma, A.; Wang, J.; Sattar, A.; Yang, Y.; Zhou, Y. Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by iterative deep learning. Sci. Rep. 2015, 5, 11476. 18. Mamoshina, P.; Vieira, A.; Putin, E.; Zhavoronkov, A. Applications of Deep Learning in Biomedicine.

Mol. Pharm. 2016, 13, 1445-1454. 19. Park, W. B.; Chung, J.; Jung, J.; Sohn, K.; Singh, S. P.; Pyo, M.; Shin, N.; Sohn, K.-S. Classification of crystal structure using a convolutional neural network. IUCrJ 2017, 4, 486-494. 20. McCulloch, W. S.; Pitts, W. A Logical Calculus of Ideas Immanent in Nervous Activity. Bull. Math.

Biophys. 1943, 5, 115-133. 21. Hebb, D. O. The Organization of Behavior; Wiley: New York, 1949. 22. Farley, B. G.; Clark, W. A. Simulation of Self-Organizing Systems by Digital Computer. Inf. Theory 1954, 4, 76-84. 23. Rochester, N.; Holland, J. H.; Habit, L. H.; Duda, W. L. Tests on a cell assembly theory of the action of the brain, using a large digital computer. IEEE Trans. Inf. Theory 1956, 2, 80-93. 24. Rosenblatt, F. The Perceptron: A Probabilistic Model for Information Storage and Organization In The Brain. Psychol. Rev. 1958, 65, 386–408. 25. Werbos, P. J. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Ph.D. Thesis, Harvard University, Cambridge, U.S.A, 1974. 26. Minsky, M. L.; Papert, S. A. Perceptrons: An Introduction to Computational Geometry ; MIT Press: Cambridge, 1969. 27. Rumelhart, D. E.; McClelland, J. L. Parallel Distributed Processing: Explorations in the

Microstructure of Cognition; MIT Press: Cambridge, 1986. 27



28. Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86, 2278-2324. 29. Graves, A.; Liwicki, M.; Fernandez, S.; Bertolami, R.; Bunke, H.; Schmidhuber, J. A Novel Connectionist System for Unconstrained Handwriting Recognition. IEEE Trans. Pattern Anal. Mach.

Intell. 2009, 31, 855-868. 30. Hinton, G. E.; Osindero, S.; Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural

Comput. 2006, 18, 1527–1554. 31. Dean, J.; Corrado, G. S.; Monga, R.; Chen, K.; Devin, M.; Le, Q. V.; Mao, M. Z.; Ranzato, M.; Senior, A.; Tucker, P.; Yang, K.; Ng, A. Y. In Large scale distributed deep networks, Advances in Neural Information Processing Systems 25, Harrah's Lake Tahoe, NV, U.S.A, Dec 03-08, 2012. 32. Deng, L.; Yu, D. Deep learning: Methods and applications; NOW Publishers: Boston, 2014. 33. Takei, K.; Takahashi, T.; Ho, J. C.; Ko, H.; Gillies A. G.; Leu, P. W.; Fearing, R. S.; Javey, A. Nanowire active-matrix circuitry for low-voltage macroscale artificial skin. Nat. Mater. 2010, 9, 821-826.

28


Page 28 of 32

Page 29 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Pressure Pressure Pressure Region II Region I Region O

29



30


Page 30 of 32

Page 31 of 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Figure 3

31



Graphic for manuscript


Page 32 of 32

A deep-learning technique to convert a crude ... - ACS Publications

Recommend Documents