An Analog Circuit Design for k -Winners-Take-All Operations Xiaoyang Liu1(B) and Jun Wang2,3(B) 1
3
School of Automation, Huazhong University of Science and Technology, Wuhan, China
[email protected] 2 Department of Computer Science, City University of Hong Kong, Kowloon Tong, Hong Kong Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China
[email protected] Abstract. This paper designs an analog circuit for k -winners-take-all (k WTA) operations. The circuit is stable and finite-time convergent. The stable state of the circuit is equivalent to the optimal solution of the k WTA. Simulation results via SPICE substantiate the efficiency of the design. Keywords: k -winners-take-all Recurrent neural network
1
· Analog circuit design
Introduction
Winner-take-all (WTA) problem is to select the largest element from a collection of inputs. k -winners-take-all (k WTA) problem is an extension of WTA problem. It is to select k largest elements from n inputs (1 ≤ k ≤ n) [11]. WTA and k WTA operations have many applications in various fields, such as associative memories [5,14], sorting [6,13], image processing [2], etc. There are many WTA and k WTA models [1,4,8,9,18,21,22]. A simple k WTA model with only one state variable is proposed in [4]. A fast-converge k WTA model with a hard-limiting step activation function is proposed in [8]. The k WTA model proposed in [21] adopts both single state variable and hard-limiting step activation function. Various WTA and k WTA circuits are proposed [3,7,12,13,15–17,19,20]. They have various precision, accuracy and speed. This paper presents an analog k WTA circuit based on the k WTA model in [21] with theoretically guaranteed Jun Wang—This work was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region of China, under Grants 14207614 and 11208517, and in part by the National Natural Science Foundation of China under grant 61673330. c Springer Nature Switzerland AG 2018 L. Cheng et al. (Eds.): ICONIP 2018, LNCS 11307, pp. 666–675, 2018. https://doi.org/10.1007/978-3-030-04239-4_60
An Analog Circuit Design for k -Winners-Take-All Operations
667
global stability, high precision and fast convergence speed. Compared with existing k WTA circuits, the proposed circuit is theoretically guaranteed stable and the operation time can be easily controlled. The rest of the paper is organized as follows. In Sect. 2, preliminaries of k WTA problem, the k WTA model used in this paper, and some basic operation circuits are introduced. Section 3 describes and analyzes the presented k WTA circuit. Section 4 illustrates the simulation results. Finally, the conclusion is given in Sect. 5.
2 2.1
Preliminaries The k -Winners-Take-All Model
The k WTA problem can be formulated as the function [10] 1, if ui ∈ {k largest elements of u} xi = f (ui ) = 0, otherwise
(1)
where xi ∈ x = (x1 , x2 , . . . , xn )T is the output vector, and ui ∈ u = (u1 , u2 , . . . , un )T is the input vector. It is shown in [21] that the k WTA problem can be formulated as the following model assuming that the solution is unique n
dy = xi − k, dt i=1 xi = g∞ (ui − y),
(2) i = 1, 2, . . . , n;
(3)
where is a time constant, y ∈ R is the state variable, and g∞ (·)is the Heaviside step activation function which is defined as 0, ρ < 0 (4) g∞ (ρ) = 1, ρ ≥ 0. This model has one neuron and 2n connections, where n is the input number. It has advantages of being global stable, finite-time convergent, and having highresolution. x corresponding to the equilibrium of the state equation (2) is a solution to the k WTA problem (1). 2.2
Basic Circuit Elements
Analog Adder. An analog adder circuit composed of four resistors R1 –R4 and one operational amplifier (OA) A is show in Fig. 1. v1 –v3 are input voltages, and vout is the output voltage. v+ and v− are positive and negative input voltages of the operational amplifier, respectively.
668
X. Liu and J. Wang
v-
v3
R4 -
R3 v+
v2 v1
A
vout
+
R2 R1 Fig. 1. An analog adder
According to Kirchhoff’s current law (KCL) and properties of OA v3 − v− v− − vout = , R3 R4 R4 R 4 v3 vout = (1 + )v− − , R3 R3
(5)
and v1 − v + v+ − v 2 = , R1 R2 R 2 v1 + R 1 v2 v+ = . R1 + R2
(6)
According to properties of OA v + = v− .
(7)
Combining (5), (6), and (7), the output voltage is expressed as vout =
(R3 + R4 )(R2 v1 + R1 v2 ) R4 v3 − . R3 (R1 + R2 ) R3
(8)
If R1 = R2 and R3 = R4 , then vout = v1 + v2 − v3 .
(9)
The output of the circumstance of any number of inputs can be computed accordingly. Analog Integrator. An analog integrator circuit composed of a capacitor, resistors, and one OA is shown in Fig. 2. According to properties of OA v+ = v− = 0, vin vin − v− ic = = . R R
(10) (11)
An Analog Circuit Design for k -Winners-Take-All Operations
669
ic
v-
vin
C -
R v+
A
vout
+
Fig. 2. An analog integrator
Assume that the initial voltage of the capacitor C is 0, then the voltage of C is 1 vc = ic dt C 1 vin dt. = (12) RC Therefore the output voltage of the integrator is 1 vout = −vc = − vin dt. RC
3
(13)
Circuit Design
The circuit is designed with available electronic devices, so it can be easily implemented with a practical circuit. The devices models will be detailed in Sect. 4. 3.1
Design
The presented k WTA circuit is shown in Fig. 3. It is composed of one capacitor, resistors, transistors, and n + 2 operational amplifiers. u = (u1 , u2 , . . . un ) is the input voltage vector, x = (x1 , x2 , . . . xn ) is the output voltage vector, k is an input voltage whose voltage amplitude represents the number of winners under the dimension of volt, and y is the state variable. There are totally n neurons and 2n connections. From the circuit diagram, when R1 = R2 = · · · = Rn+1 and R1 = R2 = · · · = Rn = Ra , following equations can be got vx = k − (x1 + x2 + . . . + xn ), 1 y=− vx dt. RC
(14) (15)
670
X. Liu and J. Wang
Cmp - C R
+
u1
P
vcmp
+
y
1V
Inv
-
x1
x2
-
...
R2
+
R1
...
...
Rn'
vx
+
R 1'
R2' 1V
Ra
xn
+ un
Rn+1 Rn
N 1V
u2
k
-
Fig. 3. The k WTA circuit
Substituting (14) into (15) 1 y= RC That is
n ( xi − k)dt.
(16)
i=1 n
RC
dy = xi − k, dt i=1
(17)
which is in accordance with (2) with RC = . Because the stable state of xi (i = 1, 2, . . . , n) is 0 V or 1 V, so the number of winners is equal to the voltage sum of the these winners under the dimension of volt. When the circuit reaches steady n state, (17) is equal to 0 which means that xi = k, and therefore k represents i=1
the winner number. A comparator Cmp and an inverter Inv execute the comparison operation and the Heaviside step activation function together. Take the first input u1 as an example: When u1 > y, vcmp = V−− , otherwise vcmp = V++ , where V++ > 0 and V−− < 0 are the high and low power voltages of the operational amplifier in the comparator, respectively. When vcmp = V−− , vcmp < 0 V and 1 − vcmp is bigger that the opening threshold voltage vth of P , so the P-channel transistor P is open and x1 = 1 V, otherwise the N-channel transistor N is open and x1 = 0 V. That is 0, ui < y xi = g∞ (ui − y) = (18) 1, ui > y, i = 1, 2, . . . , n.
An Analog Circuit Design for k -Winners-Take-All Operations
671
Equation (18) is in accordance with (3). Considering the previous conclusion that (17) is equal to (2), the presented circuit can execute k WTA operations. 3.2
Specifications Analysis
Range of Inputs. Denote power voltages of operational amplifiers in the integrator, comparators, and the adder by VI++ and VI−− , Vc++ and Vc−− , and Va++ and Va−− , where Vx++ > 0 and Vx−− < 0, the letter x stands for I, c, or a. It is known that input voltages of an operational amplifier should not exceed power voltages, therefor any ui and y must be smaller than Vc++ and larger than Vc−− . On the other hand, y is produced by an integrator, that is, |y| < VI++ and |y| < |VI−− |, therefore |y| < Vc++ and |y| < |Vc−− | are always satisfied if VI++ >= Vc++ and VI−− max(u) and Vc−− < min(u), Va++ > k, and Va++ > n. The Initial Value of the State Variable. The initial value y0 of the state variable y effects the convergence time, [21] gives initialization approaches of y under different distribution of inputs. In the presented circuit, y0 is represented by the reverse initial voltage of the capacitor C, and it is set to 0. Co-operation Speeds of Electronic Devices. The operation speeds of every components of the circuits should keep pace with each other. The integration speed of the integrator, which plays an essential role in the operation speed of the k WTA circuit, is controlled by the value of RC. By adjusting the value of R and C, the operation speed of the circuit is easily controlled. If RC is very small, high-speed operational amplifiers are needed to constitute comparators and the adder.
4
Simulation Results
High-speed OAs ML339 and LM324 are selected for comparators and the adder, and the integrator, respectively, in SPICE simulations. 4.1
Constant Inputs
Consider ten inputs with voltages increase from 0.01 V to 0.1 V with interval 0.01 V, and k = 5. Figure 4(a) shows inputs, Fig. 4(b) shows outputs of the designed k WTA circuit, and Fig. 4(c) shows the curve of the state variable y. Stable xi ’s with 1 V are winners, and those with 0 V are losers. At first y = 0, so all xi ’s are initialized with 1 V. After about 1.4 × 10− 5 s, x1 to x5 which are corresponding to five smallest inputs u1 to u5 go down to 0 V, remaining x6 to
672
X. Liu and J. Wang
x10 being 1 V, and all xi ’s are stable after that. Simulation results show that the circuit successfully determines five largest inputs which are close to each other in amplitude in about 1.4 × 10−5 s, proving that the circuit is high-speed and high-precision. y is stable at about 0.0592 V, which is between 0.05 V and 0.06 V, conforming that y¯ is a threshold. 4.2
Time-Vary Inputs
Consider an input vector of sinusoidal signals up (t) = sin[40π(5t+(p−1)π)](p = 1, 2, . . . , 10) and k = 5. Figure 5 plots simulation results. Figure 5(a) shows input signals, and for better view, ten output curves are depicted in five subfigures Fig. 5(b)–(f). There are always five xi ’s with 1 V, which represent five winners, corresponding to five largest ui ’s at that time. The circuit successfully selects five winners from time-vary inputs instantly.
ui / V
0.1
u1
u2
u3
u4
u5
u6
u7
u8
u9
u10
0.05 0
0
0.5
1
1.5
2
2.5 10-5
t/s
(a)
xi / V
1 x1
x2
x3
x4
x5
x6
x7
x8
x9
x10
0.5 0 0
0.5
1
1.5
2
2.5 10-5
t/s
(b)
y/V
0.1
0.05
0
0
0.5
1
1.5
t/s
2
2.5 10-5
(c)
Fig. 4. Simulation results for ten constant inputs with k = 5. (a) Inputs. (b) Outputs of the k WTA circuit. (c) State variable.
An Analog Circuit Design for k -Winners-Take-All Operations
ui / V
1
u1
u2
u3
u4
u5
u6
u7
u8
u9
673
u10
0
-1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 10-3
t/s
(a)
xi / V
1
x1
x2
0.5 0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 10-3
t/s
(b) 1
xi / V
x
3
x
4
0.5 0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 10-3
t/s
(c)
xi / V
1 x5
x6
0.5 0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 10-3
t/s
(d) 1
xi / V
x
7
x
8
0.5 0 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1 10-3
t/s
(e) 1
xi / V
x
9
x
10
0.5 0 0
0.1
0.2
0.3
0.4
0.5
t/s
0.6
0.7
0.8
0.9
1 10-3
(f)
Fig. 5. Simulation results for ten sinusoidal inputs with k = 5. (a) Inputs. (b)–(e) Outputs of the k WTA circuit.
674
5
X. Liu and J. Wang
Conclusion
In this paper, a k -winners-take-all circuit is designed. It is constructed with high speed, high precision, and stability with available electronic devices. Simulations on constant inputs and sinusoidal inputs substantiate the efficiency of it. The k WTA circuit may serve as a building block in many applications.
References 1. Ferreira, L., Kaszkurewicz, E., Bhaya, A.: Synthesis of a k-winners-take-all neural network using linear programming with bounded variables. In: Proceedings of the International Joint Conference on Neural Networks, vol. 3, pp. 2360–2365. IEEE (2003) 2. Fish, A., Akselrod, D., Yadid-Pecht, O.: High precision image centroid computation via an adaptive k-winner-take-all circuit in conjunction with a dynamic element matching algorithm for star tracking applications. Analog Integr. Circ. Sig. Process 39(3), 251–266 (2004) 3. Hsu, T.C., Wang, S.D.: k-winners-take-all neural net with Θ(1) time complexity. IEEE Trans. Neural Networks 8(6), 1557–1561 (1997) 4. Hu, X., Wang, J.: An improved dual neural network for solving a class of quadratic programming problems and its k-winners-take-all application. IEEE Trans. Neural Networks 19(12), 2022–2031 (2008) 5. Krogh, A., Hertz, J., Palmer, R.G.: Introduction to the Theory of Neural Computation. Addison-Wesley, Reedwood City CA (1991) 6. Kwon, T.M., Zervakis, M.: KWTA networks and their applications. Multidimension. Syst. Signal Process. 6(4), 333–346 (1995) 7. Lazzaro, J., Ryckebusch, S., Mahowald, M.A., Mead, C.A.: Winner-take-all networks of O(n) complexity. In: Advances in Neural Information Processing Systems, pp. 703–711 (1989) 8. Liu, Q., Wang, J.: Two k-winners-take-all networks with discontinuous activation functions. Neural Networks 21(2–3), 406–413 (2008) 9. Liu, S., Wang, J.: A simplified dual neural network for quadratic programming with its kwta application. IEEE Trans. Neural Networks 17(6), 1500–1510 (2006) 10. Maass, W.: Neural computation with winner-take-all as the only nonlinear operation. In: Advances in Neural Information Processing Systems, pp. 293–299 (2000) 11. Majani, E., Erlanson, R., Abu-Mostafa, Y.S.: On the k-winners-take-all network. In: Advances in Neural Information Processing Systems, pp. 634–642 (1989) 12. Opris, I.E.: Rail-to-rail multiple-input min/max circuit. IEEE Trans. Circuits Syst. II: Analog Digital Sig. Process. 45(1), 137–140 (1998) 13. Ou, S.H., Lin, C.S., Liu, B.D.: A scalable sorting architecture based on maskable WTA/MAX circuit. In: IEEE International Symposium on Circuits and Systems, ISCAS 2002, vol. 4, p. IV. IEEE (2002) 14. Pouliquen, P.O., Andreou, A.G., Strohbehn, K.: Winner-takes-all associative memory: a hamming distance vector quantizer. Analog Integr. Circ. Sig. Process 13(1– 2), 211–222 (1997) 15. Sekerkiran, B., Cilingiroglu, U.: A CMOS k-winners-take-all circuit with O(N) complexity. IEEE Trans. Circuits Syst. II: Analog Digital Sig. Process. 46(1), 1–5 (1999)
An Analog Circuit Design for k -Winners-Take-All Operations
675
16. Serrano-Gotarredona, T., Linares-Barranco, B., Andreou, A.G.: A high-precision current-mode WTA-MAX circuit with multi-chip capability. In: Adaptive Resonance Theory Microchips, pp. 89–115. Springer, Boston (1998). https://doi.org/ 10.1007/978-1-4419-8710-5 4 17. Sum, J.P.F., Leung, C.S., Tam, P.K.S., Young, G.H., Kan, W.K., Chan, L.W.: Analysis for a class of winner-take-all model. IEEE Trans. Neural Networks 10(1), 64–71 (1999) 18. Sum, J.P.-F., Leung, C.-S., Ho, K.: Analysis on Wang’s k WTA with stochastic output nodes. In: Lu, B.-L., Zhang, L., Kwok, J. (eds.) ICONIP 2011. LNCS, vol. 7064, pp. 268–275. Springer, Heidelberg (2011). https://doi.org/10.1007/9783-642-24965-5 29 19. Urahama, K., Nagao, T.: K-winners-take-all circuit with O(N) complexity. IEEE Trans. Neural Networks 6(3), 776–778 (1995) 20. Wang, J.: Analogue winner-take-all neural networks for determining maximum and minimum signals. Int. J. Electron. 77(3), 355–367 (1994) 21. Wang, J.: Analysis and design of a k-winners-take-all model with a single state variable and the heaviside step activation function. IEEE Trans. Neural Networks 21(9), 1496–1506 (2010) 22. Wolfe, W.J., Mathis, D., Anderson, C., Rothman, J., Gottler, M., Brady, G., Walker, R., Duane, G., Alaghband, G.: K-winner networks. IEEE Trans. Neural Networks 2(2), 310–315 (1991)
