Irreversible Thermodynamics of Neural Networks: Calortropy

7104

J. Phys. Chem. B 2001, 105, 7104-7114

Irreversible Thermodynamics of Neural Networks: Calortropy Production in Logic Operations Daniel Barraga´ n† and Byung Chan Eu*.‡ Department of Chemistry, McGill UniVersity, 801 Sherbrooke St. West, Montreal, Quebec H3A 2K6, Canada ReceiVed: December 14, 2000; In Final Form: March 16, 2001

Irreversible thermodynamics of chemical neural networks is formulated, and energy and matter dissipation accompanying logic operations is investigated in this paper. This formalism therefore puts the dynamics of neural networks within the framework of the laws of thermodynamics and thereby provides a mathematical basis to study neural dynamics as irreversible processes subjected to the thermodynamic principles. By using the minimal bromate oscillator, we apply the formalism of irreversible thermodynamics to study numerically the modes of energy dissipation related to logic operations. The calortropy of the system serves as an integral surface of the evolution equations for the concentrations of chemical species in the neurons of the network. It is shown that logic operations in the neural networks can be regarded as evolution of the states of neurons in the network toward the steady-state values of the calortropy production characteristic of the logic operations.

I. Introduction Neural networks are dynamical structures that are believed to have some of the properties of real biological neurons and thus provide insights into nervous systems with regard to their logic operations and computational performance. They are an interconnected assembly of logic processing units called neurons, and the information processing ability of the network is stored in the interneuronal connection strengths called weights. Weights of connections are determined by a learning process from a set of training inputs in accordance with the Hebbian learning rules.1 A system of neurons is arranged into layers, and neurons of a layer (e.g., the hidden and output layers) perform a weighted summation of all states of the neurons in the input layer; the result of the sum is compared with a certain threshold value to decide which will be the new state of the neuron in the next layer receiving the input. Thus, there are two states possible: one below the threshold and the other above the threshold. These states may be respectively assigned values 0 and 1, the former to the state below the threshold and the latter to the state above the threshold. This model of neuronal activity is called the McCulloch-Pitts neuron model.2 The two states in this model are akin to the two states identified with the levels of the electric current in a digital computer circuit.3 There are numerous studies of neural networks made on the basis of this model in recent years.4,5 Animate systems function on the basis of numerous intricately coupled chemical reactions involving many chemical species according to the predesigned rules of biology or biochemistry of the species, and some sorts of computing are involved in the living processes of animate systems. It therefore is not at all farfetched to think that chemical reactions are involved in them. Indeed, chemical neural networks have been demonstrated as a particular class of neural networks mentioned earlier. In * Corresponding author. E-mail address: [email protected]. † On leave from Departamento de Quı´mica, Universidad Nacional de Colombia, Bogota, Colombia. ‡ Also at the Centre for the Physics of Materials and Department of Physics, McGill University and the Asia Pacific Center for Theoretical Physics, Seoul, Korea. E-mail address: [email protected].

chemical neural networks the neurons are represented by nonlinear chemical systems that are coupled with each other and operate under specified conditions. They have been applied to problems related to the prediction of the dynamics of oscillatory reactions,6 pattern recognition,7,8 implementation of finite-state machines and Turing machines,9-11 self-organization,12 chemical computers performing logic operations,13,14 and simulation of action potentials.15 Specific applications to computation with chemical logic gates have been made in a number of ways. For example, Ross et al.13 constructed a chemical neural network wherein the neurons are represented by a complex cyclic enzyme mechanism that shows bistability and thus essentially two states of activity; a particular state of activity is chosen by the connections between neurons in the network. The logic state of any neuron is determined by the concentration of the catalyst of the biochemical cycle, and, if the threshold value is exceeded, the state is assigned value 1 and otherwise value 0. Schneider et al.14,16 made neural networks where the neurons are represented by a chemical oscillator (minimal bromate oscillator) operating in a continuously stirred tank reactor (CSTR). This reaction exhibits bistability and oscillatory behavior, and they used the bistability to assign the states of activity or logic states in the neuron, whereas the connections are represented by fluxes that indirectly couple reactors. They gave an experimental and numerical solution of different logic gates for specific inputs, but the weights and the threshold value must be modified for each logic operation. Showalter et al.17 computed logic operations based on chemical wave propagation in geometrically constrained excitable media in a two-dimensional (membrane) Belousov-Zhabotinsky system. In this system the excitable medium is separated into compartments by walls with one or two apertures in them. In the case of a geometrical configuration corresponding to the 2-2-1 feedforward system considered by Schneider et al. (see below), the first two walls have two apertures each and the third wall has one aperture. Depending on the ratio of the tube radius (or channel width) to the aperture in the wall, the chemical waves transmit through the aperture (“true”) or the transmission is inhibited (“false”), and the spatio-temporal synchronization

10.1021/jp004504v CCC: $20.00 © 2001 American Chemical Society Published on Web 06/22/2001

Irreversible Thermodynamics of Neural Networks of the waves also may occur; logic operations thereby can be performed. The size of apertures, however, must be changed by an external agent in this system. Regardless of the types of chemical reactions and modes used for constructing chemical neural networks, they are evidently macroscopic systems that are subject to the laws of physics, and as such they must obey, in particular, the laws of thermodynamics according to the currently held viewpoint toward macroscopic phenomena. Therefore, it is important to examine chemical neural networks within the framework of the thermodynamic principles; it is the first of the aims of the work presented in this paper. Because the dynamical processes in the chemical neural networks are irreversible, we first formulate a thermodynamic theory of irreversible processes representative of the logical operations accompanying computations performed by chemical neural networks and then apply the theory to typical neural networks and examine the deductions that one can make on the basis of the networks taken as examples. In general, the theory developed is applicable to any chemical neural network that performs logic operations. In fact, inasmuch as the thermodynamic laws are applicable to any macroscopic processes, the basic irreversible thermodynamic formalism developed here can be adapted to any neural network if appropriate relevant variables representative of logic operations by the neural network are identified and subjected to the thermodynamic laws. Because the theory of irreversible processes presented here is general, it has relevance to neural networks in general. In this work, as an application of the thermodynamic theory, we compute the energy and matter dissipation accompanying logical operations when the chemical neural network performs computation with the logic gates constructed. The quantities called the calortropy and the calortropy production are the mathematical vehicle for studying the aforementioned dissipation. Here, by the term “matter dissipation” we mean that matter transforms from a useful to a less useful form for the given task, chemical or mechanical, of interest, and a similar meaning is applicable to energy. The idea of usefulness is relative, and that which is no longer useful for a given task can be useful for another lower in a hierarchical system of tasks. In the thermodynamic theory of irreversible processes, the second law of thermodynamics is representable, in the case of irreversible processes, by the quantity called calortropy.18 We show that this calortropy provides an integral surface for chemical neural networks on which the system evolves and computation is performed. It turns out that the system evolves on this calortropy surface from one stable minimum to another as computation is performed. Hopfield4 found an energy integral for his model of neural network for which action potentials are relevant variables obeying nonlinear differential equations. His energy integral may be regarded as a special case of the calortropy for the neural networks performing computations in accordance with the thermodynamic laws. To make the desired connection of the two concepts, it will be necessary to express the calortropy and its production in terms of the action potentials and related quantities in the Hopfield theory. This question, albeit interesting, is beyond the scope of this work. This paper is organized as follows. In section II, the thermodynamic theory of irreversible processes is formulated for chemical neural networks by gleaning and generalizing to chemical neural networks the results of the theory18-20 of irreversible processes formulated by one of the authors (B.C.E). In section III, neural networks are constructed on the basis of the minimal bromate oscillator reported in the literature.21-23

J. Phys. Chem. B, Vol. 105, No. 29, 2001 7105 In section IV, the formulated theory of irreversible thermodynamics is applied to neural networks that are based on the minimal bromate oscillator and the calortropy production is studied for various logic operations by the networks by numerically calculating the calortropy production accompanying the logic operations in section V. Section VI is the discussion and concluding remarks. II. Irreversible Thermodynamics of Neural Networks To prepare ourselves for the intended formulation of theory of irreversible processes in neural networks, we first briefly review the irreversible thermodynamics24 of a finite system where various irreversible processes occur. The theory was originally formulated for a single finite system interacting with reservoirs at equilibrium. For the neural networks, it is necessary to generalize the theory to a system consisting of finite subsystems (neurons) of a macroscopic scale that interact with each other at their boundaries. A. Irreversible Thermodynamics of Processes in a Finite System. Let us imagine a macroscopic but finite system consisting of various chemical species contained in volume V. This system interacts with the surroundings (e.g., reservoirs of heat or matter) by exchanging energy or matter at the boundaries. Parts of the boundaries may be inert to matter and energy exchange with the surroundings, and some other parts may be diathermal and permeable to matter. The system may be also subject to stresses at the boundaries, which may be deformable in response to the stress. The intensive variables characterizing the (nonequilibrium) thermodynamic states may be nonuniform over the volume so that the system is generally displaced from the equilibrium state. We assume that there are r independent chemical species present in the system. Because the system is a thermodynamic system of a macroscopic size, which is displaced from equilibrium, the processes occurring in the system are clearly not only irreversible in general but also governed by the laws of thermodynamics. A mathematical theory of evolution of macroscopic processes in such a system requires representations of the thermodynamic laws in terms of measurable quantities characterizing the state of the system. The mathematical representation of the first law of thermodynamics is well established through Joule’s experiment. If the internal energy change accompanying an infinitesimal process in a cycle of changes in a system is denoted by dE then the first law of thermodynamics is expressible with the vanishing cyclic integral

IdE ) 0

(1)

which implies that E is a state function in the space of macroscopic variables characterizing the state of the system. In the case of the second law of thermodynamics, the mathematical representation was given by R. Clausius25 for reversible processes only; it gives rise to the notion of entropy. It is important to recognize that the notion of entropy is applicable only to reversible processes or to systems at equilibrium with the surroundings. In the case of irreversible processes, Clausius did not have a mathematical representation, except in the form of an inequalitysthe Clausius inequalitys which is not suitable for formulation of a quantitative theory of irreversible processes. However, after a long lacuna, a mathematical representation of the second law of thermodynamics is acquired that is capable of giving rise to an adequate theory of irreversible processes. The quantity that generalizes the notion of entropy to irreversible processes is called calortropy (heat evolution), denoted by Ψ, which gives rise to a vanishing cyclic

7106 J. Phys. Chem. B, Vol. 105, No. 29, 2001

Barragań and Eu

integral for a cycle of irreversible changes in the system.

IdΨ ) 0

(2)

Thus, for a cyclic process in a macroscopic system, the first and second laws of thermodynamics can be represented by a pair of vanishing cyclic integrals of the internal energy and the calortropy in the thermodynamic space. In thermodynamics of irreversible processes, the internal energy E and the calortropy Ψ are studied in an appropriate thermodynamic space spanned by macroscopic variables for irreversible processes in the system of interest. Clausius25 elucidated two kinds of heat, compensated and uncompensated, which are associated with any irreversible process. In this paper they are denoted by Qc and Ξ, respectively, and in the case of an infinitesimal process their rates of change are related to the calortropy just introduced as follows:18,19,24,26

(

)

dQc dΞ dΨ ) T-1 + dt dt dt

(3)

This is a mathematical representation of the second law of thermodynamics for an infinitesimal irreversible process in a macroscopic system. A theory of irreversible processes can be formulated on the basis of this differential form with elucidation of the compensated and uncompensated heats mentioned earlier. The second law requires that the rate of change in uncompensated heat is always positive

dΞ/dt g 0

(4)

for all processes, with the equality holding for reversible processes only. It must be emphasized that the differential form (eq 3) is in fact the equation that defines the calortropy in the appropriately defined thermodynamic space, where dΨ is a total differential. However, it should not be construed that Qc and Ξ are integrals of the rates dQc/dt and dΞ/dt, which are not integrable in the thermodynamic space mentioned. The T is the absolute temperature of the system at the infinitesimal process of interest. The rate of change in compensated heat dQc/dt is related to the calortropy flux, and the rate of change in uncompensated heat dΞ/dt to the calortropy production. In theory of irreversible processes, we aim to elucidate these quantities in terms of thermodynamic variables spanning the thermodynamic space mentioned and constitutive relations for the substance comprising the system. If the process is reversible, then

dΞ )0 dt

(5)

(6)

where the symbol limrev means the limit of the process becoming reversible; a reversible process is an idealized process in which there is no energy dissipation, that is, eq 5 holds valid. In the case of a single finite system interacting with the reservoirs, the rate of change in compensated heat is given by the form24

dQc dQE dQn ) + dt dt dt

dQE )dt

(7)

∫B dB‚Q

(8)

with Q denoting the internal energy flow through the boundaries B of the system, whose outward normal vector is counted positive. The integral is over the entire set of boundaries. The right-hand side of this expression may be given as a difference in the boundary values of local heat flux Q if the special configuration is assumed for the system that has two opposing diathermal faces only. The second term on the right in eq 7 is related to the heat transfer associated with the transfer of matter and other forms of energy between the system and the surroundings. The first law of thermodynamics governs the internal energy and its relation with work and heat transfer. For an infinitesimal process in a finite system, it may be expressed by the mathematical form

dE dt

)

dQE dt

dV

-p

r

+

dt

∑ µˆ a a)1

deMa

r

+

dt

∑ ∑ Xka a)1 kg1

deΓka

+

dt

diW dt

(9)

where a denotes the species, µˆ a the chemical potential per unit mass of species a, Xka the kth generalized potential of species a, Γka the “flux” (nonconserved variables) conjugate to Xka, the symbol de/dt the rate of change arising from the exchange with the surroundings of the quantity by the system; the second through the fourth terms on the right in eq 9 are the terms representing work. Finally, diW/dt is the internal work intrinsic to the system and the irreversible process in question. It must be elucidated within the framework of irreversible thermodynamics. The signs of the terms in eq 9 are chosen according to the sign conventions26 used for heat and work in thermodynamics. We note that p, µˆ a, and Xka are constitutive parameters characteristic of the substances in the system. They are given by means of an empirical equation of state, formulas for the chemical potentials, and so forth in the phenomenological approach to irreversible thermodynamics. Analysis24 of compensated heat on the basis of local theory of irreversible processes suggests that it is possible to express dQn/dt in the form

dQn

identically, and the calortropy reduces to the Clausius (reversible) entropy:

dQc dΨ dS ) limrev ) limrev T-1 dt dt dt

where the first term on the right is the internal energy transfer between the system and its surroundings, which in the local theory may be expressed as a surface integral

dt

deMa

r

)-

∑ a)1

µˆ a

dt

r

+

∑ ∑ Xka a)1 kg1

deΓka dt

+

diQ dt

(10)

where diQ/dt is the intrinsic rate of heat change arising from irreversible processes in the system. This quantity is in essence related closely to diW/dt and must be elucidated within the framework of irreversible thermodynamics. In the equations presented earlier, there appear two kinds of time derivatives, namely, the transfer time derivative de/dt and the intrinsic time derivative di/dt, which together give rise to the total time derivative of a quantity. We therefore note that the total rate of change in a quantity A is given by

dA deA diA ) + dt dt dt

(11)

where the second term on the right accounts for the intrinsic rate of change within the system because of the irreversibility

Irreversible Thermodynamics of Neural Networks

J. Phys. Chem. B, Vol. 105, No. 29, 2001 7107

of the processes involved. The transfer time derivative of A represents the exchange of A between the system and its surroundings through the boundaries between them, and thus the compensation of A from the surroundings. It may be given by the volume integral of negative divergence of the local “flux” of A that appears in local theory of irreversible processes.26 This manner of writing the total rate of change in A suggests that we are thinking in terms of the compensated and uncompensated changes in A. We note that in this notation the nonconserved variables and masses evolve according to the constitutive equations

dΓka deΓka diΓka ) + dt dt dt

(12)

dMa deMa diMa ) + dt dt dt

(13)

where the dissipation part, namely, the second term on the right in these equations, must be suitably modeled for the irreversible processes of interest. These constitutive equations basically represent what we understand about the system with regard to the evolution of nonconserved variables Γka and masses Ma of various species in the system. For the details of the local theory model for these constitutive equations, see ref 24. Suppose there are l chemical reactions r

∑νalCa ) 0

(14)

a)1

where Ca denote compounds and νal the stoichiometric coefficients of chemical reaction l, which are counted positive for the products and negative for the reactants. Then, in addition to the form of the compensated heat presented in eqs 7 and 10, the rate of change in uncompensated heat is found to be given by the form24

T-1

dΞ dt

) Σl - T-1

m

AlRlV + ∑ l)1 T-1

r

∑ ∑ Xka a)1 kg1

diΓka dt

g 0 (15)

where Al is the affinity of chemical reaction l defined by r

Al )

∑ µˆ aνal

(16)

a)1

with Rl denoting the reaction rate for chemical reaction l, and Σl is defined by the bilinear expression

Σl ) T-1

r

∑ ∑(Γkaωka + Yka$ka)

(17)

a)1 kg1

Here ωka and $ka denote the thermodynamic forces driving irreversible processes within the finite system. They may be defined by the volume integrals of the gradients of the velocity of matter and intensive thermodynamic variables

ω1a ) χu ) -V-1

∫V dr ∇u

ω2a ) χV ) -V-1

∫Vdr ∇u

∫Vdr ∇ ln T

ω3a ) χln T ) -V-1 ω4a ) χµˆ φa ) -V-1

∫Vdr ∇(µˆ a + φa) (φa ) -∇Fa)

$ka ) -χXka ) V-1

∫Vdr ∇Xka

(18)

and Yka is the flux of Γka, which is formally defined by the volume average of ψka that appears in the flux evolution equation for the local flux Φka corresponding to the nonconserved variable Γka; Γka is the volume average of the local quantity Φka in such a local theory model.18,24 These definitions imply that the system is driven thermodynamically because of the velocity, temperature, concentration, or nonconserved flux difference between the boundaries of the system. In the reference quoted earlier in connection with the formulation of irreversible thermodynamics for a finite system, the bilinear form Σl defined earlier is also identified24 with (diW/dt - dQn/dt) in the sense

Σl ) T-1

(

)

diW dQn dt dt

(19)

which, together with the eq 15 for Ξ, forms a proposition for the theory of irreversible processes in a finite system by which the nature of calortropy production is elucidated. On combining the proposition for the calortropy production (eq 15) and the proposition for the compensated heat (eqs 7 and 10) together with the identification for Σl made in eq 19, the calortropy differential form in eq 3 is given by the exact differential form24

dΨ dt

) T-1

(

dE dt

dV

+p

dMa

r

-

dt

∑

µˆ a

dt

a)1

∑∑

)

dΓka

r

+

Xka

a)1 kg1

dt

(20)

This is the extended Gibbs relation from which, with the help of the constitutive relations for the intensive variables, T-1, pT-1, µˆ aT-1, and XkaT-1, appearing in the differential form, the calortropy of the system can be determined as an integral of the differential form (eq 20). The system evolves toward a stable equilibrium state on this integral surface. The second law of thermodynamics is then mathematically represented by the differential form (eq 20) and the inequality of eq 15. The reader is referred to the original work24 for the details of this theory. The aforementioned propositions will be the basis of the formulation of irreversible thermodynamics of neural networks discussed below. B. Irreversible Thermodynamics of Neural Networks. The foregoing formalism can be generalized to a system consisting of finite subsystems (e.g., neurons) interacting (i.e., connected) with each other in the sense that they exchange matter and energy through their boundaries. To distinguish the subsystems we affix the superscript s to the global macroscopic variables presented in the previous section. The whole system is made up of ν neurons and their surroundings which will be designated as the (ν + 1)th subsystem. Thus the rate of calortropy change is

dΨ dt

ν+1

)

∑ s)1

T

(s)-1

(

dQ(s) c dt

+

)

dΞ(s) dt

(21)


Barragań and Eu

with the condition

whereas the calortropy production is given by ν+1

T ∑ s)1

(s)-1

dΞ(s) dt

g0

(22)

where the subsystem contributions are given by the equations that have the same meaning for a single finite system considered earlier. The compensated heat is given by

dQ(s) dQ(s) dQ(s) c E n ) + dt dt dt

(23)

dQ(s) n

r

)-

dt

∑

µˆ (s) a

deM(s) a

+

(24)

s

r

dt

a)1

∫B dBs‚Q(s) ∑∑

X(s) ka

deΓ(s) ka

+

diQ(s)

dt

a)1 kg1

dt

(25)

and the uncompensated heat by the expression

T(s)-1

dΞ(s) dt

m

(s)-1 ) Σ(s) l -T

Al(s)R(s) ∑ l V+ l)1 r

T

(s)-1

∑∑

X(s) ka a)1 kg1

diΓ(s) ka dt

(26)

The constitutive equations appearing in these equations are given by the evolution equations

deΓ(s) diΓ(s) dΓ(s) ka ka ka ) + dt dt dt

(27)

deM(s) diM(s) dM(s) a a a ) + dt dt dt

(28)

with various symbols denoting r

A(s) l )

∑νalµˆ (s)a

(29)

a)1 r

(s)-1 Σ(s) l )T

(s) (s) (s) (Γ(s) ∑ ∑ ka ωka + Yka $ka ) a)1 kg1

(30)

(s) Here ω(s) 1a , ‚‚‚, $ka are defined by the expressions in eq 18 for each set of boundaries of the subsystem. If the system is at a uniform temperature, free from stress, and of uniform composition, then the structure of the aforementioned formalism becomes simpler, and we obtain

dQ(s) dQ(s) c n ) dt dt

(31)

with the contribution to the right-hand side coming from mass changes only

dQ(s) n dt

r

)-

∑

a)1

µˆ (s) a

deM(s) a dt

(32)

dΞ(s)

m

) -T(s)-1

dt

(s) A(s) ∑ l Rl V l)1

(33)

since the species are uniformly distributed within the subsystems and hence there is no chemical potential gradients within the subsystems (neurons). Therefore, the calortropy change in the entire neural network is simply given by the expression

( ) dΨ dt

with the definitions

dQ(s) E )dt

T(s)-1

nn

) -T

-1

ν

r

∑ ∑ s)1 a)1

µˆ (s) a

deM(s) a dt

ν

r

∑ ∑A(s)l R(s)l V s)1 l)1

- T-1

(34)

where the contribution to the calortropy change from the surroundings, namely, the (ν + 1)th subsystem is excluded, because it is not of interest to us here. In this form, the rate of calortropy change within the neural network coincides with the rate of entropy change for a system in local equilibrium. However, it should be noted that because of the ongoing chemical reactions the system is not in thermodynamic equilibrium and the notion of entropy does not really apply in the present circumstances. Therefore, we will continue to use the notion of calortropy in this work. If we assume that some of the subsystems are fed with, and drained of, matter at a fixed rate by reservoirs, while the rest of subsystems are not connected with the reservoirs, then the part of the compensated heat corresponding to the interaction with the reservoirs can be further separated from the first term on the right of eq 34 and the remainder can be regarded as the calortropy production. We will assume that this separation has been already made in the first term on the right of eq 34 and regard the remainder as part of the calortropy production for the whole system. Thus, the first term on the right of eq 34 is the calortropy production arising from the intersubsystem (e.g., interneuron) interactions that give rise to exchange of matter between the subsystems (e.g., neurons), and the second term is the calortropy production arising from the chemical reactions within each subsystem (e.g., a neuron). Equation 34 thus provides the total calortropy production in the thermodynamic theory of the neural network under consideration. We will apply it to neural networks based on the minimal bromate oscillator in the following. We close this section with the remark that the formalism leading to the calortropy production formula (eq 26) for arbitrary irreversible processes is general. Thus, with an appropriate choice for the nonconserved variables Γ(s) ka representative of the neural network in hand, a theory of irreversible thermodynamics can be developed, and the calortropy and the calortropy production can be calculated for it. Ultimately, one may even try to quantify the efficiency of logic operations by the neural network. The present formulation of the theory provides a framework for such studies. III. The Minimal Bromate Oscillator A. Evolution Equations for the Neural Network. The system of chemical reactions used in this work is the minimal bromate oscillator,21-23 which is based on the Noyes-FieldThompson (NFT) mechanism.27 The NFT mechanism is often thought of as the inorganic part of the Field-Ko¨ros-Noyes (FKN) mechanism28 and the Gyorgyi-Turanyi-Field (GTF) mechanism29 proposed to explain the oscillatory behavior observed in the Belousov-Zhabotinsky reaction. The NFT mechanism (see reactions CR1 to CR6 and Table 1 below) describes the ceric ion oxidation by bromate ions in a sulfuric acid medium. When conducted in a continuously stirred tank



TABLE 1: Rate Constants for the NFT Model reaction

forward rate constant

reverse rate constant

CR1 CR2 CR3 CR4 CR5 CR6

k1 ) 2.0 M-3 s-1 k2 ) 3×106 M-2 s-1 k3 ) 3 × 109 M-2 s-1 k4 ) 42 M-3 s-1 k5 ) 8 × 10 M-2 s-1 k6 ) 3 × 103 M-1 s-1

k-1 ) 3.2 M-1 s-1 k-2 ) 2×10-5 M-1 s-1 k-3 ) 2.0 s-1 k-4 ) 4.2 × 107 M-1 s-1 k-5 ) 8.9 × 103 M-1 s-1 k-6 ) 1 × 10-8 M-2 s-1

the concentrations by xi and order them in the following manner: x1 ) Ce4+, x2 ) Ce3+, x3 ) Br-, x4 ) BrO3-, x5 ) H+, x6 ) HBrO2, x7 ) BrO•2, x8 ) HOBr, x9 ) Br2. Then the mass action laws for the forward and reverse reactions are given by

Λ1 ) k1x3x4x25

Λ-1 ) k-1x6x8

Λ2 ) k2x3x5x6

Λ-2 ) k-2x28

Λ3 ) k3x3x5x8

Λ-3 ) k-3x9

Λ4 ) k4x4x5x6

Λ-4 ) k-4x27

Λ5 ) k5x2x5x7 Λ6 ) k6x26

Λ-5 ) k-5x1x6 Λ-6 ) k-6x4x5x8

(35)

In this notation, the reaction rates for various species are given by R1 ) Λ5 - Λ-5 - (Λ6 - Λ-6) ≡ Λ+ 1 - Λ1

Figure 1. Bifurcation diagram for the minimal bromate oscillator. The steady states in the upper branch (a) representing the logic value “1” or “true”, and those in the lower stable branch (c) representing the logic value “0” of “false”, bifurcate to oscillating states represented by part (b). The vertical broken lines in part (b) connect the maximum and minimum amplitudes of the oscillating concentrations to guide the eyes.

reactor (CSTR), the reaction system exhibits bistability with hysteresis between two steady states and oscillations in a small region of values for the flow rate (kf), which is used as a control parameter; typically, this parameter ranges from 3.87 × 10-3 s-1 to 5.43 × 10-3 s-1; see Figure 1. If the value of kf is smaller than 3.87 × 10-3 s-1, the steady state is characterized by high concentrations in Ce4+ ion, whereas if the value of kf is larger than 5.43 × 10-3 s-1, the steady state is characterized by low concentrations in Ce4+ ion. Therefore, in the case of the former flow rate the oxidated state of cerium ion is predominant, whereas in the case of the latter flow rate the reduced state of cerium ion is predominant. The reactions for the minimum bromate oscillator are as follows:

BrO3- + Br- + 2H+ h HBrO2 + HOBr

(CR1)

HBrO2 + Br- + H+ h 2HOBr

(CR2)

HOBr + Br- + H+ h Br2 + H2O

(CR3)

BrO3- + HBrO2 + H+ h 2BrO•2 + H2O

(CR4)

Ce3+ + BrO•2 + H+ h Ce4+ + HBrO2

(CR5)

2HBrO2 h BrO3- + HOBr + H+

(CR6)

where BrO•2 denotes the free radical. The rate constants for this system of reactions are summarized in Table 1, where ki and k-i denote the rate constants for the forward and reverse reaction, respectively. To cast the evolution equations for the chemical species of the minimal bromate oscillator in a general form, it is convenient to use a unified system of notations. Let us denote

R2 ) -(Λ5 - Λ-5) + Λ6 - Λ-6 ≡ Λ+ 2 - Λ2

R3 ) -

3 (Λj - Λ-j) ≡ Λ+ ∑j)1 3 - Λ3

R4 ) -(Λ1 - Λ-1) - (Λ4 - Λ-4) + Λ6 - Λ-6 ≡ Λ+ 4 Λ4 R5 ) - 2(Λ1 - Λ-1) -

5 (Λj - Λ-j) + Λ6 ∑j)2

Λ-6 ≡ Λ+ 5 - Λ5

R6 ) Λ1 - Λ-1 - (Λ2 - Λ-2) - (Λ4 - Λ-4) + Λ5 Λ-5 ≡ Λ+ 6 - Λ6

R7 ) 2(Λ4 - Λ-4) - (Λ5 - Λ-5) - (Λ6 - Λ-6) ≡ Λ+ 7 Λ7 R8 ) Λ1 - Λ-1 + 2(Λ2 - Λ-2) - (Λ3 - Λ-3) ≡ Λ+ 8 - Λ8 R9 ) Λ3 - Λ-3 ≡ Λ+ 9 - Λ9

(36)

The evolution of species is then described by the differential equations

dxi ) kf(x0i - xi) + Ri (i ) 1, ..., 9) dt

(37)

where x0i ) 0 for i ) 1, 6, 7, 8, 9 and x0i * 0 for i ) 2, 3, 4, 5 are the concentrations of inflowing reactants and kf is the flow rate. This is the set of evolution equations for a neuron in the network. To distinguish the neurons, we affix a superscript (s) on various quantities. Thus for the whole neural network we have the evolution equations

dxi(s) (s) (s) ) kf(x0(s) i ) - xi + Ri (i ) 1, ..., 9; s ) 1, 2, ...) (38) dt where x0(s) have the same meanings as for x0i for a single i neuron.


Barragań and Eu

B. The Chemical Neural Networks. We use an architecture of neural network similar to that used by Schneider et al.14,16 The chemical neural networks studied in this work are 2-1 and 2-2-1 feedforward networks through which the information flows in just one direction. These networks have a computational capability to perform logic operations. The input layer of neurons consists of two neurons that do not directly transfer mass to the output neuron but trigger mass transfer from the reservoir into the output neuron through a circuit (e.g., an interface consisting of a PC monitoring the input). The flow rate into the output neuron is determined according to the Hebbian rules. The network is trained by using different inputs until it has learned to perform desired logic operations, and the computational capability of the trained network has been tested by using random inputs. Each neuron in the network is represented by a CSTR tank containing the aforementioned chemical system, and the connections between the neurons are realized by means of their flow rates in such a way that the flow rate in the output neuron is “controlled” according to the output signal (x1) in the input layer of neurons through the “circuit” (e.g., the interface). Therefore, mass exchange between the neurons of the input and output layers of the network is not made directly but through the interface, and the total input into the output neuron is determined by the transfer function 2

Γj )

wijx(i) ∑ 1 + Θj i)1

(39)

where Θj (normally called “bias”) is related to a threshold that has to be exceeded by the net input 2

wijx(i) ∑ 1 i)1

{

kf e 3.86 × 10 , if Γj < 0 kf g 5.44 × 10-3, if Γj > 0

IV. Calortropy Production Irreversible processes in the neural network evolve on the calortropy surface spanned by species concentrations in the case of the architecture used in the present work. In the sense that the calortropy surface is a mathematical realization of the thermodynamic laws, they are consistent with the thermodynamic laws, which in particular demand that the calortropy production19,20 be positive and vanish at equilibrium only. Since the processes evolve in the calortropy surface, it will be interesting to understand what the consequences will be for the calortropy production for the global system, which also indicates a measure of energy and matter dissipation; that is, in this particular case, matter transforming from a useful to a less useful form as a consequence of the task performed. We calculate the calortropy production by using the formula presented in the previous section under the assumption that the solutions are ideal; that is, the chemical potentials for the species in the solutions of neurons are given by the formula

µˆ a(T, p, xa) ) µˆ 0a(T, p) + RT ln xa

}

( ) dt

) -VT-1

nn

ν

) VR

(40)

If the threshold value is exceeded in this network, the output neuron is said to have fired with the value of the state variable “1” or “true” assigned to it; otherwise, the output neuron is said to have been inhibited with the value of the state variable “0” or “false” assigned to it. We exploit the bistable behavior of the minimal bromate oscillator by assigning the logic value “1” or “0” to the high (kf e 3.86 × 10-3 s-1) or low (kf g 5.44 × 10-3 s-1) steady-state values of the output signal (x1), respectively. Operationally, the Ce4+ ion concentration in the neuron (CSTR) is measured by means of the redox potential, and the so-measured values of Ce4+ ion concentrations are used to compute kf for the subsequent layer of neurons (CSTR) according to eqs 39 and 40, and the resulting value of kf is used by the interface to control the flow rate into the next layer of neurons. In the present neural network, the value of the bias Θj is fixed at a value as is generally required in theory of neural networks, whereas in the case of neural networks examined by Schneider et al.14,16 the different values were, by necessity, used for different logic operations. The cause for this difference is in using the rule for mass transfer (eq 40) together with the Hebbian rule (eq 39) in the neural network considered in this work, whereas in the case of the neural networks of Schneider

(41)

where µˆ 0a(T, p) is the reference chemical potential of species a, xa is the mole fraction of a (the total mole number is set equal to unity in this manner of writing the chemical potentials), and R is the gas constant. The reference chemical potentials are defined such that the Gibbs free energy change for an isolated neuron is equal to zero at chemical equilibrium. With eq 41, the calortropy production arising from chemical reactions in the neural network is given by the formula

dΞ

The magnitude of this transfer function determines the mass transfer from the reservoir according to the rule -3

et al., only the Hebbian rule (eq 39) is used; that is, kf is simply set equal to Γj.

ν

m

(s) A(s) ∑ ∑ l Rl s)1 l)1

[

m

∑ ∑ s)1 l)1

(Λ+ l

-

Λl )

( )] Λ+ l

ln

Λl

(42)

s

where the quantity in the square brackets on the right represents the calortropy production in neuron s. To derive this equation, we have made use of the relation between the chemical equilibrium constant, which is equal to the ratio of the forward and reverse rate constants of the reaction. On summing the neuronal contributions over the neural network the total calortropy production arising from the system of chemical reactions is obtained. In the case of the neural network in which neurons are coupled or, put in other words, interact by exchanging matter, there is also a contribution to the calortropy production to the neural network that arises from the compensated heat. This contribution is given by

( ) dQc dt

) -T

-1

nn

ν

r

∑ ∑ s)1 a)1

µˆ (s) a

deM(s) a dt

(43)

Because at chemical equilibrium ν

r

∑ ∑ s)1 a)1 we obtain

µˆ 0(s) a

deM(s) a dt

)0

(44)


( ) dQc dt

ν

r

∑ ∑ s)1 a)1

) -R

nn


[

kf(x0a - xa) ln xa]s

(45)

Finally, the total calortropy production in the neural network is given by

() dΨ dt

nn

ν

) -R

TABLE 2: Logic Table to Compute the Output in the Logic Gates input1

input2

AND

NAND

OR

NOR

XOR

XNOR

1 1 0 0

1 0 1 0

1 0 0 0

0 1 1 1

1 1 1 0

0 0 0 1

0 1 1 0

1 0 0 1

r

kf(x0a - xa) ln xa]s + ∑ ∑ s)1 a)1 [

ν

VR

[

m

∑ ∑ s)1 l)1

(Λ+ l

-

Λl )

( )] Λ+ l

ln

Λl

≡ σnet (46)

s

It should be noted that (dΨ/dt)nn ≡ σnet is not the same as the total rate of calortropy change dΨ/dt for the whole system including the surroundings, because the contribution from the surroundings has been subtracted out; see eqs 21 and 34. If the concentrations oscillate with a period τ, then it is useful to define the mean value of σnet by the time average

σosc )

1 τ

∫0τ dτ σnet

(47)

It should be noted that because σnet is not a state function in the thermodynamic space, its cyclic integral does not vanish. The calortropy production will be examined for various logic operations performed by the neural network considered in this work.

Figure 2. Architectures for the chemical networks: (a) architecture of the 2-1 feedforward network to compute the logic operations AND, NAND, OR, and NOR; (b) architecture of the 2-2-1 feedforward network to compute the logic operations XOR and XNOR. The different values for the weights wij are given in Table 3.

V. Logic Operations and Irreversible Thermodynamics The neural network described in the previous section is capable of performing logic operations AND, NAND, OR, NOR, XOR, and XNOR. The logic table is given in Table 2, where relations between the inputs and the logic functions are summarized. The 2-1 feedforward network is capable of logic operations AND, NAND, OR, and NOR, whereas to perform logic operations XOR and XNOR in addition to the AND, NAND, OR, and NOR logic operations, it is necessary to have the 2-2-1 feedforward network. A. The 2-1 Feedforward Network. The architecture of a 2-1 feedforward network, which is basically that used by Schneider et al.,14,16 is depicted in Figure 2a. This neural network is capable of computing the logic functions AND, NAND, OR, and NOR. The shaded squares within the broken lined box in Figure 2a denote input neurons. These neurons receive only one constant input from the exterior world (surroundings). The left half in the unshaded square for the output neuron in the box, which comprises the interface made up of a PC monitoring the total input, represents the total input 2

wijx(j) ∑ 1 i)1 which would have to exceed the Θj threshold value (called the bias in the literature) to give an output of value 1 or an output “true” or it will give no output, that is, a value 0 or “false”. Depending on the value of the input relative to the threshold value, the output neuron (i.e., the right half of the unshaded square on the right) puts out a “true” (upper line) or “false” output. The training of the network consists of determining the weights wi,j, which allow to compute the correct output by the network according to the logic operation intended. In this work we show that for a single threshold value for Θj there can be

different weights wij that allow to compute the different logic operations AND, NAND, OR, NOR, XOR, and XNOR; see Table 3 for the values of weights. B. The 2-2-1 Feedforward Network. To perform logic operations XOR and XNOR, a 2-2-1 feedfoward network is used. In Figure 2b the architecture of the 2-2-1 network is depicted. In Figure 2a, the shaded squares in the broken lined box represent the input neurons, and unshaded squares represent two layers of output neurons, with the meanings of the symbols in the squares being the same as for Figure 2a. This network is more complex so that to compute one output it is necessary to compute two additional logic operations internally: XOR ) [(AND,NOR),NOR] and XNOR ) [(AND,OR),NOR]. By this notation we mean the following: for XOR the inputs from the shaded neurons (layer 1) are integrated (or summed) to yield “true” for the output by the first neuron and “false” by the second neuron in layer 2. These outputs are integrated, and the output neuron in layer 3 yields NOR. The output for different logic operations is summarized in Table 2. It must be noted that the logic state of a neuron is not given by a single definite value of the Ce4+ concentration but by a range of values which is set to value “1” or “0”. Therefore, different concentrations for inputs can be representing the same logic output. The values of weights in Table 3 are for Figure 2a and 2b. Because the threshold value is fixed at a value for a network and consequently the weights are different from those chosen by Schneider et al., the present neural networks, despite the same architectures as theirs, are different in the neural dynamics. C. Numerical Simulations. The systems (eq 38) of ordinary differential equations corresponding to the 2-1 and 2-2-1 neural networks described earlier are solved numerically with the LSODE30 subroutine based on Gear’s method31 for stiff ordinary differential equations. This method uses a numerically estimated Jacobian matrix. The following initial conditions are taken for


Barragań and Eu

TABLE 3: Weights and Bias logic operation AND NAND OR NOR XOR XNOR

w13

w23

0.61 -0.61 0.97 -0.97 0.61 0.61

0.61 -0.61 0.97 -0.97 0.61 0.61

w14

-0.97 -0.97

w24

-0.97 -0.97

w35

-0.97 0.97

w45

bias1

bias2

bias3

-0.97 0.97

-2 × 10-4 2 × 10-4 -2 × 10-4 2 × 10-4 -2 × 10-4 -2 × 10-4

2 × 10-4 2 × 10-4

2 × 10-4 -2 × 10-4

Figure 3. Logic operation (1,0)AND ) 0 and the calortropy production: (a) Logic state “0” for the output neuron; (b) logic state “1” for one input neuron, (c) logic state “0” for the other input neuron; (d) dissipation in the network. The scale for curve (d) is given by the secondary y axis, where σ* is given in units of R according to the definition σ* ) σnet/R.

Figure 4. Logic operation (1,0)OR ) 1 and the calortropy production:. (a) logic state “1” for the output neuron; (b) logic states “0” for one input neuron; (c) logic states “1” for the other input neuron; (d) dissipation in the network. The scale for curve (d) is given by the secondary y axis, where σ* is given in units of R according to the definition σ* ) σnet/R.

the concentrations of chemical species: x4 ) 0.1, x5 ) 0.75, x2 ) 3 × 10-4, x3 ) 3 × 10-4, which are the same as the concentrations of the inflowing reactants. Because the global behavior of a network is determined by the dynamics of neurons and their outputs, once the network has been trained, the capability to process information and to produce the output desired must be independent of the inputs provided from the external world. In this work, the dynamics of the networks is investigated as follows. Initially, each one of the input neurons is provided with a random value for the control parameter kf. If kf e 3.86 × 10-3, the system evolves to a steady state where the logic state of the neuron is interpreted as “1” or “true”; if kf g 5.44 × 10-3, the system evolves to a steady state where the logic state of the neuron is interpreted as “0” or “false”. After this and every 10 000 s, the input neurons are provided with new random values for the control parameter kf, whereas the output neuron continuously processes the information and the output is assessed for assignment to the “true” or “false” value. Notice that even when a new random control parameter kf is given every 10 000 s and state values are changed thereby, the output level remains virtually constant, independently of input levels. Figure 3 shows the capability of the network (Figure 2a and Table 3) when the logic operation AND is computed, more specifically, when (1,0)AND with output equal to “0” is computed. The curve a shows that the logic state of the output neuron is at all times below value 1.5 × 10-4 for the output signal (x1). This logic state is interpreted as “0”, whereas curves b and c show that the input neurons are at all times firing values in the output signal (x1) that are interpreted as “1” and “0”, respectively. The curve d shows the profile of dissipation in the network, which is approximately constant for a logic state of the output neuron. The observed discontinuous changes in the matter dissipation are associated with the changes that occur in the input neurons.

Figure 5. Logic operation (0,0)NAND ) 0 and the calortropy production: (a) logic state “1” for the output neuron; (b) logic states “0” for one input neuron; (c) logic states “0” for the other input neuron; (d) dissipation in the network. The scale for curve (d) is given by the secondary y axis, where σ* is given in units of R according to the definition σ* ) σnet/R.

Figure 4 shows the network computing the logic operation OR, specifically, the state “1” or “0” with the output equal to “1”. The explanation for curves a, b, c, and d is similar to that given for Figure 3. In this logic state the output neuron is continuously firing a value greater than 1.6 × 10-4 in the signal output (x1), namely, curve a, which is interpreted as “1”. Figure 5 shows the network computing the logic operation (0,0)NAND with the output equal to “1”, and Figure 6 shows the network in Figure 2b computing the logic operation (1,1)XOR with the output equal to “0”. The dissipation of matter in the networks was analyzed in detail. For this purpose each one of the networks was tested with 30 different random values for the control parameter kf in the input neurons (300 000 s for running time). It is not our


Figure 6. Logic operation (1,1)XOR ) 0 and the calortropy production: (a) logic state “0” for the output neuron; (b) logic states “1” for one input neuron; (c) logic states “1” for the other input neuron; (d) dissipation in the network. The scale for curve (d) is given by the secondary y axis, where σ* is given in units of R according to the definition σ* ) σnet/R.

Figure 7. Levels of dissipation for the logic operations AND and NAND (b (1,1)AND ) 1; O (1,1)NAND ) 1; 3(1,0)NAND ) 1, 4 (1,0)AND ) 0; * (0,0)AND ) 0; . (0,0)NAND ) 1): (a) dissipation in the network when one input neuron is in an oscillatory state (kf ) 5.4 × 10-3), the other input neuron is in steady state with logic meaning “0” (kf ) 7 × 10-3) and the output neuron is in a steady state with logic meaning “0” (kf ) 7 × 10-3); (b) dissipation when both input neurons are in oscillatory states (kf ) 4.2 × 10-3 and kf ) 4.8 × 10-3); (c) dissipation in the network when one input neuron is in a steady state with logic meaning “1” (kf ) 1.5 × 10-3) and the other input neuron is in an oscillatory state (kf ) 5.4 × 10-3). σ* is given in units of R according to the definition σ* ) σnet/R.

intention to establish exact correlations between the values in the input neurons and the dissipation of matter, but to indicate the tendency in the behavior of the dissipation of matter from a useful to a less useful form. Figure 7 shows that the dissipation (calortropy production) levels are increased, as the network in Figure 2a computes for the logic operation AND with different inputs, namely, (1,1), (0,0), and (1,0), the highest level being when the network computes the logic state AND with “0” and “0” for the inputs, whereas the lowest level of dissipation is attained when it computes the logic operation AND with inputs “1” and “1”. The level of dissipation for the logic state “1” AND “0” is in the middle. In the case of logic operation NAND, the level of dissipation in the network is similar when it computes the logic states for (1,0)NAND and (1,1)NAND, whereas the highest level of dissipation for this operation results when the network computes the logic state for (0,0)NAND. As a comparison, we


Figure 8. Levels of dissipation for the logic operations OR and NOR (b (1,1)OR ) 1; O (1,1)NOR ) 0; 3(1,0)NOR ) 0; 4 (1,0)OR ) 1; * (0,0)OR ) 0; .(0,0)NOR ) 1). σ* is given in units of R according to the definition σ* ) σnet/R.

computed the dissipation when in one or both input neurons the state is oscillatory, whereas the output neuron is in the steady-state all the time; in this situation the network does not compute for any logic operation. The curve a in the figure shows the level of dissipation when the output network is in a steady state at kf ) 7 × 10-3; one input neuron is in a steady state at kf ) 7 × 10-3, and the other input neuron is in an oscillatory state at kf ) 5.4 × 10-3. For the control parameters kf ) 4.2 × 10-3 and kf ) 4.8 × 10-3, both input neurons are in an oscillatory state (these states are indicated by line b) and, in the case of line c, one input neuron is in a steady state at the control parameter value kf ) 1.5 × 10-3 and the other input neuron is in an oscillatory state at kf ) 5.4 × 10-3. It is evident that the lowest dissipation is given when both input neurons are in an oscillatory state. But then, no computation is performed by the present network in an oscillatory state. Figure 8 shows that in the case of logic operation OR the highest level of dissipation is obtained when the network computes for the logic state (0,0)OR, whereas in the case of logic operation NOR the highest levels of dissipation are obtained when the network computes for the logic states (0,0)NOR and (1,0)NOR. When the network in Figure 2b is computing for the logic operation XOR or XNOR, the behavior of the levels of dissipation is more complex, but some tendencies are shown in Figure 9. For the logic operation XOR the highest level of dissipation in the network is obtained when it computes for the logic state (0,0)XOR, whereas for the logic operation XNOR the highest levels of dissipation are obtained when the network computes for the logic states (0,0)XNOR and (1,0)XNOR. We do not know whether this trend is general or not, but it may be worth remembering for the future. The notable feature of the calculations made for the dissipation associated with various logic operations appears to be not in their magnitudes but in the fact that the neurons have characteristic steady states corresponding to logic operations and evolve toward a particular steady state that has what seems to be a characteristic level of dissipation on the calortropy production surface, which also appears to be a local minimum in the surface. This minimum, however, is not sharply defined for a given logic operation, because the true or false states do not have a constant but a varying dissipation within a limit, as is clear from the figures shown. In the sense that the calortropy production is not equal to zero at the steady states, the system is removed from equilibrium, and consequently logic operations


Figure 9. Levels of dissipation for the logic operations XOR and XNOR (b (1,1)XOR ) 0; O (1,1)XNOR ) 1; 3 (1,0)XNOR ) 0; 4 (1,0)XOR ) 1; * (0,0)XOR ) 0; . (0,0)XNOR ) 1). σ* is given in units of R according to the definition σ* ) σnet/R.

in the neural networks are irreversible processes in systems removed from equilibrium. The level of energy and matter dissipation by neurons and by the whole neural network is important if it is the aim to design an energetically and materially efficient neural network, although it is quite conceivable that because the primary function of a neural network is to perform desired logic operations and computation, it may be maintained by the system (e.g., an animate system) regardless of the cost of energy and matter. Nevertheless, the irreversible thermodynamic formalism presented will enable us to investigate the question of efficiency of neural networks, mutual relations of various processes, and the efficiency of information transfer in the networks in the future. VI. Discussion and Concluding Remarks In this work we have formulated a theory of irreversible processes in neural networks by generalizing the theory of irreversible processes in a finite system formulated previously.24 With the formalism presented, we have shown that it is possible to numerically calculate the calortropy production representing energy and matter dissipation associated with logic operations in neural networks based on the minimal bromate oscillator. Specifically, we have considered 2-1 and 2-2-1 feedforward neural networks constructed by Schneider et al.14,16 The calortropy production has a characteristic value for each logic operation performed by the neural network, and the calculation indicates that the neural network evolves to a characteristic steady state in the calortropy surface and thereby performs a logic operation and computation. Therefore, the calortropy surface provides a convenient and thermodynamically consistent means of understanding logic operations and computations in the neural network, especially from the viewpoint of energy and matter and their evolution. The theory of irreversible thermodynamics formulated here is applicable to any neural network. Because the power of thermodynamics lies in its ability to correlate various seemingly different phenomena within the framework of the thermodynamic principles, it is possible to believe that the same would hold true for the theory presented for neural networks. This aspect, of course, needs investigation in the future. We believe that the present theory of irreversible processes in neural networks provides a basis and also a framework for such investigations.

Barragań and Eu Our results also show that the highest level of dissipation in both architectures considered (Figures 2a and 2b) arises when the networks are computing, from inputs with logic state “0” and one output logic state “0”, as it is the case of the logic gates AND, OR, and XOR and, generally, logic operations appear to have characteristic levels of calortropy production. An interesting outcome of this investigation, which may be useful to note, is that if it is possible to build a network with computational capabilities and the logic states characterized by oscillatory behaviors, the dissipation levels will be low for all logic operations, compared with networks whose logic states are characterized by steady states. To perform logic operations with such an oscillatory system according to the McCullochPitts model, it is necessary to have two distinctive limit cycles between which the states of the system make transition. However, it is not clear whether such a network of two distinctive limit cycles in concentrations is possible to construct in practice. Acknowledgment. This work was supported in part by the Natural Sciences and Engineering Research Council of Canada, by FCAR of Quebec through the Centre for the Physics of Materials, McGill University, and a grant from the Universidad Nacional de Colombia, Bogota, Colombia. References and Notes (1) Hebb, D. The Organization of BehaVior; Wiley: New York, 1949. (2) McCulloch, W.; Pitts, W. Bull. Math. Biophys. 1943, 5, 115. (3) Mano, M. M. Computer System Architecture, 2nd ed.; PrenticeHall: Englewood Cliffs, NJ, 1982. (4) Hopfield, J. J. ReV. Mod. Phys. 1999, 71, S431 and references therein. (5) Hopfield, J. J.; Tank, D. W. Science 1986, 233, 625. (6) Kiss, I. Z.; Gaspar, V. ACH-Models Chem. 1995, 132, 887. (7) Hjelmfelt, A.; Schneider, F. W.; Ross, J. Science 1993, 260, 335. (8) Hjelmfelt, A.; Ross, J. J. Phys. Chem. 1993, 97, 7988. (9) Turing, A. Proc. London Math. Soc. ser. 2 1936, 42, 230. (10) Hjelmfelt, A.; Weinberger, E. D.; Ross, J. Proc. Natl. Acad. Sci. U.S.A. 1991, 88, 10983. (11) Hjelmfelt, A.; Weinberger, E. D.; Ross, J. Proc. Natl. Acad. Sci. U.S.A. 1992, 89, 383. (12) Okamoto, M.; Maki, Y.; Sekiguchi, T.; Yoshida, S. Physica D 1995, 84, 194. (13) Hjelmfelt, A.; Ross, J. Physica D 1995, 84, 180. (14) Lebender, D.; Schneider, F. W. J. Phys. Chem. 1994, 98, 7533. (15) Schinor, N.; Schneider, F. W. Biophys. Chem. 2000, 85, 199. (16) Zeyer, K. P.; Dechert, G.; Hohmann, W.; Blittersdorf, R.; Schneider, F. W.; Z. Naturforsch. 1994, 49A, 953. (17) Steinbock, O.; Kettunen, P.; Showalter, K. J. Phys. Chem. 1996, 100, 18970. (18) Eu, B. C.; Kinetic Theory and IrreVersible Thermodynamics; Wiley: New York, 1992. (19) Eu, B. C. Phys. ReV. E 1995, 51, 768. (20) Al-Ghoul, M.; Eu, B. C. Physica D 1996, 90, 119. (21) Bar-Eli, K. J. Phys. Chem. 1985, 89, 2855. (22) Bar-Eli, K.; Field, R. J. J. Phys. Chem. 1990, 94, 3660. (23) Weiner, J.; Holz, R.; Schneider, F. W.; Bar-Eli, K. J. Phys. Chem. 1992, 96, 8915. (24) Eu, B. C. J. Phys. Chem. 1999, 103, 8583. (25) Clausius, R. Ann. Phys. (Leipzig), 1865, 125, 313. (26) Eu, B. C. Nonequilibrium Statistical Mechanics; Kluwer: Dordrecht, 1998. (27) Noyes, R. M.; Field, R. J.; Thompson, R. C. J. Am. Chem. Soc. 1971, 93, 7315. (28) Field, R. J.; Koros, E.; Noyes, R. M. J. Am. Chem. Soc. 1972, 94, 8649. (29) Gyorgyi, L.; Turanyi, T.; Field, R. J. J. Phys. Chem. 1990, 94, 7612. (30) Hindmarsh, A. C. LiVermore SolVer for Ordinary Differential Equations, Technical Report No. UCID-3001; Lawrence Laboratory: Livermore, CA, 1972. (31) Gear, C. W. Numerical Initial Value Problems in Ordinary Differential Equations; Prentice-Hall: Englewood Cliffs, NJ, 1971.

Irreversible Thermodynamics of Neural Networks: Calortropy

Recommend Documents