Ind. Eng. Chem. Res. 2008, 47, 5243–5249
5243
Fisher Information, Entropy, and the Second and Third Laws of Thermodynamics Heriberto Cabezas* and Arunprakash T. Karunanithi U.S. EnVironmental Protection Agency, Office of Research and DeVelopment, National Risk Management Research Laboratory, 26 W. Martin Luther King DriVe, Cincinnati, Ohio 45268
We propose Fisher information as a new calculable thermodynamic property that can be shown to follow the second and third laws of thermodynamics. However, Fisher information is qualitatively different from entropy and potentially possesses much more structure. Hence, a mathematical expression is derived for computing the Fisher information of a system of many molecules from the canonical partition function. This development is further illustrated through the derivation of Fisher information expressions for a pure ideal gas and an ideal gas mixture. Some of the unique properties of Fisher information are then explored through the classic experiment of the isochoric mixing of two ideal gases. Note that, although the entropy of isochorically mixing two ideal gases is always positive and is dependent only on the respective mole fractions of the two gases, the Fisher information of mixing has far more structure, involving the mole numbers, molecular masses, temperature, and volume. Although the application of Fisher information to molecular systems is clearly in its infancy, it is hoped that the present work will catalyze further investigation into a new and truly unique line of thought on thermodynamics. 1. Introduction The second law of thermodynamics is based on entropy. The second law states that the total entropy of an isolated system increases over time, approaching a maximum value as time goes to infinity. Entropy in the truest classical sense applies to systems in equilibrium as defined by Clausius, Caratheodory, and others for the case of heat engines with the expression δQ (1) T where dS is the change in entropy caused by the flow of heat δQ into the system at temperature T. From the microscopic statistical thermodynamics point of view, entropy was defined by Boltzmann as dS )
S ) κ ln Ω (2) which can be shown to be equal to Shannon information, which is given by m
S ≡ -k
∑ P ln P i
i
(3)
i)1
where Pi is the probability of finding the system in a particular microstate i and k is a constant. From the information theory point of view, entropy is considered as a measure of disorder in a system. Hence, the entropy of a thermodynamic system increases if the number of microstates available to the system increases. In information theory, there is another type of information, called Fisher information (I). The Fisher information is of the form m
I)
∑ i)1
[ ]
1 dpi pi di
2
(4)
where pi is the probability density for finding the system in microstate i. This will be further discussed later. For a thermodynamic system, this quantity decreases if the number * To whom correspondence should be addressed. Tel.: 513-569-7350. Fax: 513-487-2511. E-mail address:
[email protected].
of microstates available to the system increases. For example, if the system exists in only one microstate, the slope of the probability density function is extremely high around that state, and so is the corresponding Fisher information. However, if the system exists in many states, each approximately equally probable, the probability density function becomes flat; its slope is near zero, and so is the Fisher information. Hence, Fisher information is a measure of order in the system, because order can be defined as a system that exists very predictably in one or a few microstates that have high probability. From eqs 3 and 4, we can intuitively say that Shannon entropy and Fisher information are, in some ways, analogous. However, these two quantities do not measure the same properties of the system, because the Shannon entropy is a function of the probability density function, whereas the Fisher information is dependent primarily on the slope of the probability density function. Therefore, the Shannon information is a global property whereas Fisher information is sensitive to local information in the probability density function. Thus, Fisher information provides different details, in comparison to Shannon information. In this paper, we propose Fisher information as a new thermodynamic property. We discuss entropy and the second and third law of thermodynamics. We show that Fisher information follows the second and third law, and we discuss dynamic order, which is a property captured by Fisher information. We further derive expressions of Fisher information for an ideal gas, an ideal gas mixture, and the mixing of two ideal gases. 2. The Second and Third Laws of Thermodynamics The second law of thermodynamics is possibly one of the most broadly applicable and most fundamental principles in science, and there are many interesting and very complete expositions of the topic that are widely available.1–4 Here, we proceed by revisiting several well-known results. First, for any process that occurs in an isolated system, i.e., a system that has no exchange of mass or energy with the surroundings, the entropy (S) will always approach a maximum (Smax) with time (t f ∞ as S f Smax), where Smax(t ) ∞) g S(t < ∞). The entropy
10.1021/ie7017756 CCC: $40.75 2008 American Chemical Society Published on Web 07/09/2008
5244 Ind. Eng. Chem. Res., Vol. 47, No. 15, 2008
is, thus, an increasing function of time (∂S/dt g 0). To the extent that the universe can be considered an isolated system, the second law then states that the entropy of the universe (Su) must also approach a maximum with time, and, therefore, the entropy of the universe is an increasing function of time (∂Su/dt g 0) as well. Dividing the universe into a system and the surroundings (Su ) Ssystem + Ssurr), as usually done in thermodynamic analysis, gives the well-known result dSu dSsyst dSsurr ) + g0 dt dt dt
(5)
where Ssyst is the entropy of the system and Ssurr is the entropy of the surroundings. The equality holds for quasi-static or reversible changes in the entropy that occur infinitesimally far away from thermodynamic equilibrium. Equation 5 is a general mathematical expression of the second law of thermodynamics. 2.1. Boltzmann Entropy. Ludwig E. Boltzmann5 is credited with the expression relating the entropy of a macroscopic system consisting of a large number (n) of microscopic identical particles to the state of the particles. The number n is of the order of Avogadro’s number, and there are m states available to the system. This famous expression, inscribed on Boltzmann’s tomb, is often given in the following form, S ) κ ln Ω
(6)
where S is the entropy, κ is Boltzmann’s constant (κ ≈ 1.3806 × 10-23 J/K, and Ω is the thermodynamic probability, which is defined by Ω≡
n! (n1 ! )(n2 ! )...(nm ! )
(7)
where n is the total number of particles, ni the number of particles in microscopic state i of the system, n! the factorial of the total number of particles, ni! the factorial of the number of particles in state i, and n ) n1 + n2 +... + nm. The symbol Ω denotes the number of possible unobservable microstates, consistent with the observable macrostate of the system. Note that Ω g 1 is the number of ways that n identical particles can be distributed among m different states of the system. Hence, Ω is not a statistical probability, because statistical probabilities must be 100, it can be numerically shown that, to a good approximation, x! ≈ x ln(x) - x. Using this latter result, the thermodynamic probability Ω can be expressed as
∑ (n ln(n ) - n ) ) -∑ n ln( n ) m
ln Ω ≈ n ln(n) - n -
m
i
i
ni
ni
i
i)1
i)1
(10) where, by definition, the statistical probability of finding a particle in a microstate i is given by ni/n ≡ Pi. The Boltzmann entropy then can be expressed as m
S ≡ -κ
∑ P ln(P ) i
(11)
i
i)1
which is equivalent to the Shannon expression for entropy if we let K ) κ. Recall that κ is Boltzmann’s constant. Hence, the Boltzmann and Shannon expressions for entropy are equivalent, at least within the limits of Stirling’s approximation. 2.3. Fisher Information. Another completely different form of information was derived by the statistician Ronald Fisher,9 in the context of obtaining a value to a parameter θ from observations or measurement of a related variable sˆ. For one variable, the Fisher information (I) is defined by I(θ) ≡
∫ p(sˆ1|θ) [ ∂p(s∂θˆ |θ) ]
2
ds
(12)
where p(sˆ|θ) is the probability density for observing a particular value of sˆ in the presence of the unknown parameter θ. The probability density p(sˆ|θ) is related to the statistical probability P(sˆ|θ) by the elementary relation p(s^|θ) ≡
P(s^|θ)
∫ P(s^|θ) ds
(13)
However, for the systems that will be treated here, the two are numerically the same, because the integrant above is equal to 1. However, the probability density p has units of 1/s, whereas the probability P has no units. Following the arguments of Frieden,10 Mayer et al.,11 and Plastino and Plastino12 note that, if the variable sˆ contains no information about θ (i.e., they are not related), then the derivative ∂p(sˆ|θ)/∂θ) ) 0 and the Fisher information is I ) 0. This means that the distribution of the observed values of sˆ has no relationship to the value of θ, and no information about θ can be obtained from observing sˆ. However, if ∂p(sˆ|θ)/∂θ) f ∞, much information about θ can be obtained from observing sˆ. In summary, the Fisher information is a measure of the amount of information about an unknown that can be obtained from a set of observations of reality. More philosophically, the Fisher information is the amount of information that can be transferred from reality to the observer by studying a particular set of observations. The definition of the Fisher information is often difficult to apply in practice, because there is no straightforward way of evaluating the derivative ∂p(sˆ|θ)/∂θ), with respect to an unknown parameter θ. However, for systems that obey shift invariance,
Ind. Eng. Chem. Res., Vol. 47, No. 15, 2008 5245 10,13
as treated elsewhere, simplified to I)
the aforementioned expression can be
1 dp(s) ∫ p(s) [ ds ]
2
∫ [ dq(s) ds ]
2
ds ) 4
ds
(14)
where the parameter θ conveniently disappears, and where p(s) is the simple probability density for observing a particular value of s, and ∂p(s)/∂s is the slope of the probability density curve. The expression after the second equality is obtained by substituting the probability density p(s) with the amplitude of the probability density q(s), where the two are related by p(s) ) q2(s). Hereafter, we will use the Fisher information expression in terms of q(s), because it avoids the problems associated with division by zero when p(s) becomes very small. Shift-invariant systems are those where the parameter θ is independent of the fluctuations in the data s. For illustration, consider a long time series for a cyclical system spanning many cycles, showing the variable s as a function of time. Presume that we are attempting to determine the value of the average 〈s〉 (i.e., θ ) 〈s〉) of the variable s from the time series. The value of 〈s〉 then is independent of the particular bracket of time used to estimate it, provided the bracket spans more than one cycle. Hence, the assumption of shift invariance holds for systems near steady dynamic regimes but not near a regime shift, roughly the dynamic analog of a phase transition. The discrete version of the aforementioned continuous expression is m
I≈4
∑s -s i
i)1
i+1
[
qi - qi+1 si - si+1
]
2
m
)4
∑ [q - q i
2 i+1]
(15)
i)1
where si is state i of the system, and pi is the probability density for the system being in state i. If we further set si ) i, to conform to the notation for states previously used for the Boltzmann and Shannon entropies, the aforementioned expression simplifies to that shown after the second equal sign. If the likelihood of the system being in any of the m states is the same, then qi ) qi+1 and the Fisher information is zero (I ) 0). If the likelihood of the system being in any of the m states is zero, except for a particular state k, then qk ) 1 and qi*k ) 0 and the Fisher information I ) 4 is a maximum. 2.4. Boltzmann Entropy, Shannon Entropy, and Fisher Information. The second law (see eq 5) states that the entropy of an isolated system is an increasing function of time (∂S/dt g 0), and that it eventually reaches a maximum value (as t f ∞, S f Smax). From the Boltzmann expression for entropy (eqs 6 and 7), for a fixed number of particles n, the entropy can increase only if Ω is increasing. This is only possible if the particles are occupying, with time, increasingly more of their available states m, so that the number of particles in each state ni is increasingly smaller, and the product n1!n2!...nm! also becomes increasingly smaller. Note that, at the hypothetical extreme, where every state has one particle, ni ) 1, the product n1!n2!...nm! ) 1, Ω ) n!, and S is a maximum (S ) κ ln n!). However, as already discussed, S f 0 when nk f n for any arbitrary state k and where ni*k ) 0. Hence, for the Boltzmann entropy to follow the second law, the particles must be progressively distributed over more and more of the available states as time passes. If the particles are being progressively distributed with time over many states, as previously discussed, then the number of particles ni in each state also is decreasing for a constant number of total particles n. The Shannon entropy (eqs 10 and 11), where S ) κ (n ln(n) - ∑ ln(ni)) then will increase as the value of ni decreases, reaching a maximum as ni ) 1 and S ) κ n ln(n). On the other hand, if nk for any particular state k increases so
that nk ) n and ni*k ) 0, then S ) 0, as already discussed. In summary, the Shannon entropy follows the second law of thermodynamics, as expected, because it is equivalent to the Boltzmann entropy. Fisher information is not a form of entropy, but it does have similar and very interesting properties. To explore the issue, we expand eq 15 to I ) (4/n)∑(ni - 2ni1/2ni+11/2 + ni+1), using the definitions of pi and qi, where qi ) (ni/n)1/2. In the extreme case where every state has only one particle (ni ) 1), S ) κ ln(n!) is a maximum, but the Fisher information is a minimum at I ) 0. At the other extreme case, where all of the particles are in one state, nk and nk ) n and ni*k ) 0 for any particular state k, S ) κ ln(n!/n!) ) 0 is a minimum, but the Fisher information is at a maximum. Hence, the Fisher information follows the second law of thermodynamics, but it does so in a direction opposite to that of the entropy. Between these two extremes, the entropy is, according to the second law of thermodynamics, increasing with time between the minimum (S ) 0) to the maximum (S ) κ n ln(n)), and the Fisher information is simultaneously decreasing from a maximum (I ) 4) to a minimum (I ) 0). However, the Fisher information does not simply decrease reciprocally to the entropy. To further understand the behavior of the Fisher information, we must consider in detail the process by which the entropy increases over a period of time t0 e t e tf from zero to a maximum. From eq 15, at t0, all of the particles are in some state k, nk(t0) ) n, and I(t0) ) (4/n) (n1/2 - 0)2 ) 4, as already shown. At the next step in time (t1 > t0), if the entropy has increased, then the particles must be distributed into more than one state. For the purpose of illustration, we will consider that, at t1, the particles are distributed between two arbitrary states i and i + 1. Then, nl+1 (t1) ) n - ni (t1), because the particles are conserved. Using the expansion of eq 15, the Fisher information at t1 is I(t1) ) (4/n) (ni - 2ni1/2ni+11/2 + n - ni) ) 4 - (8/n)(ni1/2(n - ni)1/2). Because (8/n)[ni1/2(n - ni)1/2] > 0 is a positive number, then I(t0) > I(t1) and the Fisher information decreases as the entropy increases with time, according to the second law of thermodynamics. At the other extreme is a situation where all n states of the system are occupied ni ≈ ni+1 ≈... nn and I ) (4/n)∑(ni1/2 - ni+11/2)2 ≈ 0, because all the elements in the sum are almost zero. Between these two extremes, as the systems progresses from occupying two states to occupying three states or from m to n states, where m < n, the entropy will still steadily increase, as shown. However, the Fisher information can, in principle, locally increase or decrease within an overall decreasing trend for the information, depending on the actual number of particles occupying the available states. This is due to the fact that the Fisher information is dependent not on the occupancy of the states of the system ni, but on the differences between the occupancies. Although this may be true for systems far from equilibrium such as biological systems, for thermal systems near equilibrium, one would expect that, as particles progressively occupy more and more states, the differences between the occupancies will smoothly decrease. This leads to smoothly decreasing Fisher information with time, according to the second law of thermodynamics. In summary, the Fisher information is not entropy, but rather a measure of dynamic order that is more sensitive to structure than entropy. 2.5. Dynamic Order. Dynamic order roughly refers to a characteristic of dynamic systems in which repeated measurement or observation over time yields the same results or patterns. Hence, an orderly system is one that is well-defined and predictable over time.11 For a system consisting of Avogadro’s number of molecules or particles, a perfectly orderly system is
5246 Ind. Eng. Chem. Res., Vol. 47, No. 15, 2008
one where all of the particles exist in one state over time, and the probability of finding any given particle in that state is 1. Such is the case for a perfect crystal lattice at 0 K. The system has perfect order, because the state of each and every particle is well-defined and well-known. As already discussed from a different perspective, for such a perfectly orderly system, the entropy is zero and the Fisher information is a maximum. By comparison, a system of many particles is perfectly disordered when the particles can exist over a great many states with equal probability. In this case, the probability of finding a particle in any particular state is very small, and the state of the particles is neither well-defined nor well-known. A gas at high temperature approaches the dynamic regime of a system that is in perfect disorder. As previously explored, for a perfectly disorderly system, the entropy is a maximum and the Fisher information is I ) 0 or at least a minimum. Systems of many particles under ordinary room conditions exist between these two extremes under conditions where the Fisher information and the entropy are finite numbers, neither 0 nor ∞. Furthermore, dynamic order is a fundamental property of dynamic systems that changes when the dynamic regime of the system changes. As an additional illustration, consider at the macroscopic scale the case of fluid flow. Fluid flowing in a laminar regime has different dynamic order and Fisher information than fluid flowing in a turbulent regime, because the patterns associated with the two flows are different. When fluid flow shifts from laminar to turbulent or vice versa, dynamic order and Fisher information change as well, and the transition between the two regimes is manifested as a sharp loss of dynamic order and Fisher information. The reason is that laminar flow has a simple pattern and turbulent flow also has a wellestablished pattern, albeit a more complicated one. However, the transition between laminar and turbulent flow has no wellestablished patterns and, hence, lower dynamic order than either laminar or turbulent flow. At the macroscopic or molecular level, changes in the dynamic regime of the particles are manifested as changes in dynamic order and Fisher information. For example, a change in phase for a fluid, for example, from gas to liquid, is a change in dynamic regime, which is manifested as change in dynamic order and Fisher information. Here, again, occupancy of the macroscopic states in a gas is different from the occupancy in a liquid (e.g., a gas has access to higher energy states than a liquid). Gases and liquids have well-established patterns, albeit different ones, to the occupancy of microscopic states that are temporarily lost during the phase transition. Hence, a phase transition would be manifested as a change in dynamic order and Fisher information, with a sharp decrease in both between the phases. In the next sections, we briefly explore some practical applications of the theory. 3. Fisher Information as a Physical Property The objective of this section is the development of Fisher information as a new thermodynamic property. The idea is to use a statistical thermodynamic molecular approach to calculate Fisher information as a physical property. We start by deriving an expression for the property, Fisher information, in the canonical ensemble. Next, we develop Fisher information expressions for a monatomic pure component ideal gas, a monatomic ideal gas mixture, and the isochoric mixing of ideal monatomic gases. 3.1. Fisher Information from the Canonical Partition Function. To examine the relation between Fisher information and the canonical partition function, consider a system of constant number of moles N, volume V, and temperature T that
is allowed to exchange energy with a constant-temperature bath. With this concept, we construct a canonical ensemble of many identical systems, where each member of the ensemble is at the same N, V, and T. The system may be in any of the quantum states of differing energies.14 For such an ensemble, the probability of finding the system in a quantum state “i” is related to the canonical partition function by eq 16.14 This expression is called the distribution law: pi(Ei) )
e-βEi Q
(16)
where Q ) ∑i e-βEi and Ei is the energy of the quantum state “i” and β ) 1/(kT), where k is Boltzmann’s constant and T is the absolute temperature (in Kelvin). Assuming that the quantum states are very closely spaced, one can invoke a continuous approximation to replace Ei with a continuous function E(s) and the sum over states in eq 16 with an integral to give e-βE(s) Q
p(s) )
(17)
where Q = ∫0∞e-βE(s) ds and s is the state of the system, and where s is continuous but covers the same span of values as i. Thus, Fisher information of a thermodynamic system as expressed by eq 14 can be related to the canonical partition function (Q) through eq 18:
∫
[
qIG t )
(
4 ∞ d -E(s)⁄(2kT) 2 e ds (18) Q 0 ds 3.2. Fisher Information of Pure Ideal Gas. The simplest system for which the aforementioned concepts can be applied is that of a monatomic, pure-component ideal gas, composed of non-interacting and indistinguishable particles that are free to move around. The translational molecular partition function for an ideal gas is I)
]
2πmkT h2
)
3⁄2
V
(19)
where m is the mass of particles, h Planck’s constant, and V the volume. The molecular partition function (qt) is related to the canonical partition function14 (Q) through eq 20:
(qtIG)N
Q ) IG
N!
)
(
V N 2πmkT N! h2
)
3N⁄ 2
(20)
where N is the number of molecules. The translational partition function for an ideal gas15 is also given by ∞
qIG t )
∑e
-βEi
∞
∑e
)
i
-3h2i2⁄(8mkTV2⁄3)
(21a)
i
or, again, invoking a continuous approximation, qIG t ≈
∫
∞ -βE(s)
0
e
ds)
∫
∞ -3h2s2⁄(8mkTV2⁄3)
0
e
ds
(21b)
where the sum or integral is taken over all the energy states. Therefore, for the case of a monatomic, pure-component ideal gas, the energy levels are given by EIG i )
3h2i2 8mV2⁄3
EIG(s) )
3h2s2 8mV2⁄3
(22a) (22b)
where it has been tacitly assumed that the contribution from electronic and nuclear energies is negligible. To compute an
Ind. Eng. Chem. Res., Vol. 47, No. 15, 2008 5247
(
expression for the Fisher information, we must evaluate the following derivative: d -EIG(s)⁄(2kT) d -3h2s2⁄(16mV2⁄3kT) e ) e (23) ds ds We proceed by inserting the solution of eq 23 into eq 18, and this result (eq 24) is integrated, giving an expression for the Fisher information (eq 25). 9h4 4 I ) 2 Q 64m V 4 ⁄ 3R2T2 IG
I ) IG
( )[ 1⁄2
33
23N+7
∫
∞ 2 -3h2s2⁄(8mV4 ⁄ 3kT) se 0
]
N! (3N-1)⁄2
π
ds
(24)
Q
)
(25)
N1 IG N2 (qIG 1 ) (q2 ) ...
n
∏ j)1
qIG j ) Nj!
n
∏ j)1
qIG tj ) Nj!
∞
QIGM )
∑ exp i
[
-
n
∏ j)1
(
2πmjkT 2
h
)
3Nj/2
V Nj Nj!
(27)
IGM (i) ij
j)1
] [ ] ∞
≈
kT
∫ exp 0
-
∑E
IG ( ) j s
j)1
kT
where
and, therefore,
[ ]
EIGM i ) exp 2kT
3h2i2 8mjV2 ⁄ 3
n
∏ j)1
[
3h2i2 exp 16mjV2 ⁄ 3kT
]
where the approximate equality results from a continuous approximation, and ( ) EIG j s )
therefore,
3h2s2 16mjV2 ⁄ 3kT
)
(29)
j
2 2
3h s 8mjV2 ⁄ 3
We obtain the desired result by inserting the solution to the aforementioned derivative in eq 18, thereby arriving at a generic expression for the Fisher information for an n-component ideal gas mixture (eq 30): n
IIGM ) 4
∏N ! j
j)1
(
h2 2πmjkT
[
)
3Nj⁄2
n
V-Nj
∫
∞
×
0
n
]
s ∑ 8m-3h ∏e V kT 2
2⁄3
j)1
2
-3h2s2⁄(16mkV2⁄3kT)
k)1
j
ds
(30)
Note that, to derive an integrated expression, one must specify n and actually expand the squared expression in the integrand. However, for the special case of a two-component (components 1 and 2) ideal gas system, the aforementioned expression reduces to
[ (
IIGM 12 ) 4 N1!
h2 2πm1kT
∫
∞
{
)
-
3N1⁄2
][ (
V-N1 N2!
h2 2πm2kT
)
3N2⁄2
]
V-N2 ×
Rs -Rs2⁄(2m1) -Rs2⁄(2m2) [e e ]+ m1 -Rs -Rs2⁄(2m1) -Rs2⁄(2m2) [e e ] m2
}
2
ds
(31)
where R ) 3h2/(8V2/3kT). Next, we proceed to develop a practical expression for the special case of a two-component monatomic ideal gas mixture by explicitly evaluating the integral in the Fisher information expression (eq 31). The integral in the aforementioned expression (denoted hereafter as “Int”) is expanded as follows:
∫
∞
0
[
R2s2 -R[(1⁄m1)+(1⁄m2)]s2 2R2s2 -R[(1⁄m1)+(1⁄m2)]s2 e + e + m1m2 m12
]
R2s2 -R[(1⁄m1)+(1⁄m2)]s2 e ds m22
ds (28)
EIG ij )
j)1
∑∏
Int )
n
∑E
∏
exp
IG IG n n IG ∂eEj (s)⁄(2kT) dEj (s) d -EIGM(s)⁄(2kT) e ) eEk (s)⁄(2kT) ds ds ∂EIG(s) j)1 k)1
0
For this n-component system, the canonical partition function (eq 17) takes the following form: n
(
n
as expected. We further proceed by evaluating the derivative in the Fisher information integral, which, for this case, under the continuous approximation, reduces to eq 29:
(26)
N1!N2!...
The development generally follows that, for a monatomic, purecomponent ideal gas, except that the canonical partition function for this system can be expressed in terms of the molecular partition function for each of the n components, as shown below: QIGM )
)
EIGM(s) ) 2kT
k*j
h3(N+1) VN+1(kTm)3(N+1)⁄2
Thus, we have developed an expression of the Fisher information for a monatomic pure-component ideal gas from the statistical mechanics perspective and the canonical partition function. 3.3. Fisher Information of an Ideal Gas Mixture. To develop an expression for Fisher information of a monatomic ideal gas mixture, consider an n-component mixture at constant T and V having N1, N2,..., Nn molecules of components 1, 2,..., n, respectively. As stated by Reed and Gubbins,14 such an ideal gas mixture will have molecules of different species that are distinguishable, and molecules of the same species which would be indistinguishable. Each separate species will have its own set of energy levels, which are a function of volume and are unaffected by the presence of other components.14 The partition function can then be written as IGM
exp -
(32)
Noting that eq 32 can be expressed as three integrals of the 2 form ∫as2 e(-bs ) ds, where a and b are constants, we proceed to use the first integral to illustrate the evaluation procedure. Hence, we first perform the following transformation. Let X ) s2, so that dX ) 2s ds and ds ) dX/(2s) ) 1/(2X1/2) dX, where the first integral now becomes
∫
∞
0
R2s2 -R[(1⁄m1)+(1⁄m2)]s2 e ds ) m12
∫
∞
0
aXe-bX
∫
∞
0
as2e-bs ds ) 2
1 a dX ) 1⁄2 2 2X
∫
∞
0
X1⁄2e-bX dX
(33)
with a ) R2/m12 and b ) R[(1/m1) + (1/m2)]. To develop a closed form of the first integral, another transformation is needed; this transformation is described as follows. Let Y ) X1/2, so that dY ) 1/(2X1/2) dX and dX ) 2/(X1/2) dY ) 2/Y dY, where the integral now becomes
5248 Ind. Eng. Chem. Res., Vol. 47, No. 15, 2008
a 2
∫
∞
0
X1⁄2e-bX dX )
a 2
∫
∞
0
Ye-bY
2
2 dY ) a Y
∫
∞ -bY2 e 0
dY )
a 2
π b (34)
Using the two transformations (eqs 33 and 34), eq 32 then is integrated to obtain the following expression: Int )
(
1 3h2 2 8V2⁄3kT
) 3⁄2
(
π
1
(1 ⁄ m1) + (1 ⁄ m2) m12
+
2 1 + m1m2 m 2 2
)
(35)
Inserting eq 35 in eq 31 and conducting further algebraic simplification gives us a practical expression for computing the Fisher information of a two-component monatomic ideal gas mixture: IIGM 12 )
(
33
)( 1⁄2
23N1+3N2+7
)
h3(N1+N2+1) × π(3N1+3N2-1)⁄2 VN1+N2+1(kT)3(N1+N2+1)⁄2 1⁄2 1 1 1 1 3⁄2 + (36) 3N1 3N2 m m m m 1 2 N1!N2!
[( )( )] ( 1
)
2
Note that, in the pure-component limit of, for example, pure component 1, where N2 ) 0 and m2 is simply deleted, the aforementioned expression simplifies to the following expression: IIG 1 )
( )( 33
N1!
1⁄2
3N1+7
2
(3N1-1)⁄2
π
)
h3(N1+1) VN1+1(m1kT)3(N1+1)⁄2
(37)
which is exactly the same expression as that derived for the case of a pure-component, monatomic, ideal gas (eq 25). 3.4. Isochoric Mixing of Ideal Gases. In this section, we develop Fisher information expressions for the isochoric mixing IG IGM of ideal gases (∆Imix ≡I12 - I1IG - I2IG) 1 and 2. The mixing process can be described as follows: visualize the classic thought experiment of a container divided into two compartments. In one compartment, we have N1 moles of ideal gas 1, and in the other compartment, we have N2 moles of ideal gas 2. Now, if we remove the partition, the gases will begin to diffuse into each other and the system will eventually reach the state where both gases are uniformly distributed throughout the container. For reasons of simplicity of illustration, we consider the special case whereN1 ) N2, N ) N1 + N2, andV1 ) V2. The purecomponent Fisher information of the individual gases (I1IG and I2IG) before mixing occurs are given by eq 25. The Fisher IGM information of the ideal-gas mixture (I12 ) after the mixing process is completed is given by eq 36. To explore further, we derive the ratio between the Fisher information of the two pure components and the mixture, which reduces to the following form: IG IIG 1 + I2
IIGM 12
)
( π2 ) ( N 2e ) ( V2πkTh ) 1⁄2
N1
(1 ⁄ 2)N1
2⁄3 2
(
3N1⁄2
1 1 + m1 m2
) (38)
Note that, in the limit of T f ∞ at constant N1 and V, the ratio IG IGM (IIG 1 + I2 )/I12 f ∞, whereas at the limits of T f 0 at constant IGM N1 and V, we also observe the ratio (I1IG + I2IG)/I12 f 0. Similarly, in the limits of V f ∞ at constant N1 and T, the ratio IGM (I1IG + I2IG)/I12 f 0, whereas at the limits of V f 0 at constant IG IGM N1 and T, the ratio (IIG 1 + I2 )/I12 f ∞. Detailed consideration of the significance of these ratios merits further investigation and probably another paper beyond the present introductory presentation. Furthermore, we develop an expression for the Fisher information of mixing (∆Imix) as follows:
IGM IG IG ∆IIG mix ≡ I12 - I1 - I2
(39)
IG Again, for the special case of N1 ) N2 and V1 ) V2, ∆Imix takes the form
( )[ ] ( )( ) ( ) [ ]( ) ( ) [ ]( ) (N1!)2
h3(2N1+1) × 2 π V (kT)3(2N1+1)⁄2 1 1 1 1 3⁄2 + 3N + 3N1 m1 m2 1 m1 m2 N1! 33 1⁄2 h3(N1+1) 23N1+1 π(3N1-1)⁄2 1 V N1+1(m kT)3(N1+1)⁄2 1 2 3 1⁄2 N ! 1 h3(N1+1) 3 (40) 23N1+1 π(3N1-1)⁄2 1 V N1+1(m kT)3(N1+1)⁄2 2 2 For comparison, note that the entropy of mixing is generally given by ∆IIG mix )
33
6N1+7
1⁄2
(6N1-1)⁄2
2N1+1
[ ( )
∆Smix ) -k
( )]
N2 N1 N1 N2 ln ln + N N N N
(41)
and, for the special case of N1 ) N2 andV1 ) V2, the entropy of mixing expression reduces to
[ 21 ln( 21 ) + 21 ln( 21 )] ) k ln 2
∆Smix ) -k
(42)
By comparing eqs 40 and 42, we can see that the entropy of mixing is constant while the Fisher information of mixing is not. It is very interesting that the Fisher information is a function of temperature(T), volume (V), and the molecular masses of the particles(m1 and m2). Hence, the Fisher information gain or loss is dependent on the conditions under which the mixing occurred and on the masses of the components being mixed. This again merits more-detailed consideration than can be given in an introductory presentation of a new subject. However, it does show that the Fisher information of mixing (and Fisher information in general) is qualitatively different than the entropy of mixing (and entropy itself) and provides different types of knowledge on the mixing process and more details of its structure. 4. Summary The aforementioned applications of Fisher information to molecular systems open new and very interesting avenues for investigations in thermodynamics. However, the chief purpose of the present work is to open such avenues and to stimulate further research, rather than provide definitive answers. Therefore, we will now proceed to explore and speculate on the significance of the developments, leaving more-definitive results to future papers. Hence, the principal new results presented here are embodied in (i) section 2.4, where it is shown that Fisher information follows the second and third laws of thermodynamics for molecular systems; (ii) eq 18, which makes it possible to compute Fisher information from the canonical partition function; (iii) eq 25, which expresses Fisher information in terms of molecular parameters and thermodynamic variables for a pure-component, monatomic, ideal gas; (iv) eq 36, which gives the Fisher information of a binary mixture of monatomic ideal gases; and (v) eq 40, which shows the Fisher information change that is associated with mixing two monatomic ideal gases. These results, assembled together, are significant for the following reasons. Given that the Fisher information follows the second and third laws of thermodynamics, then any analysis or
Ind. Eng. Chem. Res., Vol. 47, No. 15, 2008 5249
any criteria that has traditionally been done or established using entropy should be equally feasible, based on Fisher information, without loss of rigor. However, because Fisher information captures far more detail than entropy (compare eqs 40 and 42), more information about the system can potentially be learned from a Fisher information calculation than is the case for an entropy calculation. To further illustrate this point, we consider the simple example of isochorically mixing two different types of monatomic ideal gases (e.g., type 1 and type 2). The entropy of mixing calculation is based on the probability of observing a particle of type 1 or type 2 when sampling randomly, regardless of the mass or the size of the particles; therefore, the same increase in entropy will appear, regardless of the size or mass of the particles. However, the Fisher information of mixing calculation will show a loss of information, as required by the second third law of thermodynamics. But it is explicitly dependent on the mass of the particles, so that the information loss from the mixing of large and small particles will not be the same. This is simply a reflection that the mixing of very large particles and the mixing of very small particles are not quite the same, because very large particles are easier to count and locate than small particles (i.e., the information changes inherent in the mixing of large and small particles are not quite the same). Furthermore, consider that the Fisher information of mixing under the aforementioned conditions asymptotically approaches zero with increasing temperature, whereas the entropy of mixing is again constant and independent of temperature. Now, Fisher information is a measure of order or patterns in dynamic systems, and it decreases with increasing temperature, because order generally decreases as the temperature increases. There are also more subtle functional dependences, such as that with the number of moles and volume, which will be explored in the context of a future paper. In summary, we have presented Fisher information that is applied to molecular systems as a new physical property that follows the second and third laws of thermodynamics. Therefore, Fisher information can be used in many calculations where entropy is traditionally used, and where there is a need for moredetailed knowledge of the process.
Acknowledgment A.T.K. is a Postdoctoral Research Associate with the National Research Council in residence at the USEPA. H.C. acknowledges the life-long encouragement of Professor J. P. O’Connell, a man of wisdom and knowledge. Literature Cited (1) Planck, M. Treatise on Thermodynamics; Dover: New York, 1945. (2) Kestin, J. A Course in Thermodynamics, Vol. 2; Hemisphere Publishing: Washington, DC, 1979. (3) Astarita, G., Thermodynamics: An AdVanced Textbook for Chemical Engineers, Plenum Press: New York, 1989. (4) O’Connell, J. P.; Haile, J.M. Thermodynamics: Fundamentals for Applications; Cambridge University Press: Cambridge, U.K., 2005. (5) Boltzmann, L. E. Vorlesungen u¨ber Gastheorie; Barth: Leipzig, Germany, 1912. (6) Aston, G. A.; Fritz, J. J. Thermodynamics and Statistical Thermodynamics; Wiley: New York, 1959. (7) Shannon, C. E.; Weaver, W. A Mathematical Theory of Communication; University of Illinois Press: Urbana, IL, 1949. (8) Courant, R. Differential and Integral Calculus, Vol. 1, 2nd Edition; Blackie and Son: London, 1958; pp 361-364. (9) Fisher, R. A. On the mathematical foundations of theoretical statistics. Philos. Trans. R. Soc. London 1922, 222, 309–368. (10) Frieden, B. R. Science from Fisher Information: A Unification; Cambridge University Press: Cambridge, U.K., 2004. (11) Mayer, A. L.; Pawlowski, C. W.; Fath, B. D.; Cabezas, H. Applications of Fisher Information to the Management of EnVironmental Systems. Exploratory Data Analysis Using Fisher Information; Frieden, B. R., Gatenby, R. A., Eds.; Springer-Verlag: New York, 2007; pp 217244. (12) Plastino, A.; Plastino, A. R. Information and Thermal Physics. Exploratory Data Analysis Using Fisher Information, Eds: Frieden, B. R., Gatenby, R. A., Eds.; Springer-Verlag: New York, 2007; pp 119-154. (13) Fath, B. D.; Cabezas, H.; Pawlowski, C. W. Regime Changes in Ecological Systems: An Information Theory Approach. J. Theor. Biol. 2003, 222 (4), 517–530. (14) Reed, T. M.; Gubbins, K. E. Applied Statistical Mechanics; McGraw-Hill: New York, 1973. (15) McQuarrie, D. A. Statistical Thermodynamics; Harper & Row: New York, 1973.
ReceiVed for reView December 27, 2007 ReVised manuscript receiVed June 9, 2008 Accepted June 13, 2008 IE7017756
