COM MUN ICAT1ON
CONVERGENCE OF A M E T H O D OF SUCCESSIVE APPROXIMATIONS IN T H E THEORY OF OPTIMAL PROCESSES The convergence of a method of successive approximations used recently for the optimization of discrete and confinuous systems is examined. Convergence is assured within some neighborhood of the maximum for nonsingular continuous systems, but attempts to use the algorithm for discrete systems could lead to divergence from starting policies arbitrarily close to the maximum.
HE determination of optimal conditions in systems of comTplicated topological structure can generally be reduced to the solution of a boundary value problem in differential and difference equations with a subsidiary algebraic condition to be satisfied a t each point-e.g., (2, 5, 7, 8, 70, 72, 75). I n particular, consider the case of a simple process without recycle or bypass whose state p is defined over a n interval 0 by the differential equations.
where q is a decision vector function belonging to a closed set and must be chosen at each point to maximize a profit function of the final state,
(3)
p = P[P(B)I
T h e Green’s vector, a, for the linearized system is defined by the equations (The summation convention is used throughout, in which summation is carried out over all values of a n index appearing once in the upper and once in the lower position in an expression.) AT i
& = - -u 1
bp3 at
(4) (5)
and the weak maximum principle (2, 3, 75) requires that the optimal decision vector q* satisfy the algebraic condition
and the weak maximum principle (2, 5, 8) requires that the optimal qn* satisfy ,m
a,
I
0 2 1 ” =
0, qn*f interior
bqn’ tJn,Tni = maximum, qn*3 a t a boundary
(1 2)
Numerous techniques have been proposed to solve the twopoint boundary value problem which results from the specification of the initial value of the state, PO, but the final values of the Green’s vector, a, or a@)-e.g., ( 7 , 2, 5-7, 77, 73). Katz (72) and Kelley (73) have suggested a method as follows. Assume a decision Solve Equation 1 or 8 in the forward direction and then Equation 4 or 9 in the reverse direction. With the values and 5 so calculated use Equations 6 and 7 or 11 and 12 as transcendental equations defining a new value, 4. Set the new decision a t
G.
q =q
+ X ( q - q)
where X is a relaxation parameter, and repeat until convergence is obtained. This method has been used by Lee (74) for continuous systems and Fan and Wang (8) for discrete systems. I t is of interest to investigate the convergence of the successive approximations described above. We restrict our attention to interior regions of the decision space for nonsingular systems, where Equations 6 and 11 must be satisfied. (A singular system is one in which Equation 6 or 11 is independent of 9.) Dropping the stage number, n, the algorithm based on Equation 6 or 11 may be written at each point or stage as
if q*3 is an interior element of the allowable set and
u t T i = maximum with respect to 93
(7)
if g * j lies at a boundary of the region of admissibility. Similarly, for a simple system of sequential stages defined by the transformation equation pn = T n ( p n . - l , qn), n = 1,2, . . . N
b TnI --, a, 3Pn-
n
=
I
1, 2, . . . , N
4
i.
(8)
where qn is to be chosen to maximize P [ p N ] ,the Green’s vector satisfies Un-l,t =
Near the maximum we expect only small changes in the decision, and it is reasonable to expand Equation 14 about by Taylor’s theorem, retaining only first-order terms in {Thus, approximately,
or
(9)
where the elements of A D
Gkj
compose the inverse matrix of
@Ti elements of a, From Equation 13, the change in bqjbqk’ ~
VOL. 4
NO. 2 M A Y 1 9 6 5
231
decision, 6q, is then
It at rate may
’
has been fodnd previously (7. 2. 6, 7 ) that the direction in the space of decision vectors which gives the greatest of increase in P for both continuous and staged systems be written
where s is a distance parameter and rkJis a positive definite tensor depending on the metric of the decision space. For sufficiently small step sizes in the direction indicated by Equation 18 convergence can always be obtained in principle, although the amount of computing time might exceed practical limits. Conversely. the negative direction defines a half-space of locally decreasing profit and indicates divergence from the maximum. Thus, for sufficiently small X Equation 17 will converge if and only if Gk’ or, equivalentlv.
-
-
a, d2T*(P‘ -~ ’I.
is negative definite. dq’dqk For nonsingular continuous systems defined by Equation 1 the maximum principle of Pontryagin (75) requires that - d2Tz - be negative definite a t almost every interior point of O, dqJdqk the function q* which maximizes P,and this is true for systems with more general topdlogical structures as well (2. 4 ) . I t follows from the continuitv of the solution of a differential equation with respect to a function (2) and the continuity of d2TZ T with respect to its arguments that 5, __ must also be dq’dqk negative definite in some neighborhood of the maximizing function. which is consistent with the linearity assumption in Equation 15. Thus, the iterative algorithm will always converge for starting functions ?jsufficiently close to the maximizing function. For staged systems it has been shown that in general \ o m
1
0’1 n ”
cannot be expected to be negative definite at the bqn’dqnE maximizing set { qn* } , (2, 4, 9 ) . T h e exceptions are discussed in (2) and ( 4 ) ,the most important case being the linear straightchain system \vith linear profit function. Only in these exceptional cases is convergence assured, and examples can be constructed \shere successive approximations starting arbitrarily close to the maximum must diverge because of the positive definite character of Gk’ a t the maximum. Thus mnt
232
~
I&EC
FUNDAMENTALS
the algorithm should not be used, in general. for discrete systems. T h e success of Fan and Il’ang (8) in the use of the algorithm is a consequence of the linearity of their systems. and attempts to solve similar systems \+ith nonlinearities could diverge. Nomenclature
Gk,= inverse of B,
h2173
“
’\
~~
dq’bqk
~
p
=
p
= state vector corresponding to
P q,
2 q 1
t T 6q rkj
X
o -
a
state vector
q
profit decision vector optimal decision vector particular decision = decision computed from Equation 14 = parameter in decision space = timelike parameter = stage or differential transformation = change in q = positive definite tensor = relaxation parameter = Green’s vector = Green’s vector corresponding to = = = =
literature Cited
(1) Bryson. A. E., Denham, 1.V. F., Carroll, F. J.; Mikami, K.; J . Aerospac? Sci. 29, 420 (1962). (2) Denn, M. M., ”Optimization of Complex Systems.” Ph.D. thesis, University of Minnesota, 1964. (3) Denn, M. M.. Ark, R ( 4 ) Denn, M. M., Aris, R., Chern. Eng. Scz.. in press. 4, (5) Denn. M. M.. Ark, R., I N D .E N C . CIIEM.FUNDAMENTALS 7 (1965). (6) Zbid., p. 213. (7) Ibid..in press. (8) Fan, L. T.?\Yang. C. S.,“The Discrete Maximum Principle.” \Viley, New York, 1964. (9) Horn, F., Jackson, R., I K D . ENG.CHEM. FUNDAMENTALS 4,110 (1965). (10) Jackson. R., Chem. Eng. Sci. 19, 19 (1964). (11) Kalaba. R.. in “Nonlinear Differential Equations and Nonlinear Mechanics,” J . P. La Salle and S. Lefschetz, eds., .4cademic Press, New York, 1963. (12) Katz, s., I S D . ENC.C H E M . FUNDAMENTALS 1 , 226 (1962). (13) Kelley, H. J., in “Optimization Techniques with .4pplications to Aero-space System,” G. Leitmann. ed., Academic Press, New York, 1962. (14) Lee, E. S...4.Z.Ch.E. J . 10, 309 (1964). (15) Pontryagin. I,. S.,Boltyanskii, V. A . . Gamkrelidze. R. V.. Mischenko, E. F., “Mathematical Theory of Optional Processes.” 1Viley. New York. 1962.
MORTON M. D E N N Cniuersitj of DelaEare
.Yewark. Del. RECEIVED for November 2. 1964 ACCEPTED January 19, 1965
