Optimal Resource Extraction with Uncertain Reserves

Robert J. Lipshutz
Daniel H . Wagner Associates, Inc., Sunnyvale, California 94089

Lawrence D. Stone
Metron, Inc., Reston, Virginia 22090

We consider the problem of finding a plan that maximizes the expected discounted
return when extracting a nonrenewable resource having uncertain reserves. An
extraction plan specifies the rate at which the resource is extracted as a function
of time until the resource is exhausted or the time horizon is reached. The return
per unit of resource extracted may depend on the rate of extraction, time, and the
amount of resource previously extracted. We apply a new method called the gen-
eralized search optimization technique to find qualitative features of optimal plans
and to devise algorithms for the numerical calculation of optimal plans.

1. INTRODUCTION
In this article we apply the generalized search optimization (GSO) technique
described in [lo] to maximizing the discounted return when extracting a non-
renewable resource with uncertain reserves.
The GSO technique was developed from a method that has been successfully
applied to search problems, particularly those involving the detection of moving
targets. In [lo], Stromquist and Stone discuss numerous applications of the GSO
technique which are all related to search. The results in this article demonstrate
that the GSO technique can be applied to problems outside of search theory.
This application establishes a connection between GSO and control theory. We
discuss this connection at the end of this section.

1.1. Description of Results

The optimal resource extraction problem considered in this article is similar
to the one considered by Loury [8] and Liu [7]. One important difference is that
our results apply to utility or return functions that have finite derivative at 0,
whereas Loury and Liu require an infinite derivative at 0. Consequently, our
results apply to problems in which utility is measured in terms of money, where
marginal rates of return are finite. In contrast, the results of Loury and Liu
apply to problems involving the social utility of a critical resource. In addition,

Naval Research Logistics, Vol. 39,pp. 153-173 (1992)

Copyright 0 1992 by John Wiley & Sons, Inc. CCC 0894-069X1921020153-21$04.@@
154 Naval Research Logistics, Vol. 39 (1992)

we allow bounds on the rate at which resources may be extracted and finite as
well as infinite time horizons.
Deshmukh and Pliska [2-41 consider resource extraction problems in which
there is uncertainty about the level of reserves, the discovery of additional
reserves, and the future demand for the resource. Our results consider uncer-
tainty only in the level of the resource, but we do allow for utility functions
which depend on time and the amount of resource extracted. This is a more
general utility function than that considered by Deshmukh and Pliska. Gilbert
[6] also considers optimal extraction problems with uncertainty in the level of
the reserve and the discovery of additional reserves. He assumes a social utility
function with infinite derivative at 0. In [ 2 ] ,Deshmukh and Pliska give a brief
discussion of other articles related to optimal resource recovery under uncer-
tainty.
We now summarize our results. Let F be the distribution function of the total
resource available, and let r(t, u, w) be the instantaneous return we obtain at
time t when extracting at the rate u given that w amount of resource has already
been extracted. Let 2: [0, T] + [0, 031 denote an extraction plan, i.e., Z ( t ) is
the rate at which resources are extracted at time t. The time horizon T may be
finite or infinite, and we can impose a constraint B on the rate at which resources
can be extracted. Let 6 be the discount rate for income received in the future.
The expected discounted return R ( Z ) , using plan 2, is given by

where

X(t) = J]: Z ( s ) ds.

Our objective is to find Z* to maximize R .
In Section 2 we define the extraction problem for both continuous and discrete
time, and in Section 3 we use the results of [lo] to find necessary and sufficient
conditions for Z* to be an optimal plan. The necessary conditions are used to
find properties of optimal plans in Section 4. In the process of doing this we
extend the following result due to Loury.
Loury [8] considers the case of continuous time, B = m, T = m, and r(t, u ,
w ) = u ( u ) for all t , u , w (i.e., r depends only on u ) . Let u' denote the derivative
of u and E indicate expectation with respect to the distribution of total resource.
Let E be the random time at which the resource is exhausted when using the
optimal plan Z * and let T = min{T, E } . Assuming lim,, u ' ( u ) = 00, Loury shows
that an optimal plan 2" satisfies

The above equation says that for an optimal plan, the discounted marginal return
at time t is equal to the expected value of the discounted average return at the
time extraction stops.
Lipshutz and Stone: Optimal Resource Extraction 155

Let r, denote the partial derivative of r with respect to the ith variable. When
r, is bounded, we obtain a version of Loury’s result for the more general return
function r and general time horizon. We still assume B = w. For all t such that
Z * ( t ) > 0 and F ( X * ( t ) )< 1 we show in Proposition 4.0 that

Observe that when T = w and r, = 0, we obtain the analog of Loury’s result

for u’ bounded. The second term on the right-hand side of the above equation
accounts for the extra cost involved in removing an increment of resource as
the total extraction increases.
When the return function is linear and the rate of extraction is constrained,
we show that a “bang-bang’’ control (extract at the maximum rate until ex-
haustion) is optimal.
In Section 5 we present an algorithm based on the GSO technique which
computes optimal plans when time is discrete and the time horizon finite. This
algorithm is used to produce the examples presented in Section 6.
In Section 7, we compare the GSO algorithm to a dynamic programming
algorithm which could be designed to solve the same problem. The number of
steps in the GSO algorithm grows linearly with the number of time steps, whereas
the number of steps in the dynamic programming algorithm grows quadratically.
Thus for a large number of time increments, the GSO algorithm will be more
efficient. In addition, the dynamic programming approach requires one to ap-
proximate the continuous state space (amount of resource extracted) by a dis-
crete one.

1.2. Relationship to Control Theory

For discrete time, one can use the form of the discrete maximum principle
given in Theorem 1.2 of Chapter 6 of Whittle [ l l ] to obtain the necessary
conditions of Proposition 3.1. One does this by first solving explicitly for the
conjugate variables from the recursions given there. Substituting this solution
into the Hamiltonian and using the fact that the control maximizes the Hamil-
tonian yields the desired conditions. The necessary conditions for continuous
time follow in a similar manner from the form of the continuous maximum
principle given in Theorem 1.1 of Chapter 7 of Whittle [ l l ] . In both of these
cases, the assumptions of the theorems from Whittle have to be relaxed a bit
to apply them to the problem considered here, but undoubtedly that is possible.

2. PROBLEM DEFINITION
In this section we define the problem of maximizing the discounted return
when extracting a nonrenewable resource with uncertain reserves. We will con-
sider discrete and continuous time. Let the random variable Y give the quantity
156 Naval Research Logistics, Vol. 39 (1992)

of resource available. Define F(x) = Pr{Y < x} and let G(x) = 1 - F(x) for
x 2 0. Assume F has a continuous derivative F‘. Since F is a cumulative dis-
tribution function, F’ L 0. Let 6 be the continuous discount rate so that the
discount rate per unit period of time is e-b.

2.1. Discrete Time

Fix T a positive integer and suppose t E (1, . . . , T } .

D1: Let r(t, u, w)be the return from extracting u units of resource at time
t when w units of resource have already been extracted and when u + w 5 Y.

If u + w > Y, the resource is exhausted part way through the time step. For
simplicity we assume that the return is zero. Let ri and rii denote the first and
second partial derivative of r with respect to the ith variable. Assume that r(t,
0, w ) = 0, r2(t,0, w) > 0, and rzz(t,u , w) < 0 for t = 1, . . . , T , u L 0, and
w 2 0. The last two conditions imply that r(t * w) is concave and has a decreasing
marginal rate of return.
We will consider cases in which u is constrained and unconstrained.

D2: In particular we assume u E [0, B] where B is a positive real or

B = 03.

D3: Let ad= [0, B]x . . . x[O, B] where the Cartesian product is taken T
times.

D4: A discrete extraction plan Z is an ordered T-tuple, Z = {Zl, . . . , Z,}

of nonnegative real numbers such that Z E ad.

Given Z E (Rd we will interpret Z as a function from (1, . . . , T} to [0, B]

and Z ( t ) = Z f will denote the tth component of Z.

D5: Given Z E a d , define a function X z from (0, . . . , T} to [0, 00) by

1

Xz(0) = 0 and X,(t) = 2 Z(s), for t = 1, . . . , T.

s=l

Thus, Xz(t) is the cumulative extraction up to and including time t. When the
meaning is clear, we will denote X z by X.
For y L 0, define the random variable I( y) by

1, i f y s Y
0, i f y > Y

D6: Let R ( Z ) be the expected return using extraction plan Z; then

Lipshutz and Stone: Optimal Resource Extraction 157

Since

A discrete extraction plan Z* E Rd is defined to be optimal if R ( Z * ) e R ( Z )

for all Z E ad.

2.2. Continuous Time

Fix T a positive real number, or let T = + w . Let t E [0, T I , and let
r(t, u, w) be the rate of return at time t when extracting at rate u , when w has
already been extracted and w 5 Y . If w > Y , we assume the return is zero.
We will consider constrained and unconstrained extraction rates. Suppose B
is a positive real number or B = + m. A continuous extraction plan Z is a real-
valued continuous function from [0, TI to [0, B ] .

D7: Let a, be the set of all such extraction plans Z .

Given Z E R,, define a function X, from [0, TI to the nonnegative real numbers
by

Then X z ( t ) is the cumulative extraction up to and including time f. When the

meaning is clear, we will denote X, by X.

D8: Fix Z E acand define the random variable T to be the time at which
the resource is exhausted using plan 2. If the resource is not exhausted using
plan 2, then T = T .
Let R ( 2 ) be the expected return using extraction plan Z . By an argument
similar to the one which led to Eq. (l),

R(Z) = ' e-%(t, Z ( t ) , X ( t ) ) G ( X ( t ) )dt. (2)

0
In the following lemma we determine a class of return functions r for which
R is bounded. This will assure us that maximizing R is meaningful when
T = X.

LEMMA 2.0: Let r(t . w) be concave, r2(0,0, 0) < w, r3(t, u, w ) 5 0, and

e-"r(t, u , w) be a decreasing function of t for each pair (u, w). If E [ Y ] < co,
i.e., the expected quantity of the resource is finite, then R is bounded.

PROOF: Generalizing Lemma 2 of Loury [8], we let J * ( y ) be the optimal

return when Pr{Y = y} = 1. Since e-%(t, u , w ) Ir(0, u, 0) 5 ur,(O, 0, 0),
158 Naval Research Logktics, Vol. 39 (1992)

J*( y ) 5 yr2(0,0,O) < w . If Z E R, is any extraction plan, then for each possible
value of the random variable z

1: e-"/r(t,Z ( t ) , X ( t ) ) dt 5 J*(X(z)),

and it follows that

R(Z) = E [(1: 11
e-"r(t, Z ( t ) , X ( t ) dt 5 E[(J*(X(r))]

which proves the lemma.

A continuous extraction plan Z * E R, is defined to be optimal, if R (Z* ) 2

R ( Z ) for all Z E R,.

3. NECESSARY AND SUFFICIENT CONDITIONS

In this section we use the generalized search optimization (GSO) technique
of [lo] to find necessary and sufficient conditions that a plan Z be optimal. In
order to use this technique we much calculate the Gateaux differential of the
functional R. Suppose Z E SZ (R = R, or a,), Ih( E S Z , and 2 + ah E R for
all sufficiently small positive a. Following Stromquist and Stone [lo] we define
R ' ( 2 , h ) , the Gateaux differential of R at Z in the direction h by

1
R'(Z, h) = lim -
W+ot a
[R(Z + ah) - R(Z)]

whenever the above limit exists.

3.1. Discrete Time

We now calculate the Gateaux differential when time is discrete.

LEMMA 3.0: Let T < w . If r2(t, u , w), r3(t, u, w), and G ' ( w ) exist for all
t, u,and w and if Z E Rd, \hi E SZ,, and Z +
ah E R, for small positive a,then
Lipshutz and Stone: Optimal Resource Extraction 159

where

T
+ 2 e-&[r(s,~ ( s ) ,~ ( -s I))G’(X(s))
s=1

PROOF: Fix 2 E adand suppose h E adsuch that 2 + ( ~ Eh adfor small

positive (Y, then

1
R ’ ( 2 , h ) = lim
4 t
-
a
[R(Z + ah) - R ( Z ) ]

To evaluate this limit we bring the limit inside the outer summation, evaluate
the limit, and change the order of summation. Then

+ r(s, Z ( s ) , X ( s - l))G’(X(s))] - h ( t )
I .

Defining D(t, Z , X ) as in the statement of the lemma, we are done.

The following expression relates D(t, Z , X ) and D ( f + I , Z , X):

D ( t , 2, X) = e-”[r2(t,Z ( t ) , X ( t - l))G(X(t)) + r(r, X ( t ) , X ( t - l ) ) G ’ ( X ( t ) ) ]

- e-b(‘+l)(r,(t + I , Z ( t + I), X ( t ) ) - r3(t + 1, Z(t + I), X(t))]G(X(t+ 1))
+ D(t + 1, Z, X). (6)
This recursive definition of D(t, 2, x ) is used in Section 6.
We may now state the necessary and sufficient conditions that a discrete ex-
traction plan Z* be an optimal plan.
160 Naval Research Logistics, Vol. 39 (1992)

PROPOSITION 3.1: Under the assumptions of Lemma 3.0, a neces-

sary condition that Z" E R,/ be an optimal extraction plan is that for t =
1 , . . . , T,

D(r, Z " , X * ) L 0, if Z*(r) = B ,

D(r, Z * , X * ) 5 0, if Z * ( t ) = 0, and

D(r, Z " , X * ) = 0, if 0 < Z * ( t ) < B .

If R is a concave function on Q/, then these conditions are sufficient for a discrete
extraction plan to be optimal.

PROOF: With a small modification to account for the upper bound B , the
proof is a direct application of Theorems 1 and 2 of [ 101.

3.2. Continuous Time

-
We now calculate the Gateaux differential when time is continuous. In
order to do this we introduce the class F ( 6 ) , of Bore1 measurable functions
f : [O, TI (--, m) such that

Let be the subset of 0,.which lies inside F ( 6 ) , i.e.,

PROPOSITION 3.2: Let r?, r,, and G' be bounded. If Z E iy and h E F ( 6 )

are such that 2 +
ah E (1; for small positive a , then

where

D(t, Z , X ) = e-"r,(t, Z(r), X ( r ) ) G ( X ( t ) ) + 1' e+

PROOF: See Appendix A .

We may now state the necessary and sufficient conditions that a continuous
extraction plan be optimal.
Lipshutz and Stone: Optimal Resource Extraction 161

PROPOSITION 3.3: Under the assumptions of Proposition 3.2, a necessary

condition that Z* E be an optimal extraction plan is that for almost every
t E [O, TI,

D(t, Z * , X * ) 2 0, if Z * ( t ) = B ,

D(r, Z * , X * ) 5 0, if Z * ( t ) = 0, and

D(r, Z * , X*)= 0, if 0 < Z*(t) < B .

If R is a concave function on Q., then these necessary conditions are sufficient

for a continuous extraction plan to be optimal.

PROOF: With a simple modification to account for the upper bound B . the
proof is a direct application of Theorems 1 and 2 in [lo].

4. PROPERTIES OF OPTIMAL EXTRACTION PLANS

In this section we explore the properties of optimal extraction plans in con-
tinuous time with and without constraints.

4.1. Continuous-Time Plans with Constraints

Suppose 0 IZ * ( t ) 5 B , r(t, u , w ) = u, and G(w) > 0 for all w > 0. When
the conditions of Proposition 3.2 hold and Z* is optimal, then Z* is a bang-
bang control, extracting at the maximum rate until the resource is depleted.
Observe that X * ( t ) 5 t B , so that G(x*(t))> 0 for all t.
Substituting r(t, u, w ) = u into (7), we obtain

D(t, Z * . X * ) = e-"G(X*(t)) +
I' ds.
e-dsZ*(s)G'(X*(s)) (9)

Differentiating with respect to t , we see that

(f, Z*, X * ) = -de-nrG(X*(t))< 0.

at

Therefore, D ( . , Z * , X*)is a decreasing function of t. If T is finite, Eq. 9 yields

D(T, Z * , X * ) = e-6TG(X*(T))> 0. Therefore, D(t, Z * , X*)> 0 for all f E
[0, TI. If T is infinite, then

lim D(r, Z * , X * )
h r
= lim
r-i
e-"Z*(s)G'(X*(s))ds
1 = 0,

and we again have D(t, Z * , X * ) > 0 for all t E [O, TI. By Proposition 3.3,
Z * ( t ) = B for all t E [O, TI. This proves our claim.
162 Naval Research Logistics, Vol. 39 (1992)

4.2. Continuous-Time Plans Without Constraints

We will prove an analog of Loury's result in [8] which relates the present
discounted marginal return of an optimal plan and the expected discounted
average return for future time. Our results differ from Loury's in a number of
respects. Loury considers only return functions r(t, u , w ) = u(u) that depend
solely on u , the rate at which resource is extracted. In addition Loury requires
limWou ' ( u ) = m. In contrast our results apply to return functions r , which may
depend on time and total resource extracted and for which r, (which corresponds
to Loury's u ' ) is bounded. We also allow bounded time horizons, i.e., T < m.

PROPOSITION 4.0: Suppose the assumptions of Proposition 3.3 are sat-

isfied and Z* is an optimal extraction plan. Let T be the minimum of T
and the time at which the resource is exhausted using 2 " .Then for all t such that
G ( X * ( t ) )> 0 and Z * ( t ) > 0,

- E [1'e-d'r3(s,Z * ( s ) , X * ( s ) ) dslr 2
I
t .

PROOF: Since Z* is optimal, it satisfies (8). Let t be such that Z * ( t ) > 0

and G ( Z * ( t ) )> 0. Then D(t, X " , Z * ) = 0, and dividing both sides of Eq. (7)
by G ( Z * ( t ) ) ,we have

= -[f, + 121.
Recall that for t < T , F ( X * ( t ) ) = Pr{T It}. Using the method of proving Eq.
( 2 ) , but proceeding in reverse order, we find that

Since G' = - F . we have

This completes the proof.

Lipshutz and Stone: Optimal Resource Extraction 163

We may interpret (10) as stating that the discounted marginal return at time
f of an optimal plan is equal to the expected discounted average return at the
time the resource is exhausted given that the resource lasts beyond time t , plus
the marginal loss (r3 < 0) due to extracting when there is a higher level of
depletion.

5. THE ALGORITHMS
In this section we outline an algorithm for finding discrete-time extrac-
tion plans that satisfy the necessary conditions of Proposition 3.1. A complete
description of the algorithm and a convergence proof appear in the Appen-
dix. We restrict ourselves to functions r and G such that if D(t, u , x ) =
r2(t, u , x - u)G(x) then aD/au is continuous and nowhere vanishing. This
assumption will be used to prove that the algorithm converges. In addition we
assume that I*(?, 0, 0) > 0 for t = 1, . . . , T. In the following we refer to Z ( f )
by Z, and X ( t ) by X,for notational convenience.
To initiate the algorithm we choose a value for XT,the cumulative extraction
at time T. An examination of Eq. (5) shows that D ( T , Z, X)depends only on
XT and ZT. (Recall that XT-I = XT - ZT).Thus we can choose ZT E [0, B ]
such that the pair X T , ZT satisfies the necessary conditions of Proposition 3.1
for t = T. We then compute X T - , = X T - ZT. Since D(T - 1, Z, X)depends
only on X T , Z T ,X T - ,, and ZT-I, we may solve for ZT-I to satisfy the necessary
conditions for t = T - 1. Repeating this process for T - 2, T - 3, . . . , 1 we
may determine a plan ZT, . . . , Z, satisfying the necessary conditions of Prop-
osition 2.1.
The resulting extraction plan may not satisfy 2.T=lZ, = X T . To find a plan Z
that satisfies both this equality and the necessary conditions, the algorithm pro-
ceeds as follows. It determines X,+? 0 and X f L 0 such that ,2T=l Z,+ 5 X,+
and C,L, Zf L X f and C,T=, Z ; L X f , where Z ; , . . . , Zf and Z:. . . . , Z ;
are determined as above. Using a binary search between X f and X; it finds
X; such that the absolute difference IX: - CT=, Z:I is less than a tolerance c
set by the user. At this point the algorithm terminates and Z* is the resulting
plan.

6. EXAMPLES
In this section we present examples of discrete extraction plans which satisfy
the necessary conditions of Proposition 3.1. These plans were computed using
the above algorithm implemented in Fortran 77 on an Apollo computer.
In the following examples, the amount of resource Y is assumed to have a
gamma distribution with mean, ji = 1, and shape parameter, > 0, so that
11

var(Y) = , d / v . This implies

164 Naval Research Logktics, Vol. 39 (1992)

For integer v we have

for x > 0.

In each of these four examples, we vary a single parameter and observe the
effect on the solution. Each figure contains the solution in the case S = 0, r(t,
u , w ) = u , v = 2, and an unconstrained rate of extraction. This serves as a
benchmark for comparing the different examples.

6.1. Example 1: Varying the Constraint B

In this example (see Figure 1) we vary the bound B on Z ( t ) . The remaining
parameters are fixed as follows: r(t, u , w ) = u , 6 = 0, and 1' = 2. We let B =
0.2,0.3,0.4, and +m. The unbounded case is our benchmark. Note the minimum
extraction occurs when r = 3. At t = 3, the cumulative extraction has exceeded
0.5, the mode of F. Therefore, we see that Z * ( t ) is increasing when G"(x) is
negative and Z*(r) is decreasing when G"(x) is positive. One can show that this
behavior is true for all the examples considered in this section. Note that the
bound has little effect on the expected extraction. This is because most of the
differences in the extraction plans occur in the tail of the distribution.

t
O JI
I
1

1
I
I

3 4
I
I
. I
I

5 6
I
I
I
I

I
I
I

I
1
I

9
I
10

Figure 1. Varying the bound on the rate of extraction.

Lipshutz and Stone: Optimal Resource Extraction 165

I 2 3 4 5 6 7 8 9 10

Figure 2. Varying the discount rate 6.

6.2. Example 2: Varying the Discount Rate

In this example (see Figure 2) we vary the discount rate ii.The remaining
parameters are fixed as follows: r ( t , u , w ) = u , no constraint, and I' = 2. We
let ii = 0 , 0.1, and 0.2 with b = 0 being the benchmark case. An increase in
the discount rate shifts the extraction plan toward larger extractions earlier. The
overall plan is more conservative, as indicated by the decrease in the expected
extraction as the discount rate increases.

6.3. Example 3: Varying the Return Function with Time

In this example (see Figure 3 ) , we suppose a gradual increase in the return
as t increases. In particular, we let r ( t , u , MI) = u(2e'" - 1 ) where p = 0 . 0.01.
and 0.05. We set the remaining parameters as follows: 6 = 0, 1' = 2 , and n o
bound on the extraction rate. Figure 3 illustrates the example. The main effect
of increasing p is to shift most of the extraction to later times when the return
is greater. Although the solution in the case 1) = 0.05 satisfied Z ( t ) < 0.01 for
I = 1, . . . , 5, it was still the case that Z ( f )> 0. t = 1. . . . , 10.

6.4. Example 4: Varying the Shape of the Distribution

In this example (see Figure 4) we vary the shape of the distribution letting
I' = 1 , 2 , 3 , 5 , and 10. The remaining parameters are set as follows: r ( t , u . w )
= u . (5 = 0 . and Z ( r ) is unconstrained. Note that an increase in which reduces
13,

the variance, makes the plan more conservative, reducing both the maximum
166 Naval Research Logistics, Vol. 39 (1992)

41 - BwBclgD BWBCIIU) xzo

a BxIlucnoNm -
04 .. 001
O a69
863
869
943
2666
2454
rn 761 1411 1807
03 .. (dmdims 1OOo)

a4 ..
0.3 ..
u,

ai ,
p-Bl
0. I
I
I
I
I
I

Figure 3. Varying the price function.

&Bv
ExmcmD HXPBCIBD xm
a1 .. P BJmAmoNpBQHI: -
1 841 MI 3861

I 1 3 4 5 6 7 8 9 10

Figure 4. Varying the shape of the distribution.

ai

01
-
..
8
1
.n
-
Lipshutz and Stone: Optimal Resource Extraction

BxlnAQmpgpea:-
m
838
gwBcw)

870
1w9
xm
2666
1843
167

.a 759 459a mi
05 .. 3s
.1
5m
310
(--1ooo)
17168
61529
750
407 -
Y .75

OA ..
03 ..

I
I I I I I I 1 I I
I a 3 4 5 6 1 8 9 ID

Figure 5. Iso-elastic return varying y .

and the expected extraction. In addition as v increases, earlier extractions are

increased at the expense of the later extractions.

6.5. Example 5: Isoelastic Return Function

In this example (see Figure 5) we let r(t, u , w ) = (l/y)u;’foru 2 0, y = 0.1,
0.25, 0.5, 0.75. and 1.0. The remaining parameters are set as follows: 6 = 0,
v = 2, and no constraint on Z ( t ) . As y decreases, the extraction plan becomes
more conservative.

7. COMPARISON WITH DYNAMIC PROGRAMMING

In this section we compare the techniques we have used with those of dynamic
programming.

7.1. Discrete Time

Let R:T(X(s - 1))be the optimal return when we begin extracting at time s
with X ( s - 1) already extracted and continue extracting until time T. We may
recursively compute R:T by

G(Z(s) + X ( s - 1))
+ e-”RR,*,:(Z(s)+ X ( s - 1)))
G M s - 1)) 1
168 Naval Research Logistics, Vol. 39 (1992)

Using standard dynamic programming techniques, we discretize the state

space. Fix an integer n > 0 and assume Z(s) = k ( b / n ) for some k E
(0, . . . , n}. Then X ( s ) = k ( b / n ) for some k E (0, . . . , Tn}.
To compute R:T(X(s - l)),we must perform sn2 evaluations of the func-
tion r and have computed R,*,T,(X(s))for X(s) = X(s - 1) + k ( b / n ) , k =
0, . . . , n. To compute R,*T(0),the function r must be evaluated T(T + l)n2/2
times. We may conclude that the time to compute RtT(0)grows quadratically
in both the number of time periods and the precision.
Recalling the algorithm described in Section 4, we observe that the compu-
tation requires two nested searches for zeros, each of which grows linearly with
precision. We also need Tcalculations of D(t , Z * , X * ) , which may be calculated
recursively using (6). The time to compute Z* will grow linearly in the number
of time periods and quadratically in the precision. For large T , we may find
solutions more efficiently by the general search optimization algorithm.

7.2. Continuous Time

Loury [8] and Liu [7] apply continuous dynamic programming to solve this
problem. In general, the technique does not easily allow for constraints on Z * ( t ) .
This problem is discussed further in Bellman [ l , Chap. 91.

ACKNOWLEDGMENT
This work was supported by Office of Naval Research Contract No. N00014-
80-CO766.

APPENDIX A
Proof of Proposition 3.2

The proof of Proposition 3.2 proceeds as in Lemma 3.0. Suppose Z and h are chosen as in the
hypothesis of the proposition. Let

Then

1
R ' ( Z , h) = lim - [R(Z + ah) - R ( Z ) ]
o-0t a

= lim -
ll
a-m+ a r
[P(t,Z + ah) - P(r, Z ) ] dt.

From the generalized mean value theorem applied to P(r, Z ) , it follows that

1
IP(r. z + ah) - P ( f , Z)l 5 sup (I-: ,I
I.t dP
(I, z+ th)(
Lipshutz and Stone: Optimal Resource Extraction 169

for all t E [0, TI. If

we may apply the bounded convergence theorem to conclude

[ [2 1 (P(r, Z + ah)
R'(Z, H ) = - P(t, Z)) dt.
1
Evaluating the limit as in the proof of Lemma 3.0, we may complete the proof.
We now prove the inequality in (11). Fix a > 0 such that Z + ah E 0:. Define

n(t,5 ) = t.

(t, 2 + <h) , for 0 5 t 5 T and 0 5 5 a.

and

f$(f) = sup{(: H ( t . 5 ) = H * ( f ) }

Z*(r) = Z(t) + $(t)h(t), for 0 5 t 5 T

We now show that H * and 4 are Borel measurable functions. This will guarantee that Z* E fly.
and that H * ( t ) = ( ( a P / d < ) ( t Z*)I,
, for 0 5 t 5 T .
For each a > 0.

By 2.2.13 of [ 5 ] , the above set is Lebesgue measurable so that H * is almost everywhere equal to a
Borel function (see 2.3.6 of [ 5 ] ) . Since

{t: $(f) > u} = n{(r, 5 ) = H * ( t ) and < > u},

the same reasoning shows that f$ is almost everywhere equal to a Borel function.
Thus

= 1,' le-"r2(t, Z * ( t ) , X * ( t ) ) G ( x * ( r ) ) h ( t ) ldr

+ 1,' P I
le-."r,(t, Z * ( t ) , X * ( t ) ) G ( x * ( t ) ) h ( s ) ds dt
170 Naval Research Logistics,Vol. 39 (1992)

We will prove that I,, I., and I, are bounded. Since r2 is bounded, r / u is bounded for u > 0. Let
K be a uniform bound for r ( f ,u , w ) I u , r 2 ( f ,u , w ) , r , ( f , u , w ) and G ' ( x ) .Evaluating I , and I,, we
obtain

I, 5 I' dt < =,
Ke-'"(h(f)(

Evaluating 13. we find

Changing the order of integration, we have

This completes the proof.

APPENDIX B
The Algorithm

In this appendix we define and prove the convergence of the algorithm outlined in Section 5 . In
order to define the algorithm, we make use of the function Q defined below.

B.l. Definition of Q

Let r and G be chosen as above. Fix t E {l, . . . , T } , XT E R, XT 2 0 and Z, E [0, B ] ,

s = f + 1, . . . , T. Let Z , , . . . , Z , - , = 0, X , , . . . , X , - ] = 0 and X 3 - l = X , - Z , , s =
f + 1, . . . , T .
Recall Z = ( Z , , . . . , Z , ) and X = (XI, . . . , Xr).Note that D ( t , . . . , X) does not de-
pend on Z,, . . . , Z , - , , so we may define

g , ( y , Z,-I, . . . , ZT, X,) = D(t, ( Z , ,. . . 1 z,-,,

y. z,,,,
. . . ,Z,), X )
For convenience we will write g , ( y ) for g , ( y . Z , - [ , . . . , Z r , Xr).Define Q,(Z,+l, . . . , Zr, X,)
as follows:

1. If there is a y * E [0, B] such that g , ( y * ) = 0, then

.
Q , ( Z , + l ,. . . Z T , X T ) = Y * .

2 . If g , ( y ) > 0 for all 0 s y 5 B, then Q , ( Z , + , ., . . , Zr. X T ) = B .

3 . If g , ( y ) < 0 for all 0 5 y 5 B , then Q , ( Z , + , ,. . . , Z T , X r ) = 0.
Lipshutz and Stone: Optimal Resource Extraction 171

.
We will prove that Q,is a well-defined and continuous function of Z,, ,,. . . Z, and X , in Proposition
B. 1 below. This is done by setting u = y, w = (Z,,, , . . . ,Z . , X , ) ,p = Q , and r#J = g, in Proposition
B.1. The fact that dDifiu is continuous and nowhere zero guarantees that dr#J/au has the same
properties.
If Z is an allocation and X , a positive real number define

where

If X r is fixed and Z = (0, . . . , 0) then sequential application of Y , ,for t = T , . . . , 1 will

determine an allocation satisfying the necessary conditions given in Proposition 2.1. For arbitrary
X r , this allocation may not satisfy El=, Z, = X r .
Define Y = Y p . . ' 0 9 ,and A ( X T ,Z ) = X r - El=, Z,. As-defined q,,q ,and A are continuous
functions. Let 8 = (0, . . . , 0) E R'. Observe that A"Y (., 0) is a continuous function from the
nonnegative real numbers to R.This fact will be used later.

B.2. Definition of the Algorithm

The following sequence of steps defines the algorithm. Note, equality is understood as replacement,
in agreement with standard programming notation.

1. Fix E > 0.
2. Let U, =_2TB, V, = 0, and i = 0.
3. Let z = 0.
4. Let X,, = ( U , + V , ) / 2 .
5. Do step 6 for t = T to I .
6. Let ( X , , Z) = ul,(X,, Z).
7. Let X,,, = A ( X , , . Z )
8. If IX,,,l < & go to 12.
9.IfX,,,>OletU,,,=X,.V,+,=V,
else let V , + ,= X , , . U , , , = U , .
10. Let i = i + I .
I I . Go to4.
12. End, when the vector Z satisfies the conditions of Proposition 3.1 and

I X,, -
I
2 z,< c.

B.3. Convergence of the Algorithm

We first show that the algorithm converges if AOY(2TB. 6) 2 0 and AO'P(0, 6) 5 0. We then
prove these inequalities hold.
Suppose A o Y (2TB , 0) 2 0 and AOY(O,6) 5 0. Then by Step 9 above.

6) 2 0 2 A N ( V , ,6)
AoY( U,,

A o q ( U , + , 6)
, 2 Aoq(X,, 6) 2 A 0 Y ( V l i l ,0).
172 Naval Research Logistics, Vol. 39 (1992)

Our choice of U, and V, implies Iim,-= U,= Iim,-= X , = lim,+z V,. Using the continuity of A o q
(see Section 7), we have the following inequalities:

0 2 lim,+=A N ( V,, 0) = Iim,+= A o q ( X , , 0) = Iim,-= AoV( U,, 0) 2 0.

Therefore

Iim,+.AA,,\V(X,,, Ti) = o
and the algorithm converges.
To complete the proof, we need only show that AO\V(O,G) 5 0 and AO\v(2TB, 0) 2 0 . To prove
AoVI(0, 0) 5 0, set XT = 0, then since G(0) = 1, D ( T , Z , X ) = r2(frZ T , - Z T ) + r ( T , Z,, - Z T )
G'(X,). If Z , = 0, D ( T , Z , X ) = r2(T,ZT, - Z T ) + r(T, ZTr- Z T ) G ' ( X T ) If
. ZT = 0, D ( T , Z ,
X ) = rz(T , 0, 0) > 0. This contradicts the definition of Q T ; therefore ZT > 0. Computing AOV(0,
Z), we obtain
7
AO'P(0,G) = 0 - ZT 5 0.
I= I

To prove A o 9 ( 2 T B , G) 2 0 observe that since Z, 5 B for 1 = 1, . . . , T ,

T
A0'€'(2TB, 8) = 2TB - 2 Z , 2 2TB - TB = TB > 0.
,=I

This completes the proof of convergence.

B.4. Q, is Well Defined and Continuous

The following general lemma proves that Q, is a well-defined, continuous function of

(Z(1), . . . Z ( T ) , W T ) ) .
1

Let V = [a, b] be a closed interval of real numbers, and let W be any compact subset of R". Let
U be an open set in R x R" containing V x W. Let 4 be a continuously differentiable real-valued
function on 0 such that a4/au(u, w ) # 0 for all ( u , w ) E CJ. Define p, a function from W to V as
follows: Fix w E W then,

1. If there is a u* E V such that 4 ( u * , w ) = 0, then set p ( s ) = u*

2. If Q(u, w ) > 0 for all u E V, set p ( w ) = b.
3. If 4 ( u , w ) < 0 for all u E V, set p ( w ) = a.

+
PROPOSITION B. 1: If p and are defined as above, then p is a well-defined continuous function
on W.
PROOF: First we prove that p is well defined as above, then p is continuous. Fix w E W. Sup-
pose 0 is not in the range of 4(., w), then by the continuity of +,
4 ( u , w ) > 0 for a s u s b or
4 ( u . w ) < 0 for a 5 u 5 6 . In either case p is well defined. Suppose for some u* E V, 4 ( u t , w ) =
0. Since ik$/du is nonvanishing. +(., w ) is strictly monotonic and such a u* is unique. We conclude
p is well defined for all w E W.
To prove continuity, define 11,: U + W by rI,(u, w ) = w. If A and B are sets, let B - A denote
the complement of A in B. Let R' = { x E Rlx 2 0 } and R - = {x E R ( x 5 0). Let W ' = U -
Il2,,4I(R -), then W ' is the set of w E W such that + ( u , w ) > 0 for all u E V. Since R - is a closed
+
set, is continuous, and 11, is a closed mapping. we conclude that W' has constant value b and so
is continuous. We may similarly define W = U - 1 I 2 , , + - I ( R + ) ,then p is continuous on W - with
constant value a .
Let W,, = l I > & ' ( { O } ) n W. Then W,,is the set of w E W such that +(u, w ) = 0 for some u E
V. For each v E W,,,the implicit function theorem yields a continuous function pr defined on an
open neighborhood N, of v such that 4 ( p v ( w ) ,w ) = 0 for w E N,. and p , ( y ) = p ( y ) . Since p is the
Lipshutz and Stone: Optimal Resource Extraction 173

unique solution to 4 ( p ( w ) , w ) = 0 for w E W , , ,it follows that p ( w ) = p ( w ) for all w E N , f l W,,.

Thus p is continuous on W,,.
To complete the proof of continuity. we show that p is continuous at the boundaries of W4,,
W + , and W - . We may assume a < b. since continuity is obvious if a = b . Suppose y E W,,is a
limit point of W * . Then any open neighborhood of y in W has nonempty intersection with W ’ . Wc
need to prove that p ( y ) = b. Suppose to the contrary that p ( y ) < b. Since p , is continuous. there
is an open neighborhood N N , such that y E N , p , ( w ) < b, and + ( p , ( w ) . w ) = 0 for w E W . This
and the monotonicity of +(.. w ) imply that p ( w ) = max(a. p , ( w ) } < b for w E N . But this means
that N f l W ’ is empty, contrary to the assumption that y is a boundary point of W * . Thus, we must
have p ( y ) = b. A similar argument shows that if y E W,,is a boundary point of W , then p ( y ) =
a. Suppose now that y E W is a common boundary point of W * and W . Since y is in neither W *
or W - , it must be in W,,. By the above arguments, p ( y ) = a < b = p ( y ) . Thus, there are no
common boundary points of W ’ and W - and continuity is proven.

REFERENCES
[ 11 Bellman. R., Dynamic Programming, Princeton University Press, Princeton, NJ,
1957.
[2] Deshmukh, S.D., and Pliska, S.R., “Optimal Consumption and Exploration of
Nonrenewable Resources Under Uncertainty,” Econometrica, 48 177-900 (1980).
[3] Deshmukh, S.D.. and Pliska, S.R., “Natural Energy Resource Decisions and Prices
Involving Uncertainty.” Discussion Paper No. 499, Center for Mathematical Studies
in Economics and Management Science, Northwestern University, 1981.
[4] Deshmukh, S.D., and Pliska, S.R., “Optimal Consumption of a Nonrenewable
Resource with Stochastic Discoveries and a Random Environment ,” Discussion
Paper No. 500, Center for Mathematical Studies in Economics and Management
Science, Northwestern University, 1981.
[ 5 ] Federer, H., Geometric Measure Theory, Springer-Verlag, New York, 1969.
161 Gilbert, R.J., “Optimal Depletion of an Uncertain Stock.” Review of Economic
Studies, 46 47-57 (1979).
[7] Liu, P.T., “Optimum Extraction of an Exhaustible Resource: A Mathematical Anal-
ysis,” in Dynamic Optimization and Mathematical Economics, P.T. Liu, Ed.,
Plenum, New York, 1980.
[8] Loury, G.C., “The Optimum Exploitation of an Unknown Resource,” Review of
Economic Studies, 45 621436 (1978).
[9] Luenberger, D.G., Optimization by Vector Space Methods, Wiley, New York, 1969.
[lo] Stromquist, W.R., and Stone, L.D., “Constrained Optimization of Functionals with
Search Theory Applications,” Mathematics of Operations Research, 6(4), 518-529
(1981).
[I11 Whittle, P. Optimization Over Time, Wiley, New York, 1982, Vol. 1.

Manuscript received May 22, 1990

Final revised manuscript received May 6, 1991
Accepted May 21, 1991