6.1 Risk-sensitive hierarchical controls with discounted costs

Consider the production system whose dynamic is given by $$\dot x(t)=u(t)-z(t),\; x_0=a\in R^{n} \mbox{ ($a$\space is given)}.$$

Let $J^{\varepsilon,\sqrt{\varepsilon}}(u(\cdot))$ denote the risk-sensitive discounted cost function defined by

$$J^{\varepsilon,\sqrt{\varepsilon}}(u(\cdot))=\sqrt{\varepsi......\int_0^\infty e^{-\rho t} [h(x(t))+c(u(t))]dt\right\}\right].$$

The problem is to find an admissible control $u(\cdot)$ that minimizes $J^{\varepsilon,\sqrt{\varepsilon}}(u(\cdot))$.

We now specify the production constraints. For each $i\in{\cal M}=\{0,1,2,\ldots,m\}$, let

$${\cal U}(i)=\{l=(l_1,\ldots,l_{n})\geq0:p\cdot l\leq i\}\subset R^{n}.$$

With this definition, the production constraint at time t is $u(t)\in {\cal U}({\alpha^\varepsilon(t)}).$

We assume the demand rate z(t) to be a bounded process independent of $\alpha^\varepsilon(t)$.

We say that a control $u(\cdot)=\{u(t):\; t\geq0\}$ is admissible if u(t) is a $\sigma\{\alpha^\varepsilon(s),z(s):\; s\leq t\}$-adapted measurable process and $u(t)\in {\cal U}({\alpha^\varepsilon(t)})$ for all $t\geq 0$. Then our control problem can be written as follows:

$${\cal P}^{\varepsilon,\sqrt{\varepsilon}}:\left\{\begin{a......varepsilon,\sqrt{\varepsilon}}(u(\cdot)).\end{array}\right.$$

Let ${\cal Z}_t =\sigma\{z(s):\; s\leq t\}$. We consider the following control space:

$$% latex2html id marker 11252\begin{array}{l}{\cal A}^0=...... {\cal Z}_t \mbox{ adapted measurable process}\}\end{array}$$

and two control problems ${\cal P}^{0,\sqrt{\varepsilon}}$ and ${\cal P}^{0,0}$ defined as follows:

$${\cal P}^{0,\sqrt{\varepsilon}}:\left\{\begin{array}{cl}...... A}^0 }J^{0,\sqrt{\varepsilon}}(U(\cdot))\end{array}\right.$$

and

$${\cal P}^{0,0}:\left\{\begin{array}{cl}\mbox{minimize}&......{U(\cdot)\in{\cal A}^0}J^{0,0}(U(\cdot)).\end{array}\right.$$

It can be seen below that when $\varepsilon$ is small, ${\cal P}^{\varepsilon,\sqrt{\varepsilon}}$ can be approximated by ${\cal P}^{0,\sqrt{\varepsilon}}$ and ${\cal P}^{0,\sqrt{\varepsilon}}$ can be approximated further by${\cal P}^{0,0}$. Therefore, ${\cal P}^{\varepsilon,\sqrt{\varepsilon}}$ can be approximated by ${\cal P}^{0,0}$. Then, a near optimal control for ${\cal P}^{0,0}$ will be used to construct controls for ${\cal P}^{\varepsilon,\sqrt{\varepsilon}}$ that are nearly optimal.

Theorem 6.1   There exist constants $\varepsilon_0>0$ and C such that, for $0<\varepsilon\leq\varepsilon_0$,

$$\vert v^{\varepsilon,\sqrt{\varepsilon}}-v^{0,\sqrt{\varepsilon}}\vert\leq C\sqrt{\varepsilon}.$$

We show that ${\cal P}^{0,\sqrt{\varepsilon}}$ can be approximated by ${\cal P}^{0,0}$ and the value function of ${\cal P}^{0,0}$ is a viscosity solution to the Isaacs equation of a zero-sum, two-player differential game. To simplify the notation, we take $\delta =\sqrt{\varepsilon}$ and consider the following control problem ${\cal P}^{0,\delta }$.

$${\cal P}^{0,\delta }:\left\{\begin{array}{cl}\mbox{mini......ot)\in{\cal A}^0}J^{0,\delta }(U(\cdot)).\end{array}\right.$$

Theorem 6.2$v^{0,\delta }$is a monotone increasing function of $\delta >0$ and

$$\lim_{\delta \to0}v^{0,\delta }=v^{0,0}.$$

For each $U(\cdot)\in {\cal A}^0$.

$$J^{0,\delta }(U(\cdot))\uparrow J^{0,0}(U(\cdot))\mbox{ as }\delta \downarrow0.$$

We write v^0,0(x) as the value function of${\cal P}^{0,0}$ with the initial value x₀=x. Note that $\Vert\xi\vert _\infty=\inf_{P(F)=0}\sup_{\omega\in\Omega-F}\vert\xi(\omega)\vert$ for any random variable $\xi$. Let $\Gamma_u=\{U=(u^0,u^1,\ldots,u^m)\in R^{n\times (m+1)}$ such that $u^i\in{\cal U}(i)\}$ and let $\Gamma_z$ denote a compact subset of Rⁿ. We consider functions $z(t)\in\Gamma_z$ ($t\geq 0$) that are right-continuous and have left-hand limits. Let ${\cal Z}$ denote the metric space of such functions that is equipped with the Skorohod topology $d(\cdot,\cdot)$.

We assume $z(\cdot)=z(\cdot)(\omega)\in\Gamma_z$ a.s. and for each $z^0=z^0(\cdot)\in{\cal Z}$ and any $\delta _0>0$, $P(d(z(\omega),z^0)\leq\delta _0)>0$.

Theorem 6.3   v^0,0(x) is the only viscosity solution to the following Isaacs equation:

$$\begin{array}{rl}\rho v^{0,0}(x)=&\displaystyle \min_{U\i......,0}_x(x)+h(x)+\sum_{i=0}^m \nu_ic(u^i)\biggr].\end{array}$$

Theorem 6.4   The following assertions hold:
(i)

$$\lim_{\varepsilon\to0}v^{\varepsilon,\sqrt{\varepsilon}}=v^{0,0}.$$

(ii) Let$U(\cdot)=(u^0(\cdot),\ldots,u^m(\cdot))\in{\cal A}^0$denote a stochastic open loop $\varepsilon'$-optimal control for ${\cal P}^{0,0}$, i.e.,

$$0\leq J^{0,0}(U(\cdot))-v^{0,0}\leq\varepsilon'.$$

Let $$u^\varepsilon(t)=\sum_{i=0}^m 1_{\{\alpha^\varepsilon(t)=i\}} u^i(t)$$, where 1_A denotes the indicator of a set A. Then, $u^\varepsilon(\cdot)\in{\cal A}^\varepsilon$ and

$$\limsup_{\varepsilon\to0}\vert J^{\varepsilon,\sqrt{\vareps......ot))-v^{\varepsilon,\sqrt{\varepsilon}}\vert\leq\varepsilon'.$$

(iii) Let$U(\cdot)=U(z(\cdot),x(\cdot_)=(u^0(z(\cdot),x(\cdot)),\ldots,u^m(z(\cdot),x(\cdot)))$denote a feedback $\varepsilon'$-optimal control for ${\cal P}^{0,0}$, i.e.,$$0\leq J^{0,0}(U(\cdot))-v^{0,0}\leq\varepsilon'.$$Let

$$\displaystyle u^\varepsilon(\cdot)=u^\varepsilon(\alpha^\va......m 1_{\{\alpha^\varepsilon(\cdot)=i\}} u^i(z(\cdot),x(\cdot)).$$

Assume that U(z,x) is locally Lipschitz in x, i.e., for some k>0,

$$\vert U(z,x)-U(z,x')\vert\leq C(1+\vert x\vert^{k_5}+\vert x'\vert^{k_5})\vert x-x'\vert.$$

Then, $u^\varepsilon(\cdot)=u^\varepsilon(\alpha^\varepsilon(\cdot),z(\cdot),x(\cdot))\in{\cal A}^\varepsilon$ and

$$\limsup_{\varepsilon\to0}\vert J^{\varepsilon,\sqrt{\vareps......ot))-v^{\varepsilon,\sqrt{\varepsilon}}\vert\leq\varepsilon'.$$