Markov control of continuous time Markov processes with long run functionals by time discretization*
Lukasz Stettner 1 · 2025
Read Markov control of continuous time Markov processes with long run functionals by time discretization* by Lukasz Stettner 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06916.txt.
Introduction
Stettner studies control of continuous time Markov processes in twenty twenty-five. He approaches them through discrete time versions that are easier to handle. The focus is on long run goals, such as the average reward per unit time. A risk sensitive long run goal is also covered. The paper asks how well the discrete time view captures the true continuous problem, and how stable the long run values are when the chosen controls converge only point by point.
Catalog record
| CREATOR | Lukasz Stettner 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06916
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Markov control of continuous time Markov processes with long run functionals by time discretization* Lukasz Stettner 1 Abstract In the paper we study continuous time controlled Markov processes using discrete time controlled Markov processes. We consider long run functionals: average reward per unit time or long run risk sensitive functional. We also investigate stability of continuous time functionals with respect to pointwise convergence of Markov controls. 1 INTRODUCTION Assume that state space E is Polish with Borel \sigma -field {\cal E} , although in particular examples we shall consider {\cal E}=R^{d} or a bounded convex subset of R^{d} . We have also a compact set of control parameters U and a family {\cal U} of Borel measurable mappings u:E\mapsto U called later Markov controls. On a probability space (\Omega,F,(F_{t}),P) , for each u\in{\cal U} we are given a continuous time controlled Markov process (X_{t}^{u}) with transition operator P^{u}_{t}(x,dy) for x\in E and control u(X_{t}^{u}) at generic time t . We consider a natural pointwise convergence topology on {\cal U} , which means that u_{n}\in{\cal U} converges to u\in{\cal U} whenever u_{n}(x)\to u(x) as n\to\infty for each x\in E . Then we consider discrete time approximations (X_{t}^{(h),u}) of (X_{t}^{u}) which is a discrete Markov process X_{nh}^{(h),u} at generic moments nh such that X_{t}^{(h),u}=X_{\left[{t\over h}\right]h}^{(h),u} , where \left[{t\over h}\right] is the integer part of {t\over h} and X_{nh}^{(h),u} has transition operator P^{(h),u(X_{nh}^{(h),u})}(X_{nh}^{(h),u},\cdot) . This means that while process (X_{t}^{u}) is controlled at each time t using u(X_{t}^{u}) , its discrete time approximation X_{nh}^{(h),u} is controlled at moments nh using u(X_{nh}^{(h),u}) . To be more precise consider our main example. Example 1. Assume for u\in{\cal U} we have the following equation in R^{d} X_{t}^{u}=x_{0}+\int_{0}^{t}b(X_{s}^{u},u(X_{s}^{u}))ds+\int_{0}^{t}\sigma(X_{s}^{u})dW_{s}, (1) where (W_{t}) is a Brownian motion, |b(x,a)-b(y,a)|+\|\sigma(x)-\sigma(y)\|\leq K_{R}|x-y| for a\in U , |x|,|y|\leq R , |b(x,a)|^{2}+\|\sigma(x)\|^{2}\leq K(1+|x|^{2}) and \xi^{T}\sigma(x)\sigma^{T}(x)\xi\geq{1\over K_{R}}|\xi|^{2} for \xi\in R^{d} , |x|\leq R and any R>0 . By Theorem 2.2.12 of [ 1 ] for each u\in{\cal U} there is a unique strong solution to the equation ( 1 ). Our discrete approximation with discretization step h is defined as X_{(n+1)h}^{(h),u}=X_{nh}^{(h),u}+\int_{nh}^{(n+1)h}b(X_{s}^{(h),u},u(X_{nh}^{(h),u}))ds+\int_{nh}^{(n+1)h}\sigma(X_{s}^{(h),u})dW_{s} (2) for n=1,2\ldots and X_{0}^{(h),u}=x . Since we have a unique strong solution on each time interval [nh,(n+1)h] we have well defined process (X_{t}^{(h),u}) . In what follows we shall consider a general case introducing a number of assumptions which are mainly satisfied by the model considered in this example. In the paper we want to maximize the following functionals: average reward per unit time J_{x}(u)=\liminf_{t\to\infty}{1\over t}E_{x}^{u}\left\{\int_{0}^{t}c(X_{s}^{u},u(X_{s}^{u}))ds\right\}, (3) for a bounded measurable function c:E\times U\mapsto R , continuous with respect to the second (control) parameter, and its discrete time approximation J_{x}^{h}(u)=\liminf_{n\to\infty}{1\over nh}E_{x}^{u}\left\{\sum_{i=0}^{n-1}hc(X_{ih}^{(h),u},u(X_{ih}^{(h),u}))\right\}, (4) long run risk sensitive with risk factor \alpha<0 I_{x}^{\alpha}(u)=\liminf_{t\to\infty}{1\over\alpha}{1\over t}\ln E_{x}^{u}\left\{e^{\alpha\int_{0}^{t}c(X_{s}^{u},u(X_{s}^{u}))ds}\right\}, (5) and its discrete time approximation I_{x}^{\alpha,h}(u)=\liminf_{n\to\infty}{1\over\alpha}{1\over nh}\ln E_{x}^{u}\left\{e^{\alpha h\sum_{i=0}^{n-1}c(X_{ih}^{(h),u},u(X_{ih}^{(h),u}))}\right\}. (6) Risk sensitive functionals are important since they measure not only expected value of the reward but also other moments of the reward including variance with weight \alpha , which his considered as a measure of risk (see [ 15 ] , [ 19 ] , [ 20 ] ). We want to show that under suitable assumptions J_{x}^{h}(u)\to J_{x}(u) and I_{x}^{\alpha,h}(u)\to I_{x}^{\alpha}(u) as h\to 0 . Then we consider stability of continuous time functionals i.e. we using discrete approximation show that whenever u_{n}\to u then also J_{x}(u_{n})\to J_{x}(u) and I_{x}^{\alpha}(u_{n})\to I_{x}^{\alpha}(u) as n\to\infty . The paper generalizes and extends [ 16 ] , where only discrete time was considered. Usually we have a continuous time model which we control using discrete time inputs. In the paper we want to justify such procedure. Practically we use piecewise constant controls in discrete time moments, which we expect to be good, feasible approximation of real world model. Notice that such models can not be approximated using weak convergence technics considered in [ 12 ] . Average reward per unit time problem is considered in full generality considering Lyapunov function V , which allows us to have unbounded reward function c and consequently we obtain a number of results in norms weighted by V . The studies of long run risk sensitive functionals are practically restricted to compact state spaces for which we consider nondegenerate diffusions, possibly with jumps, in regular bounded sets. 2 Average reward per unit time problem We shall need the following assumption: (ER) for each u\in{\cal U} process (X_{t}^{u}) is aperiodic and ergodic in the sense that it has a unique invariant measure \mu^{u} . In what follows we shall consider discrete time approximations with h=2^{-m} , and to simplify notations we shall denote process X_{nh}^{(h),u} by X_{n2^{-m}}^{(m),u} . We assume that (ERd) for each m\in N and the process (X_{n2^{-m}}^{(m),u}) is aperiodic and ergodic. Furthermore we assume that (UEd) for each u\in{\cal U} there is \rho\in(0,1) and function V:E\to[1,\infty) such that for x,x^{\prime}\in E and m\in N \int_{E}V(y)|P_{1}^{(m),u}(x,dy)-P_{1}^{(m),u}(x^{\prime},dy)|\leq\rho\left[V(x)+V(x^{\prime})\right]. (7) Above introduced V is called sometimes a Lyapunov function . Using V we consider the norm \|f\|_{V}:=\sup_{x\in E}{|f(x)|\over V(x)} for Borel measurable functions f and define the space B_{V} as the space of Borel measurable functions f with finite norm \|f\|_{V} . Similarly in the space of finite signed measures M(E) we consider the norm \|\nu\|_{V}:=sup_{f\in B_{V},\|f\|_{V}\leq 1}|\int_{E}f(x)\nu(dx)| . The condition ( 7 ) was introduced by Kartashov (see [ 11 ] and also [ 10 ] ) and has the following important consequences Lemma 1. If there is x^{*}\in E such that P_{n}^{(m),u}V(x^{*})<\infty then under (UEd) there is a unique invariant measure \mu_{m}^{u} for the Markov process (X_{n}^{(m),u}) and {\|P_{n}^{(m),u}(x,\cdot)-\mu_{m}^{u}(\cdot)\|_{V}\over V(x)}\leq\rho^{n}[1+{P_{1}^{(m),u}V(x^{*})+\rho V(x^{*})\over 1-\rho}]. (8) Proof. It follows from Theorem 7.3.14 of [ 10 ] . \Box Assume (FPV) we have \sup_{m}\sup_{x\in E}{P_{n}^{(m),u}V(x)\over V(x)}<\infty for each x\in E We immediately have Corollary 1. If \sup_{m}P_{1}^{(m),u}V(x^{*})<\infty for some x^{*}\in E then the bound in ( 8 ) is uniform with respect to m and consequently we have (FPV). Assuming furthermore (ERd) we have that \mu_{m}(\cdot) is a unique invariant measure for the process (X_{n2^{-m}}^{(m),u}) . Denote by P(E) the set of probability measures on E and let P_{V}(E):=\left\{\nu\in P(E):\|\nu\|_{V}<\infty\right\} . In what follows we shall need the following technical Lemma Lemma 2. Assume that for \nu_{n},\nu\in P_{V}(E) we have \|\nu_{n}-\nu\|_{V}\to 0 and for f_{n},f\in B_{V} with \|f_{n}\|_{V} bounded we have {f_{n}(x)-f(x)}\to 0 for each x\in E . Then \nu_{n}(f_{n})\to\nu(f) . Proof. Without loss of generality we may assume that \|f_{n}\|_{V}\leq 1 . Then also \|f\|_{V}\leq 1 and we have \displaystyle|\nu_{n}(f_{n})-\nu(f)|\leq|\nu_{n}(f_{n})-\nu(f_{n})|+|\nu(f_{n})-\nu(f)\|\leq \displaystyle\|\nu_{n}-\nu\|_{V}+|\nu((g_{n}-g)V)|\to 0 (9) as n\to\infty , with g_{n}={f_{n}\over V} , g={f\over V} and where the last convergence follows from the dominated convergence theorem. \Box Assume (Conv) for each u\in{\cal U} and x\in E we have \|P_{1}^{(m),u}(x,\cdot)-P_{1}^{u}(x,\cdot)\|_{V}\to 0 as m\to\infty . We have Proposition 1. Under (Conv) and (FPV) for each n\in N and x\in E we have \|P_{n}^{(m),u}(x,\cdot)-P_{n}^{u}(x,\cdot)\|_{V}\to 0 (10) as m\to\infty . Proof. We use induction. For n=1 ( 10 ) is satisfied by (Conv). Assume that we have ( 10 ) for n . Then by (FPV) we have that there is K\geq 0 such that \sup_{f\in B_{V},\|f\|_{V}\leq 1}\sup_{x\in E}{|P_{n}^{(m),u}(x,f)|\over V(x)}=\sup_{x\in E}{P_{n}^{(m),u}(x,V)\over V(x)}\leq K<\infty (11) and therefore \displaystyle\sup_{f\in B_{V},\|f\|_{V}\leq 1}|P_{n+1}^{(m),u}(x,f)-P_{n+1}^{u}(x,f)|\leq\sup_{f\in B_{V},\|f\|_{V}\leq 1}\left[|\int_{E}P_{n}^{(m),u}(y,f)(P_{1}^{(m),u}(x,dy)-\right. \displaystyle P_{1}^{u}(x,dy))|+|\int_{E}(P_{n}^{(m),u}(y,f)-\left.P_{n}^{u}(y,f))P_{1}^{u}(x,dy)|\right]\leq K\|P_{n}^{(m),u}(x,\cdot)- \displaystyle P_{n}^{u}(x,\cdot)\|_{V}+\int_{E}\|P_{n}^{(m),u}(y,\cdot)-P_{n}^{u}(y,\cdot)\|_{V}P_{1}^{u}(x,dy) and by induction hypothesis and dominated convergence we have that ( 10 ) for n+1 follows. \Box Using Proposition 1 to