In this thesis, the theory of lumpability (strong lumpability and weak lumpability) of irreducible finite Markov chains was studied. The most important necessary and sufficient condition for a Markov chain to be strongly lumpable (Kemeny and Snell in 1976) is there are matrices U and V such that UV PV = PV, where P is the one-step transition matrix of the Markov chain. A necessary and sufficient condition for a Markov chain to be weakly lumpable is under some condition there are matrices U π and V such that U π PVU π = U π P , where π is the steady state probability vector.
Inspired by the theory of strong lumpability, we provide guidance on how to partition the state space in a more efficient way and defined measures on how to measure the closeness of the aggregated chain to a chain with the Markov property. Combining the guidance and the measures, a new method called lumpability approximation is proposed.
We applied the lumpability approximation methods on solving steady state probability vectors of irreducible discrete time finite Markov chains with a combination with the iterative aggregation and disaggregation. From the analysis on the computational complexity and experimental results, we show that the new method converges faster with accuracy as good as the direct methods and the simple iteration method.
We also applied the lumpability approximation methods on solving Markov decision problems in infinite horizon. A Markov decision process (MDP) is generalized when Markov chains are used in decision making. Whenever the Markov chain is observed in a given state, a decision maker chooses one of finitely many actions available in that state and earns a reward or encounters a cost depending on the given state and the action chosen. The decision maker is willing to seek maximum reward or minimum cost in finite or infinite horizon under certain criteria. We show that the new method converges to the optimal value function faster with accuracy as good as the value iteration method.
