Handout #10, Markov Chains, an example, continued

Given the same diagram of cities and transition probability matrix:

We want to see if the aperiodic, irreducible chain is stable.� In other words, does it converge to some limiting distribution?

The probability vector, p, is just the limit of the probability distribution vector, assuming the system is stable.

So, can we find a p that will satisfy:

�������������������������������������

In the example above:

�������������������������������������

(There's three states.)

We also know,

So, we can solve the equations above:

�

So, the equilibrium (stationary) state probability is:

�������������������������������������������������

What is the equilibrium (stationary) state probability of (2.233), page 83?

,

So, we get

�or

Solving:

So, we get the same answer that is on page 86:

�������������������������������������������������������������������

What is the equilibrium (stationary) state probability of 2.9.1.

,

So, we get

�or

Solving:

��� ���

�������������������������������������������������������������������������������������

I gave this as a programming assignment in Fall 2000!

Find:

� The state transition diagram.

� The conditions that will make the following chain irreducible and aperiodic.

� The equilibrium (stationary) state probability.

,

State Diagram:

Irreducible:

To be irreducible, all must be able to communicate with each other.�

Therefore, 0 < b (state 3 communicates with states 1 & 2), and 0 < a (state 1 can communicate with state 3).�

Aperiodic:

If a = 1 and b = 1 then the chain is periodic with period t = 3.

Therefore, only one or the other can be equal to 1:

0 < a and 0 < b and either a < 1 and b � 1 or a � 1 and b < 0.

The equilibrium (stationary) state probability:

Solving more:

�������������������������������������������������������������������������