The *one-step transition probability* is the probability of transitioning
from one state to another in a single step.
The Markov chain is said to be time homogeneous if the transition
probabilities from one state to another are independent of time index .

The *transition probability matrix*, , is the matrix consisting of
the one-step transition probabilities, .

The *-step transition probability* is the probability of transitioning
from state to state in steps.

The -step transition matrix whose elements are the -step transition probabilities is denoted as .

The -step transition probabilities can be found from the single-step transition probabilities as follows.

To transition from to in steps,
the process can first
transition from to in steps, and then
transition from to in steps, where .

In matrix form, this becomes:

Setting yields:

From this equation we can see that:

Substituting this back into the previous equation yields:

Continuing these substitutions, eventually we have:

Therefore, the -step transition probability matrix can be found by multiplying the single-step probability matrix by itself times.

The state vector at time can also be found in terms
of the transition probability matrix and the intial state
vector . We first observe that:

In vector and matrix form, this becomes:

We also find that, through substitution:

or,

Continuing the substitution yields:

where is the vector containing the intial probabilities of being in each state at time 0.