MCS, a student is in one of three Markov states:

0:sleeping, 1:eating, or 2:reading. (To complete this assignment, you probably

should review Problem 5.5 and its solution in the back of the MCS.) With

respect to sleeping, Problem 5.5 says

Sleeping is always followed by eating. Each time the student

goes to sleep (takes a nap), the average nap time is 4 hours.

Because sleeping (state 0) is a Markov state, at the end of each hour, the

next state is given by the transition probabilities

since \sleeping is always followed by eating,” we know

So, at the end of each hour, the student switches from sleeping to eating

with probability

random variable with success probability

time spent sleeping is 1

P2

The time spent \sleeping” is also known as the

state. In general for any Markov chain, since the system departs state

probability 1

variable with expected value 1

This review of Problem 5.5 is just to introduce the following variation on

Problem 5.5:

reading, such that

with probability 0.1; otherwise, the student begins to read.

sleep (takes a nap), the nap time is

probable than an hour of eating. However, the probability of going to

sleep equals the sum of the probabilities of reading or eating.

Just as in Problem 5.5, the time step is still 1 hour. For this modified

problem:

(a) What is the Markov chain? Hint: The Markov chain needs to keep

track of how many hours the student has slept (or how many hours of

sleep the student has remaining.)

(b) What are the stationary probabilities?

(c) After a very long time, what is the probability the student is sleeping?

Once you solve this problem, now consider the following generalization:

0:sleeping, 1:eating, or 2:reading. (To complete this assignment, you probably

should review Problem 5.5 and its solution in the back of the MCS.) With

respect to sleeping, Problem 5.5 says

Sleeping is always followed by eating. Each time the student

goes to sleep (takes a nap), the average nap time is 4 hours.

Because sleeping (state 0) is a Markov state, at the end of each hour, the

next state is given by the transition probabilities

*P*00,*P*01 and*P*02. Next,since \sleeping is always followed by eating,” we know

*P*01*>*0 but*P*02 = 0.So, at the end of each hour, the student switches from sleeping to eating

with probability

*P*01. Hence the time (in hours) spent sleeping is a geometricrandom variable with success probability

*P*01. This implies that the averagetime spent sleeping is 1

*=P*01. Since the problem statement says that the*average*nap time is 4 hours, we learn that 1*=P*01 = 4, or*P*01 = 1*=*4. SinceP2

*j*=0*P*0*j*= 1, these facts imply*P*00 = 3*=*4.The time spent \sleeping” is also known as the

*sojourn time*in the sleepingstate. In general for any Markov chain, since the system departs state

*i*withprobability 1

*–**P**ii*, the sojourn time in any state*i*is a geometric randomvariable with expected value 1

*=*(1*–**P**ii*).This review of Problem 5.5 is just to introduce the following variation on

Problem 5.5:

**Problem 1:**Consider a student who is always either sleeping, eating orreading, such that

*â€¢*If the student is eating, the student continues to eat for another hourwith probability 0.1; otherwise, the student begins to read.

*â€¢*Sleeping is always followed by eating. Each time the student goes tosleep (takes a nap), the nap time is

*exactly*4 hours.*â€¢*After an hour of reading, another hour of reading is four times moreprobable than an hour of eating. However, the probability of going to

sleep equals the sum of the probabilities of reading or eating.

Just as in Problem 5.5, the time step is still 1 hour. For this modified

problem:

(a) What is the Markov chain? Hint: The Markov chain needs to keep

track of how many hours the student has slept (or how many hours of

sleep the student has remaining.)

(b) What are the stationary probabilities?

(c) After a very long time, what is the probability the student is sleeping?

Once you solve this problem, now consider the following generalization: