Chapter 17 Summary
Markov Decision Processes and Its Applications in Healthcare
Chapter Author
Jonathan Patrick - University of Ottawa
Mehmet A. Begen - University of Western Ontario
Some Commentary
A Markov Devision Process may be the right tool, when there is a question involving uncertainty and sequential decision making.
This chapter is abridged to leave the math modelling out.
1. Queues an Introduction
In healthcare we frequently deal with incomplete information. For instance, we do not know exactly how long an operating room will be needed for, or how many days a patient needs to recover, until these events happen.
Sequential decision problems (SDP) - are multiple step scenarios, where each steps becomes contingent upon the decision made in the prior step.
Markov decision processes (MDP) - is a mathematical process that tries to model sequential decision problems.
5 components of a Markov decision process
1. Decision Maker, sets how often a decision is made, with either fixed or variable intervals.
2. The state is the decision to be tracked, and the state space is all possible states
3. Each possible state has a set of potential actions,
4. Transition probabilities estimate the chance a state will be visited based on the prior decisions
5. Every state may result in a reward or a cost, a good or a bad decision, these can be calculated.
Some Examples
- hospital admission scheduling
- booking problems (if the patient is booked today, or tomorrow, it impacts who can be booked next, but there still has to be availability of the device in case a high priority patient arrives randomly). A model that places patients into different priority groups, and assigns a standard booking date range of that priority is suggested.)
- OR surgery scheduling of elective and emergent surgeries
- MRI booking
- patient decision making
- when to start HIV treatment
- when to start statin therapy
- live-liver transplantation schedules