Contingency Planning Isn't Fun and Games. But Should it Be?

“Hope for the best, plan for the worst”. This ancient principle still applies, especially for systems with high availability requirements. Principles are easy to quote, but how does an organization implement them effectively?

In its vast compendium of requisite security controls, NIST has created an entire category of requirements for contingency planning. Federal agencies are required to have a contingency plan, but keeping the plan up-to-date, and acting on it in an emergency, are difficult. Relevant personnel – system administrators, system owners, data owners, management, and others – must all go through training to ensure they know the contingency plan, and can carry it out.

This training usually consists of a simulated or “desktop” exercise, in which the participants talk through how they would react to a hypothetical emergency. While this approach is useful, participants often find it somewhat boring, and want to finish the exercise as soon as possible.

One time, however, I worked with a client who wanted to make the training more effective, and looked to my team to make it engaging and exciting, rather than a burdensome requirement. We went to work, started to brainstorm, and came up with an idea: we turned it into a game, complete with a physical board, cards like those in Monopoly, and a scoring system.

The game pitted two teams against each other in getting “back to business” in an imaginary emergency. One team would pick a “contingency” card, and then they would take turns carrying out requisite actions commensurate with the contingency plan. Points were awarded (or taken away) for compliance with the plan, and we added an element of (bad) luck as well. As teams progressed around the board, they might have to draw a “gremlin” card, which would subject them to a real world, unforeseen occurrence: “the truck carrying the backup tapes went to the wrong address”; “the department head had a car accident, and cannot be reached in the hospital”, and so on.

Teams were very excited and motivated to win, and when the game was over, the losing team wanted to do it again, to have a chance to win. This response was roughly equivalent to people asking to do a fire drill a second time, because the first one was so much fun. The gamification approach was not frivolous; it truly worked, in a serious way, to make participants better able to respond to emergencies and system outages.