Business continuity and operational resilience plans are critical components of an organization's risk management strategy. These plans outline the procedures and resources needed to maintain essential business functions during and after a disruptive event. To ensure these plans remain effective, it's vital to regularly exercise and test them.

Where did this come from?

This control comes from the CSA Cloud Controls Matrix v4.0.10 - 2023-09-26. You can download the full matrix here. The Cloud Controls Matrix provides a comprehensive set of security controls aligned to leading standards, regulations, and best practices. It's a great resource for organizations looking to improve their cloud security posture.

For more background on business continuity in the cloud, check out the AWS Business Continuity Technical Guide. It provides an overview of core concepts as well as best practices for implementing resilient systems on AWS.

Who should care?

This control is relevant for several roles:

• Business continuity managers responsible for developing and maintaining BC/DR plans
• IT operations teams who execute failover procedures during an incident
• Compliance officers that need to demonstrate BC/DR capabilities to auditors and regulators
• Senior executives accountable for minimizing business disruption

What is the risk?

Failing to regularly exercise business continuity plans creates risk of:

• Critical personnel being unprepared to carry out their duties during a crisis
• Plans containing outdated or incorrect information (contact details, recovery procedures, etc.)
• Technical issues arising when executing failover/failback not previously identified
• Longer than necessary outages while teams scramble to figure out what to do

The impact obviously depends on the nature of the business, but could range from inconvenience and reputation damage, to data loss and significant financial costs. Regular exercising helps identify gaps and issues in a controlled manner.

What's the care factor?

For any organization that relies on IT systems to deliver products and services to customers, I'd rate the care factor as high. While it requires time and effort, having battle-tested BC plans that you're confident will work is invaluable when a real disaster strikes. It's really a business decision - how much risk and downtime can you afford? Highly regulated industries like financial services will have a greater imperative than others.

When is it relevant?

Exercising BC plans is most relevant for:

• Mission-critical workloads and systems
• Customer-facing applications where downtime immediately translates to lost revenue
• Regulated environments with strict uptime and recovery obligations

It's less relevant for:

• Non-production environments like dev/test
• Stateless, easily recreatable infrastructure
• Workloads designed with inherent resilience (e.g. serverless, multi-region active-active)

What are the trade offs?

The main costs to factor in are:

• The time spent by various teams planning and executing the tests (diverting them from other work)
• Potential lost productivity for staff involved in exercises
• Possible degraded performance or brief outages in production (if doing live failover)

How to make it happen?

1. Identify workloads and systems in scope for exercising based on criticality
2. Determine suitable interval (e.g. annually) and test scenarios (e.g. data center outage)
3. Develop step-by-step runbooks mapping out recovery procedures
4. Schedule tests in low-risk business hours with all required personnel available
5. For each workload, methodically execute recovery steps and verify operation
6. Fail back to original state
7. Document issues encountered, lessons learned and update plans accordingly

Best to start small with tabletop walkthroughs and gradually work up to live hands-on-keyboard failovers as confidence increases.

What are some gotchas?

A few things to watch out for:

• Ensure you have a complete picture of application dependencies (e.g. databases, caching layers, 3rd party integrations). Miss one and your failover likely won't work.
• Test data replication/synchronization separately. Restoring services is one thing, restoring current data is another.
• Careful with stateful systems - failing over a database and then failing back without due care can cause data corruption.
• Access and permissions - you'll need to make sure the right users have the right entitlements to perform recovery actions. Required AWS IAM permissions could include:
  • EC2 permissions like ec2:StartInstances, ec2:StopInstances for manipulating instances
  • RDS permissions like rds:FailoverDBCluster for database recovery

What are the alternatives?

Alternatives and complimentary approaches to traditional BC/DR exercises include:

• Chaos engineering - deliberately injecting failure into systems to build resilience
• Game days - organized events that simulate real-world incidents to practice response
• Penetration testing and red team exercises to stress systems and find weaknesses
• Leveraging cloud provider resilience features like AWS Availability Zones and cross-region replication

Explore Further

• NIST SP 800-34 Rev. 1 Contingency Planning Guide for Federal Information Systems
• CIS Controls v8 - Control 11: Data Recovery
• AWS re:Invent talk: Backup-and-restore and disaster recovery with AWS

‍