Having a solid change management policy and procedure is essential for keeping your organization's assets secure and stable. It ensures that any changes made to applications, systems, infrastructure, or configurations are properly tested, documented, risk assessed, and authorized before being implemented. This helps prevent unexpected issues and downtime.
Where did this come from?
CSA Cloud Controls Matrix v4.0.10 - 2023-09-26. You can download the full matrix here: https://cloudsecurityalliance.org/artifacts/cloud-controls-matrix-v4
This control (CCC-01) is part of the Change Control and Configuration Management domain. It outlines the importance of establishing policies and procedures to manage the risks associated with changes. For more context, check out the AWS docs on change management best practices.
Who should care?
- IT managers responsible for maintaining system stability
- DevOps engineers implementing changes and new features
- Security officers assessing risk of system modifications
- Compliance auditors ensuring proper change control
What is the risk?
Without a robust change management process, several things can go wrong:
- Untested changes lead to bugs, performance issues, or system outages
- Unauthorized modifications create security vulnerabilities
- Lack of documentation makes troubleshooting and auditing difficult
- Skipped risk assessments allow high-impact problems to slip through
While having proper policies won't eliminate issues entirely, they significantly reduce the likelihood and severity of change-related incidents. Catching problems before production is key.
What's the care factor?
For any organization that values system reliability and security, change management should be a top priority. Even a minor uncontrolled change by a well-meaning employee can spiral into extended downtime.
The larger and more complex your IT environment, the higher the stakes. Enterprises with rigorous uptime SLAs, compliance requirements, or sensitive data can't afford to be lax about changes.
When is it relevant?
Change management policies should be applied to any significant modification to production systems, such as:
- Application code deployments
- OS/software patch and upgrade installations
- Network configuration updates
- Adding/removing servers or infrastructure
The exception is emergency fixes for ongoing major incidents. Have a streamlined process for those, but still require prompt retrospective documentation and approvals.
Lower environments like dev/test may not need as strict of controls. Use discretion based on risk.
What are the trade-offs?
Proper change management takes time and effort:
- Extensive testing catches more bugs but slows feature velocity
- Rigorous approval processes reduce agility and frustrate engineers
- Generating all the required documentation is tedious
There's a balance between gatekeeping and guardrails. Too much red tape discourages innovation. But cut corners and you'll pay the price in incidents later.
Work to automate as much of the process as possible (e.g. automated testing, integrated approval workflows, change templates). The smoother the friction, the easier adoption will be.
How to make it happen?
- Designate change management policy owner(s)
- Draft policy covering:
- Scope - what systems/assets are covered
- Roles & responsibilities
- Required steps (e.g. test, document, review, approve)
- Normal vs. emergency change workflows
- Define detailed procedures for:
- Submitting change requests
- Conducting risk assessments
- Documenting test plans and results
- Collecting approvals from stakeholders
- Scheduling and communicating changes
- Rolling back if needed
- Conducting post-change reviews
- Setup support tools like:
- Change ticketing system (e.g. Jira, ServiceNow)
- Automated testing frameworks
- CI/CD pipelines (e.g. Jenkins, GitLab)
- Monitoring and alerting for swift rollbacks
- Train engineers on policy and procedures
- Review and update at least annually
What are some gotchas?
- Configuration management tools need broad permissions like
ec2:*to fully discover and manage assets. Tightly scope and protect the credentials. - Beware of powerful APIs
- Creating clear policies is important but the human element is most critical. Focus on training, communication, and building a culture of responsible change.
What are the alternatives?
Rather than lengthy manual approval processes, look into:
- Automated security testing and enforcement with every code push
- Immutable infrastructure - never change live systems, only replace
- Feature flags to safely roll out changes to subsets of users
- Blameless post-incident reviews to learn without discouraging transparency
