CCC-03 Change Management Technology

Summary

Managing change effectively is critical for maintaining security in cloud environments. The Cloud Security Alliance's Cloud Controls Matrix specifies the importance of implementing rigorous change management processes and technologies. This article will explore CCM Control CCC-03 which provides guidance on managing the risks associated with changes to applications, systems, infrastructure and configurations.

Where did this come from?

This control comes from the CSA Cloud Controls Matrix v4.0.10 released on 2023-09-26. You can download the full Cloud Controls Matrix here.

The goal of the CSA CCM is to provide fundamental security principles to guide cloud vendors and assist prospective cloud customers in assessing the overall security risk of a cloud provider. AWS has also published a whitepaper on change management best practices.

Who should care?

This control is relevant to:

• Cloud engineers responsible for designing and implementing cloud architectures
• DevOps teams deploying application changes frequently
• Security professionals assessing change-related risks
• IT managers overseeing cloud migration projects
• Compliance officers ensuring adherence to regulatory requirements around change management

What is the risk?

Poor change management practices can lead to:

• Unintended downtime or performance degradation of production systems
• Introduction of security vulnerabilities due to misconfiguration
• Compliance violations if required approvals are bypassed
• Inability to rollback problematic changes resulting in prolonged outages
• Lack of accountability and audit trails for changes

The likelihood and impact of these risks depends on factors like the complexity of the environment, velocity of changes, maturity of processes, and sensitivity of impacted systems. CCC-03 aims to mitigate these risks through the adoption of change management technologies.

What's the care factor?

For any organization operating in the cloud, implementing effective change management should be a high priority. Even seemingly minor misconfigurations, like leaving S3 buckets open to the public, can have severe consequences. The 2017 Equifax data breach, which exposed 147 million records, was traced back to the failure to patch a known vulnerability - something sound change management could have prevented.

Furthermore, many compliance frameworks like HIPAA, PCI DSS and ISO 27001 have strict requirements around change management. Non-compliance can result in financial penalties and reputational damage.

That said, the level of rigor and technological automation will vary based on the organization's risk appetite, scale of operations and regulatory obligations. A small startup may rely on well-documented manual processes while a large enterprise would require extensive automation.

When is it relevant?

Change management technologies are applicable whenever changes are being made to production cloud environments. This includes:

• Provisioning or terminating cloud resources
• Deploying application updates
• Modifying network configurations
• Applying security patches
• Changing access controls

However, these technologies may be less relevant or cost-effective in certain situations such as:

• Isolated dev/test environments
• Fully automated deployments with robust testing
• Legacy applications with infrequent updates
• Very small cloud footprints

What are the trade offs?

Implementing change management technologies requires an upfront and ongoing investment in:

• Procuring and configuring the tools
• Training staff on their usage
• Defining and enforcing change workflows
• Maintaining audit logs

This overhead can slow down deployments which may frustrate development teams under pressure to deliver new features quickly. Overly bureaucratic approval processes can be a hindrance.

There are also costs associated with cloud resources to host these tools and store audit logs long-term.

However, these costs need to be weighed against the potentially far greater expense of a major incident or regulatory fine resulting from undisciplined change practices. Effective change management is foundational to operating a secure and stable cloud environment.

How to make it happen?

Here's a high-level plan for implementing change management technologies in an AWS environment:

1. Define change management policies covering roles and responsibilities, approval workflows, rollback procedures and audit requirements.
2. Select a suitable change management tool. AWS provides several options:
   • AWS Systems Manager Change Manager for managing changes to AWS resources
   • AWS CloudFormation for provisioning infrastructure using templates
   • AWS CodePipeline for automated release pipelines
3. Configure the tool with your defined workflows. For example, in Change Manager:
   • Set up change templates defining required approvals and notification
   • Define change freezes to prevent changes during critical periods
   • Integrate with ITSM tools like ServiceNow
4. Restrict permissions to ensure all changes must flow through the change management tool:
   • Use AWS IAM to control who can make changes
   • Disable manual changes in production environments
   • Consider multi-account setups to separate prod and dev access
5. Develop a training and communication plan to educate teams on the new processes and tools.
6. Start with low-risk changes and incrementally expand scope as the process matures.

What are some gotchas?

Some things to watch out for when implementing change management in AWS:

• Ensure IAM permissions are tightly scoped so the tools can perform required actions but not allow privilege escalation. Key permissions to manage are:
  • iam:* for managing IAM roles, policies etc
  • sts:* for assuming IAM roles
  • cloudformation:* for provisioning stacks of resources
  • Service-specific permissions like ec2:*, rds:*, etc
• When integrating with existing enterprise tools, ensure consistent workflows are enforced. For example, if a CAB approval is required, this should be configured in the AWS change process.
• Don't set overly broad change windows that effectively bypass approval. Balance control with efficiency.
• Use AWS CloudTrail to log all management events including change approvals for auditing purposes. Ensure these logs are protected against tampering.
• Monitor for unauthorized out-of-band changes that bypass the change control process. Tools like AWS Config can help with this.
• Regularly review and optimize the change workflow based on metrics and feedback.

What are the alternatives?

Other options for change management include:

• Fully manual ITSM-driven processes - may be appropriate for very small, stable environments
• Gitops - Define infrastructure and application configs as code in Git repos, with automated deployment on merge to master. Relies heavily on automated testing.
• CI/CD pipelines - Tools like Jenkins, CircleCI, Azure DevOps can include manual approval gates as part of an automated release process.

Explore further

• CIS Benchmark 1.16 recommends implementing approval processes for IAM customer-managed policy creation to prevent overly permissive access.
• AWS whitepaper on CIS Benchmark implementation
• Tutorial on using Systems Manager Change Manager
• AWS presentation on Change Manager

‍