In the world of cloud computing, it's crucial to keep your production and non-production environments separate. This means having distinct environments for development, testing, and production, each with its own set of resources, access controls, and data. By maintaining this separation, you can minimize the risk of accidental changes or data leaks that could impact your live systems.

Where did this come from?

CSA Cloud Controls Matrix v4.0.10 - 2023-09-26. You can download the full matrix here.

The Cloud Security Alliance (CSA) is a leading organization that promotes best practices for secure cloud computing. Their Cloud Controls Matrix (CCM) provides a comprehensive set of controls that cover various aspects of cloud security. The control we're discussing here, IVS-05, falls under the "Infrastructure & Virtualization Security" domain.

For more information on AWS best practices for separating environments, check out the AWS Multi-Account Strategy documentation.

Who should care?

• DevOps engineers with responsibility for managing cloud environments
• Security architects with a need to ensure proper separation of environments
• Compliance officers with a requirement to demonstrate environment segregation

What is the risk?

Failing to properly separate production and non-production environments can lead to several risks:

• Accidental changes in production: Developers or testers might inadvertently modify live data or configurations, causing service disruptions or data corruption.
• Data leakage: Sensitive production data could be exposed in lower environments if not properly sanitized.
• Compliance violations: Many regulations (e.g., PCI DSS, HIPAA) require strict separation of environments.

Implementing IVS-05 can significantly mitigate these risks by enforcing clear boundaries between environments.

What's the care factor?

For most organizations, separating production and non-production environments should be a top priority. The consequences of not doing so can be severe:

• Service outages causing reputational damage and financial losses
• Data breaches leading to regulatory fines and loss of customer trust
• Failed audits resulting in loss of business or even legal action

The effort required to properly segregate environments is generally well worth the peace of mind it provides.

When is it relevant?

Separating production and non-production environments is relevant in most situations where an organization is running cloud workloads. This is especially true for:

• Enterprises with strict compliance requirements
• Organizations handling sensitive data (e.g., financial, health, personal information)
• Businesses with a low tolerance for service disruptions

There may be some cases where full separation is less critical, such as:

• Early-stage startups with a single small team and no sensitive data
• Purely static websites with no backend processing or data storage
• Proof-of-concept or demo environments not handling real data

What are the trade offs?

Implementing environment separation does come with some costs and considerations:

• Additional resources: Separate environments mean more cloud resources to provision and manage, which can increase costs.
• Complexity: Managing multiple environments adds operational complexity, especially around promoting changes between environments.
• Potential for drift: With separate environments, there's a risk of configurations drifting out of sync over time if not managed carefully.
• Longer development cycles: The process of moving changes through multiple environments can slow down development velocity if not streamlined.

However, for most organizations, these trade-offs are outweighed by the security and stability benefits of proper environment separation.

How to make it happen?

Here's a high-level process for separating production and non-production environments in AWS:

1. Create separate AWS accounts for each environment (e.g., dev, test, prod). This provides the strongest isolation.
2. Within each account, create separate VPCs for each environment. This adds network-level separation.
3. Use IAM roles and policies to enforce strict access controls. Developers should have no access to prod, and prod admins should have limited access to lower environments.
4. Implement network ACLs and security groups to control traffic between environments. Use a deny-all default and only allow necessary ports/protocols.
5. For data sanitization between prod and non-prod:
   • Identify sensitive data elements
   • Define sanitization rules (e.g., randomize values, null out fields, age dates)
   • Use tools like AWS Glue or Lambda to apply sanitization during data copying
6. Implement change management processes to review and approve all production changes. Use tools like AWS CodePipeline to automate promotion between environments.
7. Monitor and audit environment access and changes using tools like AWS CloudTrail and AWS Config.

What are some gotchas?

• IAM permissions: Ensure IAM policies are scoped down to the specific resources and actions needed for each environment and job role. Avoid broad * permissions.
  • Devs will need permissions like ec2:StartInstances, ec2:StopInstances in dev/test, but not prod.
  • Prod admins will need ec2:*, s3:*, etc. in prod, but read-only in lower envs.
  • See AWS docs on IAM best practices.
• VPC Peering: If you need to connect environments, use VPC peering with strict routing tables, not direct VPN or AWS Direct Connect.
• RDS Snapshots: Be careful when using manual RDS snapshots to copy data from prod to non-prod. Ensure snapshots are encrypted and access is limited.
• AMI Sharing: If using custom AMIs, share them only from prod to lower envs, never the other way around.

What are the alternatives?

While separate AWS accounts per environment is ideal, some alternatives include:

• Using separate AWS Organizations OUs for each environment, with SCP policies to enforce isolation
• Implementing third-party tools like HashiCorp Sentinel for policy-as-code guardrails across environments
• For smaller workloads, using logical naming conventions and strict IAM policies to isolate within a single account (not recommended for sensitive workloads)

Explore further

• AWS Multi-Account Strategy: https://aws.amazon.com/organizations/getting-started/best-practices/
• AWS Organizations Best Practices: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_best-practices.html
• CIS AWS Foundations Benchmark (see section 1.20 on segregating duties within AWS accounts): https://www.cisecurity.org/benchmark/amazon_web_services/

The IVS-05 control aligns with CIS Control 13 ("Segregation in Networks") and CIS Control 14 ("System Configuration Management") from the CIS Critical Security Controls framework.

‍