Testing changes thoroughly before pushing them to production is super important for keeping everything running smoothly and securely. Catching bugs early means fewer headaches later. By sticking to a solid quality assurance process, you can make updates with way more confidence.
Where did this come from?
This control comes from the CSA Cloud Controls Matrix v4.0.10, released on 2023-09-26. You can grab the full Cloud Controls Matrix from Cloud Security Alliance to dive deeper. This specific control, CCC-02, is all about quality testing as part of managing changes and configurations.
Who should care?
- DevOps Engineers: You're deploying changes to production.
- QA Testers: You're making sure changes meet requirements and don't break anything.
- Release Managers: You're coordinating deployments and making go/no-go calls.
- IT Managers: You're overseeing the change management process.
- Auditors: You're assessing how mature the organization's change control process is.
What is the risk?
If you skimp on testing, you might face:
- Buggy code that causes errors or outages.
- Security gaps that could lead to breaches.
- Slow performance that frustrates users.
- Broken features that need urgent fixes.
- Compliance issues due to poor auditing.
Testing can't eliminate these risks entirely, but it can definitely reduce them. Fixing issues before they hit production is way less costly and disruptive.
What's the care factor?
DevOps teams should be all about quality testing. In the fast-paced world of CI/CD, "move fast and break things" isn't the best approach. Companies want quick updates without sacrificing stability and security. Putting time into automating tests and improving QA processes pays off by letting you move fast with confidence. Don�t skimp on testing.
When is it relevant?
Quality testing matters whenever you're changing code, configuration, or infrastructure, like:
- Developing new features.
- Upgrading frameworks or dependencies.
- Patching servers or containers.
- Changing network or IAM configurations.
- Updating database schemas.
Not every change needs the same level of testing. Minor bug fixes might need less attention than major updates. Have clear policies on how much testing is needed for different types of changes.
What are the trade-offs?
Thorough testing takes time and effort, which can slow down deployments, especially if done manually. Other downsides might include:
- Developers feeling bogged down by detailed testing requirements.
- Innovation taking a backseat to rigorous testing.
- Unrealistic goals for 100% test coverage in complex systems.
- Ongoing work to keep test cases relevant as systems evolve.
But these costs are generally outweighed by the benefits of avoiding major incidents. Automate testing as much as possible to keep things efficient.
How to make it happen?
Here�s a basic roadmap for implementing quality testing:
- Define your testing requirements and get everyone on board.
- Identify and prioritize key test cases.
- Automate tests whenever you can (unit, integration, UI tests).
- Use CI/CD quality gates to control deployments.
- Do manual exploratory and edge case testing alongside automation.
- Conduct load and security testing before major releases.
- Document test plans and results for auditing and sharing knowledge.
- Track test coverage metrics to see how effective your testing is.
- Build a culture where quality and shared responsibility for testing are a priority.
What are some gotchas?
- Make sure testers and devs have the right permissions in CI/CD and test environments.
- Testing with prod data might need obfuscation and strict controls to prevent leaks.
- Be careful about testing in production to avoid impacting users.
- Avoid creating fragile tests that break easily. Design for maintainability.
- Educate devs on testing best practices and provide examples and templates.
- Regularly review quality gates to ensure they're still useful.
What are the alternatives?
While automated testing in preproduction environments is ideal, other methods can also help:
- Canary Testing: Roll out changes to a small % of users first and monitor for issues.
- Feature Flags: Enable new features for select users to get early feedback.
- Chaos Engineering: Introduce failures on purpose to test resilience.
- Bug Bounties: Incentivize external researchers to find vulnerabilities.
- A/B Testing: Compare different versions of a feature to see which performs better.
Explore further
- ISO/IEC/IEEE 29119 Software Testing Standard
- NIST Special Publication 800-53 "SI-11 Error Handling"
- OWASP Testing Guide
- PCI DSS v3.2.1 Requirement 6
- Related CIS Controls:
- 8.2 Activate audit logging
- 16 Application Software Security
?
