Incident response metrics provide a way to quantify and track the effectiveness of an organization's incident response processes. By establishing and monitoring key metrics around incident volume, type, severity, response timeliness, and procedural compliance, security teams can identify weaknesses and drive continuous improvement. Tracking the right incident response metrics is essential for organizations of all sizes to minimize the impact of security incidents.

Where did this come from?

This control comes from the CSA Cloud Controls Matrix v4.0.10 - 2023-09-26, which can be downloaded at https://cloudsecurityalliance.org/artifacts/cloud-controls-matrix-v4. The Cloud Security Alliance developed this control as part of their comprehensive framework for cloud security assurance.

For additional context, the AWS Well-Architected Framework also emphasizes the importance of tracking metrics to evaluate the effectiveness of incident response in the Security Pillar whitepaper: https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/welcome.html

Who should care?

Incident response metrics are relevant for:

• Security analysts with responsibility for triaging and investigating security alerts
• SOC managers with accountability for overall incident response performance
• CISOs and security leadership with a need to report on the security posture
• Compliance officers with an obligation to ensure adherence to SLAs and procedures

What is the risk?

Without robust incident response metrics, organizations may struggle to:

• Detect systemic issues in alerting logic leading to missed incidents
• Investigate incidents efficiently, resulting in longer adversary dwell time
• Prioritize incidents effectively to reduce business impact
• Enforce a consistent, repeatable incident response process
• Justify security investments and demonstrate progress to leadership

However, a well-designed incident response metrics program can significantly improve an organization's ability to minimize incident impact and demonstrate the value of the security function.

What's the care factor?

The level of care and investment put into incident response metrics should be commensurate with the organization's overall reliance on technology and exposure to threat actors.

Organizations in high-risk industries like financial services, healthcare, and government should make incident response metrics a top priority. The same goes for organizations with a low risk tolerance or high public visibility.

However, even smaller organizations with limited security resources can benefit from basic incident response metrics to keep a pulse on their security program over time.

When is it relevant?

Incident response metrics are always relevant for organizations that experience any volume of security threats and incidents, which is virtually everyone these days.

The specific metrics and level of granularity tracked may vary based on the organization's risk profile, security program maturity, and available tooling. But every organization should track at least a basic set of metrics.

Incident response metrics become less relevant in environments with very little technology footprint or exceptionally low risk. For example, a small business with a simple static website and no sensitive data may be able to get by without formal incident response metrics.

What are the tradeoffs?

Implementing incident response metrics does require some investment:

• Time spent defining metrics and building data collection mechanisms
• Effort to regularly review and analyze metrics to derive insights
• Potential friction from incident responders who may see metrics as a burden
• Opportunity cost of focusing on metrics over other security initiatives

It's important to design metrics that are easy to collect and analyze with automated tools where possible. Metrics should help drive efficiency rather than become a bottleneck.

Start small and simple, then iteratively add more sophisticated metrics over time. Don't let perfect be the enemy of good.

How to make it happen?

Here's a step-by-step approach to implementing incident response metrics:

1. Define objectives - Determine what you want to measure and why. Focus on metrics that will drive action and improvement.
2. Establish a baseline - Analyze historical incident data to set a performance baseline for key metrics. This gives you a starting point to measure against.
3. Select tools - Identify tools to help collect and analyze the necessary data. Cloud-native tools like AWS Security Hub, Amazon GuardDuty, and AWS Config can provide much of the data you need.
4. Configure data collection - Set up your tools to collect the raw data needed for your metrics. This may involve enabling certain logging or automated security scans.
5. Build dashboards - Use a tool like Amazon QuickSight to build dashboards that automatically calculate your metrics and present them in an easy to consume format.
6. Schedule reviews - Set a regular cadence (e.g. monthly) to review metrics with key stakeholders. Discuss trends, outliers, and action items.
7. Implement improvements - Update processes, tools, training, staffing etc. based on the insights gained from your metrics. The goal is to drive continuous improvement.
8. Iterate - Revisit your metrics on a periodic basis to evaluate their usefulness and make adjustments as needed. Metrics should evolve along with your incident response program.

What are some gotchas?

Here are a few things to watch out for when implementing incident response metrics:

• Metric fatigue - Be selective about the metrics you track. Trying to measure too many things can lead to overload and lack of focus.
• Garbage in, garbage out - Your metrics are only as good as the data you collect. Be diligent about data quality.
• Perverse incentives - Be careful not to create metrics that incentivize the wrong behaviors. For example, measuring the number of incidents closed could lead to responders rushing and closing incidents prematurely.
• Lack of context - Metrics can be misleading without proper context. A small number of severe incidents may be more concerning than a large volume of low severity incidents. Always strive to understand the "why" behind the metrics.
• Permissions required - Security tools need permissions to collect the necessary data for metrics. Exactly what will depend on your tools. The CloudTrail log delivery IAM policy is a good place to start: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-required-policy-for-cloudwatch-logs.html

What are the alternatives?

While there is no direct substitute for incident response metrics, there are complementary practices:

• Penetration testing and red teaming - Proactive simulation of real world attacks to stress test detection and response. See https://attack.mitre.org
• Incident response playbooks - Step-by-step guides for responding to common incident types. Playbooks complement metrics by providing prescriptive guidance.
• Customer feedback - Soliciting feedback from customers or users impacted by incidents. More subjective but provides valuable perspective.

Explore further

• CIS Top 18 Critical Security Controls: https://www.cisecurity.org/controls/cis-controls-list
• NIST Special Publication 800-61: Computer Security Incident Handling Guide: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r2.pdf

‍