Application security metrics are a critical tool for aligning technical security activities with business objectives, security requirements, and compliance obligations. By defining and tracking the right metrics, organizations can gain visibility into the effectiveness of their application security program, identify areas for improvement, and demonstrate progress to stakeholders. This article will explore why application security metrics matter, when they are relevant, and how to implement them effectively.

Where did this come from?

This article is inspired by the Cloud Security Alliance (CSA) Cloud Controls Matrix v4.0.10, released on 2023-09-26. The Cloud Controls Matrix is a cybersecurity control framework for cloud computing that provides a detailed understanding of security concepts and principles aligned to the Cloud Security Alliance guidance in 13 domains. The specific control related to application security metrics is AIS-03.

Who should care?

Application security metrics are relevant to several key roles:

• CISOs and security leaders who need to demonstrate the effectiveness of the application security program to executive leadership and the board
• Application security managers responsible for overseeing vulnerability management, penetration testing, and other security activities throughout the SDLC
• Developers and engineering leaders who want to understand the security posture of their applications and prioritize remediation efforts
• Compliance and audit professionals who need to verify that the organization is meeting its security and compliance obligations

What is the risk?

Without effective application security metrics, organizations may struggle to:

• Identify and prioritize the highest-risk vulnerabilities across the application portfolio
• Verify that critical security activities like penetration testing and security training are being completed on schedule
• Detect negative trends, like an increasing number of high-severity bugs being introduced into production
• Justify investments in application security tools, processes, and headcount
• Demonstrate compliance with relevant standards and regulations

The potential consequences include increased risk of data breaches and security incidents, failed audits, fines and penalties, and reputational damage.

What's the care factor?

For most organizations, application security metrics should be a high priority. Attacks targeting applications are extremely common, and the average cost of a data breach was $4.35 million in 2022 according to IBM.

However, the appropriate level of investment will vary based on factors like:

• The criticality and sensitivity of data processed by the organization's applications
• Compliance requirements in the organization's industry
• The maturity of the existing application security program
• The size and complexity of the application portfolio

Organizations that lack accurate data on their security performance will struggle to make informed decisions about where to focus limited resources.

When is it relevant?

Application security metrics are relevant in scenarios such as:

• Reporting security posture to leadership on a quarterly basis
• Evaluating the effectiveness of a new application security tool or process
• Identifying outlier development teams that require additional training and support
• Demonstrating compliance with standards like PCI DSS, SOC 2, HIPAA, etc.

Metrics may be less relevant for applications that process non-sensitive data, or low-risk internal tools. The appropriate frequency of measurement will also vary - some metrics should be tracked daily or weekly, while others are better suited to monthly or quarterly collection.

What are the trade-offs?

Implementing a comprehensive application security metrics program requires an investment of time and resources:

• Engineering effort is required to put instrumentation in place to collect metrics from relevant security tools and pipeline
• Ongoing work is needed to validate the accuracy of metrics and investigate anomalies
• Metrics that are burdensome to collect may face resistance from development teams and slow down the SDLC
• Over-focusing on certain metrics may lead to perverse incentives and unintended consequences

It's important to be thoughtful about choosing metrics that provide real security value and align with organizational goals. Collecting metrics is of little use if there is no clear action to be taken based on the results.

How to make it happen?

Here are some key steps to implementing an effective application security metrics program:

1. Define goals and requirements - Articulate the high-level objectives of the program and identify key stakeholders and compliance obligations. Use this context to decide which types of metrics will be most valuable.
2. Evaluate data sources - Take inventory of existing application security tools and processes that can serve as sources of metrics data. This may include SAST, DAST, SCA, bug bounty platforms, GRC systems, ticketing tools, etc. Identify gaps in visibility.
3. Choose metrics to collect - Select an initial set of metrics that align with goals and requirements. Examples include:
   • Count of vulnerabilities by severity and exploitability
   • Percentage of apps compliant with security policies
   • MTTR for different vulnerability types
   • Security training completion rate
   • Penetration test coverage
4. Put instrumentation in place - Integrate relevant tools into a centralized reporting system or data lake. Write scripts to extract and normalize the necessary data on a defined cadence. Ensure metrics are accessible via API for downstream analysis and visualization.
5. Build dashboards and reports - Use a BI tool like Tableau or PowerBI to create interactive dashboards and reports tailored to different audiences. Enable users to filter and group data as needed. Configure alerts to fire when metrics deviate from expected ranges.
6. Operationalize metrics reviews - Schedule regular meetings with development teams and leaders to review metrics and discuss follow-up actions. Make metrics highly visible by embedding in wikis, readouts, and other artifacts. Incorporate key metrics into developer IDPs.
7. Measure, iterate, and expand - Continuously evaluate the usefulness of the metrics being collected. Gather feedback from stakeholders on what data they find most valuable. Retire metrics that are not driving action, and expand the program over time as new requirements emerge.

What are some gotchas?

Here are a few potential challenges and pitfalls to watch out for:

• Data quality issues - Metrics are only as reliable as the underlying data. Put validation checks in place and investigate any suspicious-looking results. Be especially cautious when comparing metrics gathered via different methodologies.
• Gaming the system - When metrics are tied to performance evaluations and incentives, people will inevitably try to optimize their scores. Make sure that metrics are driving the right behaviors. Avoid creating a culture of blame or fear around security metrics.
• Lack of context - A vulnerability count or MTTR value in isolation may be misleading. Always provide relevant context, like application risk tiers, team size, and competing priorities. Metrics should be a starting point for discussion, not a replacement for it.
• Organizational resistance - Some people may see metrics as a form of surveillance or a burden on top of existing work. Clearly communicate the value of the program and minimize any manual toil required from development teams. Make metrics a tool for empowerment and continuous improvement.

What are the alternatives?

While there is no direct substitute for a formal application security metrics program, there are some adjacent practices that can help provide visibility into security posture:

• Regular penetration testing and red team exercises to identify gaps and weaknesses
• Tracking compliance with security policies and standards via questionnaires and audits
• Informal feedback gathering from development teams on what security support they need
• Qualitative risk assessments to understand likelihood and impact of different threats

Metrics should complement these activities, not replace them entirely. A combination of quantitative and qualitative approaches will provide the most well-rounded view.

Explore further

• OWASP Application Security Verification Standard - A framework of security requirements and testing procedures for verifying the security of applications
• Veracode State of Software Security Report - An annual report highlighting trends and benchmarks in application security based on data from thousands of organizations

This control also aligns with several other frameworks and standards:

• CIS Critical Security Controls v8 - Control 16: Application Software Security
• PCI DSS v4.0 - Requirement 6: Develop and maintain secure systems and software
• NIST Cybersecurity Framework v1.1 - PR.IP-12: A vulnerability management plan is developed and implemented

Hopefully this article has provided a comprehensive overview of why application security metrics matter and how to implement them successfully. By using data to drive decision making and continuously improving over time, organizations can build more secure applications and demonstrate the value of their security investments.