Speaker: James Tarala
Event: SANS Webcast
Date: September 17, 2024
Watch on YouTube: https://www.youtube.com/watch?v=5Vc_zcNmTZI
Introduction
The Cybersecurity Standards Scorecard is an annual analysis of cybersecurity frameworks, comparing their effectiveness, applicability, and governance structure. James Tarala, a senior SANS instructor, presents the 2024 SANS Edition, sharing a data-driven evaluation of over 70 security frameworks to help organizations make informed security decisions.
Building on previous years, this year’s study introduces a more automated and programmatic approach, refining the methodology to provide a comprehensive comparison of cybersecurity standards.
Key Takeaways
- Cybersecurity standards are not universally applicable. Each has unique strengths, weaknesses, and intended use cases.
- Standards should be evaluated based on business needs, not popularity. Many organizations adopt frameworks due to executive familiarity rather than security effectiveness.
- Threat mapping remains a critical gap. Few security frameworks explicitly link security controls to known cyber threats.
- Regulatory frameworks like NYDFS, SEC rules, and CMMC 2.0 are driving compliance shifts. Organizations must integrate governance and risk management best practices.
- Emerging security architectures (e.g., DevSecOps, SaaS, serverless computing) lack dedicated security frameworks. Existing standards do not fully address modern security challenges.
Summary of the Discussion
The Complexity of Cybersecurity Standards
With hundreds of cybersecurity frameworks available, selecting the right one is challenging. Tarala highlights how organizations often choose frameworks based on familiarity rather than effectiveness, leading to security gaps and inefficient resource allocation.
To provide clarity, the 2024 Scorecard classifies frameworks into four categories:
- Comprehensive Cybersecurity Frameworks – Broad security models covering governance and technical controls (e.g., NIST CSF, ISO 27002, CIS Controls).
- Regulatory & Compliance Frameworks – Standards tied to legal or industry-specific mandates (e.g., PCI DSS, HIPAA, CMMC 2.0).
- Cyber Hygiene Standards – Government-driven security best practices (e.g., ACSC Essential Eight, NSA Top 10).
- Governance & Risk Management Standards – Focused on security program management (e.g., COBIT, AICPA TSC, ISO 27001).
Evaluation Criteria
Each framework is assessed using 12 key factors, including:
- Governance, Operational, and Technical Controls – Does it provide a balance of policy, implementation, and technical security guidance?
- Recent Updates – Is the framework actively maintained?
- Community-Driven Development – Can organizations provide feedback and contribute to updates?
- Threat Mapping – Does it link security controls to real-world cyber threats?
- Applicability to Cloud and DevSecOps – Does it support modern architectures?
- Prioritization of Controls – Does it provide implementation sequencing?
- Metrics and Measurement Guides – Does it include tools for evaluating security effectiveness?
Each framework receives a letter grade based on performance in these areas.
Notable Framework Comparisons
NIST Cybersecurity Framework (CSF) – Version 2.0
- Strengths: Strong governance structure, popular across industries.
- Weaknesses: Lacks explicit threat mapping, minimal technical security coverage.
- Industry Adoption: Increasing use in private sector organizations.
- Final Score: B
ISO 27002:2022
- Strengths: Strong governance and compliance model, widely used for third-party risk management.
- Weaknesses: Limited technical security guidance, minimal focus on modern cyber threats.
- Final Score: B-
CMMC 2.0 (NIST 800-171 & 800-172)
- Strengths: Clear compliance structure, strong technical security coverage.
- Weaknesses: Regulatory uncertainty, frequent changes to certification requirements.
- Industry Adoption: Expanding beyond Department of Defense contractors.
- Final Score: B+
CIS Controls (Version 8.1)
- Strengths: Actionable security measures, strong technical security focus.
- Weaknesses: Limited governance and privacy considerations.
- Final Score: B
PCI DSS 4.0
- Strengths: Well-defined compliance structure, effective financial data security.
- Weaknesses: Primarily compliance-driven, lacks modern threat adaptation.
- Final Score: C+
HIPAA (U.S. Healthcare Security Rule)
- Strengths: Foundational regulation for healthcare security.
- Weaknesses: Outdated, lacks prescriptive security controls.
- Final Score: D+
COBIT (ISACA)
- Strengths: Strong governance framework for IT risk management.
- Weaknesses: Lacks technical security controls, difficult to implement without additional frameworks.
- Final Score: C
MITRE ATT&CK & Enterprise Mitigations
- Strengths: Explicitly maps controls to real-world threats, strong technical defense.
- Weaknesses: Not a full security framework—designed for reference rather than implementation.
- Final Score: B
Collective Controls Catalog (Research Initiative)
- Strengths: Aggregates over 90 security frameworks into a comprehensive control baseline.
- Weaknesses: Still new, lacks broad industry adoption.
- Final Score: A-
Actionable Insights
- Align security frameworks with business needs, not just compliance. Compliance does not always equal effective security.
- Balance governance, operations, and technical security controls. A strong security program integrates all three.
- Push for better threat mapping in security frameworks. Organizations should advocate for explicit control-to-threat alignment.
- Prioritize security investments. Implement high-impact controls first to maximize risk reduction.
- Adapt security strategies to evolving technology environments. DevSecOps, SaaS, and serverless security require dedicated security frameworks.
Conclusion
The Cybersecurity Standards Scorecard provides a data-driven evaluation of security frameworks to help organizations choose the right frameworks based on governance capabilities, technical controls, and real-world applicability.
By understanding the strengths and weaknesses of cybersecurity standards, security teams can align security programs with business objectives and threat landscapes—moving beyond reactive compliance to proactive security.
For more insights on this topic, watch the full webcast here.