Speaker: James Tarala
Event: SANS Webcast
Date: September 21, 2023
Watch on YouTube: https://www.youtube.com/watch?v=gAWnnFCy40o
Introduction
The Cybersecurity Standards Scorecard is an annual evaluation of cybersecurity frameworks, assessing their relevance, effectiveness, and applicability. James Tarala, a senior faculty member at SANS, presents the 2023 edition, offering a data-driven comparison of cybersecurity standards to help organizations make informed security decisions.
Building on the 2021 and 2022 editions, this year’s Scorecard introduces new evaluation criteria and expanded coverage, with an analysis of over 70 cybersecurity standards. The 2023 report includes a deep dive into emerging frameworks, governance models, and compliance mandates, helping security teams understand which frameworks best align with their organizational goals.
Key Takeaways
- Cybersecurity frameworks are evolving. New versions emphasize governance, policy, and risk management while streamlining technical security controls.
- Regulatory standards are gaining traction. Compliance-driven frameworks like NIST 800-171, CMMC 2.0, and NYDFS updates are influencing security programs.
- Threat mapping remains a gap. Most cybersecurity frameworks do not explicitly link security controls to specific cyber threats.
- Prioritization of security controls is fading. Many modern standards do not provide clear implementation roadmaps.
- ISO 27002 and CMMC 2.0 are gaining popularity. These frameworks are increasingly used for third-party risk management and compliance.
Summary of the Discussion
The Growing Complexity of Cybersecurity Standards
Cybersecurity frameworks have expanded significantly, with hundreds of standards worldwide. Many organizations struggle to select the right framework, often choosing based on popularity rather than effectiveness.
James Tarala challenges the notion that all cybersecurity standards are the same, emphasizing that each has unique strengths and weaknesses. The 2023 Scorecard introduces a new classification system, dividing frameworks into four categories:
- Comprehensive Cybersecurity Frameworks – Broad, all-inclusive security models (e.g., NIST CSF, CIS Controls, ISO 27002).
- Regulatory & Compliance Frameworks – Standards focused on legal mandates (e.g., PCI DSS, HIPAA, CMMC 2.0).
- Cyber Hygiene Standards – Basic security practices, often developed by government agencies (e.g., ACSC Essential Eight, NSA Top 10).
- Governance & Risk Management Standards – Focused on security program management (e.g., COBIT, AICPA TSC, ISO 27001).
Evaluation Criteria
Each framework is evaluated using 12 criteria, including:
- Governance, Operational, and Technical Controls – Does the framework balance security policy, implementation, and technical defenses?
- Recent Updates – Is the standard actively maintained to address modern threats?
- Community-Driven Development – Can organizations provide feedback and contribute to improvements?
- Threat Mapping – Does the standard link security controls to known cyber threats?
- Applicability to IT Environments – Can the framework be applied to cloud, SaaS, industrial control systems, and DevOps?
- Prioritization of Controls – Does the standard provide clear guidance on which controls to implement first?
- Metrics and Measurement Guides – Does it include tools to evaluate implementation effectiveness?
Each standard is graded on a five-point scale, with letter grades assigned based on overall performance.
Notable Framework Comparisons
NIST Cybersecurity Framework (CSF) – Version 2.0 Draft
- Strengths: Improved governance structure, widely recognized.
- Weaknesses: Lacks detailed technical controls, no explicit threat mapping.
- Industry Adoption: Increasing use in private sector organizations.
- Final Score: B
ISO 27002:2022
- Strengths: Strong governance and compliance focus, well-suited for third-party risk management.
- Weaknesses: Fewer technical security guidelines, minimal modern threat coverage.
- Final Score: B-
CMMC 2.0 (NIST 800-171 & 800-172)
- Strengths: Clear compliance structure, strong emphasis on technical controls.
- Weaknesses: Still evolving, frequent changes to certification requirements.
- Industry Adoption: Increasing use beyond Department of Defense contractors.
- Final Score: B+
CIS Controls (Version 8)
- Strengths: Practical, actionable controls, strong technical security focus.
- Weaknesses: Limited governance guidance, minimal privacy considerations.
- Final Score: B
PCI DSS 4.0
- Strengths: Well-defined compliance requirements for financial data security.
- Weaknesses: Primarily compliance-focused, lacks modern threat adaptation.
- Final Score: C+
HIPAA (U.S. Healthcare Security Rule)
- Strengths: Foundational regulation for healthcare security.
- Weaknesses: Extremely outdated, provides only basic security hygiene.
- Final Score: D+
COBIT (ISACA)
- Strengths: Strong governance framework for IT risk management.
- Weaknesses: Lacks prescriptive security controls, difficult to implement as a standalone security program.
- Final Score: C
MITRE ATT&CK & Enterprise Mitigations
- Strengths: Explicitly maps controls to real-world threats, strong technical defense guidance.
- Weaknesses: Not a full security framework—designed for reference rather than policy implementation.
- Final Score: B
Collective Controls Catalog (Research Initiative)
- Strengths: Aggregates over 70 security frameworks into a comprehensive control baseline.
- Weaknesses: Still new, not widely adopted outside of research communities.
- Final Score: A-
Actionable Insights
- Select frameworks based on your needs, not just compliance. Compliance does not always equal effective security.
- Balance governance, operational, and technical security. A well-rounded security program integrates all three.
- Push for better threat mapping in security frameworks. Organizations should demand explicit control-to-threat alignment.
- Prioritize security investments. Implement high-impact controls first to maximize risk reduction.
- Continuously update security strategies. The threat landscape evolves, and security frameworks must evolve with it.
Conclusion
The Cybersecurity Standards Scorecard highlights the need for organizations to choose the right security framework based on effectiveness, governance, and technical security measures rather than just compliance requirements.
By understanding the strengths and weaknesses of cybersecurity standards, security leaders can align their security programs with business objectives and real-world risks.
For more insights on this topic, watch the full webcast here.