Culture of Collaboration: Fanatical Testing and Instrumentation

by Edward Vasko

Introduction

As we continue our series, the third recommendation from Shannon Lietz’s Shifting Security Left concept is to employ “fanatical testing and instrumentation” to detect and resolve issues before they are exploited. In Lietz’s secure software development context, fanatical testing means testing software for common vulnerabilities (such as the OWASP Top 10) early and often in the development lifecycle, while the instrumentation is used to evaluate system performance, diagnose errors, and trigger security alerts.

We can use this idea to develop organizational controls and processes to prevent, detect, respond to and recover from cyber-attacks.

Many people conduct the “technology pivot” when thinking about security controls. They focus primarily on technology and neglect to consider how people and processes that can also mitigate risk. Discussing technical controls, while important, only addresses the areas of risk that result from misconfiguration (people) or poor implementation (process).

Holistic instrumentation and testing model that balances security concerns against other organizational constraints will produce better results for everyone involved.

Developing that model requires the security team’s knowledge of the threat environment, the IT team’s knowledge of the operating environment, and users’ knowledge of the way they work. Together, they can develop controls that effectively reduce common risks with less impact on job performance.

To explore this concept, I’ll focus on a particularly damaging and increasingly common type of attack - ransomware - and a few examples of instrumentation and testing of the technology, people, and processes to mitigate that risk.

Understanding the Risks

Ransomware attacks cripple organizations by encrypting files on computers or servers, making them unusable until the ransom is paid. It causes service disruption, financial loss, and in some cases, permanent loss of critical data. Business interruption costs can be huge: the 2017 wave of notPetya attacks caused an estimated $10 billion in damages, with some organizations, including shipping giant Maersk, losing hundreds of millions of dollars each.

Gavin Ashton, Maersk’s Identity and Access Management chief at the time, gave a first-hand description of the attack's impact and the recovery effort. On the day of the attack, Maersk “lost the lot...every single domain-joined Windows laptop, desktop, virtual machine and physical server around the planet” within just a couple of hours.  Maersk’s chairman reported that the company restored its entire infrastructure in just 10 days. Ashton revealed that the IT staff continued to work around the clock for several months, replacing outdated and unsupported hardware and implementing new security controls. In all, the attack cost Maersk between $250 million and $300 million.

Cisco published a detailed description of how notPetya was originally introduced and how it spread. The highlights are:

• The hackers used stolen credentials to deliver the malware in an update to a financial software package widely used in Ukraine.
• The malware distributed through the upgrade exploited a common vulnerability, CVE-2017-0144, in Microsoft’s Service Message Block (SMB).
• A patch for the vulnerability was available, but notPetya found and infected many vulnerable unpatched devices.
• Once inside, notPetya harvested passwords and used them to spread to other devices connected with the original point of infection.

Instrumentation

Let’s take a look at three kinds of controls that can prevent or mitigate ransomware attacks as “instrumentation” examples.

Password Security: Establish and enforce secure password policies and/or use multi-factor authentication to reduce the risk of stolen credentials. Password instrumentation examples include checking new passwords against blacklists of commonly used or compromised credentials and triggering security alerts for multiple failed login attempts.

Access Management: According to Ashton, Maersk’s failure to enforce least-privilege-based access management principles allowed notPetya to spread horizontally across the organization and vertically into servers and domain controllers very rapidly. Limiting privileged access to users with current, legitimate needs for that access and auditing access privileges regularly can reduce the risk of malware spreading unchecked.

Basic Security: An unpatched system is the cyber equivalent of an unlocked front door. Basic security, such as installing patches and upgrades as they become available, using anti-virus/anti-spam solutions, and configuring your systems correctly will prevent many attacks.

Enhanced Security: Once the basics are covered, you can consider comprehensive instrumentation solutions, such as intrusion detection/prevention systems (IDS/IPS), security information event management (SIEM), and security orchestration, automation, and response (SOAR) solutions. These systems do a much better job of alerting the security team to problems and attacks, but they do need to be configured to work within an organization’s unique environment and tuned to avoid overwhelming the response team with false-positive alerts or simply too much information.

Managed Security: Organizations that do not already have an experienced security team, may find that managed security services providers (MSSPs) are a more cost-effective option to make sure that the basics are covered efficiently and the comprehensive solutions are tailored to their needs.

Testing

“... [Maersk’s] disaster recovery processes had only ever accounted for a loss of a site or datacentre. Nothing in the plans had accounted for ‘We have lost everything, everywhere, all at once.’”  Gavin Ashton

Incident response and recovery are key to surviving a ransomware attack, which means response and recovery plans should provide detailed and tested procedures to respond to worst-case scenarios. Ransomware attacks affect the entire organization; response planning requires a collaborative effort to ensure that the people, processes, and technology needed for recovery can act quickly.

Critical tasks for ransomware recovery include isolating infected systems to prevent spread and deciding whether or not to pay the ransom. If an organization decides to pay the ransom, they will need fast access to funding, possibly a source of cryptocurrency, and an awareness of who the group behind the ransomware is. This last point is because of new guidance from the Treasury Department indicating that ransomware fines paid to terrorist organizations may result in steep fines for the impacted organization as the fines may be considered “aiding and abetting” that terrorist organization. If they choose to recover from backups, that organization will need a plan that gets critical systems, and their dependencies, up and running as quickly as possible.

All of this and more must be planned and tested as thoroughly as possible, before the attack. Testing could involve verifying the integrity of backups, practicing restoration procedures from the backups, and tabletop incident response simulations. Plans will also need to be reviewed and tested regularly to adapt to changes in both the operating environment and the threat landscape.

Not Worth the Risk

Security solutions may seem too expensive, too time-consuming, or too disruptive, but as Maersk and other notPetya victims have learned, losing everything is more expensive, consumes more time, and causes greater disruption. The risks are real. Security may not be your top priority, but you cannot let it be your last.

Edward Vasko, Senior VP DevOps, and Technology

Edward Vasko brings more than 30 years of diverse management, technical, and information security experience to drive Avertium’s overall technology strategy and platform integrations for target acquisitions. His ability to build high-caliber teams that can tackle the hardest cybersecurity challenges; and identify market opportunities that leverage service-wrapped offerings to provide value to clients have been celebrated by the industry and his peers.