Overview
As the clock struck midnight on January 1, 2022, Microsoft had an unexpected hiccup with its Exchange servers. Apparently, the servers weren’t able to process the new date, which meant the servers couldn’t process mail. Cyber security analysts noticed the issue after Microsoft scheduled a patch to allow for processing the new date, but the patch didn’t deliver.
According to Marius Sandbu, a manager for the Norwegian firm Sopra Steria, Microsoft uses a date format of "YYMMDDHHMM". When the new year’s date is converted to signed int32 the new value of 2,201,010,001 is over the max value of the ‘long’ int, which is 2,147,483,647. This resulted in an integer overflow which crashed Exchange servers, causing emails to get stuck in transport queues of on-premises Exchange servers.
The error affects Exchange Servers 2013, 2016, and 2019. Additionally, the version checking performed against the signature file caused the malware engine to crash, which means the mail is getting stuck during transport. Microsoft issued a statement addressing the issue and stated that the Exchange Server bug is a date check failure and is not an issue with malware scanning or the malware engine.
Microsoft has since resolved the date issue, but the fix requires some effort on the consumers’ part. The company stated that when the date issue occurs, you’ll see errors in the Application event log on the Exchange Server – event 5300 and 1106 (FIPFS). They have issued an automated fix and a manual fix that you can find below. Looks like Y2K22 ended before it really got started.
How Avertium is Protecting Our Customers:
How Avertium is Protecting Our Customers
Some cyber security professionals are using the stop-gap solution to mitigate this problem. They are disabling malware scanning on their Exchange Servers, but this leaves users and servers vulnerable to an attack. Avertium offers the following services to help protect your organization:
- We offer EDR (Extended Detection & Response) to prevent, detect, and respond to attacks.
- We offer DFIR (Digital Forensics and Incident Response) to mitigate damage from a successful breach.
- We offer ZTNaaS (Zero Trust Networking as-a-service). Build a security program that shrinks your attack surface with cutting-edge, technology-enabled services.
Avertium's recommendations
Microsoft recommends the following automated solution:
- Download the script here: https://aka.ms/ResetScanEngineVersion
- Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
- Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell).
Microsoft recommends the following manual solution:
To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization that downloads antimalware updates.
- Verify the impacted version is installed
Run Get-EngineUpdateInformation and check the UpdateVersion information. If it starts with "22..." then proceed. If the installed version starts with "21..." you do not need to take action.
- Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.
- Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.
- Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 (or higher)
- After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.
references
Y2K22 bug: Microsoft rings in the new year by breaking Exchange servers all around the world - Neowin
Email Stuck in Exchange On-premises Transport Queues - Microsoft Tech Community
Microsoft issues a fix for Exchange Y2K22 bug that shut down company emails - The Verge
Related Reading:
Microsoft Exchange Server Breach is Formally Attributed to China by the U.S.
Contact us for more information about Avertium’s managed security service capabilities.
