Why This Is Important

Ukrainian cyberwar has become a great platform where the US government and commercial sectors can learn the best protective measures. 

Since the Russian-Ukrainian war broke out, Russian hackers have been focusing their attention and cyber attacks on Ukrainian government institutions as well as on civilian targets, keeping Ransomware attacks more like a source of financing for their day-2-day life. 

Over the first six months of 2022 only, about 1350 cyberattacks have been detected by the Computer Emergency Response Team of Ukraine (CERT-UA).

The top sectors targeted by Russian hackers are the following:

• Government and local authorities
• Security and defense sector
• Energy sector
• Financial sector
• Commercial sector
• Telecommunication sector and developers
• Transport sector

The most widespread types of cyberattacks include:

• Malicious code and Implants
• Intrusion
• Intrusion attempts
• Violation of information properties
• Accessibility disruption
• Harmful (abusive) content
• Known vulnerability (this case)
• Ransomware

UnderDefense together with the CERT-UA team is releasing this Cybersecurity Advisory to give a deep insight into one of such attacks. In this material, we would like to share a fascinating story where EDR software detected an initial foothold but missed other TTPs which were later discovered by UnderDefense MDR Team (back then we had just started providing Pro bono services for that Ukrainian government organization) and the attacker was finally kicked out. 

This time, the attack was targeted at the mail server of the Regional Military Governance Organization, more precisely on Zimbra mail service 9.0.0 patch 23.

The attacker’s innovation was in their persistence and codebase, bypassing CrowdStrike and attempting to keep remote access as the target was very important for them (especially because it was a Mail server).

Malicious actors (in our case APT groups) were constantly scanning Domain and IP space for both well-known and recently discovered vulnerabilities.

Notable Facts & Adversary TTPs used in this Attack

• The active attack phase began at night, all noisy and risky actions were performed outside of the IT team’s working hours
• More than 40 IPs took part in the attack and multiple actors acted in parallel, cooperating in real-time
• A lot of defense evasion techniques were seen throughout the Kill Chain phases, with at least 20 backdoors left on the disk and in memory
• A really interesting and innovative C&C communication scheme was established, undetected by all of the tested EDR solutions. C&C communication through a legitimate PubNub proxy, messaging service based on the Publish/Subscribe model. This trick allowed the APT to hide their server’s IP and evade detections based on repeating outbound connections to suspicious domains

The attackers’ ultimate goal was to obtain confidential email communication data from the Regional Military Governance organization, namely employees’ credentials and mailboxes. No data wipe, drive encryption, or critical files integrity violations were performed since the adversaries knew perfectly well what they wanted to get – Military Intelligence and keep stealthy access to all the further mailings.

Attack Success & Failures

Undoubtedly, such a complex, multi-stage, and obviously targeted attack couldn’t have been prevented by a single security tool. Conversely, each security misconfiguration or vulnerability accelerated the breach, making the adversaries’ goal a lot easier to reach. Here are the main reasons why this attack was successful, at least in the initial phase:

• Legacy OS: Ubuntu 16.04
• Vulnerable Zimbra Mail Server: 9.0.0 patch 23
• No MFA enforcement, password-only access
• No 24×7 security monitoring established

And here are the main security tools and attackers’ time-wasters that stopped the breach:

• CrowdStrike EDR: real-time protection against malware
• Network Isolation: kill switch that stopped the attackers
• Web Logs in Splunk: the useful source for activity correlations
• Incident Response: manual threat containment, system recovery

Attack Breakdown

As seen in the diagram above, the attack references more than twenty MITRE Enterprise techniques. Following the timeline, the active vulnerability scan began more than a month before the attack. 

A lot of different IPs were probing the server for web vulnerabilities, hidden services, and open ports, trying password spraying, and sure enough, preparing for the attack.

Initial access

June 18: Password hijacking was performed via one of the Zimbra vulnerabilities (CVE-2022-27924). Exploiting it by sending a specially crafted HTTP request, adversaries were able to redirect all further login credentials to the controlled server. The screenshot below shows one of the crafted requests that forwards login data of [email protected] to the APT-controlled 45.14.227[.]108 IP address. 

While the concept is simple, the attack consequences were devastating. In just a few days, hackers obtained valid credentials for more than five active mail accounts, including the administrative one.

Execution & Persistence

June 19, Night: The password hijacking campaign is over and initial access is achieved.  A valid account is a required step for the following vulnerability exploitation – Zimbra unrestricted file upload (CVE-2022-27925). Interestingly enough, both of the vulnerabilities were publicly disclosed less than two months before the exploitation, indicating the importance of a rapid path management process. This is an indicator of a very good shape of Malefactor toolkit preparedness and awareness.

Using the mentioned CVE, the attackers uploaded three variations of JSP backdoors. Java Server Pages (JSP) is a technology that helps developers create dynamic web pages, basically by running an arbitrary Java code to build the page components. One of the malicious JSP files (originally named skins_bk.jsp) is shown below. All the observed backdoors required two values to operate and work as follows:

Zimbra operates a large number of JSP files in its work, and the adversaries knew it for sure. One of the most challenging tasks during the Incident Response was to distinguish between legitimate and malicious JSP files, mainly because of three observed defense evasion techniques:

• At least 5 insignificant Zimbra files like translations of help pages were replaced with the mentioned backdoors.
• A few legitimate JSP files were modified to include malicious obfuscated one-liners, which were hard to detect by keyword search.
• Backdoor’s original creation timestamps were hidden using “touch -r good.jsp malicious.jsp” command which copies the creation time of the first argument to the second one.

June 19, Morning : Soon after uploading the files, the active phase began. At least three IPs were seen using different JSP backdoors simultaneously, discovering the content of Zimbra folders, analyzing OS and network configuration, and looking through the service user’s permissions. In the meanwhile, threat actors were exploring Zimbra CLI – a set of tools to manage the entire server from the command line.

Credentials Gathering

June 20, Early morning: Threat actors managed to fetch all usernames and corresponding passwords from the mail server. Moreover, they created a few fake administrative users, changed passwords for some of the inactive ones, and generated an authentication token to retain at least programmatic access to the mail server even after the accounts’ reconfiguration.

The attack timeline is worth emphasizing – every noisy action was made at night. Threat actors were silent during the day, and stealthy at night, moving forward only when everyone was asleep. However, to achieve their primary goal, to dump and upload a huge database of historical mailings, a stable network channel had to be established from the target to the controlled server.

Exfiltration Attempts

June 21, 3 AM: The hackers chose one of the noisiest ways to offload their findings. 

A simple bash reverse shell that was immediately flagged and blocked by CrowdStrike XDR. After a few dozen attempts to create a reverse shell with Bash, Java, JSP, Python, Perl, Socat, and Netcat, adversaries finally gave up and uploaded a file called “watchdog”, the one that made this breach one of the most extraordinary cases compared to previous Incident Response projects. 

C&C Establishment

The threat actors shouldn’t be underestimated, especially considering the approach and the sophisticated methods they applied. A great example – is the above-mentioned “watchdog” file. It is a simple Python script, packed with Pyinstaller, that begins the C&C communication. Yet, there are two facts that make it “special”:

• C&C communication through a legitimate PubNub proxy, messaging service based on the Publish/Subscribe model. This trick allowed the APT to hide their server’s IP and evade detections based on repeating outbound connections to suspicious domains
• Packed with a well-known, open-source Python tool, the malware was undetected by all of the tested XDR solutions, with 0 VirusTotal score, even after the beaconing start and command execution were successful

Downloaded and immediately executed, the file bypassed CrowdStrike detection and began beaconing to the ps.pndsn.com domain, whitelisted by most of the Threat Intelligence services as a part of PubNub infrastructure (which is extremely popular).

Through the watchdog script, victims subscribe to a certain PubNub channel and periodically turn to it for new messages. Meanwhile, the hacker can broadcast commands to the channel at any time, forcing the victims to execute the payload locally.

From the initial SOC perspective, a new process is beaconing to the PubNub service. Hash is clean and the source file is located in the expected Zimbra folder, together with the legitimate “swatchdog” script used as a mail logging manager.

Moreover, as you may see on the screenshot below, the file was copied to more than 10 locations on the disk, masking as log files, zimbra updates, and legitimate binaries. Besides, it was persisting across sessions by means of the nohup technique.

 Consequently, the C&C communication was nearly undetectable without manual investigation.

Privilege Escalation

June 21, 6 AM: Multiple IPs were still hiding backdoors in the system, trying obfuscation techniques to actually open reverse shell, and, of course, dumping the mailboxes locally to the /tmp folder.

Yet, a massive dump of government mailboxes couldn’t be sent via PubNub channel, so threat actors had to search for alternative data extraction methods. Restrictions of the zimbra user rights forced them to try to elevate privileges to root by abusing an outdated Ubuntu 16.04 operating system. Having a wide choice of exploits, adversaries chose one of the most popular ones – CVE-2021-4034, also known as PwnKit.

Trying both GitHub-downloaded binaries and locally compiled exploits, attackers still failed to overcome CrowdStrike prevention. Meanwhile, our Incident Response team was already on duty reviewing suspicious CrowdStrike alerts and the investigation began.

Remediation & Containment

The first thing we noticed was the fact that there were more than 15 CrowdStrike detections regarding reverse shell initiation attempts, so network isolation approval was immediately requested. While the request was being processed, both our security team and threat actors were doing their jobs.

Luckily, we were a bit faster and managed to stop the breach before the mailbox dump was completed. Still trying to figure out the root cause of the attack, and still observing malicious actions in real-time, we were discovering and remediating the backdoors one by one: from the memory, hidden folders, legitimate Zimbra files, etc.

As it was previously mentioned, adversaries had a plan to deeply persist on the infected system, and after each new backdoor discovery, it was becoming even more clear that complete threat containment would take a lot of time.

At the same time, the IT team was asked to reset users’ passwords and update all Zimbra components immediately to avoid reinfection. After prompt patching, the system was finally isolated from the network, and our security team could breathe a sigh of relief.

Over the next few hours of Incident Response, the security team was cooperating with local administrators checking the system for file integrity violations, looking for threat actors’ traces, and security hardening on the fly.

The server became fully operational (and patched!) after three hours of Incident Response.

Hardening & Recommendations

Although some of the steps were made during the IR, most of the time-consuming, but still critically required actions were planned for the nearest future:

Issue 1: A list of all created mailboxes was compromised

Proposal: Focus on cybersecurity awareness and personnel training since targeted phishing attacks may be observed in the nearest future.

Issue 2: The password hijacking attack was successful

Proposal: Enforce multifactor authentication (MFA) on login for every mail account, configure a secure password policy and configure access controls according to the principle of least privilege.

Issue 3: Adversaries have accessed some mailboxes via UI

Proposal: Audit login and management activities, most notable logins from suspicious sources, configure access controls according to the principle of least privilege

Issue 4: Two-month-old critical Zimbra vulnerability was exploited

Proposal: Prepare and enforce patch management process for all used services, both for internal and internet-facing ones. Regularly back up critical data and make sure it is stored offline, securely on a separate server

Issue 5: Privilege escalation attempt was performed on Ubuntu 16.04 OS

Proposal: Install updates for operating system, prebuilt software, and firmware as soon as possible. Migrate to a vendor-supported OS like Ubuntu 20.04

Issue 6: Some actions were detected, but not blocked by CrowdStrike

Proposal: Configure the prevention policies, and utilize all the available features of an existing security solution

Issue 7: Initial access, as well as C&C communication, were still undetected by CrowdStrike

Proposal: Configure SIEM to ingest more data sources, enable generic correlation rules and think of the custom ones to cover attack vectors specific to the mail server. Setup real-time notifications about any abnormal activity.

Issue 8: Network-based protection was absent, critical services were publicly accessible

Proposal: Forward the incoming traffic through a firewall with configured IPS to block known attacks, limit outbound network access and think about WAF implementation.

Issue 9: A recovery plan was not in place. Most of the decisions were made on the fly

Proposal: Work on a plan that describes the needed steps for a rapid recovery from integrity or availability violations. Make sure that the workflow is clear and consistent for the IT team.

Issue 10: No 24×7 security monitoring was established, the threat could have been detected a lot sooner

Proposal: Begin implementation of a solid real-time detection, notification, and escalation workflow or entrust the available security tools to a managed SOC team.

Indicators of Compromise IoC

IP

5.8.41[.]132
103.186.159[.]133
45.14.227[.]108

Mitigation

• Implement a recovery plan that maintains and retains multiple copies of sensitive or proprietary data and servers in a physically separate, segmented, and secure location (i.e., hard drive, storage device, or the cloud)
• Regularly back up data and password protect backup copies stored offline. Ensure copies of critical data are not accessible for modification or deletion from the system where the data resides
• Install, regularly update, and enable real-time detection for antivirus software on all hosts
• Install updates for operating systems, software, and firmware as soon as possible
• Audit user accounts with administrative privileges and configures access controls according to the principle of least privilege
• Enforce multi-factor authentication (MFA)
• Focus on cybersecurity awareness and training. Regularly provide users with training on information security principles and techniques as well as overall emerging cybersecurity risks and vulnerabilities, such as ransomware and phishing scams