stack.io CEO founder Hany Fahim says that people in Internet Ops never forget the first time they deal with a hack.
That’s certainly true for Hany. As a young Systems Administrator, Hany recalls his pulse racing as he dealt with the hack which was, unusually, in real-time.
According to IBM, on average, it takes anywhere from six to nine months for a company to even discover they were hacked. Most of the time is spent assessing damage and investigating how the hackers got in.
Are We Being Hacked?
“Over my career, I’ve been on the receiving end of a hack many times,” says Hany. “You never forget your first time.”
Back in the day, when Blackberrys ruled the business world, Hany was working as a Systems Administrator at an up-and-coming tech company. The business had grown fast and organically, and was quite successful.
The hack started with a search tool, used company-wide to retrieve important information. Its speed had slowed to a crawl. Hany investigated and discovered a long list of searches waiting in the queue.
“There was a huge backlog,” explains Hany, “Hundreds and hundreds of queries were all waiting their turn in line to execute.” The culprit? The task at the head of the queue was trying to back-up the database.
Hany solved the issue by terminating the task and the backlog of search queries cleared. However, minutes later the same issue reappeared; the search tool slowed, and again, someone was trying to back-up the database.
“Are we being hacked? Is this what it looks like?” wondered Hany.
Something wasn’t right. The traffic was coming from South Korea, and the search tool was for internal use only. Hany decided the best course of action was to limit access to the search tool and inform the rest of the company.
“My palms became sweaty as I realized the magnitude of what took place. If they had waited until after hours, they probably would have gotten away with a whole database,” says Hany.
Alarms and a Vice President’s Computer
After a brief pause in the action, monitoring alarms started sounding from a highly secured back-up system inside the data centre. The hack was coming from inside the office! Hany had the hardware team track it—directly to the Vice President’s office.
Suspicious, the VP was on vacation. Someone was remotely logging into the VP’s terminal and using it to download the backup server. Hany unplugged the computer, fixing the problem… or so he thought.
Attack!
“I sat there trying to put the pieces together of what was taking place when our monitoring system flooded our inbox with alerts,” recalls Hany. “There were looks of panic and frustration on everyone’s face.”
What followed was a tit-for-tat battle with a hacker. As Hany fed the hardware team IP addresses to block, the hacker started using new IP addresses.
“This painstaking process went on for over an hour. There was no end in sight. I didn’t get it. Every time we added a new block, they stopped using the old IPs and a new wave came in. It’s like they were a step ahead,” Hany remembers.
The breach
“Are they reading our emails?” Hany wondered. No. Impossible. They had just switched to a new email system and it was secure.
After more investigation, he realized there were forwarding rules from their old email system to the new one. Hany found that the old legacy system, SquirrelMail, one of the first web-based mail programs, was still active on their network. His heart began to sink.
He sent himself an email and waited for it to appear on his screen. After several refreshes, the message went from being marked as unread to read. Their old email system had been breached; the hackers were reading all their emails in real-time.
“The system was disgustingly compromised. Post-analysis showed that the hackers were in the system reading every single email for over a year,” says Hany.
Hacks and SolarWinds
Hany explains that: “Almost every hack that I’ve ever read about or been a part of was ultimately traced to some human error.”
The recent SolarWind hacks that breached many Fortune 500 companies and US government agencies, including Homeland Security and the National Nuclear Security Administration, sidestepped the many security layers in place and focused on the weakest link – the humans.
“Like any good hacker you focus on the weakest link,” observes Hany, “Fool the humans, not the machines.” The evidence seems to suggest that one of Russia’s intelligence agencies, the SVR, was behind this hack. The investigation into how widespread the hack was is still underway.
For the full story with all the technical details be sure to listen to the podcast episode!