stack.io CEO Hany Fahim jumped online after receiving a message from his pager one evening.
His chatbot assistant showed him the alert. A customer was down. Not too alarming, right? But there were more alerts, all indicating an unexpected surge in traffic.
A decision at the fork in the road
A spike in traffic is usually the result of either an attack or something legitimate, such as a marketing event. It’s important to track down the cause behind a spike because, as Hany explains: “There’s a fork in the road ahead and a critical decision needs to be made.”
If the cause is legitimate, the surge can be accommodated by adding more resources (called scaling). If the cause is nefarious, defences need implementing.
Hunting for clues
To determine the right response, Hany performed a reverse lookup to locate the source of the traffic. The first check led back to Brazil. After a few more checks, Hany confirmed that Brazil was definitely the primary source of the traffic surge.
“In 2014, Brazil was ranked number seven in the world for source attack traffic,” Hany notes. “Based on our own internal data, Brazil was ranked as high as number five—things were looking very suspicious.”
If there’s news, it’s on Twitter
To find out what was behind this strange activity coming out of Brazil, Hany turned to Twitter.
He discovered: “Thousands upon thousands of tweets all talking about the same thing—WhatsApp was being blocked.”
WhatsApp was the most widely used app in Brazil at the time, yet the government decided to institute a block for 48 hours.
Hany noted that the Brazilian government has a long-standing “rocky relationship” with WhatsApp due to its use of end-to-end encryption.
“The government has demanded time and time again that WhatsApp, which is owned by Facebook, hand over user data in relation to criminal cases,” he explains. “WhatsApp has time and time again responded that this is impossible. That end-to-end encryption means that the data cannot be accessed.”
Creatively working around a block
As a result of being denied access to this data, the government decided to block the app altogether. Unsurprisingly, this didn’t go over well with the country’s many WhatsApp users.
Notes Hany: “The best guess we had at the time is this block forced Brazilians to find a way of circumventing. Using a VPN would certainly aid in the circumvention.”
Once Hany realized this, he started to scale up. “It was time to start building more servers, like a store expanding its location to handle more foot traffic.”
When Hany eventually called it a night several hours later, his mind still whirred. Had his solution been enough? What if it happened again?
Brazil wakes up and jumps onto Twitter
Early the next morning, traffic was even higher. Hany scaled up again and added more servers. However, there was still some bottlenecking with SSL handshakes; this is “the internet standard for communicating securely.” In one of Hany’s best elucidating moments, he says this is like “the bouncer guarding the door to a busy nightclub.”
The bouncer was taking too long to respond to users’ requests for entry. Picture a bouncer demanding an elaborate, time-consuming handshake before letting you in, rather than just giving a quick head nod.
Hany raced to build more proxy servers to address the bouncer’s bottleneck as traffic climbed. But suddenly, the traffic stopped. What had happened? A check on Twitter indicated that Brazil’s ban had been lifted sooner than expected!
Reflecting in the wake of events
Later, Hany would realize: “Just how valuable Twitter was in identifying the triggering event. Imagine what would have happened if we didn’t use it? We were heavily leaning towards this being an attack.”
Hany also wondered about the role history played in events in Brazil. Years earlier, the South American nation had refused Microsoft Windows’ high licensing fees, instead opting to cut costs by promoting the use of Linux – an open source operating system with a much steeper learning curve. Was the wide adoption of Linux the reason why Brazil had such a large population of tech-savvy VPN users in 2015?
No matter what the cause, the ingenuity of the human spirit had once again proven itself to be more powerful than government control.
“As we saw with the Arab Spring, and with this WhatsApp block in Brazil, people will find a way around, especially if it’s part of their lives,” concludes Hany.