The time we were hacked

This article is not written with pride. Getting hacked is never fun. It seems like it’s worse when you’re a cybersecurity expert. If you have ever been hacked and felt ashamed, we understand.

The article was written with the aim of raising awareness among companies about cybersecurity risks as part of Cybersecurity Month (October).

Cela dit, le présent article vous apprendra peut-être une expression nouvelle : la cyberrésilience. Mais qu’est-ce que c’est? La cyberrésilience est l’acceptation de la menace à venir et la préparation pour y répondre. En d’autres mots :

The question is not “if” you will be hacked one day, but rather “when” the event will happen.

Maxime Jobin, Cybersecurity Expert at SatelliteWP

En adoptant la cyberrésilience, cela permet de mettre en place un plan précis qui pourra être exécuté le moment venu. Évidemment, nous souhaitons tous ne jamais avoir à s’en servir, mais…

The discovery of the attack

It all started with a simple request from a team member: “Maxime, I no longer have access to my server. Can you check? ». This is the kind of request that comes in regularly. This can be a rule that’s too strict in a firewall or a user who enters the wrong password too many times. Yes, it even happens internally!

We don’t make it a big deal… fixes are put in place and the day goes on.

But not this time.

Connecting to the server, everything seemed ok. Only the site database didn’t load. By connecting to it… the shock!

The database contained only one table named “HACKED”. Its content was equally revealing: “Please go to xxxx.com and donate 0.46 Bitcoin to YYYYYYYY. Then, we will unlock your data”. In other words, pay up or you’ll be sorry.

Basically, at that point, it would have cost us about $20,000 CAD to recover our data.

That’s when we realized we had been victims of hacking and ransomware. The server databases were fully encrypted. No more data. Nothing. At all. Nada.

Say what?!?

The reaction

At SatelliteWP, we don’t half-ass when it comes to security. Our customers entrust us with the maintenance of their site and we are called upon to handle sensitive data.

We therefore have several processes in place to keep only the bare minimum in terms of data, give access rights as restricted as possible, use unique passwords, segment our activities to minimize risk, etc.

That said, the surprise was important!

If, however, there was one thing that was clear: we would NEVER pay the ransom demanded.

Our infrastructure is sound and in place is a “plan B” for everything. The compromised server only affected one developer and our backups (functional and tested) told us that we could recover the situation easily.

We were ready to begin the recovery of our digital entity.

Required steps

We followed several steps in order to get back to normal in a very short time.

Investigation

First of all, we have the technical skills to conduct the investigation. This step is crucial in order to determine the extent of the damage. If you do not have the skills to investigate and fix things, make sure you have a partner that can help you if a problem arises.

In our case, we were able to determine that the incident was not related to faulty updates and that it was an error by an individual who had not followed internal procedures.

Puis, nous avons fait le choix de se départir du serveur et de le rebâtir à partir de zéro. Les raisons : notre processus pour créer un nouveau serveur nous assurerait que le serveur est propre et les coûts de mise en place seraient beaucoup moins élevés.

Reorientation

Before rebuilding, the allocated resources must be redirected. In our case, a developer could no longer do his job. So he was redirected to a colleague’s web server and they shared a server for a while.

Votre situation est probablement différente de la nôtre, mais les impacts sont similaires. Si des gens de votre équipe ne peuvent plus faire leur travail, ils sont inutiles. Comment pouvez-vous leur redonner une partie de leur autonomie afin de ne pas perturber les activités de l’entreprise? C’est à cette question que vous devez répondre!

Repairs

De notre côté, nous avons recréé un nouveau serveur que nous avons entièrement reconfiguré. Comme nos processus sont bien documentés, recréer ce serveur aura pris une demi-journée environ. C’est vraiment ultra rapide dans les circonstances.

Remember, we never schedule time on the schedule for this kind of unforeseen event! The more ready you are, the faster your repairs will be.

In the end, we estimate we’ve lost about 2 days of productivity across the whole team. The loss is therefore below $5,000. We have not lost any data.

Communication

This is where the least pleasant step comes in: communication. What do we do? What do we say?

We opted for complete transparency.

Dans notre cas, le serveur piraté était un serveur de développement. Il contenait des sites en travail qui devaient être déployés plus tard. Autrement dit, aucun site de client en production n’a été touché par le piratage.

On the other hand, it wasn’t possible to know what the hackers did with the data.

So we contacted all customers who had a copy of their website on said server. Fortunately, since we don’t keep website copies unnecessarily on the server, we only had 3 clients to contact.

We explained the situation, the risks and the steps to follow on their end. Our communication was written as we would have liked to have been informed in such a case.

To everyone’s surprise: the 3 clients were delighted with our transparency and our management of the situation. We did not have any additional questions or complaints related to the event.

Under the circumstances, it was a great victory!

Updating the plan

Even if we had a plan, the solution to the problem wasn’t perfect! We adjusted some procedures, further reduced some access rights and insisted on best practices, once again.

All of our team members are smart. That is not the problem. The problem is to let your guard down when you feel too complacent. It is therefore important to repeat, insist, repeat and insist on best practices to follow. Oh! I forgot… it is also necessary to repeat and insist on following best practices!

Lessons Learned

Of course, we can always learn from each of our failures. Here is what we learned from this unpleasant episode:

1) Everyone is at risk. Simply. Every. One.
2) Even if you are an expert, it can happen.
3) The question is not “if” it will happen, but rather “when” it will happen.
4) Even if we have best practices in place, a single mistake from a single person can turn a fortress into a house of cards.
5) Backups save lives (and prevent having to pay ransoms).
6) Segmentation of data across multiple servers minimizes risk.
7) There is always room to improve existing practices.
8) Human coaching is necessary and must be constantly reviewed. Memory fades…
9) Transparency with our clients has solidified our relationship.
10) When you’re ready, it hurts a lot less.

In conclusion

Remember that not everyone has the skills required to get back up so quickly. An event like this can easily cost $50,000 and disrupt your business, customers, resources, credibility and more.

Do you have a plan to get back on your feet when that happens?

If your website has been hacked, you can contact us to remove all malware on your WordPress site.

Note: This episode of a pirated server is not recent. It occurred in April 2021 and affected only one development server. No sensitive data was stolen.