Server Backups: A Backup You Have Never Restored Is Not a Backup

“We have backups.” We hear this in almost every audit. When we then ask when you last tested a restore, the answer is usually silence. And that is exactly the problem: a backup is only worth something once you know it can be restored, quickly and completely. Until you have checked, it is just an assumption.

What usually goes wrong

In our experience it is always the same few situations:

• The backup doesn’t contain what matters. The disk is backed up, but the database is copied while it is being written to, so the files are inconsistent. Or configuration, certificates and secret keys are missing.
• The backup lives next to the original. A snapshot on the same disk or in the same account won’t help when the disk fails, someone deletes the account by mistake or an attacker gets in.
• The backup quietly stopped running. The script started failing three months ago, but nobody was watching the result.
• Nobody knows how to restore. The procedure lives in the head of one person, who happens to be on holiday.

The 3-2-1 rule

A proven baseline that is worth following even in a small company:

• 3 copies of your data – the original and two backups,
• on 2 different types of storage – for example a local disk and object storage,
• 1 copy off-site – another data centre, another region or another provider.

Today one more condition is often added: at least one copy should be immutable – one that nobody can delete or overwrite for a set period, not even an administrator or an attacker holding their password.

Ransomware changes the rules

Modern ransomware looks for backups first and encrypts second. If backups are reachable from the compromised server with write and delete rights, the attacker destroys them before anything else.

That is why we recommend:

• the server may only write to the backup, not delete older versions,
• backups in object storage are protected by a lock (for example S3 Object Lock) for long enough to cover the time it takes to discover an attack,
• credentials for backup storage are kept separate from everyday admin accounts.

RPO and RTO: two numbers everyone should know

• RPO (Recovery Point Objective) – how much data you can afford to lose. If you back up once a night, in the worst case you lose a whole day of work.
• RTO (Recovery Time Objective) – how long the service can be down before it is restored.

These are business decisions, not technical ones. An online shop taking orders needs a different RPO than an internal wiki. Only once you know them can you sensibly choose backup frequency, storage type and budget.

What it can look like in practice

On Linux servers we often use restic – it encrypts and deduplicates backups (storing only changed data) and can write straight to S3 or other object storage. A simplified nightly script:

#!/usr/bin/env bash
set -euo pipefail

# A consistent database dump instead of copying live files
pg_dump --format=custom app > /var/backups/app.dump

restic backup /etc /var/www /var/backups --tag nightly

# Keep 7 daily, 4 weekly and 6 monthly backups
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# Verify the integrity of part of the stored data
restic check --read-data-subset=5%

The script runs from a systemd timer or cron and reports its result to monitoring. If a backup doesn’t run, an alert fires – not only when disaster strikes.

In the cloud, native tools complete the picture, such as automated database snapshots or AWS Backup, which can copy backups to another region and another account.

Restore testing: the most important step, and the one most often skipped

A backup is only as good as your last successful restore. We recommend:

1. Restore for real, regularly – at least once a quarter, restore a whole server or database into an isolated environment and check that the application actually works.
2. Measure the time – how long did the restore take? Did you meet your RTO?
3. Write down the procedure – step by step, so that someone other than the person who set up the backups can do it.
4. Automate what you can – with Infrastructure as Code a new server is built in minutes and only the data needs restoring.

The first restore test almost always uncovers something missing. Much better to find out on a Tuesday afternoon than at three in the morning on a Saturday.

A short checklist

• Do you know which data and configuration you back up, and why?
• Is at least one copy off-site and immutable?
• Will you be alerted when a backup fails?
• Do you know your RPO and RTO?
• When did you last successfully restore a whole system?

If you can’t answer one of these, now is a good time to start.

💾 Want to be sure you can restore from your backups?

As part of our Linux server administration we set up backups following the 3-2-1 rule, ransomware protection, backup monitoring and regular restore tests with a documented procedure. Get in touch – the first consultation is free and comes with no obligation.